Site2site from hell

michmoor

@sensewolf Curious. Are the sites directly connected to the firewall? In other words are there intermediary routing devices that one has to pass through to get to the clients?
Can you provide a traceroute from each client?

stephenw10

@sensewolf said in Site2site from hell:

I was able to verify that a ping from Site A goes through the tunnel (I am logging the respective Allow FW rule) and actually reaches Server B (I am monitoring incoming packets on the server). And I know from pinging Server B locally from Site B that Server B does respond to pings.

But do you actually see that response anywhere when you ping from SiteA?

It should appear in a packet capture on all the interfaces in between. So SiteB LAN, SiteB OpenVPN, SiteA OpenVPN and SiteA LAN.

There's a good chance the server responds to pings from it's own subnet but not anywhere outside that. You would need to adjust the server config or outbound NAT the traffic if so. Generally it's better to avoid NAT if you can.

Steve

michmoor

@stephenw10 Hi Stephen. This is resolved.

Here is the reddit post where I posted my solutions

https://www.reddit.com/r/PFSENSE/comments/t7mil8/ipsec_vti_unable_to_send_any_traffic/

In summary, there are some issues with the way pfsense saw Phase2 entries. Disabled entries were causing the problem. Once deleted, P2 entries were normal.

There were other pfsense related issues but nothing has traffic breaking as the P2.

sensewolf

Sure @michmoor

So from Site B to Site A I get:

1 172.16.17.1 53.784ms 52.880ms 52.863ms
2 192.168.9.202 52.981ms 53.960ms 52.160ms

That's what I would expect.

But from Site A to Site B I get:

1  172.16.17.2  53.881 ms  53.732 ms  65.851 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *

Note that 172.16.17.0 is the OpenVPN tunnel.

While the traceroute from Site A does not identify the individual hops, it does show that there are way more than one would expect. But given that the first hop is into the tunnel, I don't know what to make of it...

sensewolf

@stephenw10

Well, this is interesting:

When I ping from Site B to Site A, I can see the ping arrive in the local network of Server A and I see an actual response from Server A.

When I ping from Site A to Site B, I can see the ping arrive in the local network of Server B but I don't see an actual response from Server B.

Now, these are both the same type of server: A Proxmox Virtual Environment. And I don't understand why one (Server A) responds to pings from any and all networks and the other (Server B) doesn't but this would seem to be a problem with the configuration of Server B, as you suggested. I shall investigate.

Thank you very much!

stephenw10

Yeah, if that actual server is able to ping the other way it must have a route so it pretty much has to be blocking the incoming pings.

Steve

sensewolf

@stephenw10

Actually, I just checked: Server B can't ping Server A.

It is only PC B that can ping Server A.

If I ping from Server B, there is a total loss of packets.

So, the issue is not resolved yet. :(

sensewolf

Okay, I found the error:

Server B has two NICs. One is connected to pfSense, one is not.

Server B's standard gateway was configured on the NIC not connected to pfSense. Once I changed this to the NIC connected to pfSense, everything started working as expected.

Thank you everyone for your help in figuring this out!!!

michmoor

@sensewolf Im getting lost in the weeds here.
So some devices can ping end 2 end and then there are some that cannot?
Would you happen to have a simple drawing for me to see what the traffic flow should be?
Also, the traceroutes you provided were for the servers but can you provide one for PCs as well?

sensewolf

@michmoor

So the problem essentially was that Server B had its standard gateway on the NIC that is not connected to pfSense.

PC B is connected to pfSense B and can ping and access everything locally and remotely fine, because PC B's standard gateway is connected to pfSense B.

Server B could not ping remotely, because its pings went out its standard gateway which wasn't connected to pfSense. But Server B could not be pinged from another local network or from a remote network. Apparently, responses to pings coming in on Server B's NIC that is connected to pfSense still went out Server B's standard gateway, rather than on the NIC the ping came in on.

It is all starting to make sense to me now. I learned a lot today!

I hope this explains it sufficiently. If not, I will make a drawing.

stephenw10

Yup that would do it! Both sides need a valid route to reach the other if there is no NAT involved.
Usually that's the default route via pfSense.

Steve