Site2site from hell
-
Hi,
(this is a renewed post of an issue I submitted in Packages>Wireguard a few days ago but it turns out that the issue is likely not wireguard).
So I set up a site2site between Site A and Site B. At first, I set up Wireguard but when I ran into an issue I could not resolve (even with the help of the forum), I tried OpenVPN - only to run into the same issue. That's why I am guessing it isn't Wireguard.
So what's the problem:
Site B dials into Site A to connect - this step works flawlessly.
Once connected, I can reach Server A from PC B but not Server B from PC A.
I was able to verify that a ping from Site A goes through the tunnel (I am logging the respective Allow FW rule) and actually reaches Server B (I am monitoring incoming packets on the server). And I know from pinging Server B locally from Site B that Server B does respond to pings.
But the response never reaches Site A.
What might cause this?
FW rules for the tunnel are Any2Any on both sides. (For Wireguard also Allowed IPs are 0.0.0.0/0). There is no manual NAT mapping on pfSense B and only one manual mapping in pfSense A specifically for my 3CX installation ("Full Cone NAT") to use a static port out the WAN interface. Other than that only the automatic rules.
I appreciate any ideas.
Many thanks in advance!
-
@sensewolf Curious. Are the sites directly connected to the firewall? In other words are there intermediary routing devices that one has to pass through to get to the clients?
Can you provide a traceroute from each client? -
@sensewolf said in Site2site from hell:
I was able to verify that a ping from Site A goes through the tunnel (I am logging the respective Allow FW rule) and actually reaches Server B (I am monitoring incoming packets on the server). And I know from pinging Server B locally from Site B that Server B does respond to pings.
But do you actually see that response anywhere when you ping from SiteA?
It should appear in a packet capture on all the interfaces in between. So SiteB LAN, SiteB OpenVPN, SiteA OpenVPN and SiteA LAN.
There's a good chance the server responds to pings from it's own subnet but not anywhere outside that. You would need to adjust the server config or outbound NAT the traffic if so. Generally it's better to avoid NAT if you can.
Steve
-
@stephenw10 Hi Stephen. This is resolved.
Here is the reddit post where I posted my solutions
https://www.reddit.com/r/PFSENSE/comments/t7mil8/ipsec_vti_unable_to_send_any_traffic/
In summary, there are some issues with the way pfsense saw Phase2 entries. Disabled entries were causing the problem. Once deleted, P2 entries were normal.
There were other pfsense related issues but nothing has traffic breaking as the P2.
-
Sure @michmoor
So from Site B to Site A I get:
1 172.16.17.1 53.784ms 52.880ms 52.863ms 2 192.168.9.202 52.981ms 53.960ms 52.160msThat's what I would expect.
But from Site A to Site B I get:
1 172.16.17.2 53.881 ms 53.732 ms 65.851 ms 2 * * * 3 * * * 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * *Note that 172.16.17.0 is the OpenVPN tunnel.
While the traceroute from Site A does not identify the individual hops, it does show that there are way more than one would expect. But given that the first hop is into the tunnel, I don't know what to make of it...
-
Well, this is interesting:
When I ping from Site B to Site A, I can see the ping arrive in the local network of Server A and I see an actual response from Server A.
When I ping from Site A to Site B, I can see the ping arrive in the local network of Server B but I don't see an actual response from Server B.
Now, these are both the same type of server: A Proxmox Virtual Environment. And I don't understand why one (Server A) responds to pings from any and all networks and the other (Server B) doesn't but this would seem to be a problem with the configuration of Server B, as you suggested. I shall investigate.
Thank you very much!
-
Yeah, if that actual server is able to ping the other way it must have a route so it pretty much has to be blocking the incoming pings.
Steve
-
Actually, I just checked: Server B can't ping Server A.
It is only PC B that can ping Server A.
If I ping from Server B, there is a total loss of packets.
So, the issue is not resolved yet. :(
-
Okay, I found the error:
Server B has two NICs. One is connected to pfSense, one is not.
Server B's standard gateway was configured on the NIC not connected to pfSense. Once I changed this to the NIC connected to pfSense, everything started working as expected.
Thank you everyone for your help in figuring this out!!!
-
@sensewolf Im getting lost in the weeds here.
So some devices can ping end 2 end and then there are some that cannot?
Would you happen to have a simple drawing for me to see what the traffic flow should be?
Also, the traceroutes you provided were for the servers but can you provide one for PCs as well? -
So the problem essentially was that Server B had its standard gateway on the NIC that is not connected to pfSense.
PC B is connected to pfSense B and can ping and access everything locally and remotely fine, because PC B's standard gateway is connected to pfSense B.
Server B could not ping remotely, because its pings went out its standard gateway which wasn't connected to pfSense. But Server B could not be pinged from another local network or from a remote network. Apparently, responses to pings coming in on Server B's NIC that is connected to pfSense still went out Server B's standard gateway, rather than on the NIC the ping came in on.
It is all starting to make sense to me now. I learned a lot today!
I hope this explains it sufficiently. If not, I will make a drawing.
-
Yup that would do it! Both sides need a valid route to reach the other if there is no NAT involved.
Usually that's the default route via pfSense.Steve
