I can ping both directions but only access servers one way...?

[Gblenn]

I have three sites, two of which are running pfsense (2.7.2 and 24.03 respectively) and one of them is a Sophos XG firewall.

Subnets are
Site A : 172.168.1.0/24 Sophos XG v20 (tunnel subnets 10.10.1.3 and 10.10.2.3)
Site B : 192.168.4.0/24 pfsense 2.7.2 (tunnel subnet 10.10.1.2/30)
Site C : 192.168.1.0/24 pfsense 24.03 (tunnel subnet 10.10.2.2/30)

I managed to set up a Routed (VTI) tunnel between sites A and B without too much hassle. I then copied the setup to use for A to C tunnel on the Sophos machine. Only changing to site C FQDN and adjusting the tunnel IP for the interface that was automatically created (somewhat similar to VTI on pfsense). Firewall rules were also amended with the subnet of site C.

I then walked through all the steps to replicate the pfsense setup from site B onto site C. Only difference being the tunnel subnet.

Site A to B is fully working but site A to C does not. I can ping from either side to any device on either of the subnets. I can access the Sophos fw and other servers from site C but not the other way around...
Meaning ICMP goes across in both directions and I can see that in pcap on site C, both directions. A trace shows the exact same and correct information regardless of which one I try A to B / C or B / C to A.

Running iperf server and testing from a client at site C gives strange results in that the very first interval shows a very slow speed (50-150 Kbps) and all following intervals are 0 = 0 bytes and 0 bits/sec.
The other way around gives me nothing... meaning that iperf just times out.

I have gone through my settings, as well as deleting and recreating the setup so many times and just can't find anything wrong, anywhere...

Anyone who has succeded in setting up a tunnel between pfsense and sophos xg who can guide me? Any ideas how to move on?

[viragomann]

@Gblenn
Possibly the packet size exceeds the limit?
If that you can try System > Advanced > Firewall & NAT > Enable Maximum MSS.

I had such an issue between pfSense and OPNsense, even there were already other IPSec connections successfully running on pfSense.

You can try out the possible packet size by playing with the data bytes option in ping.

[Gblenn]

@viragomann Thanks for the tip, turned it on set to default but no change. What do you mean playing with packet size in ping? What will that give me?
If I try ping -s 1400 I get Frag needed and DF set (mtu = 1400)
If I do -s 1500 I just get the ping times, no additional info.

[viragomann]

@Gblenn
For testing you can try a low value like 1100.If this doesn't work, there might be another reason for the problem.

[Gblenn]

@viragomann I tried various settings but no difference that I can see...

I did some PCAP when trying to access Site C from Site A and this is what it looks like in pfsense.
The problem is the same regardless of which server I'm trying to acces, be it pfsense itself or in this case a switch. I just chose it for simplicity and cleanliness of the pcap.

login-to-view

And the downloaded file from within Wireshark

login-to-view

If i google pfsense blocking SYN on port 80/443 I find an old redmine issue that clearly has been resolved since way back? So it must be something different, or? Why do I not see this in the 2.7.2 setup?

I should add that in the firewall log I see a PASS on the IPSec interface of 10.10.2.3:PORT to 10.10.177.5:80
login-to-view

[viragomann]

@Gblenn
Are the routes set properly at C?

[Gblenn]

@viragomann Yes, site C being the pfsense side, the only thing to do there is to set the static route 172.168.1.0/24 to the IPSecVTI Gateway that was automagically created.

login-to-view

On Sophos it looks pretty similar and it's been quite obvious when setting this up a few times that whenever I forgot this part, I don't get near where I am now...

[Gblenn]

I have been trying to go through the logs in Sophos as well and everything I can find on that side shows green as in all packets are passed by the correct fw and nat rules for this.

[Gblenn]

When trying to log in via SSH on a Linux machine on Site C , I get the following...
Simlar to before but it looks like there is a response initally from 192.168.1.31?

login-to-view

[viragomann]

@Gblenn
So even if the destination machine was sending an ACK, the SYN packet is re-transmitted.

I'd suspect, that the ACK packet does not arrive at the machine, which initiated the connection. But why?

BTW: I noticed, that the LAN at A is a public network range.
However, this shouldn't matter if the routes are added properly.

[Gblenn]

@viragomann said in I can ping both directions but only access servers one way...?:

@Gblenn
So even if the destination machine was sending an ACK, the SYN packet is re-transmitted.

It appears that way doesn't it... however having run the same test again several times I don't see the ACK coming back. So perhaps it was an anomaly?

I'd suspect, that the ACK packet does not arrive at the machine, which initiated the connection. But why?

BTW: I noticed, that the LAN at A is a public network range.
However, this shouldn't matter if the routes are added properly.

Ah you are right, it is in the public IP range... I made the simplest possible change from 192 to 172 and I should have changed to 16 as well. But I don't suppose pfsense would care? And it's working towards site B anyway...

I started thinking there was some sync issue since site A and C are connected via a switch. Site A is for testing purposes at the moment. But latency is below 1 ms when pinging between them. So I added a limiter in order to add a bit of latency to get it closer to what I have between the other sites, but that doesn't seem to change things.

[Gblenn]

I have sort of given up on getting this to work, for now at least. Ended up deploying Wireguard on a VM on the Sophos LAN and have a connection going that way instead...

[Gblenn]

@viragomann
So this is definitely a 24.03 thing since a change to 2.7.2 solved the issue immediately... No change in the configuration on either side, just a switch to CE.
And I guess I have to see if the patches I just found out about for IPsec VTI will fix the issue.

[Gblenn]

And now it's working also on 24.03 after implementing patches for Redmine #15449, #15430 and #15606...

[viragomann]

@Gblenn
Great info. Thanks.
I never had this issue, even running multiple IPSec site-to-site connections to other routers than pfSense and didn't apply IPSec patches.

[Gblenn]

@viragomann Well, first time I have used IPSec since Sophos doesn't do WG...

One thing to note though, this is Routed VTI connection, which is different from "Tunnel IPv4" that you have as default Mode in pfsense...

[viragomann]

@Gblenn
Ah ya, you mentioned above.
I never set up a VTI to be honest, I just have some small knowledge about from the docs.

[Gblenn]

@viragomann Well, since I couldn't get the default one to work, I tried VTI and it worked. Not sure what I did wrong with the other method but I did find VTI a bit more like the WG tunnels I have set up in the past. With the gateway and routing settings at least...