IPSEC tunnel + ISP - established connections are interrupted

mduchaine

Hello everyone,
My home network is composed of three sites, connected through IPSEC tunnels. All three have PfSense 2.4.5-p1.
PfSense SiteA uses an ISP Modem and has public IP directly on the WAN interface.
On Site B and Site C, internet connection is through an ISP AP/Router, PFSense on both sites is configured as the DMZ host.
All three sites use different ISP.

Site A and Site B are running for a few years now and no issues.
I've added Site C recently, basically a copy of the other setup.
Tunnel Ph1 and Ph2 are established, apparently everything the same as with the first setup.

The issue I'm facing is the following:
Traffic initiating from a source behind PFsense C to A and B is working fine. From a laptop in C, I can access my NAS on site A, download music and files.

Traffic initiating from either A or B towards a host on Site C stops after a few packets being exchanged. I've taken a pcap at both ends and I can see the connection being setup (TCP SYN/SYN-ACK/ACK exchanges) I get a first set of packets from the destination, then nothing comes over the tunnel anymore.

I'm at loss as to what could cause this behavior.
This is my network setup. IPs and ranges are fiction.

I'm not using NAT/BINAT translation in my PH2. Phase 1 NAT-Traversal is set to auto.

Hopefully some of you have already faced similar issues.

mduchaine

A friend suggested I check the MTU size. He was right.

running icmp's of different length, pings would stop working from 1350 bytes onwards. A bit odd if you ask me.

Anyhow, added MSS clamping at all three sites and problem fixed

johnpoz

@mduchaine said in IPSEC tunnel + ISP - established connections are interrupted:

A bit odd if you ask me.

Not really - any sort of tunnel would add its own overhead.. So yeah when tunneling traffic you need to take this overhead into account..

https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/advanced.html

MSS Clamping

Enable maximum segment size clamping on TCP flows over IPsec tunnels. This helps overcome problems with path MTU discovery (PMTUD) on IPsec VPN links.

This is useful is large TCP packets have problems traversing the VPN, or if slow/choppy connections across the VPN are observed by users. Ideally it should be set to the same value on both sides of the VPN, but traffic will have MSS clamping applied in both directions.

Enable

When set, the Maximum MSS option is available and its value is used by the firewall Configuration.

Maximum MSS

The maximum segment size set in TCP packets flowing across IPsec VPN tunnels. Defaults to 1400. Must be low enough to account for the overhead of IPsec and the MTU of the link, but no so low that unnecessarily small segments are sent as that can be inefficient.

mduchaine

It's odd for several reasons:

This is a 50 bytes more overhead than your standard 1400 IPSEC payload, meaning this ISP is encapsulating my traffic somewhere along the path to the NIX and didn't increase the MTU on the carrier network to compensate for the loss in payload.

When I encapsulate traffic, say I'm using VXLAN, or GRE, I also increase the MTU on the network equipment to keep the 1472 payload clean.
The other two ISP providers I use seem to have this right since I can traverse my IPSec tunnel with 1400 bytes packet just fine. So ISP-3 either made a configuration mistake or there is a network equipment limitation along the path.

The MTU seems to be set differently for incomming and outgoing traffic, which is not possible. I could access PFSense Site A from a host on site C - at 1400 bytes apparently - yet accessing PFSense Site C from a host on Site A was impossible untill I dropped MSS clamping at 1350. You'll admit this makes no sense, right?