Fragmentation issue on IPsec VTI tunnel

[DEHAAS]

We are migrating to IPsec VTI from an IPsec tunnel, and are experiencing some very strange issues related to packet fragmentation.

Over the old tunnel a ping with a size of 2000 goes over the tunnel just fine. After switching to VTI, the max ping size we can get across is 1372. I am testing with ethernet connected clients on both ends. The ethernet ports on both ends are standard MTU 1500, and the VTI MTU is 1400.

--

On the tunnel mode connection (which works) I see this in a packet capture on the router.

On the IPsec "interface":

15:42:07.571258 (authentic,confidential): SPI 0xcca89e5f: (tos 0x0, ttl 127, id 24630, offset 0, flags [none], proto ICMP (1), length 2028)
172.20.130.53 > 172.20.136.20: ICMP echo request, id 1, seq 474, length 2008

On the LAN interface where the 172.20.136.20 client sits

15:41:19.825056 90:ec:77:50:7a:22 > 78:8a:20:df:a6:97, ethertype IPv4 (0x0800), length 1514: (tos 0x0, ttl 126, id 24627, offset 0, flags [+], proto ICMP (1), length 1500)
172.20.130.53 > 172.20.136.20: ICMP echo request, id 1, seq 471, length 1480
15:41:19.825059 90:ec:77:50:7a:22 > 78:8a:20:df:a6:97, ethertype IPv4 (0x0800), length 562: (tos 0x0, ttl 126, id 24627, offset 1480, flags [none], proto ICMP (1), length 548)
172.20.130.53 > 172.20.136.20: ip-proto-1

The packet is fragmented correctly, according the the 1500 MTU on the LAN interface, and the client responds appropriately.

--

On the IPsec VTI connection (which does not work) I see this in a packet capture on the router.

On the IPsec VTI interface:

15:34:41.576981 AF IPv4 (2), length 1400: (tos 0x0, ttl 127, id 62903, offset 0, flags [+], proto ICMP (1), length 1396)
172.20.130.53 > 172.20.140.100: ICMP echo request, id 1, seq 464, length 1376
15:34:41.577000 AF IPv4 (2), length 656: (tos 0x0, ttl 127, id 62903, offset 1376, flags [none], proto ICMP (1), length 652)
172.20.130.53 > 172.20.140.100: ip-proto-1

On the LAN interface of the 172.20.140.100 client, this it sent as

15:35:52.961652 1a:cb:63:20:dd:3f > 00:50:56:9a:dd:a2, ethertype IPv4 (0x0800), length 2042: (tos 0x0, ttl 126, id 62905, offset 0, flags [none], proto ICMP (1), length 2028)
172.20.130.53 > 172.20.140.100: ICMP echo request, id 1, seq 466, length 2008

I suspect that something related to MTU and fragmentation is going wrong with the IPsec VTI interface. It looks like fragmentation is applied correctly over the tunnel, but it is forwarded to the client on the LAN interface without fragmenting based on the 1500 MTU?

All routers are running pfSense 23.01

Any idea what is going wrong here, and/or how I might fix this or diagnose further?

Best regards,
Christopher de Haas

[DEHAAS]

I am struggling to diagnose this problem. I found this bug from 2017, which seems to be related: https://redmine.pfsense.org/issues/7801. Unfortunately the pull request references no longer work, thus I cannot find the exact changes.

I have tried all combinations for the System / Advanced / Firewall & NAT / VPN Packet Processing / Reassemble IP Fragments until they form a complete packet, but it does not have any effect on the issue. I seems like something is wrong specifically when using a VTI interface.

I think it is related to the default scrub rule with fragment reassemble as indicated here https://forum.netgate.com/topic/26822/allow-fragments-in-rules.

So, I have now tried, in a lab, to disable Firewall Scrub in System / Advanced / Firewall & NAT. With this, packets which require fragmentation are now working correctly over the VTI link!

However, I do not really want to disable pf scrub entirely. I am also a bit unsure whether this will break a lot of over parts of the network. Any idea on a better solution here for what I must believe is a bug?

[DEHAAS]

In case anyone finds this thread while diagnosing the same problem. A fix is currently in development, and can be manually applied for testing now. Please see https://redmine.pfsense.org/issues/14396