Regression: pfSense CE 2.9.0 vs. 2.8.1

FlashBurn

WAN: PPP LTE, MTU 1492
IPsec: ESP, IPv4

2.8.1:

inner packet > ESP MTU causes outer ESP fragmentation
fragments arrive at central firewall consecutively
ping normal (~50 ms)
packet sizes >1500 work

2.9.0:

<= ping payload 1410 works (~50 ms)
payload 1411 triggers ESP fragmentation and ~1 second RTT
larger packets ~2 sec or packet loss
sufficiently large packets fail completely

Capture on 2.9.0 ppp0:
both ESP fragments leave within microseconds.

Capture on central WAN:
first fragment arrives normally,
second fragment arrives ~1.0 second later.

Normal non-IPsec IPv4 fragmentation over the same 2.9.0 PPP link works normally (~50 ms).

Same central firewall and same topology work with pfSense 2.8.1.

stephenw10

You're using tunnel mode P2 there?

The ping fails when initiated in both directions?

FlashBurn

@stephenw10 Thanks! Yes, P2 is configured as Tunnel IPv4, i.e. IPsec tunnel mode.

I have now tested the behavior in both directions:

Internal server -> central fw (pfSense 2.8.1) -> branch fw (2.9.0)
branch fw (2.9.0) -> central fw (pfSense 2.8.1)

The same behavior occurs in both directions:

<=1410 byte payload: ~64 ms
1411 byte payload: ~1081 ms
1600 byte payload: ~2080 ms

So the issue is not specific to traffic in one direction. The same packet-size threshold and the resulting ~1–2 second delay can be observed regardless of which side initiates the ping.

As a workaround, I have meanwhile set Maximum MSS to 1390 on all of the affected 2.9.0 firewalls.

Interestingly, this seems to have changed the observed behavior somewhat. This is unexpected for the ICMP tests, since MSS should only affect TCP and therefore should not directly affect these ICMP pings. For example, yesterday I was seeing complete ping failures with larger packet sizes (e.g. 1600 bytes), whereas today the same kind of tests are completing, but with the ~1–2 second delay.

So I would not consider the MSS setting a real fix at this point, but I wanted to mention the change in case it is relevant.

stephenw10

Hmm, interesting. I agree the mss clamp on the tunnel would normally only affect TCP traffic.

Are you running the new strongswan pkg? That was added to the repo after 2.9.0 release so you may not be.
Try running at the CLI: pkg upgrade and see what upgrades are offered. For example on a test box here I see:

Installed packages to be UPGRADED:
	if_pppoe-kmod: 2.9.0.1600018 -> 2.9.0.1600018_1 [pfSense]
	pfSense-pkg-System_Patches: 2.3.4 -> 2.3.5_1 [pfSense]
	pfSense-pkg-tftpd: 0.1.3_10 -> 0.2 [pfSense]
	pfSense-repoc: 20260807.184806 -> 20260828.183115 [pfSense]
	strongswan: 6.0.7 -> 6.1.0_1 [pfSense]
	unbound: 1.25.2 -> 1.26.1 [pfSense]

Upgrade stongswan to 6.1.0_1 if you have not got it already.

FlashBurn

@stephenw10 Thanks again for the quick update!

I checked the installed version directly:

# pkg info strongswan
strongswan-6.1.0_1
Name           : strongswan
Version        : 6.1.0_1
Installed on   : Wed Sep 23 11:53:10 2026 CEST
Origin         : security/strongswan
...

So it looks like I am already running the new strongSwan package.

Also pkg upgrade reports that all packages are up to date.

So the issue is reproducible with strongSwan 6.1.0_1 already installed.

stephenw10

OK great I'll try to replicate it here. Failed to do so so far. Do you have anything non-default set on the tunnel?

FlashBurn

@stephenw10 Sorry for the delay in answering your question.

Our tunnels are all policy-based IPv4 IPsec. The LTE WAN is PPP with MTU 1492. The tunnel is using native ESP (IP protocol 50), not NAT-T. Phase 1 uses AES256-GCM / 128 bit, SHA512, DH group 16. We use custom ASN.1 DN identifiers on both ends with PSK authentication. Phase 2 uses only AES256-GCM / 128 bit, no separate hash, PFS group 16. Automatically Ping Host is configured.

The important boundary appears to be outer ESP fragmentation: ICMP payload 1410 works normally, while 1411 causes the ESP packet to exceed the PPP MTU and fragment.
On 2.8.1 both outgoing ESP fragments arrive at the central firewall essentially together. On 2.9.0 they are also captured together on the LTE firewall's ppp0 interface, but the second fragment arrives at the central firewall about 1 second later. Larger packet sizes result in ~2 s latency and eventually complete packet loss.

The LTE WAN is a PPP interface (ppp0, MTU 1492). Is it possible that this is a FreeBSD 16 PPP/modem driver regression rather than an IPsec regression? On pfSense 2.9.0 both ESP fragments are captured on ppp0 only a few microseconds apart, but at the central firewall the second fragment arrives about one second later. On an otherwise equivalent pfSense 2.8.1 LTE firewall, both fragments arrive at the central firewall immediately. Plain fragmented IPv4 traffic generated directly by the 2.9.0 firewall works normally over the same PPP/LTE link.

After rebooting one affected 2.9.0 firewall the behavior changed. Instead of the previous ~1–2 second delay for fragmented ESP packets, packets above the fragmentation threshold now frequently time out completely. Successful packets have normal latency (~40–120 ms). Looks like the delay is not always happening. MSS clamping had been removed before the reboot, but since the test is ICMP, MSS clamping should not affect this traffic.

I am still not exactly sure what might be causing this. Hope this information is helpful. Please let me know what else I can provide.

stephenw10

Still failing to replicate this so far. Are you running any sort of traffic shaping that could be queing the IPSec traffic?

FlashBurn

First of all, thanks to @stephenw10 for taking the time to investigate and attempt to reproduce the issue. I really appreciate your support, and sorry for the delayed response!

We have some new findings from our side: The connections appear to be stable again, without any changes on our side. Ping latency is not outstanding, but it is consistently well below 100 ms, with no packet loss anymore.

Our current conclusion is that the LTE data allowance had been exhausted, resulting in the connection being throttled to 64 kbit/s. We were previously unaware that such a data cap and throttling policy applied to these connections. This would explain most, if not all, of the symptoms we observed, including the severe latency spikes and packet loss when ESP packets exceeded the fragmentation threshold.

Interestingly, this also fits the timeline: On October 2, a colleague reported that everything was working again after installing new antennas. However, I doubt that the antennas made a difference. The connections had been working perfectly fine before we upgraded the devices to pfSense 2.9.0, and the problems started immediately after the upgrade. It seems much more likely that the recovery coincided with the start of a new billing cycle, when the LTE data allowance was reset.

At this point, we consider LTE bandwidth throttling, potentially combined with traffic shaping or queueing by the provider, to be the most likely root cause rather than a pfSense 2.9.0 or strongSwan regression.

The fact that ordinary fragmented IPv4 traffic worked normally while fragmented ESP traffic showed such severe delays may have been a misleading clue. The additional overhead and packet characteristics of IPsec traffic could have made the effects of the bandwidth limitation more apparent.

We have not independently confirmed that throttling was actually active, so this remains our working conclusion. However, the fact that two of the three upgraded firewalls are now stable again, combined with the timing of the recovery, makes this explanation very convincing.

Thanks again for your time and effort in looking into this, and apologies for potentially sending you down the wrong path while investigating what may ultimately have been a provider-side bandwidth limitation. I really appreciate your help!

PS: Should I change the title of this topic?

stephenw10

Ah, yes that would do it. And, yes, confusing if it's not something you ever expect to see!

Yes you could change the title if you wish. Probably not necessary though.