PMTUD doesn't work at all?

rolytheflycatcher

Brand new installation of 2.5.2CE (although the problem was also present on 2.4.5).

The only interface is on my LAN and the only configuration I have done is setting the interface MTU to 1400 and creating a firewall rule to pass all traffic.

Why doesn't PMTUD work?
If I ping with 1372 bytes and with DF flag set everything is fine, but ping with 1373 bytes with the df flag results in a time out rather than the ICMP reply "fragmentation required but DF bit set".

Surely this is fundamental?

stephenw10

Where exactly are you pinging between? What interface did you set the MTU on? If it was on the LAN you just end up with an MTU mismatch. If the upstream (WAN) link in the route is a smaller MTU you will see that:

00:25:49.574886 IP 10.24.25.10 > 172.21.16.5: ICMP echo request, id 9, seq 1, length 1408
00:25:49.574990 IP 10.24.25.1 > 10.24.25.10: ICMP 172.21.16.5 unreachable - need to frag (mtu 1400), length 576
00:25:50.599490 IP 10.24.25.10 > 172.21.16.5: ICMP echo request, id 9, seq 2, length 1376
00:25:50.599531 IP 10.24.25.10 > 172.21.16.5: ip-proto-1
00:25:51.623462 IP 10.24.25.10 > 172.21.16.5: ICMP echo request, id 9, seq 3, length 1376
00:25:51.623469 IP 10.24.25.10 > 172.21.16.5: ip-proto-1

Steve

rolytheflycatcher

@stephenw10 Ultimately, I'm trying to find a way to avoid a PMTUD blackhole when using IPSec.

However, in this example, I had set the LAN MTU to 1400 (and the WAN left at default), expecting pfSense to respond with the ICMP reply. I was expecting it to, but in hindsight I realise that the pfSense LAN interface is part of the same broadcast domain as my PC and so no routing takes place until it routes traffic over to another interface. Okay, so that makes sense now.

Any insight on making it work with IPSec would be appreciated, though.

Thanks.

stephenw10

Policy based IPSec is not routed so it doesn't behave like a normal interface. If you are seeing MTU issues across the tunnel the usual solution is to enable MSS clamping in the IPSec advanced config at an appropriate value. Often 1372 or there abouts. When testing that I usually start by setting 1300 to be sure and the dial it in more accurately if that works.

Steve

rolytheflycatcher

@stephenw10 unfortunately TCP MSS doesn't help with UDP/RADIUS.

stephenw10

Indeed, MSS is TCP only.
Can you set a link at either end to be 1400 MTU?
Can you use route based IPSec? Or OpenVPN?

Steve

rolytheflycatcher

@stephenw10 the other end is a MS Azure VPN Gateway and I believe is already set to MTU 1400 (at least, that's what they recommend the remote end be set to). I'm not sure if that can be overridden at all.

I could explore route-based IPSec - but it would be nice to have it confirmed that pfSense will deal with PMTUD properly over route-based IPSec before I spend too much time on it.

stephenw10

Hmm, well I would expect Azure to send back the ICMP too large message if their internal MTU is 1400. Assuming packets reach them. I note they recommend setting MSS clamping to 1350 for policy based IPSec to Azure directly. Is that what you're doing, rather than pfSense in Azure?

Let me see if I can test route based IPSec here...

rolytheflycatcher

@stephenw10 Yes, this is a physical pfSense box which is connected to Azure over internet. I have a number of otherwise identical IPSec tunnels to Azure running on Draytek routers which all correctly pass back the ICMP too large message in response to an oversized ping with df set, and my UDP/RADIUS traffic (EAP/TLS certificates) also passes through fine at those sites.

This thread contains some further information on the specific issue, but I'd be content right now to see PMTUD working. https://forum.netgate.com/topic/162744/pfsense-ipsec-microsoft-azure-mtu/8?_=1635437503312

In the above thread, the oversized UDP packets do not make it as far as the RADIUS server running in Azure, but as they DO make it through when inbound from the Draytek routers, I can only presume it is a pfSense/Azure anomaly.

Thanks for any further advice/help. For what it's worth, I'd probably replace the Drayteks with pfSense/netgate appliances if I could get these tunnels working.

Mark

stephenw10

Ok, that does appear to work as expected with a route based IPSec tunnel:

17:22:26.125209 IP 192.168.170.11 > 192.168.78.1: ICMP echo request, id 10, seq 1, length 1308
17:22:26.125314 IP 192.168.170.1 > 192.168.170.11: ICMP 192.168.78.1 unreachable - need to frag (mtu 1300), length 176
17:22:27.155043 IP 192.168.170.11 > 192.168.78.1: ICMP echo request, id 10, seq 2, length 1280
17:22:27.155054 IP 192.168.170.11 > 192.168.78.1: ip-proto-1
17:22:27.155660 IP 192.168.78.1 > 192.168.170.11: ICMP echo reply, id 10, seq 2, length 1308
17:22:28.156440 IP 192.168.170.11 > 192.168.78.1: ICMP echo request, id 10, seq 3, length 1280
17:22:28.156452 IP 192.168.170.11 > 192.168.78.1: ip-proto-1
17:22:28.157022 IP 192.168.78.1 > 192.168.170.11: ICMP echo reply, id 10, seq 3, length 1308

That's a pcap on the LAN interface (192.168.170.1) where the client at 192.168.170.11 is ending packets to a remote subnet across a route based IPSec tunnel. The MTU on the VTI tunnel interface is set to 1300 but the subnets at each end are 1500.
So that behaves as as expect it to.

Steve

rolytheflycatcher

@stephenw10 that's promising. I will test and report back.

rolytheflycatcher

@stephenw10 I can confirm that I now have a route-based tunnel working and that I am now seeing the ICMP reply with the DF flag set and size > MTU.

My RADIUS query still isn't working but it feels like I am one step closer.

Thanks for your help.

rolytheflycatcher

RADIUS EAP/TLS now working too - YAY!

stephenw10

Nice result!