Broken UDP fragmentation when payload is between 1473 and 1479 bytes

mgaffigan

Hello!

I'm troubleshooting an MTU related issue with UDP (RADIUS and SIP), and am not sure how to proceed. We have a Netgate 6100 (Intel X553) on 25.11.1-RELEASE that seems to be silently dropping fragmented UDP packets of very particular sizes (where the completion packet is fewer than 8 bytes long).

The router is assigned IP 10.200.48.3 and is communicating with 10.200.48.41 on the same segment. I'm signed in via SSH to run tests on 10.200.48.3. ping -D -s # and netstat -rnW demonstrate the MTU is correctly determined as 1500.

Capture from 10.200.48.3 (pfsense):

# tcpdump -i ix1.248 -nvv ip host 10.200.48.41 &
# php -r '$n=1473;$s=socket_create(AF_INET, SOCK_DGRAM, SOL_UDP); socket_sendto($s, str_repeat("A", $n), $n, 0, "10.200.36.131", 12345);'
# 12:29:47.206591 IP (tos 0x0, ttl 64, id 45498, offset 0, flags [+], proto UDP (17), length 1500, bad cksum 0 (->3941)!)
    10.200.48.3.21237 > 10.200.48.41.12345: UDP, length 1473
12:29:47.206607 IP (tos 0x0, ttl 64, id 45498, offset 1480, flags [none], proto UDP (17), length 21, bad cksum 0 (->5e4f)!)
    10.200.48.3 > 10.200.48.41: ip-proto-17

Capture from 10.200.48.41 (meraki):

17:29:47.213712 IP (tos 0x0, ttl 64, id 45498, offset 0, flags [+], proto UDP (17), length 1500)
    10.200.48.3.21237 > 10.200.48.41.12345: UDP, length 1473

Note that the completion packet never hit the wire!

Sending a UDP packet from PFSense 1472 bytes long succeeds
Sending a UDP packet from PFSense 8000 bytes long succeeds
Sending a UDP packet from PFSense 1480 bytes long succeeds
Sending a UDP packet from PFSense 1473 bytes fails
Sending a UDP packet from PFSense 1479 bytes fails
Receiving a UDP packet to PFSense of any size succeeds
Sending/receiving on other hosts succeed with the same peer
The issue is reproducible between two Netgate 6100s on the same segment (same behavior and captures, so I don't think this is a Meraki issue)

The application impact is that we're seeing intermittent failures of RADIUS. RADIUS issues go away when forcing the maximum frame size to 1200 on the RADIUS server (Framed-MTU), which is ok-ish, but not a general solution. Login seems to fail otherwise when the RADIUS packet is between 1473 and 1479 bytes long.

I haven't gotten an outage window to disable offloading to see if that fixes it. Running ifconfig ix1 -rxcsum -txcsum -tso -lro has not fixed it.

Feels like a bug: I presume somewhere there's an assumption every IP packet with protocol 17 will have a UDP header (8 bytes), which is not the case when fragment offset <> 0.

TL;DR: I can't send 1473 byte UDP packets from PFSense - they never fully hit the wire.

How can I fix this?

stephenw10

Any reason you're still on 25.11?

It certainly seems like a hardware off loading issue if the pcap sees it before the NIC hardware.

So a larger fragmented packet succeeds? And you see the fragments at both ends?

Can you try testing with one of the igc ports on the 6100?

mgaffigan

@stephenw10,

Thank you for the suggestion. We will try with one of the igc ports.

Larger packets fragment normally and show up at both ends, yes. Only that 7-byte range fails. I suspect it may also fail at multiples (2945-2951 ???), but we have not tested that.

Upgrades are set to "previous stable release" for cautionary purposes. When we get an outage, we will check on the latest version.

stephenw10

Hmm, struggling to replicate that so far. I'm testing on a 6100 running 26.03.1 and I see the packet and fragment arrive:

19:52:59.597626 IP (tos 0x0, ttl 64, id 15460, offset 0, flags [+], proto UDP (17), length 1500)
    172.21.16.246.46921 > 172.21.16.222.12345: UDP, length 1475
19:52:59.597632 IP (tos 0x0, ttl 64, id 15460, offset 1480, flags [none], proto UDP (17), length 23)
    172.21.16.246 > 172.21.16.222: ip-proto-17

However I note you're using a VLAN and that 8byte padding that adds seems suspiciously close to the failure range you are seeing. You might try disabling any VLAN offloading options.

I also note that fragmented UDP is generally bad. Since UDP cannot be guaranteed to arrive it's expected that all packets do not rely on other packets.

mgaffigan

@stephenw10

It's happing on both 6100's we have - but they are indeed both using VLAN's.

Re: UDP fragmentation, I agree that it is expected to have a higher than baseline error rate. The problem here is that it is a 100% error rate. Retransmissions happen - but fail repeatedly. I didn't choose to have SIP UDP be common, but I can't change that. Radius has better fragmentation controls, I grant.

mgaffigan

@stephenw10,

Just to confirm - you are testing transmission from a 6100 of the fragmented UDP packet while capturing on a second device?

We did not see any issues with the 6100 receiving fragmented UDP packets - only that sending would show in tcpdump but never hit the wire.

stephenw10

Yes, sending o n the 6100 using the default ix3 WAN:

19:52:59.599706 IP (tos 0x0, ttl 64, id 15460, offset 0, flags [+], proto UDP (17), length 1500)
    172.21.16.246.46921 > 172.21.16.222.12345: UDP, length 1475
19:52:59.599713 IP (tos 0x0, ttl 64, id 15460, offset 1480, flags [none], proto UDP (17), length 23)
    172.21.16.246 > 172.21.16.222: ip-proto-17

[26.03.1-RELEASE][admin@6100.stevew.lan]/root: ifconfig ix3
ix3: flags=1008a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: WAN
	options=48138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,HWSTATS,MEXTPG>
	ether 00:08:a2:12:17:81
	inet 172.21.16.246 netmask 0xffffff00 broadcast 172.21.16.255
	inet6 fe80::208:a2ff:fe12:1781%ix3 prefixlen 64 scopeid 0x8
	media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
	status: active
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

I tried a few packet sizes and they all completed. That should have failed if it's hitting the issue.

Received on another pfSense device using an igb NIC:

19:52:59.597626 IP (tos 0x0, ttl 64, id 15460, offset 0, flags [+], proto UDP (17), length 1500)
    172.21.16.246.46921 > 172.21.16.222.12345: UDP, length 1475
19:52:59.597632 IP (tos 0x0, ttl 64, id 15460, offset 1480, flags [none], proto UDP (17), length 23)
    172.21.16.246 > 172.21.16.222: ip-proto-17

[26.07-BETA][admin@m470.stevew.lan]/root: ifconfig igb0
igb0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: WAN
	options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
	ether 00:90:7f:d7:a1:94
	inet 172.21.16.222 netmask 0xffffff00 broadcast 172.21.16.255
	inet6 fe80::290:7fff:fed7:a194%igb0 prefixlen 64 scopeid 0x1
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=821<PERFORMNUD,AUTO_LINKLOCAL,STABLEADDR>

I guess the clocks are not perfect since it appears to have arrived before it was sent!

No VLANs in play. Yet.

mgaffigan

@stephenw10
We confirm the issue persists after upgrade to 26.03.01. We have not gotten to test other interfaces or without VLAN's yet.

stephenw10

That's still with VLAN interfaces on ix NICs?

mgaffigan

@stephenw10
Correct. We have not yet been able to attempt the following steps:

Disabling hardware offload in the UI
Checking the issue on the igb interfaces
Checking the issue without VLAN tags

stephenw10

OK I'll see if I can replicate it with a VLAN interface.