Web Configurator traffic is not being fragmented when it should be
-
Still seems odd that it shows those frames as larger than 1500B when the connection is working. That one frame is over 9000, I wouldn't expect that to pass anywhere.
When you captured it in pfSense itself do they show as that large in the tcpdump output?
It sounds like you have devices with different MTU values in the same L2 segment which is always asking for trouble IMO.
Steve
-
@stephenw10 OK, I really have definitely figured this out this time!! This is all to do with TCP segmentation offload in the NIC. It doesn't look like a bug in pfSense, although I still can't figure out why moving the L3 gateway to the firewall has fixed the issue for me. That may be more to do with the VLAN tag, but still an unknown, and one for later.
Basically, what's happening, is in Wireshark we are seeing the packet as it is when the kernel offloads the packet to the hardware. With tcp-segmentation-offload: on - this means we see the whole application layer PDU, in all it's massiveness, unsegmented. Once the hardware has the packet it performs the TCP segmentation of the application layer PDU (plus any other hardware functions that have been enabled, like checksum calculation) and sends the multiple segments out on the wire.
Generally, higher-end NICs have lots of functions that can be offloaded to the hardware, but lower-end NICs, like the one you would get in a laptop, don't.
Both the pfSense FW, and the Sandbox server, have higher-end NICs. The pfSense has an Intel X710 2x10G NIC, and the Sandbox server on which I was doing the snooping, has an Intel X520.
I checked and, sure enough, both have offloading enabled!
Here's the setting in pfSense Advanced settings:
And here's the ethtool output from our Sandbox server:
So, in summary - The packets on the wire are compliant with the interface MTU for the network (i.e. 1500 bytes) - I just can't see them, as I'm snooping what the kernel sees, rather than what the hardware is outputting.
Sorry to have wasted time with a non-issue, but I certainly learned something along the way anyhow!
-
@othomas said in Web Configurator traffic is not being fragmented when it should be:
rather than what the hardware is outputting.
@johnpoz said in Web Configurator traffic is not being fragmented when it should be:
where your sniffing before it was handed off to the nic and the nic did the fragmentation
Just saying ;) heheheh
BTW - IMHO, jumbo have very little use in a normal network - they can make sense in a specific SAN network, etc.. where your moving large files between specific device and its storage..
@stephenw10 said in Web Configurator traffic is not being fragmented when it should be:
e devices with different MTU values in the same L2 segment which is always asking for trouble IMO.
Completely in agreement with Steve's statement here..
-
Ah, nice catch! Yeah that's disabled by default in pfSense. We hardly ever see it.
-
@stephenw10 said in Web Configurator traffic is not being fragmented when it should be:
Sounds like you have devices with different MTU values in the same L2 segment which is always asking for trouble IMO.
Slightly surprised at this as down at L2 is really the only place I would expect to accommodate jumbo frames in a mixed-traffic environment.
With the rise of 10 GbE clients running 9k frames is more normalised and, on some systems, specifically called for. I have both servers and clients using 10 GbE and 9k frames on a subnet that also hosts devices at 1 GbE and below with regular 1500 MTU packets. Never expected an issue, never experienced an issue to date, including with pfSense routing them across subnets or VLANs.
(I have seen hiccups with 'L3' switches though but that's kinda out-of-scope here.)
️ -
Well it could be that I only ever see problems.
But I have worked many tickets with users blaming pfSense for blocking traffic that never even went through pfSense and turned out to be an MTU mismatch between the clients. Maybe just cynical but I would avoid mixing MTUs if possible. -
@stephenw10 said in Web Configurator traffic is not being fragmented when it should be:
ell it could be that I only ever see problems
Its not just you.. Devices on the same L2 with different mtu is going to be problematic that is for sure.. if you have devices that never need to talk to each other - then you prob wont see a problem.
If they are on different network and through a router then the L3 should fix it up for you..
-
@stephenw10 said in Web Configurator traffic is not being fragmented when it should be:
Well it could be that I only ever see problems.
Yeah and medics only meet the sick & wounded. Who knew?
My home network above is not massive, almost fits on a page, even with working-from-home stuff. Typically there is less than 50 devices and often much lower than that; but there is a mix of old and new, IoT to servers, UPS to NTP, Windows (ugh) to 'nix, protocols to proprietary, NASes to thermostats and all working perfectly at their preferred MTUs.
Thankfully my Netgate device joined this melee without a murmur or a dropped packet. No complaints from this direction, even at L2.
️ -
Just saying ;) heheheh
Yeh fair comment, need to read replies better!
On MTU, my experience is any change from the default 1500 causes major headaches, especially when you work for a service provider network (I.e a telco network) and you're carrying user data, which always originated on a device with 1500 mtu.
The fun really starts when you start wrapping that data in lots of tunnels!
In one case in my last company we had,
User data TCP/IP over IPSEC, over IPSEC (again), then over GTP-U. We had to do MSS clamping down to 1280 bytes. -
@othomas said in Web Configurator traffic is not being fragmented when it should be:
User data TCP/IP over IPSEC, over IPSEC (again), then over GTP-U. We had to do MSS clamping down to 1280 bytes.
ugghh - yeah that isn't fun..
