Jumbo frames?

JKnott

While checking something else with Wireshark, I see some very long frames. When I checked the Ethernet config for my computer (OpenSUSE 15.3) it showed the MTU as 0, which I suspect means unlimited. While some of the frames are from addresses entirely within my network, some are not. However, I don't see any large frames on the WAN side. IPv6 doesn't fragment, so I don't know what's happening there.

Here's an example:

There is also this from the same frame:

Ideas?

johnpoz

@jknott where exactly did you sniff that? With offload you could see large frames like that when sniffed on the host sending, because the breakup of the frame is to be done by the nic. Ie offloaded - "TCP Large Segment Offload"

stephenw10

I assume that is a v6 packet?

TCP segmentation is not the same as packet fragmentation though.

JKnott

@johnpoz

It was on the computer at one end of the connection. The number that caught my eye was frame size, which I assume would be what's on the wire. With IPv6, IP packets, not frames, can be fragmented only by the source. I'll have to try a capture on the LAN side of pfSense, for the ones that go elsewhere. I wasn't familiar with that offload. Also, this is on Linux, not Windows.

JKnott

@stephenw10

I'm well aware of packet fragmentation and TCP segmentation, but had never heard of frames being fragmented. I have experimented with jumbo frames in the past and I believe my switch can pass 16K frames.

JKnott

@jknott

I just tried again. While I see jumbo frames on the Linux computer, where I'm running Wireshark, I don't see any such frames on the firewall LAN interface. However, it has the default (1500) MTU, so I'll have to try again later with my "data tap" monitoring what's on the wire.

stephenw10

If that's on the same layer2 I would expect to need the same MTU throughout.

johnpoz

@jknott said in Jumbo frames?:

so I'll have to try again later with my "data tap" monitoring what's on the wire.

As I was saying - if your doing offloading, the nic can be responsible for breaking it up to the size of the mtu on the wire.

If you sniff with a tap and actually off the wire, then you will see the size of the mtu..

I was just showing where you can enable or disable that in windows because its handy - but same thing can be done with linux.

user@NewUC:~$ ethtool --show-offload ens3 | grep offload
tcp-segmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
user@NewUC:~$

https://meet-unix.org/2017-02-19-tcp-lso.html

LSO is used by modern operating systems like GNU/Linux. LSO is a performance capability where TCP segments, when transmitted from the network stack to the network adapter, melted to a large segment, often bigger then the MTU from the network adapter. This behaviour speeds up the transition a lot. The opposite of LSO (Large Send Offload) is called LRO (Large Receive Offload).

When you sniff on a host - while you think your seeing what is put on the wire, but that is not actually the case when offloading is being done.. Features are offloaded to the nic to handle - which can have some great performance increases.

And typically what you sniff at the host is what being put on the wire - your pulling what is going to the nic, and will be put on the wire. But if the nic is doing offloading - it could look a bit different in say wireshark, say before the segmentation happens which you are offloading to the nic to handle.

https://en.wikipedia.org/wiki/TCP_offload_engine#Large_send_offload

Large send offload

In computer networking, large send offload (LSO) is a technique for increasing egress throughput of high-bandwidth network connections by reducing CPU overhead. It works by passing a multipacket buffer to the network interface card (NIC). The NIC then splits this buffer into separate packets. The technique is also called TCP segmentation offload (TSO) or generic segmentation offload (GSO) when applied to TCP. LSO and LRO are independent and use of one does not require the use of the other.

When a system needs to send large chunks of data out over a computer network, the chunks first need breaking down into smaller segments that can pass through all the network elements like routers and switches between the source and destination computers. This process is referred to as segmentation. Often the TCP protocol in the host computer performs this segmentation. Offloading this work to the NIC is called TCP segmentation offload (TSO).

For example, a unit of 64 KiB (65,536 bytes) of data is usually segmented to 45 segments of 1460 bytes each before it is sent through the NIC and over the network. With some intelligence in the NIC, the host CPU can hand over the 64 KB of data to the NIC in a single transmit-request, the NIC can break that data down into smaller segments of 1460 bytes, add the TCP, IP, and data link layer protocol headers — according to a template provided by the host's TCP/IP stack — to each segment, and send the resulting frames over the network. This significantly reduces the work done by the CPU. As of 2014 many new NICs on the market support TSO.

JKnott

@johnpoz

That's probably it then.

I'd better get back to my beer.

JKnott

@stephenw10

Or more precisely, don't send a frame that exceeds the recipients maximum size. There's nothing in an Ethernet frame that says what the MTU is.