Jumbo Frame on i211/igb adapters not working

JKnott

Have you tried any packet captures, so you know what's happening?

JKnott

@johnpoz said in Jumbo Frame on i211/igb adapters not working:

I can ping 9000 bytes to my lan just fine, when I don't set DF.

That's not a valid test for passing jumbo frames. With Linux, DF is set for everything and for TCP on Windows. So, if you don't set DF, your pings will get fragmented and pass. Of course, the WAN side of pfSense is unlikely to be able to pass jumbo frames, unless it's connected to Internet2 or equivalent.

johnpoz

@jknott said in Jumbo Frame on i211/igb adapters not working:

That's not a valid test for passing jumbo frames.

No shit ;) Which is why I brought it up - if you see his ping command he is not setting DF.

If going to test from say a linux box or freebsd, doing MTU discovery with -M, like something like this

user@NewUC:~$ ping -M do -s 8972 192.168.2.253
PING 192.168.2.253 (192.168.2.253) 8972(9000) bytes of data.
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500

If I don't do that then I can ping with that size just fine

user@NewUC:~$ ping -s 9000 192.168.2.253
PING 192.168.2.253 (192.168.2.253) 9000(9028) bytes of data.
9008 bytes from 192.168.2.253: icmp_seq=1 ttl=64 time=1.45 ms
9008 bytes from 192.168.2.253: icmp_seq=2 ttl=64 time=0.588 ms

On a work call at the moment so can't do a valid test for mtu, but I do have console access to pfsense and can move my machine cable directly to a i211 interface (lan on my sg4860) and do testing..

JKnott

@johnpoz

Shortly after I posted that, I checked and found pfSense/FreeBSD sets DF on TCP but not ping.

johnpoz

@jknott said in Jumbo Frame on i211/igb adapters not working:

pfSense/FreeBSD sets DF on TCP but not ping.

but he clearly pinging from a windows box anyway.. you can see from his c:\ prompt ;)

JKnott

@johnpoz

I just noticed something curious. Ping linux to pfsense, DF is set in both directions. Ping pfsense to linux and it's not set in either direction.

Given that fragmenting is a bad idea, I'm surprised Windows and FreeBSD use DF only on TCP.

johnpoz

@rmac1813 here I moved my pc cable directly to the lan interface (igb0 which is a i211)

I set its mtu to 9000, and then pings from my windows machine with DF set and 8972, ping works fine. If I try 8973 with DF set, see that it fails..

Also set jumbo on my windows machine to 9014

To run it through my switch would take a network outage.. Since to enable it requires reboot, but this shows setting large mtu works just fine on igb and i211 interface in pfsense.

rmac1813

@johnpoz Aye, it was the windows intel nic driver blocking the communication. The pfsense device is now responding to ping. Appreciate you guys' responses here.

(now the jumbo frame problem has moved to a lag interface on a mikrotik downstream switch in my lab..smh)

johnpoz

@rmac1813 said in Jumbo Frame on i211/igb adapters not working:

windows intel nic driver blocking the communication

Not sure would use the word "blocking" more like two people that speak different languages - might have some common words, or both know simple words of each.. Might be able to ask for a beer, or say thank you and hello.. But can't really hold a meaningful conversation.

Jumbo can be a real PITA, and needs to be really consistent with all devices. Not exactly sure what you mean when you say "(overlay networking)" but would make sure that whatever your doing actually requires jumbo.. In my experience is almost always a bad idea, and rarely worth the effort and hassle in the bigger picture.

But glad you got this specific question sorted.

If I had to sum up use of jumbo
maybe slight performance increase due to less header overhead, but more complex, config pita and then don't forget extra effort in troubleshooting when you run into x can not talk to y - since now it could be a mtu mismatch.. I don't really see where it makes sense, other than very specific use case - like on a san between your device and its storage, etc.

rmac1813

@johnpoz
"blocking" .. meaning my intel adapter advanced tab (as shown in your screen cap) had '1514 bytes', blocking frames with larger MTU from passing.

In this environment vxlan/geneve is being used for overlay networking, which requires larger L2 MTU, for TEP headers.

There are 3 switches, 2 of which are passing >1500byte frames, and now the MikroTik CRS310 is the only device not passing the larger frames.
Thanks again

JKnott

@johnpoz

Jumbo frames are often used within data centres and are default with 10Gb and faster networks. I mentioned Internet2. This is used for connecting research sites, just as the Internet was originally and it supports jumbo frames. If you're moving a lot of data, you want jumbo frames.

BTW, when I first started using the Internet, my MTU was 576 bytes on a SLIP connection with static IP.

The first time I worked at IBM we had a token ring network with 4K(?) MTU.

rmac1813

@jknott
not to beat a dead horse, but data centers is not the only place jumbo is used.
overlay is also at the edge now, and software platforms like NSX, K8's CNIs (Antrea/Calico/Flannel etc), all use overlay

JKnott

@rmac1813

I just gave that as an example. I also mentioned Internet2 as another. Many people have yet to move on from 1500 byte MTU and even struggle with frame expansion, to allow VLANs etc., which happened years ago.

My first work experience with IP was on token ring, which supported much larger MTU.

With even home Internet connections over 1 Gb (my ISP recently announced plans for 8 Gb) I wonder how long before ISPs start allowing larger MTU. A few years ago, I first came across a 10 Gb connection in my work. It was for a major bank's data centre.