Jumbo Frame on i211/igb adapters not working

rmac1813

I need to use frame sizes >1500bytes (overlay networking), and everything I've read in regards to the Intel i211 interfaces say jumbo frame is supported. However I cannot get a ping response >1500 bytes on these interfaces.

interface config (VLAN enabled):

The switch is passing jumbo frames just fine to pfsense as a VM, but this hardware isint working..

JKnott

@rmac1813 said in Jumbo Frame on i211/igb adapters not working:

However I cannot get a ping response >1500 bytes on these interfaces.

Try doing a packet capture while pinging with a large packet, perhaps 8960 bytes. Run Wireshark on the target to see if the ping gets that far. I assume you also configured jumbo frames on the target.

Also, can your switch handle jumbo frames?

rmac1813

@jknott the switch is configured for MTU size 12288, and will pass large frames to a pfsense VM all day.

i connected directly to the LAN interface (same settings) on the pfsense device. no icmp response.

Are there any settings I should check on pfsense CLI?

johnpoz

@rmac1813 Your ping command you show there isn't even setting DF.. And the interfaces you show, I don't see a 192.168.8 address on them.

I can ping 9000 bytes to my lan just fine, when I don't set DF.. It is a i211 igb interface on a sg4860

$ ping 192.168.9.253 -l 9000

Pinging 192.168.9.253 with 9000 bytes of data:
Reply from 192.168.9.253: bytes=9000 time<1ms TTL=64
Reply from 192.168.9.253: bytes=9000 time<1ms TTL=64
Reply from 192.168.9.253: bytes=9000 time<1ms TTL=64
Reply from 192.168.9.253: bytes=9000 time<1ms TTL=64

Ping statistics for 192.168.9.253:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

I would have to take down my network to test setting a mtu of 9000 on pfsense lan. My switch is currently set to not pass jumbo, and changing that would require a complete reboot of the switch.. Maybe tmrw I can plug directly into the port and set mtu on it and my machine for test what wouldn't be all that disruptive.

But your tests there don't seem to make any sense to me.. If testing if you can pass a specific size without fragmenting it, you would need to set DF on your ping.. normally with -f

$ ping 192.168.9.253 -l 9000 -f

Pinging 192.168.9.253 with 9000 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.

JKnott

@rmac1813

Have you tried any packet captures, so you know what's happening?

JKnott

@johnpoz said in Jumbo Frame on i211/igb adapters not working:

I can ping 9000 bytes to my lan just fine, when I don't set DF.

That's not a valid test for passing jumbo frames. With Linux, DF is set for everything and for TCP on Windows. So, if you don't set DF, your pings will get fragmented and pass. Of course, the WAN side of pfSense is unlikely to be able to pass jumbo frames, unless it's connected to Internet2 or equivalent.

johnpoz

@jknott said in Jumbo Frame on i211/igb adapters not working:

That's not a valid test for passing jumbo frames.

No shit ;) Which is why I brought it up - if you see his ping command he is not setting DF.

If going to test from say a linux box or freebsd, doing MTU discovery with -M, like something like this

user@NewUC:~$ ping -M do -s 8972 192.168.2.253
PING 192.168.2.253 (192.168.2.253) 8972(9000) bytes of data.
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500

If I don't do that then I can ping with that size just fine

user@NewUC:~$ ping -s 9000 192.168.2.253
PING 192.168.2.253 (192.168.2.253) 9000(9028) bytes of data.
9008 bytes from 192.168.2.253: icmp_seq=1 ttl=64 time=1.45 ms
9008 bytes from 192.168.2.253: icmp_seq=2 ttl=64 time=0.588 ms

On a work call at the moment so can't do a valid test for mtu, but I do have console access to pfsense and can move my machine cable directly to a i211 interface (lan on my sg4860) and do testing..

JKnott

@johnpoz

Shortly after I posted that, I checked and found pfSense/FreeBSD sets DF on TCP but not ping.

johnpoz

@jknott said in Jumbo Frame on i211/igb adapters not working:

pfSense/FreeBSD sets DF on TCP but not ping.

but he clearly pinging from a windows box anyway.. you can see from his c:\ prompt ;)

JKnott

@johnpoz

I just noticed something curious. Ping linux to pfsense, DF is set in both directions. Ping pfsense to linux and it's not set in either direction.

Given that fragmenting is a bad idea, I'm surprised Windows and FreeBSD use DF only on TCP.

johnpoz

@rmac1813 here I moved my pc cable directly to the lan interface (igb0 which is a i211)

I set its mtu to 9000, and then pings from my windows machine with DF set and 8972, ping works fine. If I try 8973 with DF set, see that it fails..

Also set jumbo on my windows machine to 9014

To run it through my switch would take a network outage.. Since to enable it requires reboot, but this shows setting large mtu works just fine on igb and i211 interface in pfsense.

rmac1813

@johnpoz Aye, it was the windows intel nic driver blocking the communication. The pfsense device is now responding to ping. Appreciate you guys' responses here.

(now the jumbo frame problem has moved to a lag interface on a mikrotik downstream switch in my lab..smh)

johnpoz

@rmac1813 said in Jumbo Frame on i211/igb adapters not working:

windows intel nic driver blocking the communication

Not sure would use the word "blocking" more like two people that speak different languages - might have some common words, or both know simple words of each.. Might be able to ask for a beer, or say thank you and hello.. But can't really hold a meaningful conversation.

Jumbo can be a real PITA, and needs to be really consistent with all devices. Not exactly sure what you mean when you say "(overlay networking)" but would make sure that whatever your doing actually requires jumbo.. In my experience is almost always a bad idea, and rarely worth the effort and hassle in the bigger picture.

But glad you got this specific question sorted.

If I had to sum up use of jumbo
maybe slight performance increase due to less header overhead, but more complex, config pita and then don't forget extra effort in troubleshooting when you run into x can not talk to y - since now it could be a mtu mismatch.. I don't really see where it makes sense, other than very specific use case - like on a san between your device and its storage, etc.

rmac1813

@johnpoz
"blocking" .. meaning my intel adapter advanced tab (as shown in your screen cap) had '1514 bytes', blocking frames with larger MTU from passing.

In this environment vxlan/geneve is being used for overlay networking, which requires larger L2 MTU, for TEP headers.

There are 3 switches, 2 of which are passing >1500byte frames, and now the MikroTik CRS310 is the only device not passing the larger frames.
Thanks again

JKnott

@johnpoz

Jumbo frames are often used within data centres and are default with 10Gb and faster networks. I mentioned Internet2. This is used for connecting research sites, just as the Internet was originally and it supports jumbo frames. If you're moving a lot of data, you want jumbo frames.

BTW, when I first started using the Internet, my MTU was 576 bytes on a SLIP connection with static IP.

The first time I worked at IBM we had a token ring network with 4K(?) MTU.

rmac1813

@jknott
not to beat a dead horse, but data centers is not the only place jumbo is used.
overlay is also at the edge now, and software platforms like NSX, K8's CNIs (Antrea/Calico/Flannel etc), all use overlay

JKnott

@rmac1813

I just gave that as an example. I also mentioned Internet2 as another. Many people have yet to move on from 1500 byte MTU and even struggle with frame expansion, to allow VLANs etc., which happened years ago.

My first work experience with IP was on token ring, which supported much larger MTU.

With even home Internet connections over 1 Gb (my ISP recently announced plans for 8 Gb) I wonder how long before ISPs start allowing larger MTU. A few years ago, I first came across a 10 Gb connection in my work. It was for a major bank's data centre.