How change NIC MTU to 9000?

yon 0

I want to try setup NIC MTU to 9000 for fast.

but i try gui and ifconfig setup this not work. How i do it right? re0 is LAN igb0 is wan

re0: flags=88843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,STATICARP> metric 0 mtu 1500
options=82198<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,LINKSTATE>
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4407b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,TXCSUM_IPV6>

stephenw10

There's a high chance of hitting some sort of issue doing this. But if you're just testing...

How exactly are you applying this?

Do you see any log entries when you do?

The behaviour is the same on both interfaces? re NICs do not enjoy a good reputation in pfSense/FreeBSD.

Steve

JKnott

@stephenw10 said in How change NIC MTU to 9000?:

There's a high chance of hitting some sort of issue doing this.

Why should there be issues? Any gigabit gear should be capable of supporting jumbo frames. I've seen some that supports up to 16,000 MTU,

stephenw10

Well I could biased by the fact people only ever come to me with problems!

If one thing in that segment doesn't support that MTU or, worse, claims to support it but is actually broken then you'll see problems and could have many happy hours tracking them down. Or it could go completely smoothly...

If it just a test then go for it.

Steve

tim.mcmanus

@stephenw10 said in How change NIC MTU to 9000?:

Well I could biased by the fact people only ever come to me with problems!

If one thing in that segment doesn't support that MTU or, worse, claims to support it but is actually broken then you'll see problems and could have many happy hours tracking them down. Or it could go completely smoothly...

If it just a test then go for it.

Steve

MTU also has to be supported by every device on the chain. I worked with a customer once who set their device for no fragmentation and they couldn't understand why everything stopped working when it hit a device that didn't support a larger MTU.

I also find that most people don't understand or have a good reason for a high MTU. Would you send a water truck to be filled by a glass of water, move it to its destination? No, that's absurd. So why set high MTUs to move tiny bits of data? Unless you're moving giant loads of data, there's no reason for a larger MTU.

JKnott

@stephenw10 said in How change NIC MTU to 9000?:

Well I could biased by the fact people only ever come to me with problems!
If one thing in that segment doesn't support that MTU or, worse, claims to support it but is actually broken then you'll see problems and could have many happy hours tracking them down. Or it could go completely smoothly...

As I mentioned, any gigabit gear should handle it, though WiFi, will get in the way of 9000. The 802.11 spec supports 2304, but the equipment might not.

If something's broken, toss it.

JKnott

@tim-mcmanus said in How change NIC MTU to 9000?:

MTU also has to be supported by every device on the chain. I worked with a customer once who set their device for no fragmentation and they couldn't understand why everything stopped working when it hit a device that didn't support a larger MTU.
I also find that most people don't understand or have a good reason for a high MTU. Would you send a water truck to be filled by a glass of water, move it to its destination? No, that's absurd. So why set high MTUs to move tiny bits of data? Unless you're moving giant loads of data, there's no reason for a larger MTU.

The trend is to not fragment. That's manatory on IPv6 and IPv4 is moving that way. If the DF flag is set, the router is supposed to send an ICMP error message back to the source, specifying the maximum supported MTU.

One common application for jumbo frames is in data centres, where there is a lot of data moving around the LAN. Of course, at the moment, the Internet doesn't support anything beyond standard MTUs, but that's a configuration issue, as modern hardware generally supports jumbo frames.

Years ago, token ring was often used for LANs and it supported over 4K MTU.

The 1500 byte MTU originated with Ethernet back in the dark ages. One of the reasons for that limit was hardware cost, at a time when the controllers were built with discrete logic. There was also the trade off between performance and reliability in collision type networks. i.e. larger lost frames means more retransmission. Neither of these is a factor now.

tim.mcmanus

@jknott said in How change NIC MTU to 9000?:

One common application for jumbo frames is in data centres, where there is a lot of data moving around the LAN. Of course, at the moment, the Internet doesn't support anything beyond standard MTUs, but that's a configuration issue, as modern hardware generally supports jumbo frames.

I agree. But even then you have to understand what the traffic you're passing looks like. If it's a bunch of web traffic or messages between systems, do you really need a high MTU? Even database traffic should be scrutinized to determine whether or not you're moving significant chunks of data between servers. It could be tiny bits of data.

Good discussion, thanks!

JKnott

@tim-mcmanus said in How change NIC MTU to 9000?:

If it's a bunch of web traffic or messages between systems, do you really need a high MTU? Even database traffic should be scrutinized to determine whether or not you're moving significant chunks of data between servers. It could be tiny bits of data.

The thing about MTU is it means maximum size. It doesn't say you can't send smaller. In fact the smallest supported frame has a payload of 46 bytes. With Ethernet and IP, you can use whatever packet size is appropriate for the data. So, if you're sitting at a keyboard, connected via ssh, you will be sending out a lot of very small packets. But there is nothing to stop those small packets from sharing a network with those huge packets in the data centre. The advantage of the large packets is better performance with large amounts of data, be it in routers or end computers.

A few years ago, I was reading an article about that CERN accelerator, where that Higgs bosson was discovered. Due to the huge amounts of data being shipped around the world, they were using jumbo frames over very high speed links. So, there's no technical reason why IP with 9000 byte MTU couldn't eventually be made available on the Internet. The hardware supports it now and has for several years.

JKnott

Incidentally, here's something about a research network in Ontario, called "Orion". It's part of the CANARIE network that connects Canada and is in turn connected to similar networks around the world. This is a second Internet that is built to support up to 9000 MTU.

JKnott

@jknott said in How change NIC MTU to 9000?:

The trend is to not fragment. That's manatory on IPv6 and IPv4 is moving that way. If the DF flag is set, the router is supposed to send an ICMP error message back to the source, specifying the maximum supported MTU.

One thing I've noticed is Linux sets DF for TCP, UDP and ICMP. Windows only does for TCP.

yon 0

@tim-mcmanus said in How change NIC MTU to 9000?:

如果它是系统之间的一堆网络流量或消息，你真的需要高MTU吗？应仔细检查数据库流量，以确定您是否在服务器之间移动了大量数据。它可能是微小的数

ubuntu can easy setup it, but i don't know the Pfsense how do it. i use these server for BGP transit, so need fast network

and by the way, i have test setup the txqueuelen 10000 config, helpful fast network speed. test in ubuntu and openvpn.

but the pfsense has like this txqueuelen setup ?

stephenw10

So what have you tried that doesn't work? You should just be able to set the interface MTU in the gui.

Steve

JKnott

I just tried setting the MTU in the GUI and found it works. Here's what ifconfig shows for the interface before I changed it:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether 90:e2:ba:4d:d6:b3
hwaddr 90:e2:ba:4d:d6:b3
inet6 fe80::92e2:baff:fe4d:d6b3%em0 prefixlen 64 scopeid 0x2
inet6 fd48:1a37:2160:4::1 prefixlen 64
inet 172.16.4.1 netmask 0xffffff00 broadcast 172.16.4.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier

And after

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether 90:e2:ba:4d:d6:b3
hwaddr 90:e2:ba:4d:d6:b3
inet6 fe80::92e2:baff:fe4d:d6b3%em0 prefixlen 64 scopeid 0x2
inet6 fd48:1a37:2160:4::1 prefixlen 64
inet 172.16.4.1 netmask 0xffffff00 broadcast 172.16.4.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier

As you can see, the MTU changed from 1500 to 9000.

yon 0

@jknott ..

i have try setup in interfaces.php?if=opt3 but it not work. still MTU is 1500

stephenw10

Ok, we are going to need to see the opt3 config section and the ifconfig output for that interface. And any logs you have showing the change or any errors.

Steve

yon 0

@stephenw10
alt text
now i have try setup LAN nic change to MTU 9000, but still show 1500 MTU no log show.

alt text

Grimson

From the FreeBSD manual at: https://www.freebsd.org/cgi/man.cgi?query=re&sektion=4&manpath=FreeBSD+11.1-RELEASE

The 8169, 8169S and 8110S also support jumbo frames, which can be configured via the interface MTU setting. The MTU is limited to 7422, since the chip cannot transmit larger frames. Selecting an MTU larger than 1500 bytes with the ifconfig(8) utility configures the adapter to receive and transmit jumbo frames.

So check what kind specific Realtek chipset you have, but 9000 will simply not work with crappy hardware like this.

yon 0

@grimson

i have try intel NIC still the same wrong.

stephenw10

I would also guess an re issue there. Can you set the MTU from the CLI?:

[2.4.4-DEVELOPMENT][root@apu.stevew.lan]/root: ifconfig re2 mtu 9000
[2.4.4-DEVELOPMENT][root@apu.stevew.lan]/root: ifconfig re2
re2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
	options=82098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
	ether 00:0d:b9:37:30:12
	hwaddr 00:0d:b9:37:30:12
	inet 192.168.35.1 netmask 0xffffff00 broadcast 192.168.35.255 
	inet 19.168.35.254 netmask 0xffffff00 broadcast 19.168.35.255 vhid 1 
	inet6 fe80::1:1%re2 prefixlen 64 scopeid 0x3 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (none)
	status: no carrier
	carp: INIT vhid 1 advbase 1 advskew 0

Steve