Is pfSense handling jumbo frames correct !??

[johnpoz]

@JKnott This is what he has from my understanding

pc 9k --- 1500 pfsense 1500 --- 9k truenas

but simple ping from pc to nas, unless the ping size was adjusted and set to not fragment should work..

But trying to send 9k through something like that is never going to work.

And he also says his ping from pfsense to the devices don't work - he got something else going on for sure..

Why anyone thinks jumbo makes any sense expect for specific use cases is beyond me..

Leave your stuff on default 1500, what do you get for your 10ge performance?

[louis2]

@johnpoz

I did find a IMHO very good description about MTU's from HUAWEI

https://support.huawei.com/enterprise/en/doc/EDOC1100202534

And IMHO if TrueNAS and pfSense behave in the optimal way it should work.

However, it actually does not .... (as described)
.
One of the things is that pfSense is blocking the traffic even with the following extreme test rules

[louis2]

@JKnott

Yep I agree I should set the MTU size for the involved LAN's to 9000 and to e.g. the WAN on 1500.
However ..... HOW ???

I can set / reduce the MTU and MSS in the VLAN defs. But I have the raise the TRUNK MTU to 9000 first, and I do not have any Idea how to do that

[johnpoz]

@louis2 this rule is not even being triggered

And your rule from source to the same network is pretty pointless.. pfsense has nothing to do with traffic between devices on the same network.. That is not very explicit rule that would only allow traffic to pfsense IPs on that interface..

And I fully understand what jumbo are, and how they work - and they make no sense unless its a specific scenario.. And what your wanting to accomplish is pointless.. If you want to use jumbo between your pc and your nas, create a SAN network where they talk to each other over different interfaces.

Some box wanting to use jumbo to talk to other stuff on other network, or say the internet is only going to cause huge problems and at best cause your L3 router to do a lot of work for no reason, changing some jumbo that comes in trying to go out to the internet.. Because your internet sure isn't jumbo..

[louis2]

@johnpoz

YEP very strange strange !!!!
I think that the packages are not handled due to unexpected MTU size

Related to your earlier post

Also a ping does not work, see my original mail

• does not work from the PC (IPV4 and IPV6)
• does not work using pfsence ping test option, but with strange reaction on IPV6

Note that as expected, I can ping between equipment on the same LAN.
AND I can also ping from the 9K PC to e.g. google (like it should)

For info:
the 10G card is a Mellanox ConnectX3 at the PC side
an Intel X520 in pfSense
and an X520 in TrueNas Scale system
and an other Mellanox in my TrueNas core system
the switch is a TP-Link SX3008F

[louis2]

@johnpoz

I did make mistakes in the TEST rules. Ping is working now from the PC (IP4 and IP6)

GUI etc not yet

[JKnott]

@louis2 said in Is pfSense handling jumbo frames correct !??:

But I have the raise the TRUNK MTU to 9000 first, and I do not have any Idea how to do that

I assume you're referring to a switch. There generally isn't a setting for that. It's whatever the switch is capable of supporting. I have a TP-Link switch that can support up to 16K, but my Cisco switch is only 1518, though some Cisco switches can be set to larger.

[louis2]

@JKnott

No, I was not referring to the switches. Switches usually have a clear setting for max frame size. I simply have no idea where to change the MTU-sizes for the pfSense NIC's / Lagg's !!

In the Interfaces/Interfaces menu you can set the MTU and MSS sizes, but of course only to a lower value than those defined for the carrier interfaces above those lan/vlan's

Next problem is what those values should be .......

If the famous "1500" and "9000" values are intended as netto values than 1518 and 9018 sounds logical .(+ ethernet header = 14 and crc = 4) ..... however, every device has its own options, where it is completely unclear if the value should be read as:

• net MTU
• net MTU + header
• net MTU + header + crc

And than we have:
My MX3 cards have a default value of 1514 could be set to e.g. 9014 (or something else
My linksys switch has as option 1518 - 9216
My Realtek NIC's have a default of "off I assume 1514) and a choice for 9014
My TrueNas system has a default of 1500 and suggest 9000 as option
Etc
Next what value to take for MSS (default?).
Whatever, the huge number of packages transferred per second via a 10G-link is ‘extreme’. With as consequence that even very small delay, as introduced by the firewall, will have a dramatic impact on the possible transfer speed. So reducing the package count with a factor six will probably lead to a very significant higher throughput.
For that reason, I sincerely consider to change the package size to ^jumbo^.
Note that

• All switches between the PC/NAS/etc and pfSence do support jumbo frames
  (which they do in my case)
• An that pfSense and other devices should be capable to negotiate the max-mtu size with the destination device.
• For the packages coming / generated by the destination device there is no problem as long as the packages a smaller than the package size the switches and pfSense can handle.

[A Former User]

@louis2

Here's my understanding of it.

When it says MTU, it would be usually either 1500 or 9000. When it says Frame Size, it would be MTU size + 18, so 1518 or 9018. However, some NIC drivers let you select 9014 instead of 9018, but that means really the same. Also, some NIC drivers try to make it user-friendly and show 9K instead. Again, it means the same.

pfSense says MTU, so it would be 1500 or 9000. If MTU is set on a parent interface (NIC), all child interfaces inherit that setting by default. That can be changed per child interface to a lower value, but I think it takes effect after pfSense is restarted.

In general, MTU can be set up on a L2 or L3 interface. For L2, it would be a port/NIC. For L3, it would be a SVI/VLAN interface.

I don't think there is a thing like a LAG MTU.

[louis2]

@kjk54

I have the same feeling, but its stays confusing !

• Realtek one choose 9014 => 9014
• Mellanox .... default is 1514 => 9014
• Netgear one size 16349 => 16349
• TP-link ^jumbo 1518 – 9216^ => 9018
• zyxel (most switches) Jumbo frame is enabled by default, with support for frame sizes of up to 9K.
• TrueNas free format however the text is suggesting 1500 or 9000 => 9000
• windows depending om the used interface
• etc (if not clear => 9018)

That does not take away that will work or is peace of cake or will instantly work.
Show my testing in the past days .....

On 1-GB vlan's and vlans which do not require high performance and the WAN, I will set the MTU to 1500
on the others 9000

Problem is that I still have no idea how to set the MTU on my pfSense physical interfaces.
(appart from the WAN-interface all other interfaces shown are vlans not the underlaying physical interfaces)

[A Former User]

@louis2 said in Is pfSense handling jumbo frames correct !??:

Problem is that I still have no idea how to set the MTU on my pfSense physical interfaces.

In pfSense WebUI, go to Interfaces>Interface Assignments and select a parent interface, say LAN. You can enter MTU for it in the General Configuration section. The interface configuration will also show you the corresponding NIC. If you want to see how it really plays, SSH into pfSense shell and enter ifconfig | grep mtu.

I have jumbo frames enabled on all my switches, CISCO and Netgear, but I do not actually have any practical use-case for it. Although I have two NAS servers, they are of a general kind. At one point, I thought about setting up a TrueNAS server for my ESXi host but opted out for a hardware RAID instead. That was the time I was actually thinking about using jumbo frames.

[JKnott]

@louis2 said in Is pfSense handling jumbo frames correct !??:

If the famous "1500" and "9000" values are intended as netto values than 1518 and 9018 sounds logical .(+ ethernet header = 14 and crc = 4)

No, they don't include header & CRC. Frame expansion started in the late 90s, to allow for things like VLAN tags, etc.. Later on came jumbo frames. As I have never worked with LAGG, I can't say for certain, but I wouldn't be surprised if it's MTU is dependent on the underlying interface, as is the case with VLANs. So, you set the interface to 9K or whatever and the LAGG follows suit. Give it a try and see what happens. The ifconfig command shows the MTU.

[louis2]

@JKnott

To take away some misunderstanding, the situation is as follows:

• The PPOE-interface is on em0. I I think that one should use the normal 1500 MTU. So nothing to do there
• Then there is a lag using two 1G-Intel NIC's. That Lagg is the trunk for all 1G-VLAN's
  (I probably leave the MTU to 1500 there)
• And last but not least there is a X520-dual-10G intel card in favor of the 10G-lagg. The trunk for all 10G-VLAN's
  (I will change the MTU to 9000 for this LAGG)

So all interfaces are VLAN's. There is no ^physical LAN^.

I have no idea how / where to set the MTU's as related to the four NIC's carrying the two LAGG's

[JKnott]

@louis2 said in Is pfSense handling jumbo frames correct !??:

So all interfaces are VLAN's. There is no ^physical LAN^.

When you create a VLAN, you have to specify a parent interface. That's where you set the MTU.
With LAGG, you also have to specify the parent interface.

I use VLAN3 for my guest WiFi. It's parent interface is LAN on igb1. In the configuration for LAN, I can set the MTU and so also for VLAN3.

BTW, here's some history on Ethernet frames:

The Ethernet

Extended Ethernet Frame Size Support

[louis2]

@JKnott

I does not work that way

The problem is that if you assign an physical interface to a lagg, it is ofcourse not longer available as an interface.
So there is no way to reach that setting.

There are only two very dirty way I could try:

method-1 (should not work)

• assign an interface to the physical nic,
• change the MTU value
• delete the interface and ^hope^ that the MTU value stays in the DB (it sohould not!!!!)
• assign the physical nic to the lagg

method-2 (could perhaps work)

• save the config
• look for the involved interface with an editor
• change the MTU
• store the file
• load the file

Both methods are not OK of course

[skogs]

With nice fluke gear I've been able to not only measure very obvious hardware cpu load reductions with jumbo, but also vastly improved speed. 10G jumbo/9k is indeed noticeably faster than 10G normal MTU.

LAGG on the other hand, almost never gives the theoretical improvements expected. I'm guessing manual assignment will be your answer if it isn't hiding on interface lagg settings like it was a few years back.
I'm curious if you still need to tweak kernel settings afterward.

[JKnott]

@louis2 said in Is pfSense handling jumbo frames correct !??:

method-1 (should not work)

Have you tried it?

Here's what the pfSense manual says:

"Link aggregation is handled by lagg(4) type interfaces (LAGG) on pfSense software. LAGG combines multiple physical interfaces together as one logical interface. There are several ways this can work, either for gaining extra bandwidth, redundancy, or some combination of the two."

As I mentioned before, you create the LAGG from physical interfaces. I expect it would inherit whatever MTU the physical interfaces had.

It's simple enough to check this. Change the MTU and then create the LAGG. You can then use ifconfig to check the MTU.

This is so simple to try, arguing about it is wasting our time.

[louis2]

@JKnott

I had a look inside the configfile. There you see this

<laggs>
	<lagg>
		<members>igb0,igb1</members>
		<descr><![CDATA[LAGG TO 1G MAIN SW (GS1920)]]></descr>
		<laggif>lagg0</laggif>
		<proto>lacp</proto>
		<lacptimeout>slow</lacptimeout>
		<lagghash>l2,l3,l4</lagghash>
	</lagg>
	<lagg>
		<members>ix0,ix1</members>
		<descr><![CDATA[LAGG to 10G MAIN Switch (SX3008F)]]></descr>
		<laggif>lagg1</laggif>
		<proto>lacp</proto>
		<lacptimeout>slow</lacptimeout>
		<lagghash>l2,l3,l4</lagghash>
	</lagg>
</laggs>

AND

	<opt17>
		<descr><![CDATA[Emerg_Mngt]]></descr>
		<if>igb2</if>
		<spoofmac></spoofmac>
		<enable></enable>
		<ipaddr>192.168.9.1</ipaddr>
		<subnet>24</subnet>
		<mtu>9000</mtu>
	</opt17>

However there is no config block as show above for igb2 in favor of
igb0 / igb1 / ix0 / ix1

Neither is there such a config set for em0 .

The only situation where I see an ^<op117> like control blok, is in case of a "Physical LAN"

So adding not yet existing control block types, feels very hazzy

I think I will open a ticket. Lets see how the developers react ...