Lagg - Make Jumbo Frames Persistent?
-
I have a working jumbo frames config on LAN interface lagg0. To do this, I set the MTU in each NIC with ifconfig, then I created the aggregation.
Unfortunately, a reboot sets the nics to default 1500 MTU, which makes the lagg unable to transmit larger frames, and I can't change MTU on a lagg or its members via ifconfig while it's up.
What's the best way make this configuration persistent across reboots? A startup script? A different command? <shellcmd>tags in /cf/conf/config.xml?</shellcmd>
-
-
What is wrong with using the MTU parameter on the appropriate interfaces pages: Interfaces -> LAN, Interfaces -> OPTx etc?
-
What is wrong with using the MTU parameter on the appropriate interfaces pages: Interfaces -> LAN, Interfaces -> OPTx etc?
Because that setting has no effect on a lagg.
The ONLY way I know to get a pfSense lagg to sling jumbo frames is to configure the individual members of the lagg with the appropriate MTU before configuring the lagg. But the only way I know how to do that is via ifconfig, which is NOT persistent across reboots, and can not use ioctl to change the MTU values of a running lagg.
As it is now, I have to reconfigure pfSense and the switch before and after a reboot of pfSense, or just not use a lagg if I want to route jumbo frames.
And since laggs are configured at boot, it looks like I'll have to write a boot script to run the obsolete ifconfig commands sometime between the time the NIC drivers are loaded, and the LAN lagg is configured.
So be it.
-
What is wrong with using the MTU parameter on the appropriate interfaces pages: Interfaces -> LAN, Interfaces -> OPTx etc?
Because that setting has no effect on a lagg.
But have you:
1. Configured the lagg members to have the MTU you want? then
2. Configured the lagg interface to have the MTU you want? (maybe this step is unnecessary; I don't have gear to test it) then
3. rebooted and checked the relevant MTUs?The startup code appears to check the MTU of each lagg member and sets the MTU of each member interface to the smallest MTU of the member interfaces.
-
What is wrong with using the MTU parameter on the appropriate interfaces pages: Interfaces -> LAN, Interfaces -> OPTx etc?
Because that setting has no effect on a lagg.
But have you:
1. Configured the lagg members to have the MTU you want? then
2. Configured the lagg interface to have the MTU you want? (maybe this step is unnecessary; I don't have gear to test it) then
3. rebooted and checked the relevant MTUs?The startup code appears to check the MTU of each lagg member and sets the MTU of each member interface to the smallest MTU of the member interfaces.
Right - The MTU of a lagg can not exceed the lowest common of its members. But the ifconfig command required to change a NIC's MTU (ie #ifconfig em0 mtu 9022…) is lost on shutdown, and can't be applied to a running lagg...
So when the startup code checks the MTU of each member, it gets the default value of 1500 built into the kernel module.
I want 9022 so that lagg0 will accept the value of 9014 entered in the space provided in Interfaces>LAN in pfSense Webconfigurator, or via ifconfig from root shell. (Yes, both the NIC and the em(4) driver are capable of up to 16362 MTU, and every other port, NIC, and aggregation on the LAN are happy with 9014 MTU on IPv4.)I can accomplish this happy and efficient state of affairs by booting pfSense with no lagg, manually configuring each future member NIC MTU via ifconfig, then configuring the lagg via GUI or CLI. Then, and only then, can I get a pfSense lagg that can transmit jumbo frames.
(Moreover, if I want LACP, I have to take 802.3ad off those switch ports before pfSense boots and tries to communicate with the switch in something other than 802.3ad,… and then reapply LACP to both the switch and to pfSense afterwards... ::) )
Right now, a boot script appears the only way to make this "persistent" in pfSense - short of recompiling the em module - as I've tried <shellcmd>and <earlyshellcmd>with ifconfig, and I'm running out of userland hooks… :'(</earlyshellcmd></shellcmd>
-
But the ifconfig command required to change a NIC's MTU (#ifconfig nicname mtu 9022) is lost on shutdown, and can't be applied to a running lagg.
I wasn't discussing an MTU value configured by you issuing a shell ifconfig command, I was discussing an MTU value configured on the lagg member interfaces (and possibly the lagg interface itself) through the web GUI and saved (by clicking on the Save button) in the configuration database.
-
But the ifconfig command required to change a NIC's MTU (#ifconfig nicname mtu 9022) is lost on shutdown, and can't be applied to a running lagg.
I wasn't discussing an MTU value configured by you issuing a shell ifconfig command, I was discussing an MTU value configured on the lagg member interfaces (and possibly the lagg interface itself) through the web GUI and saved (by clicking on the Save button) in the configuration database.
I was discussing why that approach does not work. If it were as simple as typing 9014 into Interfaces>LAN>MTU field in the GUI, or editing config.xml, I wouldn't need to ask the hivemind for suggestions.
I've tried <shellcmd>and <earlyshellcmd>tags around the appropriate ifconfig commands in config.xml, but the lagg is configured by the time those commands are read, so they fail.
Config.xml seems to merely assign interfaces after they are configured by the driver modules one can't modify in pfSense userland.The only way to configure a lagg MTU is via the individual member NIC MTUs.
The only way to configure a NIC's MTU in pfSense is to change it with ifconfig after it is plumbed with MTU 1500 by the driver module, which is not configurable in pfSense userland.tl;dr: Web config doesn't work with a lagg. Ifconfig doesn't work with a booted lagg or survive a reboot. Config.xml is as powerless over a lagg as ifconfig.</earlyshellcmd></shellcmd>
-
Have you tried it?
-
Have you tried it?
Yes, I have tried it. Interestingly enough, this is precisely how I know it does not work.
;D
-
The actual problem here is that you have to define the MTU on the member interfaces of the LAGG and there is no way to do that via the GUI because the member interfaces are not assigned so there are no config options for them. Is that it?
Steve
-
The actual problem here is that you have to define the MTU on the member interfaces of the LAGG and there is no way to do that via the GUI because the member interfaces are not assigned so there are no config options for them. Is that it?
Steve
Somewhat…
I can set a MTU value in Interfaces>LAN. This works fine for a single NIC say, em0.
I can find the undesignated NICs in Interfaces>Assign>LAGG, Cntrl+clik to designate the multiple interfaces - say em0 and em1 - select a protocol, and build a LAGG, and assign it to LAN.BUT:
Unless em0 and em1 have been set to a nondefault MTU beforehand, the LAGG will have an MTU of only 1500, regardless of what is entered into the MTU field in Interfaces>LAN>MTU.
In case of reboot, pfSense loses any MTU setting made to LAGG members beforehand, then proceeds to configure the LAGG with members set to default MTU, which results in a LAGG set to default MTU. No more Jumbo Frames. -
Excactly.
You can only set an MTU value in the GUI to, say, em0 once it has been assigned but you can only add unassigned interfaces to the LAGG.
If you do it manually in the right order it works because behind the scenes it's just doing ifconfig and unassigning the interface does not reset the MTU. However it does remove it from the config file so after a reboot it doesn't come back up.
Is it possible to use a manual hack on the config file to assign attributes to LAGG member interfaces?
Edit: Doesn't look like it is but I'm open to suggestions.This seems a bit like overkill but you could probably use the record/playback feature in the php shell to generate a script that configures the MTU of each member and then reconfigures them as LAGG. Have it playback at boot. I've never tried that. ;)
Steve
-
Excactly.
You can only set an MTU value in the GUI to, say, em0 once it has been assigned but you can only add unassigned interfaces to the LAGG.
If you do it manually in the right order it works because behind the scenes it's just doing ifconfig and unassigning the interface does not reset the MTU. However it does remove it from the config file so after a reboot it doesn't come back up.
Is it possible to use a manual hack on the config file to assign attributes to LAGG member interfaces?
Edit: Doesn't look like it is but I'm open to suggestions.This seems a bit like overkill but you could probably use the record/playback feature in the php shell to generate a script that configures the MTU of each member and then reconfigures them as LAGG. Have it playback at boot. I've never tried that. ;)
Steve
Thanks for taking an interest, and thanks for the PHP shell suggestion. I'll let you know how it works. Hopefully, it won't hinge on just when in the boot process the playback script needs to run…
;D
-
PhP shell playback just modifies config.xml after the LAGG is built.
As such, it's no more effective than anything else done in config.xml, because it's read too late in the boot to change anything about a LAGG. -
But you can create a script that removes the LAGG, assigns the interfaces noamally, sets the MTU and then removes them and replaces the LAGG.
Yes, a horrendous hack! ::)Steve
-
…
Yes, a horrendous hack! ::)Steve
A Parable:
The '69 Chevelle Malibu SS with the 396ci High Output engine had a thing with spark plugs. That big block was shoehorned into the engine compartment so tight that it was difficult, if not impossible, for a grown man to get his hand into a place where he could pull the plug wire and put a deep socket on the plug.
When these cars inevitably found their way into the service bays of dealer techs and other experienced professionals, the first thing that happened was the pro loosened all the motor mounts, jacked up the front of the engine, took a metal bar and a 5-lb sledgehammer and dented in the car's firewall to allow a human being to apply a wrench to number seven and eight @#$% spark plugs.FF to present:
Is there anyone reading this who wouldn't want a '69 Chevelle SS with a 396 and a slightly dimpled firewall?
The End
-
But you can create a script that removes the LAGG, assigns the interfaces noamally, sets the MTU and then removes them and replaces the LAGG.
Yes, a horrendous hack! ::)Steve
"We had to burn LAGG0 in order to save it."
Easiest way:
1. Backup config.xml from Webconfigurator to desktop.
2. Open config in text editor, "Save As"
3. Add appropriate ifconfig commands inside <earlyshellcmd></earlyshellcmd>tags placed directly above the existing tag in the .xml. The existing LAGG must be destroyed first, then an appropriate MTU can be applied to the former/future members, then a new LAGG0 is configured manually, ie<earlyshellcmd>ifconfig lagg0 destroy</earlyshellcmd> <earlyshellcmd>ifconfig em0 mtu 9022</earlyshellcmd> <earlyshellcmd>ifconfig em1 mtu 9022</earlyshellcmd> <earlyshellcmd>ifconfig lagg0 laggproto roundrobin laggport em0 laggport em1 \ 172.16.1.1 netmask 255.255.255.0</earlyshellcmd>4. Save newly edited .xml.
5. In Webconfigurator>Backup/Restore, restore with newly edited config.xml you just saved.
6. pfSense reboots
7. Get a shell and check your work:ifconfig lagg0 lagg0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 9022 ...</up,broadcast,running,simplex,multicast>ping -s 8900 -D 172.16.1.112 PING 172.16.1.112 (172.16.1.112): 8900 data bytes 8908 bytes from 172.16.1.112: icmp_seq=0 ttl=128 time=0.957 ms 8908 bytes from 172.16.1.112: icmp_seq=1 ttl=128 time=0.814 ms 8908 bytes from 172.16.1.112: icmp_seq=2 ttl=128 time=0.848 ms ^C --- 172.16.1.112 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.814/0.873/0.957/0.061 ms8. Sit back and bask in the higher transport speeds and lower resource utilization of your jumbo aggregation. ;D
Thanks stephenw10! Your suggestion to destroy the LAGG first was the key.
-
Nice! :)
pfSense still shows the correct interfaces in the dash etc?
Not pretty but if it works…It should be possible to do this in the webgui IMHO. There was a change to the way bridge interfaces are handled that would seem similar to this problem. Previously the bridge itself could not be assigned as an interface. It seems to me quite likely that if you need to create a lagg to increase bandwidth you might also want to use jumbo frames. Something for the future perhaps.
Steve
-
Nice! :)
pfSense still shows the correct interfaces in the dash etc?Yep. :)
Not pretty but if it works…
Indeed…
It should be possible to do this in the webgui IMHO. There was a change to the way bridge interfaces are handled that would seem similar to this problem. Previously the bridge itself could not be assigned as an interface. It seems to me quite likely that if you need to create a lagg to increase bandwidth you might also want to use jumbo frames. Something for the future perhaps.
Steve
Tried to edit in GUI, but it hung on "Save", and wouldn't refresh the page. Probably a browser thing on my end.
I don't see why it wouldn't work to edit /cf/conf/config.xml in the GUI.Also, this may well work with the preferred <shellcmd>tags rather than <earlyshellcmd>. I tried "Early" first, and it worked, so I quit winners. :D
I think the "glitch" is systemic with ifconfig(8), and originates in ifconfig not having the capacity to pass a flag for persistence across boots. The FreeBSD ifconfig(8) man never even considers it, but it's common enough knowledge so that we pretty much all know we have to make an entry in rc.conf, either manually or via sysctrl, by the time most of us develop the courage to pop the hood on pfSense.
But pfSense is hardened, and so it can't work like that.
This is the definition of an "ugly" hack in that it has to destroy and recreate something after the fact, but what the hell, it works.
</earlyshellcmd></shellcmd>
