System tunables not being applied after restart

[michael984]

I have noticed some strange behavior recently regarding system tunables. Particularly, some of them do not seem to be applied after a reboot specifically for flow control. First, note that I am fairly new to pfSense, so it is entirely possible that this is user error in some way, but I can't figure out what I would do differently.

I recently converted a Dell Optiplex 920 (i7-4770k with 16 GB of RAM) to a router running pfSense 2.4.5-p1 using an Intel I350-T2 card (verified with a Yottamark). I have a 1Gbps symmetric internet connection, but I could only get about 800Mbps down (wire speed up). I was able to temporarily solve this be setting the following in system tunables:

hw.igb.fc_setting=0
dev.igb.0.fc=0
dev.igb.1.fc=0

However, after a reboot the behavior returned. I mistakenly thought this was related to the NIC, so I replaced it with an Intel X520-DA1 (10 Gbps SFP+ NIC) for the LAN and the onboard NIC (I217-LM) for the WAN.

I then realized that the GUI was showing under interfaces that rxpause and txpause was enabled, so I set the following in system tunables and that went away:

dev.ix.0.fc=0

However, when I rebooted the rxpause and txpause was in the interface list (on the status dashboard) again. That's when I realized it seems like the flow control variables are not persisting across reboots. I confirmed that my system tunable setting was still there, but the interface list was suggesting that it was not applied.

To be clear, I am setting them by going to "System -> Advanced -> System Tunables" and setting them there. It's not all system tunables that behave this way because I set

kern.vt.enable_bell=0

And it seems to be working fine.

Is this a known issue with flow control? That it doesn't persist across reboots? Is there anything that can be done? Thanks for any insight!

[stephenw10]

Did they all get set when you applied them at run time?

Some of those are probably loader variables that should be set in /boot/loader.conf.local. The sysctls just reflect the current state for some values.
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#flow-control

Steve

[michael984]

@stephenw10 They seemed to get set when I applied them at run time (as evidenced by rxpause and txpause disappearing from the "Interfaces" status widget), and from what I can tell, the pfSense documentation is actually out of date in terms of how to disable flow control. It seems like the correct way at this point is to put dev.<INTERFACE_NAME>.<INTERFACE_NUMBER>.fc=0 in system tunables. Though there is some conflicting information around that.

I did try to put everything in /boot/loader.conf.local and I still observed problems with the Intel I350-T2, though.

[stephenw10]

@michael984 said in System tunables not being applied after restart:

dev.ix.0.fc=0

Mmm, I would certainly expect that to work.

Maybe you have some other invalid value in the table above it?
I would still expect subsequent lines to be parsed though.

[michael984]

@stephenw10 Thanks for taking time to respond! Sorry for the delay. I have just now gotten a chance to run some more tests.

I have done some more testing, and I can firmly isolate and reproduce the problem. I've gone back to the Intel I350-T2 NIC. I've set igb0 to be WAN and igb1 to be LAN. If I don't have any system tunables set (that aren't standard), then my download speed on SpeedTest never goes above 770Mbps.

If I add to system tunables and hit apply

dev.igb.0.fc: 0

then my download speed immediately goes up to wire speed (940 Mbps). If I check sysctl -a in the shell I see that they are set correctly.

However, if I reboot, the download speed reverts to a max of 770Mbps. If I check sysctl -a, though I see that

dev.igb.0.fc: 0
dev.igb.1.fc: 0

so the system thinks that flow control is set correctly (in fact it sets it for igb1 even though I don't specify that in system tunable). However, my download speed stays artificially low until I apply any system tunable (i.e., I don't need to re-add the dev.igb.0.fc=0, I added hw.syscons.bell=0 and speeds returned to wire speed as long as the above flow control tunables were also present). If the flow control tunable is not present and I added another tunable, nothing changed with the speeds, so it is the flow control tunables that are responsible for the speeds.

I tried adding

dev.igb.0.fc=0

to /boot/loader.conf.local, but that did nothing (sysctl still showed dev.igb.0.fc: 3).

I find this behavior to be very weird, but somehow the flow control setting is not being applied on reboot even though the system tunable is correctly being reflected using sysctl -a. Do you have any idea how to continue diagnosing the problem? Is there some way that the system tunables is being applied and then the card is being reset/re-initialized in some way that gets rid of system tunables?

I've looked in the system logs for any indication of problems with flow control, but I couldn't find anything (though that may be just an oversight on my part).

Thanks for your help!

[stephenw10]

Hmm, interesting.

Do you see flow-control actually set or unset in ifconfig at those times?

I.e. is the sysctl simply not being applied at boot you think?

Steve

[michael984]

@stephenw10 I’m not entirely sure how I would check using ifconfig. I ran if config igb0 both right after rebooting and after re-applying the system tunable (as described above) and I saw no difference in output. Is there some specific way to check flow control using ifconfig?

I honestly have no idea why it’s not taking effect. Like I said, it looks (based on sysctl-a) that the tunable is being applied, but I can certainly observe that something is different based on very reproducible Speedtest numbers. I’m happy to try whatever is needed to diagnose the issue, so if you have some suggestions, please let me know.

I appreciate you taking time to help!

[stephenw10]

Hmm, yeah I think I conflated the output with ix, which does report it.
Reviewing what igb offers in 2.4.5 (and 2.5) there is nothing useful shown.

Are any custom system-tunables being set at boot?

For ix did you try: hw.ix.flow_control ?

Steve

[michael984]

@stephenw10 No other custom system tunables are being set at boot (either through /boot/loader.conf.local or system tunables). I have in the past set kern.vt.enable_bell=0` and that seemed to work, but I currently have no other tunables configured.

I did just try putting hw.ix.flow_control in /boot/loader.conf.local and that does seem to disable flow control for the X520-DA1 (ix0), at least according to ifconfig as evidenced by the media type changing from:

media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)

to

media: Ethernet autoselect (10Gbase-Twinax <full-duplex>)

However, just like igb0, adding dev.ix.0.fc=0 to system tunables does not survive reboot.

I haven't been able to detect any firm differences due to flow control being enabled or not enabled on ix0, so I can't say definitively that it has been disabled, but ifconfig at least says it has been disabled.

I will make the caveat that for this test, I am using a virtual machine (running on qemu-kvm) with the X520-DA1 passed through to it for LAN and an I210-AT passed through for WAN. This is a change from my previous posts where I had been running bare metal on a Dell 920 SFF because I moved the X520-DA1 to my server. I also confirmed that I see the same flow control not being enabled at boot behavior with my I210-AT as the WAN port (since the I210-AT also uses the igb driver). The hypervisor is not the Dell 920 (it's a Xeon E3-1231 v3 based platform), so the issue with flow control not being enabled at boot is not specific to the Dell 920 I was using.

I tried adding hw.igb.fc_setting=0 and hw.igb.flow_control=0 (one at a time, of course) to the /boot/loader.conf.local to see if it disabled flow control on the VM (even though it didn't on the physical machine) and after rebooting it did not disable flow control. I.e., I still saw the issues I am seeing and sysctl -a shows dev.igb.0.fc: 3.

As best as I can tell, there does not seem to be a reliable way to turn off flow control on the igb driver that survives reboot. Moreover, dev.*.fc.* system tunable does not seem to survive reboot for either igb or ix, even though it works when being applied at run time.

[michael984]

Actually, I am no longer sure if flow control is still enabled on the passed through I210-AT. The behavior that I thought was due to flow control is much more subtle on the VM (Speedtest shows wire speed for download with or without flow control disabled), and I thought I was seeing a difference between manually re-setting the system tunable dev.igb.0.fc=0 after reboot. However, I’ve seen the intermittent behavior after definitively disabling flow control, so that behavior may be unrelated.

So, it’s still possible that the inability to reliably disable flow control is due to the physical hardware in the Dell 920 (either the I350-T2 NIC or something else).

[stephenw10]

It's a driver option so if you have hardware pass-through I would expect it to be configurable.

[yswery]

We are seeing the same as @michael984 thing that the flow control in both "System -> Advanced -> System Tunables" and in /boot/loader.conf.local does not survive a reboot on pfsense 2.5.2

After a reboot we see

$ sysctl dev.ix.3.fc
dev.ix.3.fc: 3

However if I go and "reapply" System Tunables everything goes to as it should:

$ sysctl dev.ix.3.fc
dev.ix.3.fc: 0

What should we do in order to have FC disabled after a reboot?

[togelonline]

This post is deleted!

[stephenw10]

What values do you actually have in either of those places?

[yswery]

@stephenw10

in /boot/loader.conf.local

dev.ix.3.fc="0"

and in System Tuneables:

dev.ix.3.fc	Disable Flow Control	0

[stephenw10]

In loader.conf.local add: hw.ix.flow_control=0

That will disable flowcontrol for all ix NICs at boot.

Steve

[stephenw10]

Yeah, for some reason, you can disable it globally with the loader variable and then enable it again per interface with a sysctl but not the other way around. For ix at least.

Steve