System tunables not being applied after restart

michael984

@stephenw10 I’m not entirely sure how I would check using ifconfig. I ran if config igb0 both right after rebooting and after re-applying the system tunable (as described above) and I saw no difference in output. Is there some specific way to check flow control using ifconfig?

I honestly have no idea why it’s not taking effect. Like I said, it looks (based on sysctl-a) that the tunable is being applied, but I can certainly observe that something is different based on very reproducible Speedtest numbers. I’m happy to try whatever is needed to diagnose the issue, so if you have some suggestions, please let me know.

I appreciate you taking time to help!

stephenw10

Hmm, yeah I think I conflated the output with ix, which does report it.
Reviewing what igb offers in 2.4.5 (and 2.5) there is nothing useful shown.

Are any custom system-tunables being set at boot?

For ix did you try: hw.ix.flow_control ?

Steve

michael984

@stephenw10 No other custom system tunables are being set at boot (either through /boot/loader.conf.local or system tunables). I have in the past set kern.vt.enable_bell=0` and that seemed to work, but I currently have no other tunables configured.

I did just try putting hw.ix.flow_control in /boot/loader.conf.local and that does seem to disable flow control for the X520-DA1 (ix0), at least according to ifconfig as evidenced by the media type changing from:

media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)

to

media: Ethernet autoselect (10Gbase-Twinax <full-duplex>)

However, just like igb0, adding dev.ix.0.fc=0 to system tunables does not survive reboot.

I haven't been able to detect any firm differences due to flow control being enabled or not enabled on ix0, so I can't say definitively that it has been disabled, but ifconfig at least says it has been disabled.

I will make the caveat that for this test, I am using a virtual machine (running on qemu-kvm) with the X520-DA1 passed through to it for LAN and an I210-AT passed through for WAN. This is a change from my previous posts where I had been running bare metal on a Dell 920 SFF because I moved the X520-DA1 to my server. I also confirmed that I see the same flow control not being enabled at boot behavior with my I210-AT as the WAN port (since the I210-AT also uses the igb driver). The hypervisor is not the Dell 920 (it's a Xeon E3-1231 v3 based platform), so the issue with flow control not being enabled at boot is not specific to the Dell 920 I was using.

I tried adding hw.igb.fc_setting=0 and hw.igb.flow_control=0 (one at a time, of course) to the /boot/loader.conf.local to see if it disabled flow control on the VM (even though it didn't on the physical machine) and after rebooting it did not disable flow control. I.e., I still saw the issues I am seeing and sysctl -a shows dev.igb.0.fc: 3.

As best as I can tell, there does not seem to be a reliable way to turn off flow control on the igb driver that survives reboot. Moreover, dev.*.fc.* system tunable does not seem to survive reboot for either igb or ix, even though it works when being applied at run time.

michael984

Actually, I am no longer sure if flow control is still enabled on the passed through I210-AT. The behavior that I thought was due to flow control is much more subtle on the VM (Speedtest shows wire speed for download with or without flow control disabled), and I thought I was seeing a difference between manually re-setting the system tunable dev.igb.0.fc=0 after reboot. However, I’ve seen the intermittent behavior after definitively disabling flow control, so that behavior may be unrelated.

So, it’s still possible that the inability to reliably disable flow control is due to the physical hardware in the Dell 920 (either the I350-T2 NIC or something else).

stephenw10

It's a driver option so if you have hardware pass-through I would expect it to be configurable.

yswery

We are seeing the same as @michael984 thing that the flow control in both "System -> Advanced -> System Tunables" and in /boot/loader.conf.local does not survive a reboot on pfsense 2.5.2

After a reboot we see

$ sysctl dev.ix.3.fc
dev.ix.3.fc: 3

However if I go and "reapply" System Tunables everything goes to as it should:

$ sysctl dev.ix.3.fc
dev.ix.3.fc: 0

What should we do in order to have FC disabled after a reboot?

togelonline

This post is deleted!

stephenw10

What values do you actually have in either of those places?

yswery

@stephenw10

in /boot/loader.conf.local

dev.ix.3.fc="0"

and in System Tuneables:

dev.ix.3.fc	Disable Flow Control	0

stephenw10

In loader.conf.local add: hw.ix.flow_control=0

That will disable flowcontrol for all ix NICs at boot.

Steve

stephenw10

Yeah, for some reason, you can disable it globally with the loader variable and then enable it again per interface with a sysctl but not the other way around. For ix at least.

Steve