Kernel: msk1: watchdog timeout

[Gabri.91]

Hi, sometimes my WAN connections fails with this error in system logs:
Jun 15 08:03:49 kernel: msk1: watchdog timeout
Jun 15 08:03:49 kernel: msk1: link state changed to DOWN
Jun 15 08:03:49 check_reload_status: Linkup starting msk1
Jun 15 08:03:51 check_reload_status: Linkup starting msk1
Jun 15 08:03:51 kernel: msk1: link state changed to UP

It stays in this continuous loop until I reboot the PFsense.

I've read some old post but are all referred to version 1.
How could be the problem? The strange thing is that LAN interface stays up, I can login to web interface (where I reboot the machine) and all LAN services (DNS,DHCP) are ok, but WAN is down..
I think that isn't a driver problem because the LAN and WAN interface have the same motherboard chip network card (Commel 673)
Thanks

[stephenw10]

I've seen those errors on my box. They usually happen only when I stress the interface with a lot of traffic. There should be code in 2.1 to prevent it happening. Some people have reported success by disabling MSI or MSI-X. There is a tunable for the msk driver:

hw.msk.msi_disable=1

Or you can disable it for everything:

hw.pci.enable_msi=0
hw.pci.enable_msix=0

These lines should be put in /boot/loader.conf.local create the file if it doesn't exits.

Steve

[Gabri.91]

Thanks, I'll try that!

[Gabri.91]

Hi, I've created the file, but how can I verify that it was applied correctly?
This night I'll see if it works..

[stephenw10]

You can use the sysctl command to check from the console:

[2.0.1-RELEASE][root@pfsense.fire.box]/root(1): sysctl hw.pci
hw.pci.usb_early_takeover: 1
hw.pci.honor_msi_blacklist: 1
hw.pci.enable_msix: 1
hw.pci.enable_msi: 1
hw.pci.do_power_resume: 1
hw.pci.do_power_nodriver: 0
hw.pci.enable_io_modes: 1
hw.pci.default_vgapci_unit: -1
hw.pci.host_mem_start: 2147483648
hw.pci.mcfg: 1
hw.pci.irq_override_mask: 57080

Steve

[Gabri.91]

I've made the command and the output is exactly as your, so I think that /boot/loader.conf.local doesn't works (in fact this night crashes again)..
How is it possible? I've created the file /boot/loader.conf.local with this lines:

hw.pci.enable_msi=0
hw.pci.enable_msix=0

I need to change also /boot/loader.conf or some other file?

[stephenw10]

Hmm,
Some tunables can be set by adding them to the table in the webGUI, System: Advanced: System Tunables:
Just create new entries for hw.pci.enable_msi etc.
However I had thought these needed to be set as boot loader entries.  :-
In fact that is the instruction given here: http://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards
Does your file look exactly like the example given in the docs?

Steve

[wallabybob]

@Gabri.91:

I've made the command and the output is exactly as your, so I think that /boot/loader.conf.local doesn't works (in fact this night crashes again)..

/boot/loader.conf.local is processed only at boot time. Hence your change to it won't take effect until the system is rebooted.

When you wrote "I think that /boot/loader.conf.local doesn't works" did you mean (assuming you rebooted so the changed file would take effect)
1. the change didn't stop the crash?
2. the change didn't seem to affect the sysctl values?

[Gabri.91]

@wallabybob:

/boot/loader.conf.local is processed only at boot time. Hence your change to it won't take effect until the system is rebooted.
When you wrote "I think that /boot/loader.conf.local doesn't works" did you mean (assuming you rebooted so the changed file would take effect)
1. the change didn't stop the crash?
2. the change didn't seem to affect the sysctl values?

Logically I've rebooted the pfSense..
I mean all two things: the change didn't stop the crash and the change didn't seem to affect the sysctl values

[Gabri.91]

There is no solution? The problem still remains..

[stephenw10]

I have a patched msk driver I made for the firebox NICs you could try. I also modified the LED register settings but that won't effect the operation. See:
http://forum.pfsense.org/index.php/topic,20095.msg273691.html#msg273691
I have not yet had any reports of it working or not.

Steve

[Gabri.91]

Ok, I can try with that but why loader.conf.local settings aren't loaded?
I prefer to try with that before patch driver..

[stephenw10]

Some random weirdness!  ::)
Seriously I have no idea, it should work. Permissions problem?
Try putting it in loader.conf instead. That file will definitely be read and acted upon. It may be overwritten with a firmware update later but it will prove the method.

Steve

[Gabri.91]

Ok, putting them in loader.conf from sysctl the settings were loaded correctly:

hw.pci.enable_msix: 0
hw.pci.enable_msi: 0

I'll let you know if it solves the problem..

[Gabri.91]

With this settings never crashes and it's perfect! I suggest them before change driver or trash your firewall.. :)

[stephenw10]

Good to know. Thanks for reporting back.
Disabling MSI like that will presumably increase the CPU interrupt load. Have you noticed any increased load?

Steve

[Gabri.91]

I don't know because it's on a Pentium M @ 1,7Ghz and it will be never heavy load by a normal ADSL\Hyperlan connection.
I'll let you now in the next weeks\month when I'll add another NIC for DMZ and CPU will be more loaded..

[stephenw10]

It would be interesting to see if you can achieve the same thing using instead:

hw.msk.msi_disable=1

This would disable MSI only for the msk NIC rather than all PCI devices. A cleaner solution.

Steve

[Gabri.91]

I prefer to don't touch a things that works good.. ;)
If it helps I never see a delay or slowing down in ping, speedtest, navigation etc after modify the value..