pfsense sporadically gets stuck and needs manual reboot to recover

luas

Since a couple of months, our pfsense stops working unexpectedly without recognizable reason.
We were on 2.6.0 when it started and updating to 2.7.0 brought no improvement. Hardware is a discarded desktop computer (Fujitsu Esprimo D556) with an Intel Quad port nic. I must admit that I sometimes suspect this device as not adequately compatible, and I can't tell for sure if issues started with using these.

The issue sometimes occurs daily, sometimes not for weeks. pfsense will no longer answer internally - rebooting from local console is the only recovery I've seen up to now. We have two identical devices which show the same behaviour, so a hardware failure seems unlikely.

I checked the logs but I'm too unexperienced to find what is really important. I'd greatly appreciate if someone could check the attached logs.
Background info: I switched over to this hardware yesterday 13:30. First reports of internet outage came today around 16:27. The first suspect lines I see in the logs are:

Nov  8 16:04:13 pfSense check_reload_status[410]: Linkup starting re0
Nov  8 16:04:13 pfSense kernel: re0: watchdog timeout
Nov  8 16:04:13 pfSense kernel: re0: link state changed to DOWN
Nov  8 16:04:13 pfSense kernel: re0.4: link state changed to DOWN
Nov  8 16:04:13 pfSense kernel: re0.5: link state changed to DOWN
Nov  8 16:04:13 pfSense kernel: re0.6: link state changed to DOWN
Nov  8 16:04:13 pfSense check_reload_status[410]: Linkup starting re0.4
Nov  8 16:04:13 pfSense check_reload_status[410]: Linkup starting re0.5
Nov  8 16:04:13 pfSense check_reload_status[410]: Linkup starting re0.6
Nov  8 16:04:14 pfSense php-fpm[71069]: /rc.linkup: Hotplug event detected for GAESTE(opt5) static IP address (4: 192.168.1.1)
Nov  8 16:04:14 pfSense php-fpm[77090]: /rc.linkup: Hotplug event detected for ZEGG_IOT(opt6) static IP address (4: 192.168.6.1)
Nov  8 16:04:14 pfSense php-fpm[77090]: /rc.linkup: DEVD Ethernet detached event for opt6
Nov  8 16:04:14 pfSense php-fpm[71069]: /rc.linkup: DEVD Ethernet detached event for opt5
Nov  8 16:04:14 pfSense php-fpm[26342]: /rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 192.168.100.101)
Nov  8 16:04:14 pfSense php-fpm[26342]: /rc.linkup: DEVD Ethernet detached event for lan
Nov  8 16:04:14 pfSense php-fpm[70610]: /rc.linkup: Hotplug event detected for ZEGGVW(opt4) static IP address (4: 192.168.0.1)
Nov  8 16:04:14 pfSense php-fpm[70610]: /rc.linkup: DEVD Ethernet detached event for opt4

From our side, there was no action that would explain these entries.
Thanks in advance!

system1.log.txt
system0.log.txt

stephenw10

@luas said in pfsense sporadically gets stuck and needs manual reboot to recover:

Nov 8 16:04:13 pfSense kernel: re0: watchdog timeout

That is a Realtek NIC that stopped responding. If you have a 4 Intel NICs in that box the best solution would be to use those instead and not use the Realtek NIC at all.

Otherwise you can try using the alternative Realtek driver. See: https://forum.netgate.com/post/1131896

Steve

luas

@stephenw10 Oh I see! That's the onboard NIC.
I have spare ports on the Intel NIC. Which steps should I take to eliminate the Realtek NIC? Disable in BIOS?

stephenw10

You don't need to disable it, just don't use it. You will have to move the VLANs onto one of the igb ports too. It could be easier to edit the config directly to make that change. Though that's obviously more risky!

luas

@stephenw10 Indeed, was done by search&replace within a minute. I'm confident we have a stable system now again. Thank you!