[SOLVED] pfsense box suddenly dies



  • Hello guys,

    I have some issues with my current setup. It has happend a few times now that my pfsense box just suddenly dies or freezes. My entire network gets unavailable. The information I have found so far when this occours is "re0: watchdog timeout". re0 is the LAN interface. Do you guys have any idea what could be the issue here? A faulty NIC? Bad drivers?

    Im running the following
    2.4.2-RELEASE (amd64)
    built on Mon Nov 20 08:12:56 CST 2017
    FreeBSD 11.1-RELEASE-p4

    Intel(R) Core(TM) i5-6400 CPU @ 2.70GHz
    Current: 2700 MHz, Max: 2701 MHz
    4 CPUs: 1 package(s) x 4 core(s)
    AES-NI CPU Crypto: Yes (active)

    Here is the log from when the issue started untill i rebooted the box with the power button.

    Dec  9 23:01:50 pfsense kernel: re0: watchdog timeout
    Dec  9 23:01:50 pfsense kernel: re0: link state changed to DOWN
    Dec  9 23:01:50 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:01:51 pfsense php-fpm[97949]: /rc.linkup: DEVD Ethernet detached event for lan
    Dec  9 23:01:51 pfsense check_reload_status: Reloading filter
    Dec  9 23:01:54 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:01:54 pfsense kernel: re0: link state changed to UP
    Dec  9 23:01:55 pfsense php-fpm[97949]: /rc.linkup: DEVD Ethernet attached event for lan
    Dec  9 23:01:55 pfsense php-fpm[97949]: /rc.linkup: HOTPLUG: Configuring interface lan
    Dec  9 23:01:55 pfsense check_reload_status: Restarting ipsec tunnels
    Dec  9 23:02:00 pfsense check_reload_status: updating dyndns lan
    Dec  9 23:02:00 pfsense check_reload_status: Reloading filter
    Dec  9 23:02:12 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:02:12 pfsense kernel: re0: watchdog timeout
    Dec  9 23:02:12 pfsense kernel: re0: link state changed to DOWN
    Dec  9 23:02:13 pfsense php-fpm[10509]: /rc.linkup: DEVD Ethernet detached event for lan
    Dec  9 23:02:13 pfsense check_reload_status: Reloading filter
    Dec  9 23:02:16 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:02:16 pfsense kernel: re0: link state changed to UP
    Dec  9 23:02:17 pfsense php-fpm[10509]: /rc.linkup: DEVD Ethernet attached event for lan
    Dec  9 23:02:17 pfsense php-fpm[10509]: /rc.linkup: HOTPLUG: Configuring interface lan
    Dec  9 23:02:17 pfsense check_reload_status: Restarting ipsec tunnels
    Dec  9 23:02:21 pfsense check_reload_status: updating dyndns lan
    Dec  9 23:02:21 pfsense check_reload_status: Reloading filter
    Dec  9 23:02:22 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:02:22 pfsense kernel: re0: watchdog timeout
    Dec  9 23:02:22 pfsense kernel: re0: link state changed to DOWN
    Dec  9 23:02:23 pfsense php-fpm[13246]: /rc.linkup: DEVD Ethernet detached event for lan
    Dec  9 23:02:23 pfsense check_reload_status: Reloading filter
    Dec  9 23:02:26 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:02:26 pfsense kernel: re0: link state changed to UP
    Dec  9 23:02:27 pfsense php-fpm[13246]: /rc.linkup: DEVD Ethernet attached event for lan
    Dec  9 23:02:27 pfsense php-fpm[13246]: /rc.linkup: HOTPLUG: Configuring interface lan
    Dec  9 23:02:27 pfsense check_reload_status: Restarting ipsec tunnels
    Dec  9 23:02:31 pfsense check_reload_status: updating dyndns lan
    Dec  9 23:02:31 pfsense check_reload_status: Reloading filter
    Dec  9 23:02:32 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:02:32 pfsense kernel: re0: watchdog timeout
    Dec  9 23:02:32 pfsense kernel: re0: link state changed to DOWN
    Dec  9 23:02:33 pfsense php-fpm[28635]: /rc.linkup: DEVD Ethernet detached event for lan
    Dec  9 23:02:33 pfsense check_reload_status: Reloading filter
    Dec  9 23:02:36 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:02:36 pfsense kernel: re0: link state changed to UP
    Dec  9 23:02:37 pfsense php-fpm[28635]: /rc.linkup: DEVD Ethernet attached event for lan
    Dec  9 23:02:37 pfsense php-fpm[28635]: /rc.linkup: HOTPLUG: Configuring interface lan
    Dec  9 23:02:37 pfsense check_reload_status: Restarting ipsec tunnels
    Dec  9 23:02:42 pfsense check_reload_status: updating dyndns lan
    Dec  9 23:02:42 pfsense check_reload_status: Reloading filter
    Dec  9 23:02:43 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:02:43 pfsense kernel: re0: watchdog timeout
    Dec  9 23:02:43 pfsense kernel: re0: link state changed to DOWN
    Dec  9 23:02:44 pfsense php-fpm[80725]: /rc.linkup: DEVD Ethernet detached event for lan
    Dec  9 23:02:44 pfsense check_reload_status: Reloading filter
    Dec  9 23:02:47 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:02:47 pfsense kernel: re0: link state changed to UP
    Dec  9 23:02:48 pfsense php-fpm[95351]: /rc.linkup: DEVD Ethernet attached event for lan
    Dec  9 23:02:48 pfsense php-fpm[95351]: /rc.linkup: HOTPLUG: Configuring interface lan
    Dec  9 23:02:48 pfsense check_reload_status: Restarting ipsec tunnels
    Dec  9 23:02:52 pfsense check_reload_status: updating dyndns lan
    Dec  9 23:02:52 pfsense check_reload_status: Reloading filter
    Dec  9 23:03:07 pfsense kernel: re0: watchdog timeout
    Dec  9 23:03:07 pfsense kernel: re0: link state changed to DOWN
    Dec  9 23:03:07 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:03:08 pfsense php-fpm[97209]: /rc.linkup: DEVD Ethernet detached event for lan
    Dec  9 23:03:08 pfsense check_reload_status: Reloading filter
    Dec  9 23:03:12 pfsense kernel: re0: link state changed to UP
    Dec  9 23:03:12 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:03:13 pfsense php-fpm[97209]: /rc.linkup: DEVD Ethernet attached event for lan
    Dec  9 23:03:13 pfsense php-fpm[97209]: /rc.linkup: HOTPLUG: Configuring interface lan
    Dec  9 23:03:13 pfsense check_reload_status: Restarting ipsec tunnels
    Dec  9 23:03:17 pfsense check_reload_status: updating dyndns lan
    Dec  9 23:03:17 pfsense check_reload_status: Reloading filter
    Dec  9 23:04:15 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:04:15 pfsense kernel: re0: watchdog timeout
    Dec  9 23:04:15 pfsense kernel: re0: link state changed to DOWN
    Dec  9 23:04:16 pfsense php-fpm[15077]: /rc.linkup: DEVD Ethernet detached event for lan
    Dec  9 23:04:16 pfsense check_reload_status: Reloading filter
    Dec  9 23:04:19 pfsense kernel: re0: link state changed to UP
    Dec  9 23:04:19 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:04:20 pfsense php-fpm[15077]: /rc.linkup: DEVD Ethernet attached event for lan
    Dec  9 23:04:20 pfsense php-fpm[15077]: /rc.linkup: HOTPLUG: Configuring interface lan
    Dec  9 23:04:20 pfsense check_reload_status: Restarting ipsec tunnels
    Dec  9 23:04:24 pfsense check_reload_status: updating dyndns lan
    Dec  9 23:04:24 pfsense check_reload_status: Reloading filter
    Dec  9 23:05:27 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:05:27 pfsense kernel: re0: watchdog timeout
    Dec  9 23:05:27 pfsense kernel: re0: link state changed to DOWN
    Dec  9 23:05:28 pfsense php-fpm[86326]: /rc.linkup: DEVD Ethernet detached event for lan
    Dec  9 23:05:28 pfsense check_reload_status: Reloading filter
    Dec  9 23:05:31 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:05:31 pfsense kernel: re0: link state changed to UP
    Dec  9 23:05:32 pfsense php-fpm[25940]: /rc.linkup: DEVD Ethernet attached event for lan
    Dec  9 23:05:32 pfsense php-fpm[25940]: /rc.linkup: HOTPLUG: Configuring interface lan
    Dec  9 23:05:32 pfsense check_reload_status: Restarting ipsec tunnels
    Dec  9 23:05:36 pfsense check_reload_status: updating dyndns lan
    Dec  9 23:05:36 pfsense check_reload_status: Reloading filter
    Dec  9 23:06:51 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:06:51 pfsense kernel: re0: watchdog timeout
    Dec  9 23:06:51 pfsense kernel: re0: link state changed to DOWN
    Dec  9 23:06:52 pfsense php-fpm[41550]: /rc.linkup: DEVD Ethernet detached event for lan
    Dec  9 23:06:52 pfsense check_reload_status: Reloading filter
    Dec  9 23:06:55 pfsense check_reload_status: Linkup starting re0
    Dec  9 23:06:55 pfsense kernel: re0: link state changed to UP
    Dec  9 23:06:56 pfsense php-fpm[41550]: /rc.linkup: DEVD Ethernet attached event for lan
    Dec  9 23:06:56 pfsense php-fpm[41550]: /rc.linkup: HOTPLUG: Configuring interface lan
    Dec  9 23:06:56 pfsense check_reload_status: Restarting ipsec tunnels
    Dec  9 23:06:58 pfsense syslogd: exiting on signal 15
    Dec  9 23:08:42 pfsense syslogd: kernel boot file is /boot/kernel/kernel
    
    

    Im thankfull for any help that could push me in the right direction to fix this.

    Thanks!



  • I would stop using those Realtek NICs and throw an Intel NIC card in the box.  I've personally had trouble with them and I know others have as well.



  • @jamesonp:

    I would stop using those Realtek NICs and throw an Intel NIC card in the box.  I've personally had trouble with them and I know others have as well.

    Ok, thanks! I will get an intel NIC instead.



  • Here's the issues I had with RealTek..while the video is no longer there, you should be able to get the gist of it:

    https://forum.pfsense.org/index.php?topic=103199



  • @jamesonp:

    Here's the issues I had with RealTek..while the video is no longer there, you should be able to get the gist of it:

    https://forum.pfsense.org/index.php?topic=103199

    Thanks again!

    I just noticed that the other NIC I got in the box is Intel. The RealTek NIC is the onboard NIC on the motherboard. But I have another Intel NIC in the box that is not bieng used at the moment. So I will switch over to that one instead and see if i can get this issue fixed.

    Thanks again!



  • Ok, so now I have changed to a Intel NIC instead.

    Since I allready had a Intel NIC in the box that was unused, I just changed the settings in Interfaces for the the LAN network port from the RealTek to the Intel card. I dont think i need to do any other changes.

    So now both WAN and LAN go through Intel NIC's.

    Hopefully this will solve the issue.



  • Is there anyway I can try to see if this issue is fixed by reproducing what caused the "re0: watchdog timeout" ? Does anyone have a clue what actually makes the NIC to timout?

    Thanks!



  • You might try reproducing the issues I had in the post I linked. The RealTek issue seems to be an issue with the quality of the adapter and/or a driver regression.



  • I just wanted to thank everyone for the help with this issue. Changing to a Intel NIC did the trick for me. No more RealTek NIC's running in my pfsense and 80 days uptime without a single issue.

    Thanks again!



  • Well am glad u were able to resolve this with easily replaceable NICs and not some embedded soldered on NICs.  There are some system/advance/network parameters that users can turn off to deal with problematic NICs to try things out as alternative.