All of a sudden my SG-5100 started randomly rebooting



  • I've had an SG-5100 that's been 100% stable since the day I got it—which was just over 2 months ago. Not a single crash or unexpected reboot the entire time. I even messed around with enabling the hardware watchdog which isn't officially supported out of the box, and that was working fine too.

    Fast forward to yesterday, and suddenly this box has destabilized. Not sure what could be causing it. I haven't updated to 2.4.4-p1 yet. Nothing on the box has changed. No updates installed, last config change was 11/19.

    The unit first rebooted at 7:59pm, then again at 8:30pm, 9:13pm, 10:33pm, 2:50am, 4:22am, 5:50am, and 7:58am. At that point, I suspected maybe something had gone haywire with the hardware watchdog, so I booted into the BIOS, disabled it and then rebooted pfSense and manually unloaded the wbwd.ko kernel module. It's been stable since but I'm waiting to see if it makes it through the rest of the day.

    Any idea why all of a sudden this thing destabilized? I've checked for any loose power cables etc. There is nothing noteworthy in the system log at all at the time of the reboots.

    The other thing I noticed is that the HDD LED on the box has started blinking steadily. I don't remember it doing that before - I think all 3 LEDs were solid / steady green. Not sure if this is a clue, or related somehow. Disk is an Intel 2.5" SSD and SMART status is passing/green. Here's a short clip of what it looks like:
    https://vimeo.com/304392499



  • You know what, I just took another look. Seems I recorded the video from a bad angle, it's actually the middle LED ("System Status") that's blinking, not the HDD LED. According to Appendix A: "LED Indicator Explanations" on p.76 of the manual, that is undefined. Clue?


  • Netgate Administrator

    The blinking green status means an update is available.

    See: /usr/local/sbin/pfSense-led.sh

    Steve



  • @stephenw10 Ok thanks. I don't have that file on my system though. 😐


  • Rebel Alliance

    Anything in the logs or are you talking about hard reboots?
    Hard Reboots are mostly caused by Power or Hardware issue. You running on UPS?

    -Rico


  • Netgate Administrator

    Hmm, I guess I'm running 2.4.4p1 but I didn't think anything had moved there....

    For reference:

    #
    # SG-5100
    #
    sg5100_led_booting() {
    	# Booting (red)
    	/usr/local/sbin/SG-5100led 1 >/dev/null 2>&1
    }
    
    sg5100_led_ready() {
    	# Boot finished (green)
    	/usr/local/sbin/SG-5100led 3 >/dev/null 2>&1
    }
    
    sg5100_led_update() {
    	# updates, green flashing
    	/usr/local/sbin/SG-5100led 4 >/dev/null 2>&1
    }
    
    sg5100_led_update_off() {
    	# No updates, green
    	/usr/local/sbin/SG-5100led 3 >/dev/null 2>&1
    }
    
    sg5100_led_updating() {
    	# Upgrade running during boot (LED blinking red)
    	/usr/local/sbin/SG-5100led 2 >/dev/null 2>&1
    }
    
    
    

    Steve