All of a sudden my SG-5100 started randomly rebooting

luckman212

I've had an SG-5100 that's been 100% stable since the day I got it—which was just over 2 months ago. Not a single crash or unexpected reboot the entire time. I even messed around with enabling the hardware watchdog which isn't officially supported out of the box, and that was working fine too.

Fast forward to yesterday, and suddenly this box has destabilized. Not sure what could be causing it. I haven't updated to 2.4.4-p1 yet. Nothing on the box has changed. No updates installed, last config change was 11/19.

The unit first rebooted at 7:59pm, then again at 8:30pm, 9:13pm, 10:33pm, 2:50am, 4:22am, 5:50am, and 7:58am. At that point, I suspected maybe something had gone haywire with the hardware watchdog, so I booted into the BIOS, disabled it and then rebooted pfSense and manually unloaded the wbwd.ko kernel module. It's been stable since but I'm waiting to see if it makes it through the rest of the day.

Any idea why all of a sudden this thing destabilized? I've checked for any loose power cables etc. There is nothing noteworthy in the system log at all at the time of the reboots.

The other thing I noticed is that the HDD LED on the box has started blinking steadily. I don't remember it doing that before - I think all 3 LEDs were solid / steady green. Not sure if this is a clue, or related somehow. Disk is an Intel 2.5" SSD and SMART status is passing/green. Here's a short clip of what it looks like:
https://vimeo.com/304392499

luckman212

You know what, I just took another look. Seems I recorded the video from a bad angle, it's actually the middle LED ("System Status") that's blinking, not the HDD LED. According to Appendix A: "LED Indicator Explanations" on p.76 of the manual, that is undefined. Clue?

stephenw10

The blinking green status means an update is available.

See: /usr/local/sbin/pfSense-led.sh

Steve

luckman212

@stephenw10 Ok thanks. I don't have that file on my system though.

Rico

Anything in the logs or are you talking about hard reboots?
Hard Reboots are mostly caused by Power or Hardware issue. You running on UPS?

-Rico

stephenw10

Hmm, I guess I'm running 2.4.4p1 but I didn't think anything had moved there....

For reference:

#
# SG-5100
#
sg5100_led_booting() {
	# Booting (red)
	/usr/local/sbin/SG-5100led 1 >/dev/null 2>&1
}

sg5100_led_ready() {
	# Boot finished (green)
	/usr/local/sbin/SG-5100led 3 >/dev/null 2>&1
}

sg5100_led_update() {
	# updates, green flashing
	/usr/local/sbin/SG-5100led 4 >/dev/null 2>&1
}

sg5100_led_update_off() {
	# No updates, green
	/usr/local/sbin/SG-5100led 3 >/dev/null 2>&1
}

sg5100_led_updating() {
	# Upgrade running during boot (LED blinking red)
	/usr/local/sbin/SG-5100led 2 >/dev/null 2>&1
}

Steve

SebM

Did you ever get to the bottom of this?

I had a similar problem after upgrading the RAM to 16GB, but I have the feeling it was related to the Watchdog feature.

stephenw10

It could be if the extra ram has increased the boot time beyond the watchdog timeout perhaps.
Try disabling that in the BIOS.

Steve