How to debug random reboots of m1n1wall running 2.1.5-RELEASE?
-
Hello – I'd love to get some help figuring out how to debug a m1n1wall running 2.1.5 that's begun rebooting itself once or twice a day without any apparent cause.
I have a m1n1wall (a 2D13) that I've been running for a few years now, and I've started to see nearly-daily reboots that I can't figure out. It's been running pfSense 2.1.5-RELEASE for a few months, and I'm pretty sure I've only started seeing the reboots since upgrading to 2.1.5... but I can't swear to it. And since the system logs don't persist across a reboot, I can't figure out what's happening immediately before the system decides to reboot.
The reboots are not at the same time every day; sometimes it reboots once in the early morning, other times it's in the evening, other times it's a few times a day.
I've tried to use remote syslog monitoring, but nothing gets pushed out to the remote syslog immediately prior to the reboots.
In the end, I'm not entirely sure what the next steps are... I guess at a minimum it'd be helpful to figure out how to write the system logs to non-ephemeral memory so that I can see what happens immediately before a reboot.
Thanks in advance for any help!
-
The best thing to do is to connect a serial console client up (null modem serial cable, etc) and set it for a huge scrollback buffer – maybe 30,000-50,000 lines -- or to log output to a file. Then leave it connected and wait. If it crashes it should print some info which can be looked over. If it seems to reboot with no output, then the next time leave it at a shell prompt running:
clog -f /var/log/system.log
And then see what happens when it reboots.
If nothing unusual is logged at all, then it probably is either power or hardware.
Some more ideas here, but most are only relevant to a full install:
https://doc.pfsense.org/index.php/Unexpected_Reboot_Troubleshooting -
From the missing-the-forest-for-the-trees department, I hadn't even thought about dumping the serial console. I have a machine hooked up to it now, and will report back if/when I figure this one out. Thanks!
