pfSense "Random" reboots
-
Recently we have noticed a rash of reboots. In October we updated to 22.05 from 2.4.5. Prior to that upgrade it had been many month since we had rebooted. (over a year)
We rebooted. Upgraded and rebooted twice for changes to take place. Since then we had some configuration issues we worked through and rebooted several times manually and on purpose.At least three times since those problems were resolved we have seen what appears to be random reboots of the firewall. What causes this? logging doesn't show anything...
From /var/log/system.log: Jan 14 18:39:00 fwpfng sshguard[77161]: Exiting on signal. Jan 14 18:39:00 fwpfng sshguard[18131]: Now monitoring attacks. Jan 14 21:32:00 fwpfng sshguard[18131]: Exiting on signal. Jan 14 21:32:00 fwpfng sshguard[43015]: Now monitoring attacks. Jan 15 01:13:00 fwpfng sshguard[43015]: Exiting on signal. Jan 15 01:13:00 fwpfng sshguard[56467]: Now monitoring attacks. Jan 15 03:01:00 fwpfng root[30342]: rc.update_bogons.sh is starting up. Jan 15 03:01:00 fwpfng root[31979]: rc.update_bogons.sh is sleeping for 51391 Jan 15 07:08:00 fwpfng sshguard[56467]: Exiting on signal. Jan 15 07:08:00 fwpfng sshguard[79354]: Now monitoring attacks. Jan 15 15:35:12 fwpfng syslogd: kernel boot file is /boot/kernel/kernel Jan 15 15:35:12 fwpfng kernel: ---<<BOOT>>--- Jan 15 15:35:12 fwpfng kernel: Copyright (c) 1992-2021 The FreeBSD Project. Jan 15 15:35:12 fwpfng kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Jan 15 15:35:12 fwpfng kernel: The Regents of the University of California. All rights reserved. Jan 15 15:35:12 fwpfng kernel: FreeBSD is a registered trademark of The FreeBSD Foundation. Jan 15 15:35:12 fwpfng kernel: FreeBSD 12.3-STABLE plus-RELENG_22_05-n202700-3ddaea61055 pfSense amd64 Jan 15 15:35:12 fwpfng kernel: FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303) Jan 15 15:35:12 fwpfng kernel: VT(vga): resolution 640x480 Jan 15 15:35:12 fwpfng kernel: CPU: Intel(R) Atom(TM) CPU C2558 @ 2.40GHz (2400.06-MHz K8-class CPU) -
Do you see a crash report after rebooting?
Anything in the system monitoring logs look like it's being exhausted?
-
@stephenw10 said in pfSense "Random" reboots:
Do you see a crash report after rebooting?
I have not found that in the system. Where should I look?
Anything in the system monitoring logs look like it's being exhausted?
The log snippet above and an email that gets sent to us is the only indication so far it rebooted. What else can I check?
-
@jc2it @stephenw10 I was reading through Obtaining Panic Information for Developers and it did not generate a panic I guess. At least there are no textdump files in /var/crash.
I see this note near the end. "On hardware with a serial console, connect to the serial console and record the scrollback buffer when a crash happens, or there may not be a way to retrieve the crash dump otherwise." What if the serial console is disconnected will it save a crash dump?
-
Possibly C2000 series bug:
https://www.servethehome.com/intel-atom-c2000-series-bug-quiet -
@adriftatlas C2000 bugs generally prevent a device from booting at all ... it generally does not cause reboots
-
Yup, that. Unlikely it would boot again if it was hitting AVR54.
Sorry I typo'd earlier, check the System Monitoring Graphs in Status > Monitoring. That can show things like memory exhaustion.
It's possible this is a drive issue. That will prevent logging any errors but usually presents as a cascade of service failures rather than an reboot without any issues before that.
Steve
-
@stephenw10 @AdriftAtlas @heper
I looked around in the System Monitoring Graphs section. I don't think it is an utilization issue. Here are two examples and the system was barely doing anything on Sunday when it went down. I am going to verify the UPS is functioning correctly. I normally get notified when the power drops but perhaps this ups is misconfigured or malmunctioning and it is a coincidence it began about a week or two after the upgrade. The System finished rebooting at around 15:38 local time on 1/15
Memory Utilization for the last week is less than 6%
System Utilization for the last week is less than 7% peak
-
Yeah, not seeing anything that looks like a memory leak or CPU use.
If you can hook up a console and log that you might catch something that doesn't get entered into the logs.
