pfSense repeatedly rebooting at exactly 12:30
-
Do you have any packages installed, filtering any traffic or creating log data from filtering, that would be filling up either your RAM or storage, and crashing your pfsense box?
Also, what kind of hardware is this running on - DIY hardware, VM, or an official Netgate machine?
Jeff
-
I have iperf, openvpn-client-export, and suricata.
I am not using suricata right now and I am not creating logs.
Current storage utilization is 45% of 120GB and ram utilization is hovering right around 6%
Its an HP 1U server I re-purposed, Dual Xeon X5550, 24gb ram, 120gb ssd
Thank you for the reply Jeff I really appreciate it!
-
How long has this hardware been running? The fans spinning up suddenly before crashing might be a heat problem with your hardware, but it's a tough issue to diagnose over a forum post. Might be time to power it down, pop the cover and check for dust.
Also, most crashes with pfsense installations are either hardware related - bad RAM, failing power supplies or hard drives, other stuff, or the other reason is because of FreeBSD kernel panics.
https://docs.netgate.com/pfsense/en/latest/hardware/unexpected-reboot-troubleshooting.html
Jeff
-
Do you by chance have a UPS associated with this server connected with a USB cable? If so, it's possible the UPS is trying a self-test once per day and experiencing a battery problem. That could falsely trigger the UPS monitoring daemon (if you have one installed) to reboot the machine.
I would look in the pfSense system log to see what messages, if any, are being logged around that time period. The very first place you should look when any kind of strange behavior occurs is the system log to see what may be getting logged that could be hints as to what the problem might be.
-
Just popped the cover off. It's basically dust free in there. I've been running it for about a year.
I do have it plugged into a UPS but I have another server managing it (usb is connected to the other server)
Took a look at the logs. There are too many of them for me to see what may have happened at 12:30. However everything that comes after that seems to indicate a restart just occurred. I'm increasing the maximum log entries and ill report back after 12:30
Thanks Guys!
-
Have you looked at your crontab? What runs at 12:30?
-
@jwj Really stupid question but how would I do that? Is that something I can access through the gui or should I ssh?
Thanks
-
Install the Cron package.
-
Sorry for the delay responding. There is a cron package you can install. That's the easiest way.
From mine I think it's the update URL tables job that is biting you.
-
@RonpfS Excellent thank you. I think I found the issue. it was running this
/usr/bin/nice -n20 /usr/local/bin/php-cgi -f
/usr/local/pkg/suricata/suricata_check_for_rule_updates.phpat exactly 12:30 every day
And that you to everyone else that helped! I really appreciate it!
-
@TBBZ8X8 said in pfSense repeatedly rebooting at exactly 12:30:
@RonpfS Excellent thank you. I think I found the issue. it was running this
/usr/bin/nice -n20 /usr/local/bin/php-cgi -f
/usr/local/pkg/suricata/suricata_check_for_rule_updates.phpat exactly 12:30 every day
And that you to everyone else that helped! I really appreciate it!
That job should not cause a reboot. It will restart Suricata at the end of the task, and if you have Suricata running with Inline IPS Mode enabled (which uses the kernel
netmapdevice), that will cause the physical NIC interface to be disabled and then re-enabled during the restart. But it should not cause a physical reboot of the box itself.How do you have Suricata configured? Is it using Inline IPS Mode? If so, you can try enabling "Live Rule Swap" on the GLOBAL SETTINGS tab. That will not cause the Suricata daemon to stop and restart itself. Instead it will load new rules into memory and then swap over to using them. That will then prevent the
netmapdevice from restarting the physical NIC interface. The downside of this option is that for a small interval Suricata will consume nearly twice normal memory as it will keep two copies of your enabled rules in memory until it can get pointers updated to use the new rules so the old ones can be deleted from memory. -
@bmeeks Thanks for the reply!
I had switched the time of the job to when I knew no one was on the network but turns out it resets that when suricata updates. So i changed it in the suricata settings and enabled live rule swap like you suggested.
Finger crossed it stays working this time!
Thanks again!
-
@TBBZ8X8 said in pfSense repeatedly rebooting at exactly 12:30:
@bmeeks Thanks for the reply!
I had switched the time of the job to when I knew no one was on the network but turns out it resets that when suricata updates. So i changed it in the suricata settings and enabled live rule swap like you suggested.
Finger crossed it stays working this time!
Thanks again!
You must change the update job time on the GLOBAL SETTINGS tab. Suricata rewrites its configuration, including things like the cron task start time, each time a change is made and saved in the GUI or the "resync packages" command is called by pfSense itself.
As mentioned in my earlier post (after coming back and fixing some terrible typos I made ...
), Suricata updating should never reboot the entire firewall. If that happens, something is really bad wrong. However, when using the netmapdevice in Inline IPS Mode, thenetmapdevice itself will restart the NIC interface when Suricata is stopped and then restarted during the rules update. So swapping over to the Live Swap option will prevent the physical stop/start cycle of the Suricata daemon and thus also the cycling of the NIC interface bynetmap. -
Is it really rebooting? What's the system uptime?
