PFsense reboots at the same time every day

bennetbj11

We have a weird issue with a PFSense 7100U, it reboots at exactly 04:05 every morning. This started happening about 10 days ago, there is nothing in the logs about any issues, I dialed into this morning at 04:00 and there was no unusual CPU, MEM spikes, ETC. in the system log it just shows boot time.
I have copied the config off it and applied it to my lab PFSense and it works perfectly as in no reboots.
Anyone know why this might be happening? I am stumped.

Gertjan

@bennetbj11

That's not the OS who starts to execute a boot sequence. The 'user' or OS initiated shutdown leaves traces in the system log.
You've none.

Power issue ?
If you have UPS : remove it for a day or two.
You have no UPS , use one for a day or two.

Worst case solution : be present, connected at the console, at 04h05.
( or stop/kill the NTP sync process, advance the system clock to 04h00 so you make pfSense believe it's 04h00 and see what happens ).
if the system still goes belly up at the real 04h05, while the pfSense says it some other hour, you know the reboot cause comes from the exterior.

Normally, LAN(s) or WAN(s) cable disconnect (or the switch or device on the other side going down) should not reset/reboot pfSense.

bennetbj11

@Gertjan
Thanks for the reply, we have tried different power leads, and also plugged into a another UPS with no luck, we have no other power issue in the Comms room this is the only device that reboots.
I will try the NTP sync process and see what happens, will keep you posted.
Cheers

Gertjan

@bennetbj11

Change the 'admin' password and don't tell anyone ;)

SteveITS

@bennetbj11 If you install the cron package you can see all scheduled jobs even if you don’t edit any.

stephenw10

Can't see from that screenshot but does it show a shutdown or reboot entry above that?

If nothing is shown then it seems like a power/hardware issue. If that's at exactly the same time everyday I'd look for something starting at that time causing voltage spikes. Assuming no UPS.

Gertjan

@SteveITS said in PFsense reboots at the same time every day:

@bennetbj11 If you install the cron package you can see all scheduled jobs even if you don’t edit any.

It can't be a cron. I mean, @bennetbj11 wouldn't have to put a "reboot cron task @04h05" and then come here asking why his pfSense restarts at @04h05
Right ?
That's would be ... strange.

( I exclude split personality issues, or 'the entire family is admin' situation. )

SteveITS

@Gertjan Not to reboot, I'm just thinking of "some task running completely off the rails" which would of course be hard to do. If nothing runs at 4:05 then OP can move on.

Gertjan

@SteveITS

I get it, pretty sure myself also there is actual 'reboot cron' there.
Could be anything, like the pfSense package pfBlockerng (just an example) or what ever else that has some maintenance task like 'updating DNSBL' or log rotation or something else.

That said, a process that 'dies' at the same moment every day, and take the system with it, that's a rare case.

bennetbj11

@SteveITS sorry for the long delay, Cron installed and nothingin the CRON jobs to suggest any reboots, it is just got the standard list of jobs with no adjustments.

Also, as it has been a while the reboot has moved back exactly 1 hour to 03:05 each day now since the clocks have changed.

Any other Ideas? I am lost as to why this device is doing it.

mephib

In your original post, it is unclear by your wording if you monitored the system activity (cpu, mem, etc.) at 04:00 or if you only monitored the system log at that time. You said dialed in, so I'm assuming your remote connection was dropped right at 04:05? I would prefer to be directly at the console monitoring system activity when this occurs if at all possible.

I see the log is in descending order and 04:03 seems normal, but It would be helpful if you could post perhaps the previous hour leading up to that.

It also might be helpful if you could post what the 7100U interfaces are connected to.

keyser

@bennetbj11 said in PFsense reboots at the same time every day:

Also, as it has been a while the reboot has moved back exactly 1 hour to 03:05 each day now since the clocks have changed.

Any other Ideas? I am lost as to why this device is doing it.

Okay that really is a strange case. Since the reboots moved back exactly one hour with daylight saving i assume, then the cause has to be external. Since the hardware/pfsense clock moved with daylight saving, then the reboot should remain at 4:05 if it was some internal service/timebased scripts or such.

since it didn’t it must be external. So I would look at @stephenw10’s suggestions about power surges. But I don’t think you’ll diagnose this unless you attempt sitting at the console on the box when it happens. Does it throw any errors before hardbooting - fx. No diskstorage found/as in dead SSD/eMMC.