pfSense freezes on Saturdays in the morning and shortly after 4pm.

globus243

Yes, that is exactly what I am going through here :D

One week after the update to 2022.5-Release, on a Saturday, in the morning, I notices a complete outage of routing in my network. Only devices in the same vlan were able to communicate. after further investigation it seemed to be the pfSense that freezed up completely. accessing the display through esxi, I was unable to make it react to any input. After a simple reboot the pfSense started to work again as expected. I thought it was a fluke and brushed it off. The same day at 16:10ish the same thing happened again.
Now I wanted to investigate but no internal log had a clue on what was going on. So i opened this thread to make sure that the next time this happens I'll get a log: https://forum.netgate.com/topic/173351.

Today, Saturday, again in the morning the same issue occurred. And again at 16:30ish. Booth times the serial log did not contain anything of value, just the same as the "screen", the menu and maybe a log entry for the last login try.
The days between, everything works fine and no issues.

I am completely at loss now, I still have the backup of the older version and I will probably just play it back and wait this out, if no one has an idea what could be done here. Though I am intrigued to see if it does the same thing next week also.

Any input highly appreciated, best,
Tim

NollipfSense

@globus243 Without showing us a log, we can only speculate...maybe it's allergic to Saturdays...it would be normal if it was 4:20.

johnpoz

@globus243 Do you have something scheduled to happen every 7 days - like pfblocker update maybe?

Off the top I am not aware of anything schedule every 7 days out of the box?

Only thing that comes to mind is dhcp, what is the lease you get from your isp? But not getting an IP from dhcp shouldn't cause a lock up.

Do you have anything setup in esxi that happens, like a vm backup or snapshot or something?

stephenw10

Check the monitoring graphs for anything that might be climbing to exhaustion during the week.

Try hitting 'ctl+t' at the console when it seems to be unresponsive.

Steve

globus243

@nollipfsense said in pfSense freezes on Saturdays in the morning and shortly after 4pm.:

@globus243 Without showing us a log, we can only speculate...

As I said, all logs look normal, the ones in the UI as well as the serial log, everything is behaving normal in one moment and in the next we need to reboot. Their is no event in my logs indicating what's going on.

globus243

@johnpoz said in pfSense freezes on Saturdays in the morning and shortly after 4pm.:

@globus243 Do you have something scheduled to happen every 7 days - like pfblocker update maybe?

Off the top I am not aware of anything schedule every 7 days out of the box?

Only thing that comes to mind is dhcp, what is the lease you get from your isp? But not getting an IP from dhcp shouldn't cause a lock up.

Do you have anything setup in esxi that happens, like a vm backup or snapshot or something?

There sure are some jobs that run regularly but on much tighter schedules than 1week. (pfBlocker runs every hour, snort every 6 for ex.). Your suspected DHCP lease is also much more frequently renewed.

yaminb

I've noticed 2 unexplained lockups recently. It's very much unlike pfsense. I tossed them into the random lockup bucked and ignored them. Both required me to manually powercycle the unit.

One happened just this morning at around 9:30 AM. Why I found myself here. Only activities going on were my kid was watching Netflix and I was logging in to work (VPN). So I assume fairly light load. I don't have anything scheduled to run at this time. I tried looking at the logs, but didn't see anything unusual. However, I am not the best at analyzing the logs. I could definitely have missed something.

I have a Netgate 3100 that has been rock solid for a quite a well.

I'm trying to think of changes in recent history that might be at play

I did update to 22.05
I switched ISP providers. In Canada. I went from Rogers to Bell. Only odd thing here was to get the Bell modem (Homehub 4000) to integrate with my pfsense, I configured the HH4000 to use the pfsense in a DMZ. One setting changed, I didn't play around with anything else.

stephenw10

Do you see anything in the logs at that time? Or is the first thing that is logged after the reboot?

globus243

Hey @yaminb

glad i am not the only one.
while it did not freeze last saturday, it did so yesterday and today at random times.

Today my ESXI also threw out one of the drives in the array as faulty, so while it seems verrrry unlikely I have to replace the drive and wait it out another week.

Also the serial log does not contain a clue. last entry in my serial log was the pfsense console menu, its frustrating.

yaminb

@stephenw10
Nothing in the logs, first thing is a fresh boot.
I also looked at the Monitoring (CPU, memory, temperature...) Everything normal for the past week.

stephenw10

Hmm, if it's happening frequently you can try connecting a serial console and logging it to see anything is shown there when it resets.

globus243

@stephenw10, @yaminb
that would interest me aswell, as said I have nothing in my (serial-)logs which makes diagnosing this pretty much guess work.

globus243

Update:
It still happens, now more randomly also on weekdays.
I was fed up enough now to re-deploy my old backup and see if that solves it for now.

A Former User

@globus243

ml350p gen8 with 2x E5-2690 ~200Gb ram, VMware
ESXi, 7.0.3, 19482537

Is this your rig (hardware)? Were you taking the ESXi image from HP or the normal one?

Did you set up much apps such as snort, clamav (squid-http)
and/or pfBlocker-NG? And when yes, what is the time to get fresh, rules, signatures or lists? (Update I mean)

And the Bell Home Hub 4xxx is a real modem or a router?
Do you set it up in bridged mode?

globus243

@dobby_ T

Yep it ist-
The HP Image is pretty outdated so I used the official one from vmware.
I disabled most of the monitoring stuff but still have: pfBlocker, Snort, Telgraf.

My Modem is a Telekom Speedport Smart 3 in modem-mode directly connected to the pfSense.

Also since I reverted to my old backup 10 days ago I had no freezes.

So the error was either introduced during the update or the newest pfSense version has a bug.

A Former User

@globus243

So the error was either introduced during the update
or the newest pfSense version has a bug.
I don´t know I was really testing and using them here for a week or some days plus or less. All were running fine.

APU4C4 as test device, together with;

2.6 CE to 22.01 to 22.05 - without any problems
2.6 CE to 2.7 Devel to 22.09 Devel - without any problems

The HP Image is pretty outdated so I used the official
one from vmware.

Often or better not seldom from this problems came.

My Modem is a Telekom Speedport Smart 3 in
modem-mode directly connected to the pfSense.

In Germany it is for a home account or internet connection
normal to have a break every 24 hours, perhaps and if not running in modem mode it comes from there! Something in the set up let your say at witch day or time the break will com or ocours. Have a look there too please.

globus243

@dobby_ said in pfSense freezes on Saturdays in the morning and shortly after 4pm.:

@globus243

So the error was either introduced during the update
or the newest pfSense version has a bug.
I don´t know I was really testing and using them here for a week or some days plus or less. All were running fine.

APU4C4 as test device, together with;

2.6 CE to 22.01 to 22.05 - without any problems
2.6 CE to 2.7 Devel to 22.09 Devel - without any problems

The HP Image is pretty outdated so I used the official
one from vmware.

Often or better not seldom from this problems came.

My Modem is a Telekom Speedport Smart 3 in
modem-mode directly connected to the pfSense.

In Germany it is for a home account or internet connection
normal to have a break every 24 hours, perhaps and if not running in modem mode it comes from there! Something in the set up let your say at witch day or time the break will com or ocours. Have a look there too please.

Thing is non of these issues appeared one pfSense version prior. It only started after the update and it stopped after reverting is. I am reluctant to see a hardware issue on my end.

Also, yes modem is definitly in modem mode and I am aware of the reconnect and they never coincided with the freezes

JKnott

@nollipfsense said in pfSense freezes on Saturdays in the morning and shortly after 4pm.:

Without showing us a log, we can only speculate...maybe it's allergic to Saturdays...

Probably likes to sleep in too.

globus243

@jknott said in pfSense freezes on Saturdays in the morning and shortly after 4pm.:

@nollipfsense said in pfSense freezes on Saturdays in the morning and shortly after 4pm.:

Without showing us a log, we can only speculate...maybe it's allergic to Saturdays...

Probably likes to sleep in too.

yep, as I said, no logs that indicate what's going on. I even activated serial logging to see whats happening but even here: 1 sec it's fine, next we are seeing the output of a booting OS

Gertjan

@globus243 said in pfSense freezes on Saturdays in the morning and shortly after 4pm.:

next we are seeing the output of a booting OS

To have an OS to boot, it needs to jump through the boot vector.
That's a call that might exist in the OS - and can only be executed by the OS in 'root ' (ring 0) mode. Freezing up is far more common. If it's a OS controlled reboot, there will be a message.

That is : if it can store a message on the disk. If the file system is dirty, the disk might even not be writeable ( check that with the help of the official Netgate "fsck" Youtube video ).

It's always good to look at the logs, for 'OOM' messages and the like.
Or other "normally never seen" messages.
If there are none, then there is only one thing to do :
Lower the code foot print : remove all extras, all not really needed software, just the bare bone basics.
Just to be sure that, after a while, the freeze up only happens when you use 'x' (like a package).

It still happens ? It's most probably a hardware issue. Or a bad power rail.

I know, this won't help you, but it is important : if I don't touch my pfSense, it runs for weeks if not months ... and I have the logs, better : the images to share with you.

You'll say : wtf, you system restart often !! True, as I reboot often to test things out - or just to be sure a reboot (in my absence) will bring back the Internet online.