2x pfsense 24.11 hard crashes in under a week - Netgate 1537

Gertjan

@joekislo said in 2x pfsense 24.11 hard crashes in under a week - Netgate 1537:

Both units probably at some point OOMED and ran out of swap before the kernel killed bsnmpd to survive

As long as disk (partition) space isn't an issue, these kind of events can be found in the system log.
For memory (RAM), as soon as the system is running again, look at Status > Monitoring ans select System => Memory.

joekislo

This post is deleted!

joekislo

@Gertjan Gertjan No evidence of OOMs before the event, disk space was plenty good as well. FWIW the OOMs were months ago, and I sent all the details to Netgate. Memory has been stable since we started the weekly bnsmpd restarts.

The dip is the unit hard reset.

stephenw10

Did you try entering ctl+t at the console when it wasn't responding? That can sometimes show output when nothing else will.

Has this only happened once? On each node?

SteveITS

@joekislo said in 2x pfsense 24.11 hard crashes in under a week - Netgate 1537:

Jul 24 00:19:20 fw1 syslogd: exiting on signal 15

Just to confirm you think this entry is when you hit the Reset button to reboot?

We have a 4200 that put itself in standby (according to its LED) on its own, and logged that at the time, like it shut itself down. Haven't seen that anywhere/anytime else, though.

jcleaves

@SteveITS Was there anything in the logs regarding going to Standby? I didn't see any power event logged.

SteveITS

@jcleaves Nothing mentioned standby, shutdown, etc., in fact nothing for the previous hour before the "exiting." However we were running a RAM disk on it which we normally do. But the 4200 has a standby LED pattern. Our occurrence was in early June. TAC had us reinstall it.

Edit: I understand our situation may not be relevant here.

stephenw10

If you press the ATX power button that's what you would see logged:

Jul 28 21:57:35 	php-fpm 	8456 	/index.php: Successful login for user 'admin' from: 172.21.16.8 (Local Database)
Jul 29 00:21:11 	syslogd 		exiting on signal 15
Jul 29 00:23:07 	syslogd 		kernel boot file is /boot/kernel/kernel
Jul 29 00:23:07 	kernel 		---<<BOOT>>---
Jul 29 00:23:07 	kernel 		Copyright (c) 1992-2024 The FreeBSD Project.
Jul 29 00:23:07 	kernel 		Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994

You can disable that by setting the sysctl hw.acpi.power_button_state=none

SteveITS

@stephenw10 In our case a button was not pressed, per the one person in the office. It would be nice if it logged a "button push" event.

jcleaves

@stephenw10 This was definitely not a button push on ours either. Both units are in locked cabinets in a colo. Any access to the facility is logged.

@SteveITS As for it going to standby or hibernating, the person who went on site the LEDs were normal. Nothing indicating a state change or issue.