Automatic reboot in case of problem or non-manageable access
-
We use pfsenses for remote sites and we use OOB devices to manage them in case of emergency.
For most management tasks, web and SSH interface is enough but there are some situation where OOB device is needed. OOB device is connected via serial port and has its own 4G connectivity.
So we can manage devices even if there is some problem with upgrade/connectivity.Unfortunately, there are still some situations where firewall get "stuck" and we cannot manage it even with OOB device.
- When there is an panic and firewall needs to be physically turned off and on
- When we loose init shell. This happens if you break boot process, run some bg job and exit from shell. It than waits for bg job but shell is not accessible anymore.
It would be great feature if firewall can recover from this automatically.
We found parameter debug.debugger_on_panic=0 which can be added to boot options. This should help with panic and reboot automatically. Unfortunately, this is not configurable and it is 1 by default.Second problem has probably no easy solution but this happens only in rare cases when boot process is stopped manually and init shell is dead. Probably watchdog can help here (even software watchdog).
In general, my opinion is that it is better to reboot firewall if it is not functioning than wait for management access and would be nice to cover all possible states so this will be automatic behavior.
Any suggestions here please?
Thank you
Lukas -
@limos said in Automatic reboot in case of problem or non-manageable access:
We found parameter debug.debugger_on_panic=0 which can be added to boot options. This should help with panic and reboot automatically. Unfortunately, this is not configurable and it is 1 by default.
It's a loader variable, you would need to enter in /boot/loader.conf.local. Though entering the debugger should not prevent you rebooting it.
There are hardware watchdog drivers built into pfSense if you have hardware that supports it.
Steve
-
@stephenw10 The main problem is not that we can reboot in case of kernel debugger in action. Main problem is that we want this to happen automatically, without any activity from the user. I think this is almost equivalent to panic=x settings on linux which is used widely on network devices. There is no reason for device to wait for management in case of crash.
I know we can change this boot parameter in file. We did on one firewall as a test. I just wanted to discuss this solution and ask if this should not be standard (or at least configurable) behavior. But for now, we will just update /boot/loader.conf.local and it should be enough.
Regarding watchdog, I did not find any documentation how exactly it is used within pfsense.
Which process "pings" to it? Is this process configurable? Or is this kernel based watchdog without any userspace pings?
Is there any possibility to use software watchdog where hardware watchdog is not available? There is such implementation in Linux but not sure about FreeBSD.Thank you
-
If you disable the debugger like that my understanding is that you will get no backtrace or crashreport of any kind so solving issues becomes far more difficult.
Watchdogd is used so it does have some software watchdog capability.
Steve
