Can't access pfSense VM after power failure
-
We had a bad power failure and after bringing everything back online, I had issues with pfSense, which is a Hyper-V VM. This is not the first power failure, and every time it happens the pfSense VM restarts into an endless run of error messages of the same basic format:
"Cannot use a scalar value as an array in /etc/inc/config.lib.inc on line [number]"
Based on searches, this type of error is the result of a corrupt filesystem. So I revert the VM to a previous state, and it works fine, excpet I have to reconfigure a bunch of stuff. I got wise after a while and saved the configuration from within the pfSense GUI. So this time, when I got the error above, I reverted the VM, then went into the GUI and reloaded a saved configuration. Here's where the trouble really began….
After loading the configuration file, it told me that pfSense would reboot the firewall. It went to the following screen in the Hyper-V window, and stopped there. I also no longer had access to the web GUI, yet I still could browse the internet.
I rebooted the host machine, restarted Hyper-V, and got the exact same window as I did before. That's right, the exact same screen saying that I logged in at 20:12:37 (which was before the full reboot) and that pfSense is shutting down. The Hyper-V window does not respond to the keyboard, even the CTRL-ALT-DEL from the drop-down menu. I can still browse the web, but I cannot access the webGUI for pfSense.
-
a) get yourself a decent UPS
b) make a copy of a freshly installed pfSense VM and apply your config to that machine.
c) keep the freshly installed pfSense VM at a safe place for future use
d) delete a corrupted VM and don't look back.
e) get yourself a decent UPS!Ah, did I mention to get a UPS?
-
a) get yourself a decent UPS
b) make a copy of a freshly installed pfSense VM and apply your config to that machine.
c) keep the freshly installed pfSense VM at a safe place for future use
d) delete a corrupted VM and don't look back.
e) get yourself a decent UPS!Ah, did I mention to get a UPS?
Hahaha, I knew the UPS issue would come up. Unfortunately, my budget is zero only because it can't be less than zero; that's why I'm using pfSense in the first place– I couldn't afford to buy a new router.
I figured I should probably create a new image, which I intend to do over the weekend. Any chance I can crack the existing one and gain access to the GUI again?
What sort of filesystem integrity checks does the underlying BSD system have? It seems very sensitive to unexpected shutdowns, which are more likely as a VM because of the dual threat of host machine power failure and Hyper-V software crashes. I can't recall a single time since I started using pfSense that I didn't see some sort of boot message on the VM about filesystem errors. Is there a magic combination of BSD filesystem (UFS, ZFS, ext_) and physical drive formatting (NTFS, ReFS)? I use a dedicated SATA drive for the VM repository. Yes, I have checked it for errors several times with no issues.
-
I have had power outages, ups ran out without proper shutdown, etc. Have never had such issues.. I have seen pfsense report unclean, but then just cleans it up without any issues.
If you can not afford a ups, then just make sure you have a full backup of your VMs - so that if such an issue occurs you can redeploy your VMs.
UPSes are not that expensive to be honest… I can not stress enough the use of UPS!!! And then setup of your devices to correctly shutdown when battery power is low.. You have no budget but clearly you have a box that runs hyper-v, and internet access, etc. etc.. Your problem is that you have not put enough priority on the need of a ups, and spent that money on other things ;) Adjust your budget to allow for a ups purchase sooner vs later would be my advice..
Cyberpower makes some very budget friendly models... It doesn't have to be all that large, just needs to provide enough runtime to proper shutdown your stuff based upon the load, etc.
-
You risk file corruption if writing is going on when power is removed from the pfSense VM. If you can't afford a UPS then you should definitely make a valid backup and take a snapshot or whatever they're called under Hyper-V so that you can revert to a working firewall if your current instance gets corrupted.
-
Hahaha, I knew the UPS issue would come up.
With all due respect - I would never run a hypervisor without a UPS (if my time is worth something to me)
According to your post you had these problems more than once. May I suggest you look for an embedded install where the file system is usually mounted read-only and more robust to power outages.
