Help! pfSense died...

deanfourie

So, I need help! There was a power cut last night and since booting this morning, I am faced with the following error.

panic: mountroot: unable to (re-)mount root.
cpuid= 0
time= 111
KDB: enter: panic
stopped at kdb_enter+0x32: movq            $0,0x2344e43(%rip)

Any ideas? Im not super good with BSD.

Thanks!

SteveITS

@deanfourie you can try this if you’re not using ZFS:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html#manual-filesystem-check

Or reinstall and restore your config file from backup.

deanfourie

@SteveITS but can’t install pfsense + anymore.

deanfourie

@SteveITS does a config restore restore my pfsense + licence?

USSZulu

@deanfourie I am not sure if your Home and Lab license is still allowing downloading Pfsense+, however, as far as I am aware, your Pfsense+ configuration file should restore fine to Pfsense CE 2.7 if you were using Plus 23.05.1 or to CE 2.7.1 if you were using Plus 23.09

deanfourie

You know, I’ve always been a great pfsense fan.

But honestly, why is updating such a nightmare! I’ve had to reinstall 3 times trying to update to the latest version because the updates are so flakey.

You really need to sort this out.

Gertjan

@deanfourie said in Help! pfSense died...:

You really need to sort this out.

I understand what you mean.
Don't be offended, but let me get this out of the way right away :
pfSense didn't die, it was the device, probably the disk, the file system that went bad.
I'm taking my 'pfSense' very serious, so, as the PABX (company), is battery back-upped since day 1.
And even if the power doesn't go down, its the disk(s) that go bad.
So, for critical devices like your main firewall, have a plan B and C ready. if you don't, you'll be needing it (Murphy's law)

I'm updating (mostly) pfSense since version 0.99. Ones in a while, an install was needed, as I changed hardware. The last time I installed a fresh pfSense was because I had to go from UFS to the more resilient ZFS : a file system change.

The last update, from 23.05.1to 23.09, could be resumed to "a single click". But it always takes 10 minutes or so to prepare the upgrade, as I have build a check list that contains all the knowledge acquired over the last 10+ years.
Be assured : the list contains only logic steps like : reboot the system before the upgrade, get a copy of the system configuration. Check the UPS status. etc etc.
And right after the upgrade, over 48 hours or so, I check the most important functionalities.

NollipfSense

@Gertjan said in Help! pfSense died...:

I have build a check list that contains all the knowledge acquired over the last 10+ years.

My younger brother have been encouraging me to do just that, but me, the stubborn older one foolishly wants to trust memory out of laziness...

mer

@NollipfSense Of course if one writes it all down, then one forgets where you left the notebook or paper :)

stephenw10

You can still reinstall and upgrade to Plus again as long as your NDI didn't change (which it wouldn't if you're just reinstalling).

But, yes, if you're running ZFS just roll back to the last snapshot.

If you're running UFS then run a manual filesystem check:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html#manual-filesystem-check

deanfourie

Yea, I suspect the device died due to a power cut. Looks like the file system was corrupted as a result.

Have done a full restore from backup now, and upgraded to the latest version of Pfsense +

Upgrading was a bit of a nightmare but I think it just continues to take some time to “sync” after the update is complete

Thanks guys

SteveITS

@deanfourie On the plus side a reinstall would use ZFS which is less susceptible to file system corruption.

mer

I'm going to put my opinions out there; agree, disagree, all good.

UPS/Battery backup. I have always had critical systems on UPSes. Firewall device, cable modem, wifi AP? all on UPSes.
Had times where wife watching TV while power out asks "what is all this beeping?"

Beyond that, again my opinion, pfSense updating has been relatively easy.

SteveITS

@mer said in Help! pfSense died...:

watching TV while power out

TV is "critical system" (anything a spouse is doing is "critical")

I tried that and immediately found my UPS was too small for the power draw from the TV (along with everything else but TV was by far the most).

mer

@SteveITS Yep. TV is a vizio flatscreen (basically a big monitor) on it's own generic UPS from walmart. pfSense, WiFi, Cablemodem all on different UPS :)