pfSense/ROOT/default failed to mount with error 5

deanfourie · Dec 17, 2023, 8:49 AM

I am having a VERY frustrating issue with pfSense on a Sophos XGS117 hardware. This issue seems to be new as I dont recall having this issue prior to the latest pfSense version.

I'm not sure if this is directly related to pfSense or more a FreeBSD 12 issue, but everytime the firewall is not recoverable and results in the need to completely reinstall. This seems to only be the issue if the power is lost suddenly.

Running latest version of pfSense Plus. if I manually shutdown, the I can restart without issues, however, pull the power cable out and its toast.
I am using ZFS filesystem.

pfSense actually boots, but it then fails to MOUNT the pfSense/default partition? See screenshots below.

Any ideas to a fix is greatly appreciated.

Sophos XGS116
64GB SSD (m2)

login-to-view

stephenw10 · Dec 17, 2023, 9:02 PM

What devices are shown if you enter ? there?

What value do you have set (if any) for kern.cam.boot_delay in the loader?

deanfourie · Dec 17, 2023, 9:42 PM

@stephenw10 Thanks, I have reflashed the device as I need it for internet. Reinstalled and restored from backup.

Is there a way I can check the boot loader for this with a running device?

stephenw10 · Dec 18, 2023, 2:19 AM

Yes you can see what it's set to by running: sysctl kern.cam.boot_delay

deanfourie · Dec 18, 2023, 2:19 AM

@stephenw10 Thanks output is,

kern.cam.boot_delay: 10000

stephenw10 · Dec 18, 2023, 2:58 AM

Hmm, not that then.

So it only does this if you hard reset the device? And it does it every time?

deanfourie · Dec 18, 2023, 3:25 AM

@stephenw10 Yes, correct.

I have hard reset 3 times, mostly mistakes on my part, and everytime now I have had to reinstall pfSense from scratch as I CANNOT for the life of me get past the mountroot screen.

Seems like if I shut it down gracefully, all is good.

stephenw10 · Dec 18, 2023, 3:33 AM

Hmm, I've never seen ZFS fail like that without something else in play It's normally very resilient to that.

deanfourie · Dec 23, 2023, 10:15 AM

@stephenw10 This just happened again today.

Forced me to throw a Mikrotik in temporarily.

I really need to fix this but not sure where to start.

What could be causing this? Any idea on how to troubleshoot?

stephenw10 · Dec 23, 2023, 3:55 PM

What pfSense version are you using? 23.09.1/2.7.2 included ZFS fixes.

deanfourie · Dec 23, 2023, 10:08 PM

@stephenw10 I was using latest version of pfSense +.

Fully up to date.

deanfourie · Dec 24, 2023, 11:40 AM

@stephenw10 Ok, i've had a play with this today and its back in production.

I changed from legacy boot to UEFI, not sure if this will help.

I powered it up and pulled the power several times trying to force the failure, didn't fail to boot once. I do have a feeling its only once there is some data written to the drive, such as firewall logs, trafiic and host data etc. Any way we will see.

Is there any logs while i is running that I can get that may shed more light on this issue?

Thanks!

stephenw10 · Dec 24, 2023, 8:14 PM

Nothing other than the system log. Any sort of filesystem issue should log an error.