Mounting failed with error 5

[signalz]

My pfSense 2.7 VM suddenly won't boot. Last shut down was clean, though the host (Ubuntu 22.04) has crashed with the pfSense VM running before. I never noticed any apparent problems because of that. The hard drive the disk image is on doesn't seem to be having problems either.

What do I do now? Is this recoverable? I'm having a hard time getting anywhere is this shell.

[stephenw10]

I would try to boot into single user mode and run a ZFS scrub:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html#troubleshooting-disk-and-filesystem-issues

Steve

[signalz]

Is there something else going on, or am I just very bad at FreeBSD? What out of anything on the screen is the pool name?

[stephenw10]

Run: zpool status to see it. By default it's pfSense.

[stephenw10]

For example:

23.09-RELEASE][admin@5100.stevew.lan]/root: zpool status
  pool: pfSense
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
	The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(7) for details.
config:

	NAME        STATE     READ WRITE CKSUM
	pfSense     ONLINE       0     0     0
	  ada0p3    ONLINE       0     0     0

errors: No known data errors
[23.09-RELEASE][admin@5100.stevew.lan]/root: zpool scrub pfSense
[23.09-RELEASE][admin@5100.stevew.lan]/root: zpool status
  pool: pfSense
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
	The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(7) for details.
  scan: scrub in progress since Thu Nov 16 01:58:20 2023
	5.20G / 12.1G scanned at 1.73G/s, 0B / 12.1G issued
	0B repaired, 0.00% done, no estimated completion time
config:

	NAME        STATE     READ WRITE CKSUM
	pfSense     ONLINE       0     0     0
	  ada0p3    ONLINE       0     0     0

errors: No known data errors

[signalz]

Both zpool status and zpool scrub pfSense return Invalid file system specification

[stephenw10]

Hmm, probably going to be easier to just reinstall but you could try geom part list just to make sure it sees the partition at all.
Very unusual to a ZFS failure like that. I'd have to assume it's some filesystem issue at the hypervisor level.

[signalz]

Thanks for your help. geom part list says the same thing. I'll reinstall.

[GlennNZ]

Hi

I suffered pretty much the same issue,and given this is first/second google hit thought would update solution for me here in case it helps others.

Pfsense Plus power cut (backup exhausted) bad shutdown.
Then failed to start.
Root mount waiting for: CAM (lots)
Then:
Mounting from zfs:Pfsense/ROOT/Default failed with error 5
Error 5, as per screen shots here.

Any zpool command gave error - as above.
“Invalid file system specification”

I pulled PC, pulled drive, booted from USB Pfsense stick.
Pulled hair.
Could not seem to get to bottom of.
Was going to reinstall Pfense (not plus) from USB stick - no drive to be found other than USB (?). Was going to replace drive altogether.

Then discovered one post, in a forum, somewhere, which I will link to if ever can find again, talking about BIOS drive settings changed in power cut.

Solution:
Somehow power surge/power cut had changed BIOS to RAID for drives from AHCI. (interestingly did so for another PC I discovered later - Pfsense was first job - this was Windows PC and failed to Boot altogether)
Oddly here though - Pfsense partially booted - as above so for drive was readable for parts…. (Hence hadn’t considered drive issue)

Solution:
Check BIOS Settings
Hard Drives - Selection - AHCI. (not RAID which it had become somehow)

Hopefully this helps someone else in the future.

[Gertjan]

@GlennNZ said in Mounting failed with error 5:

Somehow power surge/power cut had changed BIOS

More common : the CMOS battery has gone for a holiday, or is about to die.
In such a case, BIOS defaults are load and this will surely change the boot order, boot designation etc. The result will always be the same : system can't boot anymore.
Solution : get a new battery.

[GlennNZ]

@Gertjan
Thats a great suggestion that CMOS battery died and defaults now in play.
Will check it out/replace batteries.

The purpose of the post was to point out that this error messaging and “partial” startup can happen with this issue. (AHCI vs RAID setup) Even though 1/2 starts Pfsense boots and get this messaging - problem lies elsewhere. (BIOS as here)

I suspect in this old case the same issue applied. Instinctively you would think that any boot would fail completely (such as with Windows with same issue), instead of the above messaging.

[Gertjan]

@GlennNZ

When the way how the system should access the 'disk' changes from AHCI (== 'normal SATA access') to RAID (= probably some software raid solution) then partitions, the file system need to be changed also from the ground up.
The BIOS won't be able to 'see' or even 'know' if the partitions contains a "pfSense OS", as it would be able to read anything from the boot partition. Imho, it wouldn't even be possible to boot half way. When the access method is set to RAID, the BIOS is going to search for a hardware RAID solution - an extension card, or a software solution from 'from somewhere'. It wouldn't be able to read the 'RAID' drive (which isn't a RAID drive here) by itself.
Maybe the BIOS of your system somewhat smarter ?

@GlennNZ said in Mounting failed with error 5:

Instinctively you would think that any boot would fail completely (such as with Windows with same issue)

Most of my PCs have that AHCI or RAID switch, and I can confirm ...
A switch means, afaik, a complete re install.

[GlennNZ]

@Gertjan

I have just done it and lived through it.

The messaging at the top (from the originator) is the same as mine with an Intel system set incorrectly to RAID access for single HDD, with a Pfsense install. (install of the system was with AHCI)

Invalid file system configuration for every Zpool based command.

Changing BIOS back to the correct setting (AHCI) resolved this startup issue, and the system continued to boot as usual.

So, whilst I agree, you would think such a change would lead to a completely non bootable system (and different trouble shooting) I can 100% confirm that it does not and lead you down a Pfsense install troubleshooting path (when issue is nothing to do with Pfsense)

Glenn