upgrade 2.6.0 to 2.7.0 not booting: UFS cannot be mounted anymore (superblock CGSIZE)

dneuhaeuser

Hello,

I observed multiple systems not booting after system updates from 2.6.0 to 2.7.0.
They cannot mount the root ufs anymore:

UFS2 superblock failed: CGSIZE(fs) (16388) > fs->fs_bsize (16384)
Attempted recovery for standard superblock: failed
Attempted extraction of recovery data from standard superblock: failed
Attemp to find boot zone recovery data
Finding an alternate superblock failed.
Check for only non-critical errors in standard superblock
UFS2 superblock failed: CGSIZE(fs) (16388) > fs->fs_bsize (16384)
Failed, superblock has critical errors
Mounting from ufs:/dev/da0s1a failed with error 22; retrying for 3 more seconds
[repeating]

I have many identical virtual systems of this kind, so I can reproduce this problem on multiple machines.
They were set up from a common image some years ago, underwent several sucessfull updates since then and worked flawlessly up till 2.6.0.
So this is definitely not only a faulty drive in a single system.

I found this thread on FreeBSD bugzilla talking about similar problems: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=264450
Apparently there have been changes how FreeBSD 14.0 calculates CGSIZE !?
Is the mentioned fix already in pfsense 2.7.0 ?

Does anyone have an idea how this can be corrected without reformatting the drives?
Probably I don't need to tell you that all these firewalls are remote and not easy to reach...

dneuhaeuser

On FreeBSD Bugzilla someonce commented that the available patch apparently is not yet in pfsense 2.7.0
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=264450

But, looking, 14.0-CURRENT@0c59e0b4e581 is from 2023-Mar-30, long
predating the fix reported in comment #15 . The issue is a known
problem with the old release involved, one that has been fixed.

My guess here is that pfsense needs to be based on an updated
FreeBSD that has the fix reported in #15 . Time for a 2.7.1 ?

stephenw10

@dneuhaeuser said in upgrade 2.6.0 to 2.7.0 not booting: UFS cannot be mounted anymore (superblock CGSIZE):

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=264450

I think you're out by a year.

That fix was commited in May 2022. It's in pfSense 2.7. Along with numerous more recent fixes.
https://github.com/pfsense/FreeBSD-src/commit/076002f24d35962f0d21f44bfddd34ee4d7f015d

Steve

stephenw10

My mistake this commit: https://github.com/pfsense/FreeBSD-src/commit/800a53b445e7eb113ba193b1ac98631299178529

In June 2022 as shown in the bug.

sbingner

@dneuhaeuser I encountered this same issue - it's definitely a kernel bug, if I boot the old kernel it can read the FS but there are significant issues that causes it to not be safe (like copy doesn't work). I suspect that if you were to copy a 2.7.2 kernel off and boot that on the 2.7.0 system it would work properly. I went through and upgraded from 2.7.0 to 2.7.2 and it could read everything fine once again. It seems to be some error in how the 2.7.0 UFS is calculating cgsize. After I upgraded by booting the old kernel, lots of files were zero sized because of the failure to copy files (for example all the nginx config files were empty)

I ended up just reinstalling from a 2.7.2 DVD, but I'll probably try the 2.7.2 kernel on 2.7.0 route on the next one I try this on.

stephenw10

You were also upgrading from 2.6 to 2.7 when you hit it?

sbingner

@stephenw10 yes that is correct - I posted a lot of details in slack but the important bits are above.

sbingner

@stephenw10 FYI I had to upgrade another system and the same thing happened. I booted the 2.7.2 install cd and mounted the old root. Copied /boot/kernel to /mnt/boot/kernel.new then booted that. Everything worked fine. Would be nice if the installer could have the kernel updated to one that's not bugged

stephenw10

The 2.7.2 installer? It seems like that is good if you're able to mount the old root from it no?

sbingner

@stephenw10 sorry my message was unclear, I meant the kernel in the upgrade that the UI applies. Yes the 2.7.2 installer is good… but you have to go to 2.7.0 first because…. Reasons?

stephenw10

Oh, I see so if you have this specific UFS install the 2.7.0 kernel will not boot from it. The 2.7.2 kernel will boot from it but you have to go through 2.7.0 because that was the first version that added dynamic repos. Hmm. Yeah I'm not sure what we can do about that at this point. You can always install 2.7.2 clean and restore the config from 2.6.