Upgrading pfSense 2.4.5.1 to 2.5.1 unknown file system

jimp

Curious. I've upgraded several ZFS systems from 2.4.5-p1 to 2.5.x without issue in the past.

How were these installed? Is there just one disk or a mirror?

Exocomp

@jimp interesting. It's just one disk.

During original installation (v2.4.1) my notes show I selected

Auto (ZFS) Guided Root-on-ZFS
Leave everything as defaults
Stripe - No Redundancy

jimp

It's possible there was some difference in how the filesystem was made back on 2.4.1 that is part of the issue. Though I have a couple long-lived ZFS installs I'm not sure any were from that long ago at the moment since most get reinstalled now and then as they're in my lab.

What shows up in zpool status on these?

Exocomp

@jimp Here is the result:

zpool status

jimp

Maybe try updating the pool it to get around those errors:

zpool upgrade -a
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

Though I'd still say your safest bet is to reinstall and restore the configuration. You should even be able to use the "Recover config.xml" option in the installer to make it simpler.

Exocomp

@jimp after running those commands its not able to boot at all.

As you mentioned sounds like a complete reinstall is in order. Bummer.

jimp

I just installed a VM with 2.4.1 ZFS and upgraded to 2.4.5-p1 then 2.5.1 and it went without a hitch. There must be something off with those. Not sure what it might be, though.

Exocomp

@jimp I'll do some more testing out of curiosity.

v2.4.1
SHA256: 738C47B549284B158CC3F174288A5B47AD6F112728D5BAFD11412A1B77DA32F9

Above was the sha256 of v2.4.1 I installed from.

I don't have sha256 of v2.4.5.1 (since I used the web installer), can you provide it here and I'll download and do the same test as you but with my configuration.

Exocomp

@jimp I reinstalled 2.4.1 just to test, how I can I upgrade it to 2.4.5.1, is that possible through the system update web interface? Can you please direct me to the steps to do so.

jimp

It is kind of tricky due to the age of it, the easiest way is to go to a shell prompt and run pfSense-upgrade -c a few times in a row until it offers 2.4.5-p1, you may also need to pick the 2.4.5 update branch after running it a couple times.

The installer hashes wouldn't have anything to do with this, if it was wrong it would be much more broken.

Exocomp

@jimp I'm able to reproduce the issue with a fresh install of v2.4.5.1 then upgrade to v2.5.1.

I installed using pfSense-CE-2.4.5-RELEASE-p1-amd64.iso (sha256 0a09a7748419c86c665eb8d908f584e96d54859aa13f4eeb175a60548c70e228).

Then I used the web interface to upgrade to 2.5.1 which was successful then on first boot I get the same issue.

Is this the same way you're upgrading?

Exocomp

@jimp I did the same test but using ufs and the upgrade and boot up is fine.

So definitely related to zfs and fresh install of 2.4.5.1 to 2.5.1, at least for me.

Exocomp

@jimp I tried a fresh install of v2.5.1 and now I get a boot loop.

The following is broken on v2.5.1:

VM: Hyper-V Gen2 ZFS

Let me know what you need to file this as a bug.

I noticed others have reported a similar issue: https://forum.netgate.com/topic/161130/upgrading-to-2-5-0-on-hyperv-boot-loops

jimp

Probably nothing for us to do there, it's likely between FreeBSD and Hyper-V.

I have no problems with ZFS and that scenario on KVM or ESX.

Exocomp

@jimp fair point, if I can find some time I will test FreeBSD Hyper-V Gen2 ZFS.

Exocomp

@jimp Looks like someone else has already done the testing and found it works with FreeBSD 12.2 release and Hyper-V Gen2 ZFS so sounds like something in pfSense.

Here are relevant bugs filed:

Exocomp

@jimp I was able to get v2.5.1 to load, can you please make sure the necessary engineers get this.

What works: On the boot screen choose option 6 and load from "kernal.old"

Summary:

System config: pfSense v2.5.1 on Hyper-V Gen2 ZFS

The issue is not that the following is missing from /boot/loader.conf

opensolaris_load="YES" 
zfs_load="YES"

The problem is loading with the default kernal in v2.5.1, however, using kernal.old works.

jimp

Loading kernel.old is not a valid path forward. It means you're running an old 2.4.x kernel on a 2.5.x base and any number of problems may result.

The symptoms don't quite line up with the errors on the linked Redmine issues, but if using either a Gen 1 VM or UFS works, that would be the best thing to do on Hyper-V now.

Exocomp

@jimp my point simply was that since "kernel.old" works that means that the problem is with pfSense on the new kernel and not with FreeBSD.

It appears the conclusion of those issues is that the necessary parameters were missing (opensolaris_load & zfs_load) however the issue here is that the new pfsense kernel has the problem. So I agree this issue is slightly different.

The points you make are currently the only way forward.

From my standpoint and others (it is common sense) it leaves us with a less than optimal solution and leaves us with a broken upgrade path forward.

I hope that at least a bug can be opened on this so that is looked at by the respective product owners and engineers, so that in a future release this can be fixed.

Are you able to file the bug?

jimp

@exocomp said in Upgrading pfSense 2.4.5.1 to 2.5.1 unknown file system:

@jimp my point simply was that since "kernel.old" works that means that the problem is with pfSense on the new kernel and not with FreeBSD.

That doesn't hold logically because the kernel is the FreeBSD kernel, pfSense code comes into play after that. So if loading kernel.old works, it's more likely a FreeBSD issue than a pfSense issue.

I hope that at least a bug can be opened on this so that is looked at by the respective product owners and engineers, so that in a future release this can be fixed.

Are you able to file the bug?

I can't replicate the bug so I don't have all of the details available to do so. It's possible that https://redmine.pfsense.org/issues/11469 is close to what you are seeing but the person who opened that bug linked to a thread for a 2.4.x issue and didn't give a complete error -- your error is different than the 2.4.x error, but since the original reporter of #11469 did not specify the complete message, it was closed as a duplicate given the other details they posted.

I still suggest trying a recent FreeBSD 12.2-STABLE snapshot with ZFS in a Hyper-V Gen2 VM for yourself and see if it works -- The 12.2-RELEASE of FreeBSD was a while ago and pfSense is using code from after that point so using 12.2-RELEASE is not a proper comparison.