Upgrading pfSense 2.4.5.1 to 2.5.1 unknown file system

Exocomp

@jimp I reinstalled 2.4.1 just to test, how I can I upgrade it to 2.4.5.1, is that possible through the system update web interface? Can you please direct me to the steps to do so.

jimp

It is kind of tricky due to the age of it, the easiest way is to go to a shell prompt and run pfSense-upgrade -c a few times in a row until it offers 2.4.5-p1, you may also need to pick the 2.4.5 update branch after running it a couple times.

The installer hashes wouldn't have anything to do with this, if it was wrong it would be much more broken.

Exocomp

@jimp I'm able to reproduce the issue with a fresh install of v2.4.5.1 then upgrade to v2.5.1.

I installed using pfSense-CE-2.4.5-RELEASE-p1-amd64.iso (sha256 0a09a7748419c86c665eb8d908f584e96d54859aa13f4eeb175a60548c70e228).

Then I used the web interface to upgrade to 2.5.1 which was successful then on first boot I get the same issue.

Is this the same way you're upgrading?

Exocomp

@jimp I did the same test but using ufs and the upgrade and boot up is fine.

So definitely related to zfs and fresh install of 2.4.5.1 to 2.5.1, at least for me.

Exocomp

@jimp I tried a fresh install of v2.5.1 and now I get a boot loop.

The following is broken on v2.5.1:

VM: Hyper-V Gen2 ZFS

Let me know what you need to file this as a bug.

I noticed others have reported a similar issue: https://forum.netgate.com/topic/161130/upgrading-to-2-5-0-on-hyperv-boot-loops

jimp

Probably nothing for us to do there, it's likely between FreeBSD and Hyper-V.

I have no problems with ZFS and that scenario on KVM or ESX.

Exocomp

@jimp fair point, if I can find some time I will test FreeBSD Hyper-V Gen2 ZFS.

Exocomp

@jimp Looks like someone else has already done the testing and found it works with FreeBSD 12.2 release and Hyper-V Gen2 ZFS so sounds like something in pfSense.

Here are relevant bugs filed:

Exocomp

@jimp I was able to get v2.5.1 to load, can you please make sure the necessary engineers get this.

What works: On the boot screen choose option 6 and load from "kernal.old"

Summary:

System config: pfSense v2.5.1 on Hyper-V Gen2 ZFS

The issue is not that the following is missing from /boot/loader.conf

opensolaris_load="YES" 
zfs_load="YES"

The problem is loading with the default kernal in v2.5.1, however, using kernal.old works.

jimp

Loading kernel.old is not a valid path forward. It means you're running an old 2.4.x kernel on a 2.5.x base and any number of problems may result.

The symptoms don't quite line up with the errors on the linked Redmine issues, but if using either a Gen 1 VM or UFS works, that would be the best thing to do on Hyper-V now.

Exocomp

@jimp my point simply was that since "kernel.old" works that means that the problem is with pfSense on the new kernel and not with FreeBSD.

It appears the conclusion of those issues is that the necessary parameters were missing (opensolaris_load & zfs_load) however the issue here is that the new pfsense kernel has the problem. So I agree this issue is slightly different.

The points you make are currently the only way forward.

From my standpoint and others (it is common sense) it leaves us with a less than optimal solution and leaves us with a broken upgrade path forward.

I hope that at least a bug can be opened on this so that is looked at by the respective product owners and engineers, so that in a future release this can be fixed.

Are you able to file the bug?

jimp

@exocomp said in Upgrading pfSense 2.4.5.1 to 2.5.1 unknown file system:

@jimp my point simply was that since "kernel.old" works that means that the problem is with pfSense on the new kernel and not with FreeBSD.

That doesn't hold logically because the kernel is the FreeBSD kernel, pfSense code comes into play after that. So if loading kernel.old works, it's more likely a FreeBSD issue than a pfSense issue.

I hope that at least a bug can be opened on this so that is looked at by the respective product owners and engineers, so that in a future release this can be fixed.

Are you able to file the bug?

I can't replicate the bug so I don't have all of the details available to do so. It's possible that https://redmine.pfsense.org/issues/11469 is close to what you are seeing but the person who opened that bug linked to a thread for a 2.4.x issue and didn't give a complete error -- your error is different than the 2.4.x error, but since the original reporter of #11469 did not specify the complete message, it was closed as a duplicate given the other details they posted.

I still suggest trying a recent FreeBSD 12.2-STABLE snapshot with ZFS in a Hyper-V Gen2 VM for yourself and see if it works -- The 12.2-RELEASE of FreeBSD was a while ago and pfSense is using code from after that point so using 12.2-RELEASE is not a proper comparison.

Exocomp

@jimp gotcha, I was under a different understanding. What you say makes sense now that I think about it more.

I'll test with a recent FreeBSD 12.2 stable release and see what happens.

Exocomp

@jimp I just tested with the following:

FreeBSD-12.2-STABLE-amd64-20210527-r369870-disc1.iso

I selected the same settings as pfSense:

Auto (ZFS) Guided Root-on-ZFS
ZFS Configuration - Leave on default settings
ZFS Configuration - Stripe - No Redundancy

I was able to boot to login without issue.

Please let me know what else to try and how to proceed.

jimp

Not sure what it might be then, could be the size of the pfSense kernel. Might be worth trying a 2.5.2 or 2.6.0 snapshot to see if they behave differently.

May just have to switch to UFS or Gen1 for now, eventually we'll be making the kernel more modular which will reduce its size, and if that is the problem with Hyper-V then it will resolve itself when that happens.

Exocomp

@jimp I'm not really worried about now just as long as it is recognized as a bug and addressed in a future release.

I'll try 2.5.2 / 2.6.0 and see what happens.

jimp

I'm afraid it wouldn't be recognized as a bug in pfSense as there is nothing we can directly do to solve it. It's an incompatibility between Hyper-V and the FreeBSD kernel configuration.

If it does get solved, it will be as a consequence of other FreeBSD-related changes, such as a version upgrade or changing the kernel options as I mentioned. But none of that is being done to address this, it's for other ongoing work that may also happen to resolve it.

Exocomp

@jimp said in Upgrading pfSense 2.4.5.1 to 2.5.1 unknown file system:

I'm afraid it wouldn't be recognized as a bug in pfSense as there is nothing we can directly do to solve it.

Logic would dictate that since I tested with FreeBSD and it was fine that you folks can look into this and determine what is causing it and directly take action and use a kernel/configuration which works.

Now if you folks are not willing to do that based on your reasons then that is another matter.

I tested with 2.5.2 and it has the same problem.

I feel I have provided sufficient information here to have an engineer track this problem on your side to start looking into this further if someone really does care and wants to fix the problem.