PSA: Update bootloaders after 2.7.0 dist-upgrade

dlow

Just a reminder that you need to manually update the bootloader after an in-place update (ie. gptzfsboot and friends). Otherwise it is a nasty trap awaiting a reboot after an unwary zpool upgrade zroot.

Thank goodness for pfSense HA clustering!

stephenw10

Hmm, what exactly was the issue you hit here? I haven't seen anything like that in testing.

Steve

dlow

@stephenw10 After an in-place update, I was saw some ZFS messages on the console, did some tidying up and ran zpool upgrade zroot not thinking much about it until I rebooted later. The node was dead, never came back from the reboot. From IPMI, the bootloader complained about an unsupported ZFS feature flag and refused to boot!

I assumed that since the update was updating the boot kernel, that it would update the bootloader as well. Apparently it doesn’t! The new OpenZFS 2.0 in FreeBSD needs a new matching bootloader.

Thankfully, I have an HA Cluster and I didn’t reboot my primary node yet.

It was easily fixed by booting the new 2.7.0 iso and using the rescue shell. But it got a bit tense, as it later turned out the HA primary would have died on reboot too! Then I would have angry folks shouting at me.

Overall, it is a landmine waiting to be stepped on. After the upgrade, everything will look fine, even reboot fine… until you mess with zroot! Then, kaboom! Especially if you don’t have a backup node to keep you online!

clarion

I got bitten by this. Thankfully my initial work around was to reinstall 2.7.0 using my xml backup, back working in under 10 mins. I then tested 2.6.0 to 2.7.0 and read the message zfs provides after 2.7.0 loads. I will not claim to understand the line I used to fix the bootloader, I found it in a FreeBSD NAS forum.

[2.7.0-RELEASE][root@pfSense.home.arpa]/root: zpool status
  pool: pfSense
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:00:04 with 0 errors on Tue Jul 11 20:16:15 2023
config:

        NAME        STATE     READ WRITE CKSUM
        pfSense     ONLINE       0     0     0
          ada0p3    ONLINE       0     0     0

errors: No known data errors
[2.7.0-RELEASE][root@pfSense.home.arpa]/root: zpool upgrade pfSense
This system supports ZFS pool feature flags.

Enabled the following features on 'pfSense':
  edonr
  userobj_accounting
  encryption
  project_quota
  resilver_defer
  bookmark_v2
  redaction_bookmarks
  redacted_datasets
  bookmark_written
  log_spacemap
  livelist
  device_rebuild
  zstd_compress
  draid
  zilsaxattr
  head_errlog
  blake3

Pool 'pfSense' has the bootfs property set, you might need to update
the boot code. See gptzfsboot(8) and loader.efi(8) for details.
[2.7.0-RELEASE][root@pfSense.home.arpa]/root: zpool status pfSense
  pool: pfSense
 state: ONLINE
  scan: scrub repaired 0B in 00:00:04 with 0 errors on Tue Jul 11 20:16:15 2023
config:

        NAME        STATE     READ WRITE CKSUM
        pfSense     ONLINE       0     0     0
          ada0p3    ONLINE       0     0     0

errors: No known data errors
[2.7.0-RELEASE][root@pfSense.home.arpa]/root: gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
partcode written to ada0p1
bootcode written to ada0
[2.7.0-RELEASE][root@pfSense.home.arpa]/root:

dlow

@clarion said in PSA: Update bootloaders after 2.7.0 dist-upgrade:

I got bitten by this. Thankfully my initial work around was to reinstall 2.7.0 using my xml backup, back working in under 10 mins. I then tested 2.6.0 to 2.7.0 and read the message zfs provides after 2.7.0 loads. I will not claim to understand the line I used to fix the bootloader, I found it in a FreeBSD NAS forum.
[2.7.0-RELEASE][root@pfSense.home.arpa]/root: gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
partcode written to ada0p1
bootcode written to ada0
[2.7.0-RELEASE][root@pfSense.home.arpa]/root:

Bummer that you had to reinstall. For others reading this, you can just boot the 2.7.0 install iso / USB image and drop to the recovery shell to fix the bootloaders.

For completeness, there is also an EFI variant that looks like:

mount -t msdosfs -o longnames /dev/ada0p1 /mnt
mkdir -p /mnt/EFI/BOOT
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi
umount /mnt

You can use gpart show to see if you have an EFI partition. Both MBR and EFI variants are adapted from:
https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot

Oh, and if you have mirrored root drives, remember to update all of them!

dlow

@stephenw10 I know this is a general caveat of upgrading Freebsd 12.x -> 13.x. So, it’s not exactly a pfSense problem.

But if more folks stub their toes in this, can we add a “warning” in the release notes? We already have warnings about PHP and IPsec changes in FreeBSD, why not another warning for updating boot loaders?

stephenw10

Yes, we can probably do that until we have a more robust solution in place. Though most users will never see the ZFS status or try to upgrade the pool. It could be argued that if you are manually running that command at the CLI you should understand the consequences of doing so first.

https://redmine.pfsense.org/issues/14573

clarion

@dlow said in PSA: Update bootloaders after 2.7.0 dist-upgrade:

Bummer that you had to reinstall.

It was drastic, but I knew it would work and be quick. I never want to be rebuilding without confidence my processes work and to get that confidence, the process needs to be tested.

provels

LOL, this smarty pants tried upgrading the pool, too, with equivalent results. My user (me) was very angry with the sysadmin (me)!