Latest update script failure
-
I am aware of this particular failure , although it shouldn't be catastrophic...meaning, the update should complete "successfully." The cause of this error stems from the new system update mechanism we are rolling out for ZFS based installations.
We now download AND install packages in a separate boot environment while the system remains up and running. This means we no longer defer package extraction ane installation to the initial reboot, which naturally just keeps the system offline for a longer period of time. Now, system updates are completed prior to the reboot and the reboot is just as fast as any normal reboot, because that's all it is...a normal reboot.
This mechanism means faster (less downtime) and safer upgrades. With the old update path, packages were cached locally and then installed while the system was starting up. This mechanism was great for ensuring that the system was "quiet" while we upgraded everything (e.g. no contention for open files, locks, etc.). However, it was very common for users to have enough space to store the package cache but not enough space to actually extract and install the update. This would inevitably leave you with a nonfunctional or only partially updated system...which always produces interesting side effects.
Now updates are truly atomic.
It's pretty neat, expect a technical video from me this week talking about the new approach.
(back to the issue at hand, the fix here is I need to nullfs mount the ESP into the chroot environment so that pkg can reach in and update the loader successfully)
-
@cmcdonald Not that the old method of upgrading was unbearable in any way, I really appreciate the time and effort to implement this new method which sounds like great idea. Determining there are issues before the reboot will save countless hours of troubleshooting for many in the future
-
I am seeing this same error, but it does not seem to cause any issues with the upgrade itself.
-
Thanks for the feedback!
-
This has also started happening recently:
Boot Environment
Current: upgrade-upgrade-upgrade-upgrade-upgrade-default-20240221223437-20240222100045-20240223161555-20240224222137-20240226110515
Next: upgrade-upgrade-upgrade-upgrade-upgrade-default-20240221223437-20240222100045-20240223161555-20240224222137-20240226110515 -
@behemyth I think that's by design since you have to verify the new boot environment on next boot before it automatically reverts back to previous boot environment. Despite the naming convention changing, I think this is a great new feature. It's not something you would need to rename very often unless you're installing the nightly snapshots.
-
We could certainly clean up the name post upgrade / verification. This was discussed last week internally
-
@cmcdonald Even though it's an extra post-install manual step, I kind of like it this way with the new safely verification feature. I didn't like it at first, but it makes sense why you did it this way. You know how it is though - people generally don't like change.
-
I'm stuck at version 24.03.a.20240215.0600, each time I try to do the upgrade it just reboots to the same version. Do I need to do a manual step? It looks like it's creating a boot environment but will not boot into it. Should I manually boot into the new environment?
-
@boessi You have to "verify" it within a few minutes or it will revert to the previous boot environment. This is a new safety feature. You will see the prompt after you logon after the upgrade and reboot.
-
That is manual verification which can be enabled/disabled in Update Settings.
You can also defer the reboot entirely so it won't automatically reboot after completing the upgrade.
-
@cmcdonald Hmm I don't get this prompt, lets check if I can find the settings option
EDIT: Manually boot into the 0226 environmet also just reverts back to 0215, no option and no prompt
-
@boessi ok setting the boot environment not just for next boot but setting it as new default looks like it was now booting correctly into new version
-
FYI, I'm still seeing this error during the last week or two daily updates, but it still doesn't seem to impact the upgrade; however, my disk usage grows after each upgrade even after deleting the previous boot environment. For the past year or two, my 6 100 MAX ZFS pool has maintained a very consistent 2.5GB worth of space, but now it's grown another gig since installing each 24.03 nightly snapshots - and again I delete the previous boot environment after each successful upgrade and testing as well as clear all system logs.
[1/7] Fetching pfSense-base-24.03.a.20240229.0246.pkg: .......... done [2/7] Fetching snort-2.9.20_8.pkg: .......... done [3/7] Fetching pfSense-24.03.a.20240229.0246.pkg: .......... done [4/7] Fetching pfSense-boot-24.03.a.20240229.0246.pkg: .......... done [5/7] Fetching pfSense-repo-24.03.a.20240229.0246.pkg: . done [6/7] Fetching pfSense-default-config-serial-24.03.a.20240229.0246.pkg: . done [7/7] Fetching pfSense-kernel-pfSense-24.03.a.20240229.0246.pkg: .......... done Checking integrity... done (0 conflicting) [1/7] Upgrading pfSense-boot from 24.03.a.20240228.0600 to 24.03.a.20240229.0246... ******[1/7] Extracting pfSense-boot-24.03.a.20240229.0246: .......... done No such file or directory****** -
This is a good catch, my on-board storage is almost 50% used now, I don't ever remember seeing it this high.
-
Tomorrow's build will properly cleanup the pkg cache.
-
@cmcdonald said in Latest update script failure:
Tomorrow's build will properly cleanup the pkg cache.
Package cache was cleared, ZFS pool has returned to it’s normal size, and zero install errors logged with today’s build! Nice work!
-
@DefenderLLC Awesome thanks for following up!
-
Mine is failing this morning, apparently on the renaming of the boot environment.
Updating repositories metadata...
Updating pfSense-core repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: . done
Processing entries: . done
pfSense-core repository update completed. 5 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
pfSense repository update completed. 736 packages processed.
All repositories are up to date.
Setting vital flag on pfSense...done.
Renaming current boot environment from upgrade-upgrade-upgrade-upgrade-upgrade-upgrade-default-20240221223437-20240222100045-20240223161555-20240224222137-20240226110515-20240228113322 to upgrade-upgrade-upgrade-upgrade-upgrade-upgrade-default-20240221223437-20240222100045-20240223161555-20240224222137-20240226110515-20240228113322_previous...failed. -
This has been fixed.
But you need to manually rename the current BE to something more sensible and then try the upgrade again.
