Best way to upgrade 2.5.2ce to 22.05 plus

JeGr

Adding some quick screens:

Filelist in preview:


 0) Logout (SSH only)                  9) pfTop
 1) Assign Interfaces                 10) Filter Logs
 2) Set interface(s) IP address       11) Restart webConfigurator
 3) Reset webConfigurator password    12) PHP shell + pfSense tools
 4) Reset to factory defaults         13) Update from console
 5) Reboot system                     14) Disable Secure Shell (sshd)
 6) Halt system                       15) Restore recent configuration
 7) Ping host                         16) Restart PHP-FPM
 8) Shell

Enter an option: 13

>>> Updating repositories metadata...
Updating pfSense-core repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: . done
Processing entries: .. done
pfSense-core repository update completed. 14 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries:
Processing entries............. done
pfSense repository update completed. 540 packages processed.
All repositories are up to date.
>>> Upgrading pfSense-upgrade... done.
>>> Setting vital flag on pfSense-upgrade... done.
Migrating /cf to ZFS dataset pfSense/ROOT/default/cf... done.
Migrating /var/cache/pkg to ZFS dataset pfSense/ROOT/default/var_cache_pkg... done.
Migrating /var/db/pkg to ZFS dataset pfSense/ROOT/default/var_db_pkg... done.
>>> Updating repositories metadata...
Updating pfSense-core repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: . done
Processing entries: .. done
pfSense-core repository update completed. 14 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries:
Processing entries............. done
pfSense repository update completed. 540 packages processed.
All repositories are up to date.

The following 93 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        libpsl: 0.21.1_3 [pfSense]
        php74-libbe: 0.1.4 [pfSense]
        py38-libzfs: 1.1.2022021400 [pfSense]

Installed packages to be UPGRADED:
        bash: 5.1.12 -> 5.1.16 [pfSense]
        bind-tools: 9.16.23 -> 9.16.26 [pfSense]
        ca_root_nss: 3.71 -> 3.76 [pfSense]
        curl: 7.80.0 -> 7.83.1 [pfSense]
        cyrus-sasl: 2.1.27_2 -> 2.1.28 [pfSense]
        devcpu-data: 20211109 -> 20220221 [pfSense]
        devcpu-data-amd: 20211115 -> 20220221 [pfSense]
        devcpu-data-intel: 20210608 -> 20220207 [pfSense]
        dnsmasq: 2.86,1 -> 2.86_3,1 [pfSense]
        dpinger: 3.0 -> 3.2 [pfSense]
        expat: 2.4.1 -> 2.4.7 [pfSense]
        filterdns: 2.0_5 -> 2.0_6 [pfSense]
        gdbm: 1.22 -> 1.23 [pfSense]
        glib: 2.70.2,2 -> 2.70.4_1,2 [pfSense]
        hostapd: 2.9_4 -> 2.10 [pfSense]
        ldns: 1.7.1_2 -> 1.8.1 [pfSense]
        libedit: 3.1.20210216,1 -> 3.1.20210910,1 [pfSense]
        libgpg-error: 1.43 -> 1.44 [pfSense]
        libinotify: 20180201_2 -> 20211018 [pfSense]
        libssh2: 1.9.0_3,3 -> 1.10.0,3 [pfSense]
        libunistring: 0.9.10_1 -> 1.0 [pfSense]
        lighttpd: 1.4.63 -> 1.4.64_1 [pfSense]
        links: 2.20.2_1,1 -> 2.25,1 [pfSense]
        luajit-openresty: 2.1.20210510 -> 2.1.20220310 [pfSense]
        mpd5: 5.9_6 -> 5.9_7 [pfSense]
        mtr-nox11: 0.94_1 -> 0.95 [pfSense]
        mysql57-client: 5.7.36 -> 5.7.37_1 [pfSense]
        nginx: 1.20.2_1,2 -> 1.20.2_9,2 [pfSense]
        ntp: 4.2.8p15_3 -> 4.2.8p15_5 [pfSense]
        openvpn: 2.5.4_1 -> 2.6.0_8 [pfSense]
        pfSense: 2.6.0 -> 22.05 [pfSense]
        pfSense-base: 2.6.0 -> 22.05 [pfSense-core]
        pfSense-default-config: 2.6.0 -> 22.05 [pfSense-core]
        pfSense-kernel-pfSense: 2.6.0 -> 22.05 [pfSense-core]
        pfSense-pkg-Cron: 0.3.7_5 -> 0.3.8_1 [pfSense]
        pfSense-pkg-Filer: 0.60.6_5 -> 0.60.6_6 [pfSense]
        pfSense-pkg-sudo: 0.3_6 -> 0.3_7 [pfSense]
        pfSense-rc: 2.6.0 -> 22.05 [pfSense-core]
        pfSense-repo: 2.6.0 -> 22.05_2 [pfSense]
        php74: 7.4.26 -> 7.4.28 [pfSense]
        php74-bcmath: 7.4.26 -> 7.4.28 [pfSense]
        php74-bz2: 7.4.26 -> 7.4.28 [pfSense]
        php74-ctype: 7.4.26 -> 7.4.28 [pfSense]
        php74-curl: 7.4.26 -> 7.4.28 [pfSense]
        php74-dom: 7.4.26 -> 7.4.28 [pfSense]
        php74-filter: 7.4.26 -> 7.4.28 [pfSense]
        php74-ftp: 7.4.26 -> 7.4.28 [pfSense]
        php74-gettext: 7.4.26 -> 7.4.28 [pfSense]
        php74-intl: 7.4.26 -> 7.4.28 [pfSense]
        php74-json: 7.4.26 -> 7.4.28 [pfSense]
        php74-ldap: 7.4.26 -> 7.4.28 [pfSense]
        php74-mbstring: 7.4.26 -> 7.4.28 [pfSense]
        php74-opcache: 7.4.26 -> 7.4.28 [pfSense]
        php74-openssl: 7.4.26 -> 7.4.28 [pfSense]
        php74-pcntl: 7.4.26 -> 7.4.28 [pfSense]
        php74-pdo: 7.4.26 -> 7.4.28 [pfSense]
        php74-pdo_sqlite: 7.4.26 -> 7.4.28 [pfSense]
        php74-pear-HTTP_Request2: 2.5.0,1 -> 2.5.1,1 [pfSense]
        php74-pecl-rrd: 2.0.1_1 -> 2.0.3 [pfSense]
        php74-pfSense-module: 0.76 -> 0.81 [pfSense]
        php74-posix: 7.4.26 -> 7.4.28 [pfSense]
        php74-readline: 7.4.26 -> 7.4.28 [pfSense]
        php74-session: 7.4.26 -> 7.4.28 [pfSense]
        php74-shmop: 7.4.26 -> 7.4.28 [pfSense]
        php74-simplexml: 7.4.26 -> 7.4.28 [pfSense]
        php74-sockets: 7.4.26 -> 7.4.28 [pfSense]
        php74-sqlite3: 7.4.26 -> 7.4.28 [pfSense]
        php74-sysvmsg: 7.4.26 -> 7.4.28 [pfSense]
        php74-sysvsem: 7.4.26 -> 7.4.28 [pfSense]
        php74-sysvshm: 7.4.26 -> 7.4.28 [pfSense]
        php74-tokenizer: 7.4.26 -> 7.4.28 [pfSense]
        php74-xml: 7.4.26 -> 7.4.28 [pfSense]
        php74-xmlreader: 7.4.26 -> 7.4.28 [pfSense]
        php74-xmlwriter: 7.4.26 -> 7.4.28 [pfSense]
        php74-zlib: 7.4.26 -> 7.4.28 [pfSense]
        postgresql13-client: 13.5 -> 13.6 [pfSense]
        protobuf: 3.17.3,1 -> 3.19.4,1 [pfSense]
        python38: 3.8.12_1 -> 3.8.12_2 [pfSense]
        readline: 8.1.1 -> 8.1.2 [pfSense]
        smartmontools: 7.2_3 -> 7.3 [pfSense]
        socat: 1.7.4.2 -> 1.7.4.3 [pfSense]
        sqlite3: 3.35.5_4,1 -> 3.37.2,1 [pfSense]
        strongswan: 5.9.4 -> 5.9.5 [pfSense]
        sudo: 1.9.8p2 -> 1.9.10 [pfSense]
        unbound: 1.13.2 -> 1.15.0_1 [pfSense]
        wpa_supplicant: 2.9_11 -> 2.10 [pfSense]
        wrapalixresetbutton: 0.0.8 -> 0.0.13 [pfSense]
        xxhash: 0.8.0 -> 0.8.1 [pfSense]
        zstd: 1.5.0 -> 1.5.2 [pfSense]

Installed packages to be REINSTALLED:
        libgcrypt-1.9.4 [pfSense] (options changed)

Number of packages to be installed: 3
Number of packages to be upgraded: 89
Number of packages to be reinstalled: 1

The process will require 18 MiB more space.
174 MiB to be downloaded.

**** WARNING ****
Reboot will be required!!
Proceed with upgrade? (y/N)

stephenw10

Hmm, looks like that was an unexpected change and has been reverted. You should see 22.01 offered via the Plus Upgrade repo. Then when you upgrade from there to 22.05 the full pkg reinstall is triggered.

Steve

JeGr

@stephenw10 Quick checking shows yes, update step is now back on 22.01 with only a minor few packages to update for the switch. That is a bit more comforting now :)

Thanks for having a look!

stephenw10

No problem, thanks for pointing it out.

It should be possible to go directly to 22.05 once the quirks are resolved.

Steve

JeGr

@stephenw10 Seems that - sad enough - something was really bonkers with the upgrade. Don't know if it had something to do with the system having already seen the 22.05 repo or sth alike but after selecting "pfSense Plus Upgrade" what should have been a quick update and reboot to 22.01 turned into a nightmare with a reboot, a "corrupt config.xml - 0 bytes" message and no config in /cf/conf and NO backups anymore in /cf/conf/backups. In fact the backups folder was erased completely. So sth. went very wrong with that.

I don't know if it's related to the repo pointing to 22.05 before or if theres another thing messy with switching from 2.6 to 22.01 but we got 2x the "corrupt config" and "broken installation" after reboot so cut our losses as long as we were still in our maintenance window and did a reinstall of 2.6 on the machine (glad it was the standby node) and reinstalled the config from our backup.

Would be nice if you could check if there's another thing that leaves upgrades from normal HW stranded with no config and wiped backups. The system also was still on 2.6 after the boot, so somehow it didn't even install the 22.01 kernel or meta packages. Very weird!

Cheers
\jens

stephenw10

Hmm, that's odd. I tested it here after reverting that and several systems upgraded no problem.

Do you have any logs from the upgrades?

JeGr

@stephenw10 I had to look if the logserver did still receive something of interest but as we had to abandon and reinstall, no logs on the device itself of course.

Otherwise only found 2 messages from pkg-static before the reboot that -kernel and -rc were updated to 22.01 then reboot by root. Just after the rebooting the system "hang" in limbo state due to /cf/conf being completely empty as if that ZFS dataset had been wiped clean. And as the system hang without configuration no logging to logserver of course :/

stephenw10

@jegr said in Best way to upgrade 2.5.2ce to 22.05 plus:

2 messages from pkg-static before the reboot that -kernel and -rc were updated to 22.01 then reboot by root

That was after the update to 22.01?
If that was during the upgrade to 22.05 that looks unexpected.

JeGr

@stephenw10 That was indeed after selecting "pfSense Plus ugprade" and hitting option 13 via SSH to upgrade. It showed around 9?10? packages to do, downloaded and then rebootet to complete it but was then stuck with no config anymore.

stephenw10

Hmm, any history on those units? They were on 2.6 and failed at the upgrade to 22.01?

But they did 'see' the 22.05 repo for a short time? Were they ever set to use the 2.7 repo branch?

JeGr

@stephenw10

No history on that. Both boxes were freshly installed with 2.6 because of ZFS changes beforehand (and had problems with the large filter set bug that was hotfixed with system_patches). So we installed both nodes from scratch, installed system_patches for the hotfixes and then did a restore of the configs. Worked very well and the cluster is up and running well since that.

As for the upgrade, we registered both (the primary one actually as the secondary node still had active TAC pro as we had to diagnose the 2.6/pf ruleset bug with support) and I switched the secondary to "Upgrade" as it's written in the Update menu. After switching, we had 22.05 as target instead of 22.01 but no package or update was triggered at that point.
That was when I got in contact here in the forums and you guys switched back to 22.01. Just beore the upgrade I made sure the target is still 22.01 by switching back to stable (2.6) and back to upgrade (22.01 was shown). Then got into the box via SSH and did "13" (upgrade) via console, got the list of ~12 packages that would be upgraded and they were downloaded. Then the system did the 10s downtime and reboot.
Afterwards after waiting for more then 10m for coming back (it's not the fastest booting those UEFI things...) it still was missing and I checked via the internal IPMI to find a console in a "broken" state. E.g. it looked pretty much like that:

FreeBSD/arm64 (Amnesiac) (ttyu0)

Config.xml is corrupted and is 0 bytes.  Could not restore a previous backup.
 0) Logout (SSH only)                  9) pfTop
 1) Assign Interfaces                 10) Filter Logs
 2) Set interface(s) IP address       11) Restart webConfigurator
 3) Reset webConfigurator password    12) PHP shell + Netgate pfSense Plus tools
 4) Reset to factory defaults         13) Update from console
 5) Reboot system                     14) Enable Secure Shell (sshd)
 6) Halt system                       15) Restore recent configuration
 7) Ping host                         16) Restart PHP-FPM
 8) Shell

Enter an option:

So menu shown but already via the "(Amnesiac)" in the first line one could see it didn't correctly boot up pfSense core setups.
Checking /cf/conf showed only a text file but no content whatsoever. config.xml was gone, backup folder nonexistent, other text files or rules.debug.old nowhere to be found.

After much daddling with the box I got it manually configured on an interface and ssh started up so I could get the last config.xml on the box and rebootet. Afterwards the box booted up fine on first glance but doing anything package or repo related was broken as hell. No packages were listed as installed, it couldn't find updates or update repos at all etc.
I did a reset and refresh (as per the docs with repo problems) and any pkg-static magic that I could think of but the most I got was that the 3 packages that were installed by re-boostrapping the repos/pkg-package were shown as installed. Any other package wasn't even shown, no FreeBSD base, no pfSense specifics, none. The whole installed package list was empty (until I bootstrapped, then it was the 3 pkgs from bootstrapping).

So somehow the OS forgot as much as every package installed by switching the repos. And it seems that particular problem or something very related is also still happening even with Netgate Boxes, as a german forum member describes almost the same problem happened to him after changing out his UPS. He thought he had the box shutdown correctly but after changing the UPS and repowering his Netgate 2100 the box came up exactly the same way as our secondary firewall HW - with Amnesiac / corrupt config.xml menu, no configs on the device and broken package paths. /topic/173402/hilfe-config-xml-is-corrupted-and-is-0-bytes-could-not-restore-a-previous-backup (german) His device also was running on the new ZFS scheme and after having the problem and getting the last config back on the device had problems with the package repos although he was somehow getting it back to running via console only with the GUI still bonkers.

As his was also running on ZFS and 22.01 (not 22.05 yet but had the update path set to 22.05 for a bit), perhaps something strange is going on related to switching either repos or perhaps with the new ZFS datasets and the preparations for Boot Environments?

/cf/conf completely missing every single file and directory gives a bit of "snapshot or dataset gone wrong" vibe as it is a distinct dataset in the ZFS setup, so I'm leaning towards that and perhaps something with the /var/xy dataset going very wrong?

Thats just an educated guess though but I'm happy to help in debugging it as far as I'm able to without compromising the box/failover of our DC cluster.

Cheers
\jens

stephenw10

@jegr said in Best way to upgrade 2.5.2ce to 22.05 plus:

/cf/conf completely missing every single file and directory gives a bit of "snapshot or dataset gone wrong" vibe

Mmm, I agree. Hard to see what else could cause something like that.
Were you able to try switching Boot Environments?

Steve

JeGr

@stephenw10 No, as the system came up with the config again, it still showed 2.6 instead of 22.01
That's why I thought something must have happened before the system could actually process all the update packages. /var/cache etc. or /cf/conf gone missing could remedy that. Whatever the case, it was a very strange thing. And as it wasn't yet on 22.01, no snapshot or BE was created (that I knew of). Also no possibility to choose one at boot time (I stopped the menu once to check).

stephenw10

Mmm, yes it wouldn't have had the boot menu option.
Hmm, OK, thanks we are looking into this.

JeGr

@stephenw10 If I can apply further information, I'd be happy to help