Netgate 4200 - Upgrade to 24.03 fails due to a packet upgrade/downgrade loop.

Fretboard36 · Apr 27, 2024, 1:14 PM

Hi All,

It seems i'm running into issues when i try to upgrade my Netgate 4200 device from 23.09.1 to 24.09. The earlier upgrades from 23.05 to 23.09 and then to 23.09.1 went smooth and without issues.

A quick look into the syslog points at the pfSense-upgrade package being upgraded and downgraded indefinitely. And that's where the upgrade process stops. Aside from the steps described in the Troubleshooting upgrades documentation what can be done to fix this? Anyone ran into the same issue?

At this moment my Netgate 4200 system update and update settings are back at the 23.09.1 branch.

I do have full configuration backups, however a complete factory reset and restoring the backup seems a last resort option to me.

3 php-fpm[412]: /pkg_mgr_install.php: Configurationchange: (Local Database): Creating restore point before upgrade.
3 php-fpm[31120]: /pkg_mgr_install.php: Beginning configuration backup to https://acb.netgate.com/save
5 check_reload_status[459]: Syncing firewall
3 php-fpm[31120]: /pkg_mgr_install.php: Configurationchange: (Local Database): Creating restore point before upgrade.
5 pkg-static[99099]: pfSense-upgrade upgraded: 1.2.1_1 -> 1.2.20
5 pkg-static[42976]: pfSense-upgrade reinstalled: 1.2.1_1 -> 1.2.1_1
5 pkg-static[38742]: pfSense-upgrade downgraded: 1.2.20 -> 1.2.1_1
3 php[1057]: /usr/local/sbin/acbupload.php: End of configuration backup to https://acb.netgate.com/save (success).
3 php[1057]: /usr/local/sbin/acbupload.php: End of configuration backup to https://acb.netgate.com/save (success).
5 pkg-static[50526]: pkg upgraded: 1.20.8_2 -> 1.20.9_1
5 pkg-static[31615]: pfSense-upgrade upgraded: 1.2.1_1 -> 1.2.20
5 pkg-static[78468]: pfSense-upgrade downgraded: 1.2.20 -> 1.2.1_1
5 pkg-static[78468]: pkg downgraded: 1.20.9_1 -> 1.20.8_2
5 pkg-static[41502]: pfSense-upgrade upgraded: 1.2.1_1 -> 1.2.20
5 pkg-static[90773]: pfSense-upgrade downgraded: 1.2.20 -> 1.2.1_1
5 pkg-static[54580]: pfSense-upgrade upgraded: 1.2.1_1 -> 1.2.20
5 pkg-static[2999]: pfSense-upgrade downgraded: 1.2.20 -> 1.2.1_1
5 pkg-static[51157]: pfSense-upgrade upgraded: 1.2.1_1 -> 1.2.20
5 pkg-static[2415]: pfSense-upgrade downgraded: 1.2.20 -> 1.2.1_1
5 pkg-static[12581]: pfSense-upgrade upgraded: 1.2.1_1 -> 1.2.20
5 pkg-static[755]: pfSense-upgrade downgraded: 1.2.20 -> 1.2.1_1
5 pkg-static[2707]: pfSense-upgrade upgraded: 1.2.1_1 -> 1.2.20
5 pkg-static[54839]: pfSense-upgrade downgraded: 1.2.20 -> 1.2.1_1
5 pkg-static[6073]: pfSense-upgrade upgraded: 1.2.1_1 -> 1.2.20
5 pkg-static[57449]: pfSense-upgrade downgraded: 1.2.20 -> 1.2.1_1
5 pkg-static[77784]: pfSense-upgrade upgraded: 1.2.1_1 -> 1.2.20
5 pkg-static[26997]: pfSense-upgrade downgraded: 1.2.20 -> 1.2.1_1
5 pkg-static[80621]: pfSense-upgrade upgraded: 1.2.1_1 -> 1.2.20
5 pkg-static[32327]: pfSense-upgrade downgraded: 1.2.20 -> 1.2.1_1
5 pkg-static[95746]: pfSense-upgrade upgraded: 1.2.1_1 -> 1.2.20
5 pkg-static[46681]: pfSense-upgrade downgraded: 1.2.20 -> 1.2.1_1
5 pkg-static[246]: pfSense-upgrade upgraded: 1.2.1_1 -> 1.2.20
5 pkg-static[50918]: pfSense-upgrade downgraded: 1.2.20 -> 1.2.1_1

stephenw10 · Apr 27, 2024, 2:01 PM

Hmm, I once saw that during testing but not from 23.09.1 is was due to a dev version. Did you ever run 24.03-dev?

Try setting the branch back to 23.09.1 that force reloading the key upgrade pkgs:

pkg-static clean -ay; pkg-static install -fy pkg pfSense-repoc pfSense-upgrade

Then set he branch back to 24.03 and try to upgrade again.

Steve

Fretboard36 · Apr 27, 2024, 3:30 PM

Hi Steve,

Since this 4200 device is my production system, i never ran any development version of pfsense on this machine.

I set back the branch the branch to 23.09.1 , waited untill the gui fully updated and ran:

pkg-static clean -ay; pkg-static install -fy pkg pfSense-repoc pfSense-upgrade

Afterwards i set the branch to 24.03 and tried to upgrade again. Unfortunately with the same loop as a result.

Not giving up, i reversed back to the 23.09.1 branch once more and ran the pkg-static clean command. I reloaded the device and then tried another attempt to upgrade.

This time from the CLI with the debug option enabled.

pfSense-upgrade -d

The same behaviour can be spotted with this method. While the webgui upgrade keeps up- and downgrading the pfSense-update package, the CLI process does this once and then stops.

Updating pfSense-core repository catalogue...
Fetching meta.conf: 
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be DOWNGRADED:
	pfSense-upgrade: 1.2.20 -> 1.2.1_1 [pfSense]

Number of packages to be downgraded: 1
[1/1] Downgrading pfSense-upgrade from 1.2.20 to 1.2.1_1...
[1/1] Extracting pfSense-upgrade-1.2.1_1: ...... done

At the end of the process it simply tells me all repositories and packages are up to date.

stephenw10 · Apr 28, 2024, 1:43 PM

Do you have the complete log from that? Does it show an ABI error?

Fretboard36 · Apr 28, 2024, 2:37 PM

The output and log doesn't show an ABI error.

syslog:

5	pkg-static[96248]: pfSense-upgrade downgraded: 1.2.20 -> 1.2.1_1
5	pkg-static[46355]: pfSense-upgrade upgraded: 1.2.1_1 -> 1.2.20

And the complete pfSense-upgrde -d output.

[23.09.1-RELEASE] pfSense-upgrade -d

Updating repositories metadata... 

pkg-static: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
Updating pfSense-core repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: . done
Processing entries: . done
pfSense-core repository update completed. 5 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: ......... done
Processing entries: .......... done
pfSense repository update completed. 736 packages processed.
All repositories are up to date.

Locking package pkg... 

pkg-static: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
Locking pkg-1.20.8_2

Upgrading pfSense-upgrade... 

pkg-static: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
	pfSense-upgrade: 1.2.1_1 -> 1.2.20 [pfSense]

Number of packages to be upgraded: 1

27 KiB to be downloaded.
[1/1] Fetching pfSense-upgrade-1.2.20.pkg: . done
Checking integrity... done (0 conflicting)
[1/1] Upgrading pfSense-upgrade from 1.2.1_1 to 1.2.20...
[1/1] Extracting pfSense-upgrade-1.2.20: ........ done

Unlocking package pkg... 

pkg-static: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
Unlocking pkg-1.20.8_2

Setting vital flag on pfSense-upgrade...

Updating repositories metadata...

Updating pfSense-core repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: . done
Processing entries: . done
pfSense-core repository update completed. 5 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .... done
Processing entries: .......... done
pfSense repository update completed. 739 packages processed.
All repositories are up to date.

Upgrading pfSense-upgrade...

Updating pfSense-core repository catalogue...
Fetching meta.conf: 
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be DOWNGRADED:
	pfSense-upgrade: 1.2.20 -> 1.2.1_1 [pfSense]

Number of packages to be downgraded: 1
[1/1] Downgrading pfSense-upgrade from 1.2.20 to 1.2.1_1...
[1/1] Extracting pfSense-upgrade-1.2.1_1: ...... done

Setting vital flag on pfSense-upgrade... 

Updating repositories metadata... 

Updating pfSense-core repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: . done
Processing entries: . done
pfSense-core repository update completed. 5 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: ....... done
Processing entries: .......... done
pfSense repository update completed. 739 packages processed.
All repositories are up to date.

Setting vital flag on pkg... 

Your packages are up to date
[23.09.1-RELEASE][root

stephenw10 · Apr 29, 2024, 3:21 PM

What version of repoc do you have? Try: pkg info -x pfSense

Fretboard36 · Apr 29, 2024, 4:59 PM

Output of pkg info -x pfSense

[23.09.1-RELEASE]:pkg info -x pfSense
pfSense-23.09.1
pfSense-Status_Monitoring-php82-1.8_3
pfSense-base-23.09.1
pfSense-boot-23.09.1
pfSense-composer-deps-0.1
pfSense-default-config-serial-23.09.1
pfSense-kernel-pfSense-23.09.1
pfSense-pkg-Netgate_Firmware_Upgrade-23.05.00
pfSense-pkg-System_Patches-2.2.10_1
pfSense-pkg-WireGuard-0.2.1
pfSense-pkg-acme-0.7.5
pfSense-pkg-arping-1.2.2_4
pfSense-pkg-aws-wizard-0.10
pfSense-pkg-iperf-3.0.3
pfSense-pkg-ipsec-profile-wizard-1.2
pfSense-pkg-lldpd-0.9.11_2
pfSense-pkg-mtr-nox11-0.85.6_3
pfSense-pkg-net-snmp-0.1.5_11
pfSense-pkg-nmap-1.4.4_7
pfSense-pkg-ntopng-0.8.13_10
pfSense-pkg-nut-2.8.2_1
pfSense-pkg-pfBlockerNG-3.2.0_8
pfSense-pkg-snort-4.1.6_17
pfSense-pkg-sudo-0.3_8
pfSense-repo-23.09.1
pfSense-repoc-20230912
pfSense-upgrade-1.2.1_1
php82-pfSense-module-0.95

stephenw10 · Apr 29, 2024, 8:17 PM

Ok, I managed to recreate this locally but the command I suggested above allowed me to escape it.

Try force re-installing both repo and repoc. So set the repo back to 23.09.1 then:
pkg-static clean -ay ; pkg-static install -fy pkg pfSense-repo pfSense-repoc pfSense-upgrade

Then retest.

Fretboard36 · Apr 30, 2024, 6:58 PM

@stephenw10

The inclusion of the pfSense-repoc package in the command below + the temporary removal of Snort and pfblockerNG did the trick.

pkg-static clean -ay ; pkg-static install -fy pkg pfSense-repo pfSense-repoc pfSense-upgrade

Happily running on version 24.03 now. Cheers!