1100 Upgrade from 23.09.1 to 24.04 Failing
-
Read over a few others issues with the 1100, and in my case it doesnt' look to be related to zfs snapshots or space.
[23.09.1-RELEASE][root@<REMOVED>]/root: zfs list -t snapshot no datasets available [23.09.1-RELEASE][root@<REMOVED>]/root:Filesystem Size Used Avail Capacity Mounted on pfSense/ROOT/default 5.9G 1.2G 4.7G 20% /Though I have gotten a few different errors, and the installs seems to fail at different places. Most recent it got to extracting files.
>>> Updating repositories metadata... pkg-static: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended Updating pfSense-core repository catalogue... Fetching meta.conf: . done Fetching packagesite.pkg: . done Processing entries: . done pfSense-core repository update completed. 5 packages processed. Updating pfSense repository catalogue... Fetching meta.conf: . done Fetching packagesite.pkg: ......... done Processing entries: .......... done pfSense repository update completed. 710 packages processed. All repositories are up to date. >>> Renaming current boot environment from default to default_20240424184218...done. >>> Cloning current boot environment default_20240424184218...done. >>> Removing vital flag from php82...done. >>> Upgrading packages in cloned boot environment default... Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... pfSense repository is up to date. All repositories are up to date. Checking for upgrades (168 candidates): .......... done Processing candidates (168 candidates): .......... done The following 280 package(s) will be affected (of 0 checked): Installed packages to be REMOVED: aws-sdk-php82: 3.273.3 pfSense-Status_Monitoring-php82: 1.8_3 php82: 8.2.11 php82-bcmath: 8.2.11 php82-bz2: 8.2.11 php82-ctype: 8.2.11 php82-curl: 8.2.11 php82-dom: 8.2.11 ...skipping... smartmontools-7.4 [pfSense] (ABI changed: 'freebsd:14:aarch64:64' -> 'freebsd:15:aarch64:64') ssh_tunnel_shell-0.2_1 [pfSense] (ABI changed: 'freebsd:14:aarch64:64' -> 'freebsd:15:aarch64:64') uclcmd-0.2.20211204 [pfSense] (ABI changed: 'freebsd:14:aarch64:64' -> 'freebsd:15:aarch64:64') voucher-0.1_3 [pfSense] (ABI changed: 'freebsd:14:aarch64:64' -> 'freebsd:15:aarch64:64') vstr-1.0.15_1 [pfSense] (ABI changed: 'freebsd:14:aarch64:64' -> 'freebsd:15:aarch64:64') whois-5.5.7 [pfSense] (ABI changed: 'freebsd:14:aarch64:64' -> 'freebsd:15:aarch64:64') wol-0.7.1_4 [pfSense] (ABI changed: 'freebsd:14:aarch64:64' -> 'freebsd:15:aarch64:64') xinetd-2.3.15_2 [pfSense] (ABI changed: 'freebsd:14:aarch64:64' -> 'freebsd:15:aarch64:64') xxhash-0.8.2 [pfSense] (ABI changed: 'freebsd:14:aarch64:64' -> 'freebsd:15:aarch64:64') zip-3.0_1 [pfSense] (ABI changed: 'freebsd:14:aarch64:64' -> 'freebsd:15:aarch64:64') zstd-1.5.5 [pfSense] (ABI changed: 'freebsd:14:aarch64:64' -> 'freebsd:15:aarch64:64') Number of packages to be removed: 53 Number of packages to be installed: 60 Number of packages to be upgraded: 69 Number of packages to be reinstalled: 98 The operation will free 30 MiB. [1/280] Reinstalling indexinfo-0.3.1... [1/280] Extracting indexinfo-0.3.1: .... done [2/280] Reinstalling mpdecimal-2.5.1... [2/280] Extracting mpdecimal-2.5.1: .......... done [3/280] Upgrading gettext-runtime from 0.22_1 to 0.22.3... [3/280] Extracting gettext-runtime-0.22.3: .......... done [4/280] Upgrading readline from 8.2.1 to 8.2.7... [4/280] Extracting readline-8.2.7: .......... done [5/280] Reinstalling libffi-3.4.4... [5/280] Extracting libffi-3.4.4: .......... done [6/280] Reinstalling expat-2.5.0... [6/280] Extracting expat-2.5.0: .......... done [7/280] Reinstalling libiconv-1.17... [7/280] Extracting libiconv-1.17: .......... done [8/280] Upgrading python311 from 3.11.4 to 3.11.7... [8/280] Extracting python311-3.11.7: ....>>> Unlocking package pkg...done. __RC=1 __REBOOT_AFTER=10Any thoughts or ideas on why it may be failing? Above is the contents from the /conf/upgrade_log.txt file.
-
@defunct78 Pretty sure there was a post yesterday on an 1100 where deleting some old boot environments let it succeed, even though it had space. I don't have a link, but you might try deleting some old BEs anyway.
-
@SteveITS Ya, BE are the same as ZFS snapshots. As you can see from the UI, i don't have any BE's or snapshots
-
Adding more details, showing the same usage per
https://docs.netgate.com/pfsense/en/latest/backup/zfsbe/space.html?highlight=zfs[23.09.1-RELEASE][root@<REMOVED>]/root: df -h / Filesystem Size Used Avail Capacity Mounted on pfSense/ROOT/default 5.9G 1.2G 4.7G 20% / [23.09.1-RELEASE][root@<REMOVED>]/root: zfs list / NAME USED AVAIL REFER MOUNTPOINT pfSense/ROOT/default 1.34G 4.70G 1.18G / [23.09.1-RELEASE][root@<REMOVED>]/root: -
Hmm, not a failure I've seen.
Try upgrading from the CLI (from the console if you can) using:
pfSense-upgrade -d -
@stephenw10 I did try Option 13 via CLI also, with no luck before. That being said, I went ahead an ran your suggested command. No luck and failed at the same place. Attaching the console log file, which matches the /conf/upgrade_log.txt file.
End of /conf/upgrade_log.txt
[8/267] Upgrading python311 from 3.11.4 to 3.11.7... [8/267] Extracting python311-3.11.7: ......>>> Unlocking package pkg... Unlocking pkg-1.20.8_2Console Log
pfsense upgrade.txt -
Nothing shown after that? It panics? reboots?
There seems to be less output there. It never reports 'done' on the python pkg.
-
@defunct78 Ya, nothing, just stops.
May be unrelated but I did see this. I don't think it is a problem since I am not seeing any errors anywhere else.
: mmc extcsd read /dev/mmcsd0rpmb | egrep 'LIFE|EOL' eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x08 eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x0b eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01 -
Mmm, I'd expect to see more errors there if the eMMC wouldn't allow writing. And I'd expect it to fail in different places.
So after that at the console you had to manually power cycle it?
You might try that again and hit 'ctl+t' at the command line when it appears to stop responding. It may still respond to that and show you what it's waiting for.
-
@defunct78 Sorry, when I said it just stops, the install stop. Connectivity is not lost. Just keeps going with no issues to the network flow. I can even attempt another install right after.
I have just rebooted though (after 28 days) and attempting another upgrade. I will update the thread shortly.
-
Ok then running 'ctl+t' at the command line if you try to upgrade there should show why it's stopped.
-
@stephenw10 OK, I will have to console in. All these attempt have been done over the network, as my firewall is not next to my laptop.
btw, after the reboot, it go farther. I will update shortly once I console in.
tail /conf/upgrade_log.txt [37/267] Upgrading log4cplus from 2.1.0 to 2.1.1... [37/267] Extracting log4cplus-2.1.1: .......... done [38/267] Installing libpfctl-0.8... [38/267] Extracting libpfctl-0.8: ...... done [39/267] Reinstalling pcre-8.45_3... [39/267] Extracting pcre-8.45_3: .......... done [40/267] Reinstalling libxslt-1.1.37... [40/267] Extracting libxslt-1.1.37: .......... done [41/267] Upgrading boost-libs from 1.82.0_1 to 1.83.0_1... [41/267] Extracting boost-libs-1.83.0_1: .>>> Unlocking package pkg...done.SG-1100 24.03 (ZFS)
-
@defunct78 said in 1100 Upgrade from 23.09.1 to 24.04 Failing:
[41/267] Upgrading boost-libs from 1.82.0_1 to 1.83.0_1...
[41/267] Extracting boost-libs-1.83.0_1: .>>> Unlocking package pkg...done.Hmm that's interesting. That
Unlocking package pkgmessage seems to be in the wrong place. Like it maybe has more than one upgrade process running? -
@stephenw10 I would not think so, unless one of the upgrade attempts left something behind because I just rebooted.
: ps -awx -l UID PID PPID C PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 0 0 1 -16 0 0 1232 swapin DLs - 0:46.39 [kernel] 0 1 0 1 24 0 11316 812 wait ILs - 0:00.16 /sbin/init 0 2 0 0 -60 0 0 32 - WL - 0:00.96 [clock] 0 3 0 0 -16 0 0 48 crypto_w DL - 0:00.00 [crypto] 0 4 0 0 -16 0 0 48 - DL - 0:00.00 [cam] 0 5 0 0 -60 0 0 16 - DL - 0:00.00 [busdma] 0 6 0 0 20 0 0 720 t->zthr_ DL - 0:05.70 [zfskern] 0 7 0 0 -16 0 0 16 pftm DL - 0:01.85 [pf purge] 0 8 0 1 -16 0 0 16 - DL - 0:00.76 [rand_harvestq] 0 9 0 0 -16 0 0 16 mx25jq DL - 0:00.00 [task: mx25l flash] 0 10 0 0 -16 0 0 16 audit_wo DL - 0:00.00 [audit] 0 11 0 0 187 0 0 32 - RNL - 41:19.52 [idle] 0 12 0 1 -60 0 0 320 - WL - 2:08.93 [intr] 0 13 0 0 -16 0 0 32 sleep DL - 0:00.00 [ng_queue] 0 14 0 1 -8 0 0 48 - DL - 0:01.09 [geom] 0 15 0 0 -16 0 0 16 seqstate DL - 0:00.00 [sequencer 00] 0 16 0 0 -60 0 0 160 - DL - 0:00.04 [usb] 0 17 0 1 -16 0 0 16 mmcsd di DL - 0:07.13 [mmcsd0: mmc/sd card] 0 18 0 0 -16 0 0 16 mmcsd di DL - 0:00.00 [mmcsd0boot0: mmc/sd] 0 19 0 0 -16 0 0 16 mmcsd di DL - 0:00.00 [mmcsd0boot1: mmc/sd] 0 20 0 1 -16 0 0 48 psleep DL - 0:07.86 [pagedaemon] 0 21 0 1 -16 0 0 16 psleep DL - 0:00.89 [vmdaemon] 0 22 0 0 -16 0 0 32 psleep DL - 0:00.04 [bufdaemon] 0 23 0 1 16 0 0 16 syncer DL - 0:00.03 [syncer] 0 24 0 0 -16 0 0 16 vlruwt DL - 0:00.02 [vnlru] 0 25 0 1 -16 0 0 16 aldslp DL - 0:00.00 [ALQ Daemon] 0 404 1 1 20 0 109516 3716 kqread Ss - 0:00.15 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm) 0 405 404 0 68 0 111628 0 accept IW - 0:00.00 (php-fpm) 0 406 404 1 68 0 111628 0 accept IW - 0:00.00 (php-fpm) 0 453 1 1 52 20 13288 0 kqread IWNs - 0:00.00 /usr/local/sbin/check_reload_status 0 454 453 0 68 20 13288 0 kqread IWN - 0:00.00 check_reload_status: Monitoring daemon of check_reload_status (check_reload_status) 0 726 1 0 20 0 14456 1724 select Is - 0:00.14 /sbin/devd -q -f /etc/pfSense-devd.conf 0 728 404 0 68 0 114764 0 accept IW - 0:00.00 (php-fpm) 0 741 0 1 -16 0 0 16 WAIT DL - 0:00.00 [iimb0] 0 3799 1 1 20 0 12580 1076 kqread Is - 0:00.21 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d <REMOVED> -p /var/r 0 5998 1 1 20 0 48312 37196 bpf Ss - 0:00.40 /usr/local/sbin/arpwatch -Z -f /usr/local/arpwatch/arp_mvneta0.4091.dat -i mvneta0.4091 -w 0 6510 1 1 20 0 48312 37136 bpf Ss - 0:00.39 /usr/local/sbin/arpwatch -Z -f /usr/local/arpwatch/arp_<REMOVED>.dat -i <REMOVED> -w <REMOVED> 0 6724 1 0 20 0 48312 37160 bpf Ss - 0:00.39 /usr/local/sbin/arpwatch -Z -f /usr/local/arpwatch/arp_<REMOVED>.dat -i <REMOVED> -w <REMOVED> 0 7152 1 0 20 0 48312 37144 bpf Ss - 0:00.39 /usr/local/sbin/arpwatch -Z -f /usr/local/arpwatch/arp_mvneta0.4092.dat -i mvneta0.4092 -w 0 8699 70096 0 24 0 0 16 - RE - 0:00.01 [kldstat] 65 13404 1 0 20 0 13140 1744 select SCs - 0:00.02 dhclient: mvneta0.4090 (dhclient) 0 15314 404 1 68 0 109580 0 accept IW - 0:00.00 (php-fpm) 168 17055 1 1 20 0 47372 9544 select Ss - 0:00.36 /usr/local/sbin/zebra -d 0 17701 48064 1 20 0 22492 4300 select Ss - 0:01.29 sshd: root@pts/0 (sshd) 168 18668 1 0 20 0 21140 5152 select Ss - 0:00.16 /usr/local/sbin/mgmtd -d 168 20400 1 0 20 0 21044 4220 select Ss - 0:00.16 /usr/local/sbin/staticd -d 168 23082 1 1 20 0 22096 4076 select Ss - 0:00.73 /usr/local/sbin/ospfd -d 0 23501 1 0 20 0 12724 1552 select Ss - 0:00.91 /usr/sbin/syslogd -O rfc5424 -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f / 0 25301 404 1 68 0 109580 0 accept IW - 0:00.00 (php-fpm) 0 29130 1 0 68 0 12572 0 wait IWs - 0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh 0 29456 29130 1 20 0 12572 340 nanslp I - 0:00.00 minicron: helper /usr/local/bin/ping_hosts.sh (minicron) 0 29786 1 1 68 0 12572 0 wait IWs - 0:00.00 /usr/local/bin/minicron 300 /var/run/ipsec_keepalive.pid /usr/local/bin/ipsec_keepalive.php 0 29968 29786 0 44 0 12572 344 nanslp I - 0:00.00 minicron: helper /usr/local/bin/ipsec_keepalive.php (minicron) 136 30069 1 0 20 0 26680 7844 select Ss - 0:00.53 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf / 0 30761 1 0 68 0 12572 0 wait IWs - 0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /usr/local/sbin/fcgicli -f /etc/r 0 31175 1 1 20 0 13468 2284 bpf Ss - 0:01.28 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid 0 31370 30761 0 68 0 12572 0 nanslp IW - 0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts (minicron) 0 31645 1 0 68 0 12572 0 wait IWs - 0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /usr/local/sbin/fcgicli -f 0 32109 31645 1 68 0 12572 0 nanslp IW - 0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data (minicron) 0 36621 1 0 20 0 18388 2604 select Ss - 0:00.43 /usr/local/sbin/watchfrr -d -r /usr/local/etc/rc.d/frrbBrestartbB%s -s /usr/local/etc/rc.d/ 0 38638 1 0 20 0 36268 2548 usem Is - 0:00.05 /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1 136 44658 1 1 20 0 23736 6176 select Ss - 0:00.52 /usr/local/sbin/dhcpd -6 -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpdv6.conf 0 46101 1 1 52 0 12576 324 kqread Ss - 0:00.00 /usr/local/sbin/dhcpleases6 -c /usr/local/bin/php-cgi -f /usr/local/sbin/prefixes.php -l /v 0 46113 1 0 20 0 278140 221872 select Ss - 0:09.15 /usr/sbin/bsnmpd -c /var/etc/snmpd.conf -p /var/run/snmpd.pid 0 48064 1 1 20 0 22288 2468 select Is - 0:00.01 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups (sshd) 0 49437 1 1 20 0 18048 1208 select Ss - 0:00.29 /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/run/miniupnpd.pid 0 63499 1 0 68 0 12904 956 nanslp Ss - 0:00.11 /usr/sbin/cron -s 0 70096 1 0 23 0 73708 25988 - Rs - 0:02.13 php_wg: WireGuard service (php_wg) 0 72103 1 1 20 0 23088 3276 select Ss - 0:00.57 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid 0 74133 1 1 39 0 30948 0 pause IWs - 0:00.00 nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx) 0 74300 74133 1 37 0 30948 0 kqread IW - 0:00.00 nginx: worker process (nginx) 0 74609 74133 1 39 0 30948 0 kqread IW - 0:00.00 nginx: worker process (nginx) 59 82677 1 0 20 0 59752 29520 kqread Ss - 0:02.99 /usr/local/sbin/unbound -c /var/unbound/unbound.conf 0 84008 1 1 68 0 13460 952 uwait Is - 0:00.81 /usr/local/bin/dpinger -S -r 0 -i <REMOVED> -B <REMOVED> -p /var/run/dpinger_<REMOVED> 0 84418 1 1 68 0 13460 956 uwait Is - 0:00.82 /usr/local/bin/dpinger -S -r 0 -i <REMOVED> -B <REMOVED> -p /var/run/dpinger_<REMOVED> 0 84851 1 1 68 0 13460 964 uwait Is - 0:00.91 /usr/local/bin/dpinger -S -r 0 -i <REMOVED> -B <REMOVED> -p /var/run/dpinger_<REMOVED> 0 85866 1 1 68 0 17688 1028 uwait Is - 0:00.65 /usr/local/bin/dpinger -S -r 0 -i <REMOVED> -B <REMOVED> -p /var/run/dpinger_<REMOVED> 0 86489 1 1 20 0 12644 1452 select Ss - 0:00.07 /usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog 0 90587 404 1 68 0 111628 0 accept IW - 0:00.00 (php-fpm) 558 90961 1 0 20 0 14468 2040 select S - 0:03.06 avahi-daemon: running [<REMOVED>] (avahi-daemon) 62 91736 1 0 20 0 12916 244 kqread Ss - 0:00.02 /usr/sbin/ftp-proxy -v 0 95182 1 0 20 0 13136 1432 select Is - 0:00.01 dhclient: system.syslog (dhclient) 0 99246 1 0 4 0 13136 1636 select Is - 0:00.00 dhclient: mvneta0.4090 [priv] (dhclient) 0 4597 1 1 68 0 13116 0 wait IWs u0 0:00.00 login [pam] (login) 0 7791 4597 0 50 0 13356 0 wait IW u0 0:00.00 -sh (sh) 0 10800 7791 1 68 0 13356 732 ttyin I+ u0 0:00.02 /bin/sh /etc/rc.initial 0 8484 52075 1 21 0 13464 3024 - R+ 0 0:00.01 ps -awx -l 0 28827 17701 1 68 0 13356 0 wait IWs 0 0:00.00 -sh (sh) 0 30076 28827 1 20 0 13356 944 wait I 0 0:00.04 /bin/sh /etc/rc.initial 0 52075 30076 0 20 0 13820 3952 pause S 0 0:00.08 /bin/tcsh [23.09.1-RELEASE] -
@defunct78 Well that is strange. So i connect via serial console, preformmed the upgrade, and all is successful. No issues. Have no idea why it worked this way an not over the TCP.
*** Welcome to Netgate pfSense Plus 24.03-RELEASE (arm64) on <REMOVED> *** -
Huh, did not expect that!
-
@stephenw10 Neither did I. Thanks for the help though.
-
I saw one report yesterday where the amount of available RAM on the 1100 was a factor. Running the upgrade from the shell likely ended up using less resources which allowed it to complete.
Anyone else hitting similar issues on an 1100 could try stopping or disabling some non-critical services (especially packages), trying the upgrade again, and then re-enabling them.
-
@stephenw10 Using this procedure is the only thing that worked for me. Thanks for posting it!
-
Glad i found this thread.
My upgrade was failing for a few days. Saw the comment above and I was able to kill all non-critical services and upgrade from the GUI.
