pfSense+ 24.11 to 25.07 upgrade failed - Netgate 6100 Max appliance

sandie

@SteveITS 19 at the moment. Some created manually, some created automatically. The oldest is 22.05.

I would give my hand that I did not see such a message prior upgrade. It made me concerned, because after reverting to manual BE snapshot I am having it (?).

PS. I just checked 7100 DT (still 24.11) and message is: "Your Netgate device has pfSense+ as part of your device purchase.". I think it was like that prior 6100 upgrade attempt. Now it is no longer displayed like that [in 6100] even I reverted to previous snapshot (?).

PS2. I will check more on zfs list -t snapshot. It is very late now. I reverted to previous snapshot and will continue tommorow evening or the day after tommorow. Maybe message about disk space was true as you said, but it looked like nonsense to me :)

SteveITS

@sandie unless you really want to go back that far I’d delete the older ones. The space shown is totally misleading IMO…it’s more like 1 GB between versions.

There are posts about that library error I just don’t have a link handy.

sandie

@SteveITS I can not put full command output because comment is flagged as SPAM.

Output of: zfs list -t snapshot in attached txt file.output.txt

Deleting some BE snapshots in GUI is not really working well to me - eg. I attempted to remove quick-20230818011712 and getting error:

"The following input errors were detected:
Unable to unmount boot environment "quick-20230818011712". Error 14"

once BE screen is refreshed. Entries are disappering from BE menu, but they do not release disk space (?) - I am getting "unmount" errors. What should I do in terminal - zfs destroy ? What is safe to be destroyed from my list?

Still something is messed with disk space (?), because having 17 x 1,5 GB is not 70 GB. Is there any command to check how much disk space is truly available?

SteveITS

@sandie The dashboard should show the amount of used space. or "df -hi" command.

check:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-usage.html

@stephenw10 may be able to help further...

sandie

@SteveITS So the situation looks like that - even more weird.

I attempted to remove some BootEnv as suggested. None removed, but entries disappeared from BootEnv list. Therefore I run [in sequence] commands from output: output2.txt

As you see I must have used "-R" to remove some snapshot and dependant snapshots (?). Entries in menu were always disappearing, but always displayed error that snapshot is in use or so (message pasted in previous comment).

Whatever I try now (eg. upgrade again) I am always getting:
"Another instance of pfSense-upgrade is running. Try again later"
in System / Update / System Update in "Updating System" iframe. System does not want to update and seems to be stuck.

Screenshot 2025-08-13 at 21-20-33 router2.t24cc.com - System Update System Update.png

Manual removal of some snapshots did not help. "df -hi" does not show much diskspace occupied (see command output at the very bottom of in output2.txt file).

@SteveITS I think message about diskspace running out was nonsense message.

sandie

SSH is not working - I mean when I am attempting to connect SSH client goes to the "password prompt" then after giving correct password it immediately disconnects.

I find following info in system.log:

Aug 13 22:22:07 router2 sshd[3841]: warning: cannot open /etc/hosts.allow: Input/output error
Aug 13 22:22:07 router2 sshd[3841]: warning: cannot open /etc/hosts.deny: Input/output error
Aug 13 22:22:07 router2 sshd[22965]: in openpam_parse_chain(): /etc/pam.d/sshd: Input/output error
Aug 13 22:22:07 router2 sshd[22965]: fatal: PAM: initialisation failed
Aug 13 22:22:24 router2 sshd[3841]: warning: cannot open /etc/hosts.allow: Input/output error
Aug 13 22:22:24 router2 sshd[3841]: warning: cannot open /etc/hosts.deny: Input/output error
Aug 13 22:22:24 router2 sshd[31861]: in openpam_parse_chain(): /etc/pam.d/sshd: Input/output error
Aug 13 22:22:24 router2 sshd[31861]: fatal: PAM: initialisation failed

I can not load /etc/hosts.allow and /etc/hosts.deny in 6100 Diagnostics / Edit File (anymore?). /etc/pam.d/sshd seems to be empty upon loading too (!).

Screenshot 2025-08-13 at 22-38-45 router2.t24cc.com - Diagnostics Edit File.png

Screenshot 2025-08-13 at 22-40-21 router2.t24cc.com - Diagnostics Edit File.png

In theory some files exist with non-zero/correct size, but when attempted to be loaded nothing loads. Filesystem is screwed or what? :( I also see ntopng is being constantly restarted.

How it is possible that dir displays correct filesizes, but they simply do not load content in GUI?

24.11 is having similar filesizes and loads files in GUI properly.

There is more fun: Reboot is not rebooting:
Uptime 37 Days 10 Hours 30 Minutes 24 Seconds

I "rebooted" few times already, but in fact reboot is not rebooting. Filesystem or something must be screwed seriously.

sandie

Sorry guys but upgrade must have messed with BE very well. I run "shutdown -r now" to check if router will reboot.
It did (truly) reboot, but now it complains about expired SSL LE cert (expired on 19.07.2025). There was HSTS enabled, so I am not able to open GUI due to expired cert.

Cert must have been switched back to some previous snapshot (?) or what, because router had properly renewed and non-expired LE cert before first 2507 upgrade attempt.

I am not able to SSH-in to router2 (now router completely refuses SSH connections for some reason).

It looks like I will have to drive 380 km tommorow and I am generally angry because BE brought nothing but mess in my case (???). Instead of saving my ass things went super crazy.

I have 3 routers (1 already upgraded to 2507 without hiccup), but screwed one must always be located in most distant place ;) Ergh......

7100 DT is having most complicated setup and I am seriously worried about its failure during upgrade. How can I make full disk backup? What did you do guys earlier to secure upgrade when you did not have BE ? (I must not rely on BE when upgrading 7100 DT - it even has more snapshots created and 1 TB disk)

PS. I know that in 2507 you have introduced BE 2.0 but I lost trust to BE just now.

BTW. This distant location has PPPoE WAN connection, so please instruct me how I can recover to 2507 in most quickly manner.

PS2. I hope that $config['system']['webgui']['disablehsts']=true;write_config();exit; in Console option 12) PHP shell + Netgate pfSense Plus tools followed by 11) Restart GUI can disable HSTS. I am also counting that maybe ACME cert will autorefresh in few hours.

sandie

I was able to connect via HTTPS and public IP address and it looked like some older 24.11 snapshot was activated (?????, => old LE certs) and reboot generally worked/helped to liquidate weirdnesses.
Suprisingly after reboot Boot Environments shown again all BootEnv entries (?) - even though I already removed few of them, even with running backend commands.
I renewed LE certs, removed 22.xx and 23.xx images in BootEnv (they removed without noting any problems/warnings!), restarted box and initiated 2507 upgrade (screen was not anymore showing ".so problem" weird message).

Here is the log from this (again) Failed upgrade:

>>> Updating repositories metadata...
Updating pfSense-core repository catalogue...
Fetching meta.conf: . done
Fetching data.pkg: . done
Processing entries: . done
pfSense-core repository update completed. 5 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: . done
Fetching data.pkg: .......... done
Processing entries: 
Processing entries............. done
pfSense repository update completed. 734 packages processed.
All repositories are up to date.
>>> Setting vital flag on pkg...done.
>>> Setting vital flag on pfSense...done.
>>> Renaming current boot environment from default to default_20250814014604...done.
>>> Cloning current boot environment default_20250814014604...done.
>>> Removing vital flag from php83...done.
>>> Downloading Netgate Nexus...
The following packages will be fetched:

New packages to be FETCHED:
	pfSense-pkg-Nexus: 25.07_1 (43 MiB: 100.00% of the 43 MiB to download)

Number of packages to be fetched: 1

The process will require 43 MiB more space.
43 MiB to be downloaded.
Fetching pfSense-pkg-Nexus-25.07_1.pkg: .......... done
>>> Upgrading packages in cloned boot environment default...
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking for upgrades (92 candidates): .......... done
Processing candidates (92 candidates): .......... done
The following 94 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	bind920: 9.20.6 [pfSense]
	if_pppoe-kmod: 25.07.1500029 [pfSense]

Installed packages to be UPGRADED:
	bind-tools: 9.20.2 -> 9.20.6 [pfSense]
	boost-libs: 1.85.0 -> 1.86.0 [pfSense]
	bwi-firmware-kmod: 3.130.20 -> 3.130.20.1500029 [pfSense]
	c-ares: 1.34.2 -> 1.34.4 [pfSense]
	ca_root_nss: 3.104 -> 3.104_1 [pfSense]
	cpu-microcode-amd: 20240810 -> 20241121 [pfSense]
	cpu-microcode-intel: 20240910 -> 20250211 [pfSense]
	curl: 8.10.1 -> 8.11.0_1 [pfSense]
	dhcpcd: 10.1.0 -> 10.2.0 [pfSense]
	dnsmasq: 2.90_2,1 -> 2.90_4,1 [pfSense]
	expat: 2.6.3 -> 2.7.1 [pfSense]
	glib: 2.80.5,2 -> 2.80.5_1,2 [pfSense]
	graphviz: 12.0.0 -> 12.2.1 [pfSense]
	groff: 1.23.0_3 -> 1.23.0_4 [pfSense]
	harfbuzz: 10.0.1 -> 10.1.0 [pfSense]
	iperf3: 3.17.1 -> 3.18 [pfSense]
	kea: 2.6.1_1 -> 2.6.2 [pfSense]
	libfido2: 1.15.0 -> 1.15.0_1 [pfSense]
	libmaxminddb: 1.11.0 -> 1.12.2 [pfSense]
	libnghttp2: 1.63.0 -> 1.64.0 [pfSense]
	libpfctl: 0.13 -> 0.15 [pfSense]
	libssh2: 1.11.0_1,3 -> 1.11.1,3 [pfSense]
	libuv: 1.49.1 -> 1.49.2 [pfSense]
	luajit-openresty: 2.1.20240815 -> 2.1.20241104 [pfSense]
	miniupnpd: 2.3.7,1 -> 2.3.7_1,1 [pfSense]
	nettle: 3.10_1 -> 3.10.1 [pfSense]
	nginx: 1.26.2_6,3 -> 1.26.3,3 [pfSense]
	ntp: 4.2.8p18_1 -> 4.2.8p18_5 [pfSense]
	openldap26-client: 2.6.8 -> 2.6.9 [pfSense]
	opensc: 0.25.1 -> 0.26.0 [pfSense]
	openvpn: 2.6.12 -> 2.6.14 [pfSense]
	pciids: 20240920 -> 20241024 [pfSense]
	pfSense: 24.11 -> 25.07 [pfSense]
	pfSense-base: 24.11 -> 25.07 [pfSense-core]
	pfSense-boot: 24.11 -> 25.07 [pfSense-core]
	pfSense-composer-deps: 0.1 -> 0.3 [pfSense]
	pfSense-default-config-serial: 24.11 -> 25.07 [pfSense]
	pfSense-kernel-pfSense: 24.11 -> 25.07 [pfSense-core]
	pfSense-pkg-Cron: 0.3.8_5 -> 0.3.8_6 [pfSense]
	pfSense-pkg-System_Patches: 2.2.20_5 -> 2.2.22 [pfSense]
	pfSense-pkg-WireGuard: 0.2.9 -> 0.2.9_5 [pfSense]
	pfSense-pkg-acme: 0.9_1 -> 1.0 [pfSense]
	pfSense-pkg-bind: 9.18_2 -> 9.20_1 [pfSense]
	pfSense-pkg-ntopng: 5.6.0_1 -> 6.2.0 [pfSense]
	pfSense-pkg-pfBlockerNG: 3.2.0_16 -> 3.2.7 [pfSense]
	pfSense-repo: 24.11 -> 25.07 [pfSense]
	pftop: 0.10_1 -> 0.13 [pfSense]
	php83: 8.3.12 -> 8.3.19 [pfSense]
	php83-bcmath: 8.3.12 -> 8.3.19 [pfSense]
	php83-bz2: 8.3.12 -> 8.3.19 [pfSense]
	php83-ctype: 8.3.12 -> 8.3.19 [pfSense]
	php83-curl: 8.3.12 -> 8.3.19 [pfSense]
	php83-dom: 8.3.12 -> 8.3.19 [pfSense]
	php83-filter: 8.3.12 -> 8.3.19 [pfSense]
	php83-ftp: 8.3.12 -> 8.3.19 [pfSense]
	php83-gettext: 8.3.12 -> 8.3.19 [pfSense]
	php83-gmp: 8.3.12 -> 8.3.19 [pfSense]
	php83-intl: 8.3.12 -> 8.3.19 [pfSense]
	php83-ldap: 8.3.12 -> 8.3.19 [pfSense]
	php83-mbstring: 8.3.12 -> 8.3.19 [pfSense]
	php83-opcache: 8.3.12 -> 8.3.19 [pfSense]
	php83-pcntl: 8.3.12 -> 8.3.19 [pfSense]
	php83-pdo: 8.3.12 -> 8.3.19 [pfSense]
	php83-pdo_sqlite: 8.3.12 -> 8.3.19 [pfSense]
	php83-pfSense-module: 0.99 -> 0.105 [pfSense]
	php83-posix: 8.3.12 -> 8.3.19 [pfSense]
	php83-readline: 8.3.12 -> 8.3.19 [pfSense]
	php83-session: 8.3.12 -> 8.3.19 [pfSense]
	php83-shmop: 8.3.12 -> 8.3.19 [pfSense]
	php83-simplexml: 8.3.12 -> 8.3.19 [pfSense]
	php83-sockets: 8.3.12 -> 8.3.19 [pfSense]
	php83-sqlite3: 8.3.12 -> 8.3.19 [pfSense]
	php83-sysvmsg: 8.3.12 -> 8.3.19 [pfSense]
	php83-sysvsem: 8.3.12 -> 8.3.19 [pfSense]
	php83-sysvshm: 8.3.12 -> 8.3.19 [pfSense]
	php83-tokenizer: 8.3.12 -> 8.3.19 [pfSense]
	php83-xml: 8.3.12 -> 8.3.19 [pfSense]
	php83-xmlreader: 8.3.12 -> 8.3.19 [pfSense]
	php83-xmlwriter: 8.3.12 -> 8.3.19 [pfSense]
	php83-zlib: 8.3.12 -> 8.3.19 [pfSense]
	protobuf: 28.2,1 -> 28.3,1 [pfSense]
	protobuf-c: 1.4.1_6 -> 1.4.1_7 [pfSense]
	py311-libzfs: 1.1.2023020700_2 -> 1.1.2023020700_3 [pfSense]
	py311-packaging: 24.1 -> 24.2 [pfSense]
	py311-sqlite3: 3.11.10_8 -> 3.11.11_8 [pfSense]
	python311: 3.11.10 -> 3.11.11 [pfSense]
	radvd: 2.19_4 -> 2.20 [pfSense]
	rsync: 3.3.0 -> 3.4.0 [pfSense]
	socat: 1.8.0.1 -> 1.8.0.2 [pfSense]
	tailscale: 1.76.1 -> 1.82.5 [pfSense]
	unbound: 1.22.0 -> 1.23.0 [pfSense]
	zip: 3.0_2 -> 3.0_3 [pfSense]

Number of packages to be installed: 2
Number of packages to be upgraded: 92

The operation will free 11 MiB.
347 MiB to be downloaded.
[1/94] Fetching unbound-1.23.0.pkg: .......... done
[2/94] Fetching pfSense-pkg-Cron-0.3.8_6.pkg: . done
[3/94] Fetching nginx-1.26.3,3.pkg: .......... done
[4/94] Fetching php83-filter-8.3.19.pkg: . done
[5/94] Fetching php83-pdo_sqlite-8.3.19.pkg: . done
[6/94] Fetching pciids-20241024.pkg: ........ done
[7/94] Fetching groff-1.23.0_4.pkg: .......... done
[8/94] Fetching libuv-1.49.2.pkg: .......... done
[9/94] Fetching nettle-3.10.1.pkg: .......... done
[10/94] Fetching cpu-microcode-amd-20241121.pkg: ....... done
[11/94] Fetching pfSense-pkg-System_Patches-2.2.22.pkg: ......... done
[12/94] Fetching php83-ldap-8.3.19.pkg: .. done
[13/94] Fetching libnghttp2-1.64.0.pkg: .......... done
[14/94] Fetching dnsmasq-2.90_4,1.pkg: ....... done
[15/94] Fetching php83-simplexml-8.3.19.pkg: . done
[16/94] Fetching php83-bz2-8.3.19.pkg: . done
[17/94] Fetching php83-sockets-8.3.19.pkg: ... done
[18/94] Fetching if_pppoe-kmod-25.07.1500029.pkg: ... done
[19/94] Fetching php83-pfSense-module-0.105.pkg: ... done
[20/94] Fetching opensc-0.26.0.pkg: .......... done
[21/94] Fetching protobuf-c-1.4.1_7.pkg: ..... done
[22/94] Fetching php83-pcntl-8.3.19.pkg: . done
[23/94] Fetching php83-xmlreader-8.3.19.pkg: . done
[24/94] Fetching php83-sqlite3-8.3.19.pkg: . done
[25/94] Fetching iperf3-3.18.pkg: ...... done
[26/94] Fetching pfSense-repo-25.07.pkg: . done
[27/94] Fetching graphviz-12.2.1.pkg: .......... done
[28/94] Fetching php83-session-8.3.19.pkg: .. done
[29/94] Fetching pfSense-composer-deps-0.3.pkg: .......... done
[30/94] Fetching php83-zlib-8.3.19.pkg: . done
[31/94] Fetching pfSense-kernel-pfSense-25.07.pkg: .......... done
[32/94] Fetching libpfctl-0.15.pkg: ... done
[33/94] Fetching c-ares-1.34.4.pkg: .......... done
[34/94] Fetching php83-dom-8.3.19.pkg: ... done
[35/94] Fetching zip-3.0_3.pkg: .......... done
[36/94] Fetching php83-sysvmsg-8.3.19.pkg: . done
[37/94] Fetching protobuf-28.3,1.pkg: .......... done
[38/94] Fetching glib-2.80.5_1,2.pkg: .......... done
[39/94] Fetching php83-shmop-8.3.19.pkg: . done
[40/94] Fetching cpu-microcode-intel-20250211.pkg: .......... done
[41/94] Fetching libssh2-1.11.1,3.pkg: .......... done
[42/94] Fetching php83-bcmath-8.3.19.pkg: . done
[43/94] Fetching expat-2.7.1.pkg: ..... done
[44/94] Fetching bind920-9.20.6.pkg: .......... done
[45/94] Fetching py311-sqlite3-3.11.11_8.pkg: ... done
[46/94] Fetching pfSense-base-25.07.pkg: .......... done
[47/94] Fetching php83-curl-8.3.19.pkg: .... done
[48/94] Fetching boost-libs-1.86.0.pkg: .......... done
[49/94] Fetching php83-gmp-8.3.19.pkg: .. done
[50/94] Fetching pfSense-pkg-pfBlockerNG-3.2.7.pkg: .......... done
[51/94] Fetching pfSense-pkg-WireGuard-0.2.9_5.pkg: .. done
[52/94] Fetching pfSense-pkg-bind-9.20_1.pkg: . done
[53/94] Fetching tailscale-1.82.5.pkg: .......... done
[54/94] Fetching openvpn-2.6.14.pkg: .......... done
[55/94] Fetching php83-xmlwriter-8.3.19.pkg: . done
[56/94] Fetching bind-tools-9.20.6.pkg: .......... done
[57/94] Fetching py311-packaging-24.2.pkg: ...... done
[58/94] Fetching php83-pdo-8.3.19.pkg: .... done
[59/94] Fetching php83-ftp-8.3.19.pkg: . done
[60/94] Fetching pfSense-25.07.pkg: .......... done
[61/94] Fetching radvd-2.20.pkg: .. done
[62/94] Fetching ntp-4.2.8p18_5.pkg: ......... done
[63/94] Fetching php83-opcache-8.3.19.pkg: ........ done
[64/94] Fetching php83-sysvshm-8.3.19.pkg: . done
[65/94] Fetching php83-posix-8.3.19.pkg: . done
[66/94] Fetching pfSense-pkg-ntopng-6.2.0.pkg: . done
[67/94] Fetching libfido2-1.15.0_1.pkg: .......... done
[68/94] Fetching pfSense-boot-25.07.pkg: .......... done
[69/94] Fetching php83-readline-8.3.19.pkg: . done
[70/94] Fetching harfbuzz-10.1.0.pkg: .......... done
[71/94] Fetching miniupnpd-2.3.7_1,1.pkg: . done
[72/94] Fetching ca_root_nss-3.104_1.pkg: ....... done
[73/94] Fetching python311-3.11.11.pkg: .......... done
[74/94] Fetching pfSense-default-config-serial-25.07.pkg: . done
[75/94] Fetching kea-2.6.2.pkg: .......... done
[76/94] Fetching dhcpcd-10.2.0.pkg: ......... done
[77/94] Fetching php83-mbstring-8.3.19.pkg: .......... done
[78/94] Fetching php83-gettext-8.3.19.pkg: . done
[79/94] Fetching rsync-3.4.0.pkg: ....... done
[80/94] Fetching socat-1.8.0.2.pkg: ........ done
[81/94] Fetching php83-ctype-8.3.19.pkg: . done
[82/94] Fetching curl-8.11.0_1.pkg: .......... done
[83/94] Fetching php83-tokenizer-8.3.19.pkg: . done
[84/94] Fetching php83-8.3.19.pkg: .......... done
[85/94] Fetching bwi-firmware-kmod-3.130.20.1500029.pkg: .. done
[86/94] Fetching php83-xml-8.3.19.pkg: . done
[87/94] Fetching pfSense-pkg-acme-1.0.pkg: ....... done
[88/94] Fetching php83-intl-8.3.19.pkg: ...... done
[89/94] Fetching libmaxminddb-1.12.2.pkg: ... done
[90/94] Fetching openldap26-client-2.6.9.pkg: .......... done
[91/94] Fetching luajit-openresty-2.1.20241104.pkg: .......... done
[92/94] Fetching pftop-0.13.pkg: .... done
[93/94] Fetching php83-sysvsem-8.3.19.pkg: . done
[94/94] Fetching py311-libzfs-1.1.2023020700_3.pkg: .......... done
Checking integrity... done (1 conflicting)
  - bind920-9.20.6 [pfSense] conflicts with bind918-9.18.30 [installed] on /usr/local/bin/named-checkconf
Checking integrity... done (0 conflicting)
Conflicts with the existing packages have been found.
One more solver iteration is needed to resolve them.
The following 96 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
	bind918: 9.18.30

New packages to be INSTALLED:
	bind920: 9.20.6 [pfSense]
	if_pppoe-kmod: 25.07.1500029 [pfSense]

Installed packages to be UPGRADED:
	bind-tools: 9.20.2 -> 9.20.6 [pfSense]
	boost-libs: 1.85.0 -> 1.86.0 [pfSense]
	bwi-firmware-kmod: 3.130.20 -> 3.130.20.1500029 [pfSense]
	c-ares: 1.34.2 -> 1.34.4 [pfSense]
	ca_root_nss: 3.104 -> 3.104_1 [pfSense]
	cpu-microcode-amd: 20240810 -> 20241121 [pfSense]
	cpu-microcode-intel: 20240910 -> 20250211 [pfSense]
	curl: 8.10.1 -> 8.11.0_1 [pfSense]
	dhcpcd: 10.1.0 -> 10.2.0 [pfSense]
	dnsmasq: 2.90_2,1 -> 2.90_4,1 [pfSense]
	expat: 2.6.3 -> 2.7.1 [pfSense]
	glib: 2.80.5,2 -> 2.80.5_1,2 [pfSense]
	graphviz: 12.0.0 -> 12.2.1 [pfSense]
	groff: 1.23.0_3 -> 1.23.0_4 [pfSense]
	harfbuzz: 10.0.1 -> 10.1.0 [pfSense]
	iperf3: 3.17.1 -> 3.18 [pfSense]
	kea: 2.6.1_1 -> 2.6.2 [pfSense]
	libfido2: 1.15.0 -> 1.15.0_1 [pfSense]
	libmaxminddb: 1.11.0 -> 1.12.2 [pfSense]
	libnghttp2: 1.63.0 -> 1.64.0 [pfSense]
	libpfctl: 0.13 -> 0.15 [pfSense]
	libssh2: 1.11.0_1,3 -> 1.11.1,3 [pfSense]
	libuv: 1.49.1 -> 1.49.2 [pfSense]
	luajit-openresty: 2.1.20240815 -> 2.1.20241104 [pfSense]
	miniupnpd: 2.3.7,1 -> 2.3.7_1,1 [pfSense]
	nettle: 3.10_1 -> 3.10.1 [pfSense]
	nginx: 1.26.2_6,3 -> 1.26.3,3 [pfSense]
	ntp: 4.2.8p18_1 -> 4.2.8p18_5 [pfSense]
	openldap26-client: 2.6.8 -> 2.6.9 [pfSense]
	opensc: 0.25.1 -> 0.26.0 [pfSense]
	openvpn: 2.6.12 -> 2.6.14 [pfSense]
	pciids: 20240920 -> 20241024 [pfSense]
	pfSense: 24.11 -> 25.07 [pfSense]
	pfSense-base: 24.11 -> 25.07 [pfSense-core]
	pfSense-boot: 24.11 -> 25.07 [pfSense-core]
	pfSense-composer-deps: 0.1 -> 0.3 [pfSense]
	pfSense-default-config-serial: 24.11 -> 25.07 [pfSense]
	pfSense-kernel-pfSense: 24.11 -> 25.07 [pfSense-core]
	pfSense-pkg-Cron: 0.3.8_5 -> 0.3.8_6 [pfSense]
	pfSense-pkg-System_Patches: 2.2.20_5 -> 2.2.22 [pfSense]
	pfSense-pkg-WireGuard: 0.2.9 -> 0.2.9_5 [pfSense]
	pfSense-pkg-acme: 0.9_1 -> 1.0 [pfSense]
	pfSense-pkg-bind: 9.18_2 -> 9.20_1 [pfSense]
	pfSense-pkg-ntopng: 5.6.0_1 -> 6.2.0 [pfSense]
	pfSense-pkg-pfBlockerNG: 3.2.0_16 -> 3.2.7 [pfSense]
	pfSense-repo: 24.11 -> 25.07 [pfSense]
	pftop: 0.10_1 -> 0.13 [pfSense]
	php83: 8.3.12 -> 8.3.19 [pfSense]
	php83-bcmath: 8.3.12 -> 8.3.19 [pfSense]
	php83-bz2: 8.3.12 -> 8.3.19 [pfSense]
	php83-ctype: 8.3.12 -> 8.3.19 [pfSense]
	php83-curl: 8.3.12 -> 8.3.19 [pfSense]
	php83-dom: 8.3.12 -> 8.3.19 [pfSense]
	php83-filter: 8.3.12 -> 8.3.19 [pfSense]
	php83-ftp: 8.3.12 -> 8.3.19 [pfSense]
	php83-gettext: 8.3.12 -> 8.3.19 [pfSense]
	php83-gmp: 8.3.12 -> 8.3.19 [pfSense]
	php83-intl: 8.3.12 -> 8.3.19 [pfSense]
	php83-ldap: 8.3.12 -> 8.3.19 [pfSense]
	php83-mbstring: 8.3.12 -> 8.3.19 [pfSense]
	php83-opcache: 8.3.12 -> 8.3.19 [pfSense]
	php83-pcntl: 8.3.12 -> 8.3.19 [pfSense]
	php83-pdo: 8.3.12 -> 8.3.19 [pfSense]
	php83-pdo_sqlite: 8.3.12 -> 8.3.19 [pfSense]
	php83-pfSense-module: 0.99 -> 0.105 [pfSense]
	php83-posix: 8.3.12 -> 8.3.19 [pfSense]
	php83-readline: 8.3.12 -> 8.3.19 [pfSense]
	php83-session: 8.3.12 -> 8.3.19 [pfSense]
	php83-shmop: 8.3.12 -> 8.3.19 [pfSense]
	php83-simplexml: 8.3.12 -> 8.3.19 [pfSense]
	php83-sockets: 8.3.12 -> 8.3.19 [pfSense]
	php83-sqlite3: 8.3.12 -> 8.3.19 [pfSense]
	php83-sysvmsg: 8.3.12 -> 8.3.19 [pfSense]
	php83-sysvsem: 8.3.12 -> 8.3.19 [pfSense]
	php83-sysvshm: 8.3.12 -> 8.3.19 [pfSense]
	php83-tokenizer: 8.3.12 -> 8.3.19 [pfSense]
	php83-xml: 8.3.12 -> 8.3.19 [pfSense]
	php83-xmlreader: 8.3.12 -> 8.3.19 [pfSense]
	php83-xmlwriter: 8.3.12 -> 8.3.19 [pfSense]
	php83-zlib: 8.3.12 -> 8.3.19 [pfSense]
	protobuf: 28.2,1 -> 28.3,1 [pfSense]
	protobuf-c: 1.4.1_6 -> 1.4.1_7 [pfSense]
	py311-libzfs: 1.1.2023020700_2 -> 1.1.2023020700_3 [pfSense]
	py311-packaging: 24.1 -> 24.2 [pfSense]
	py311-sqlite3: 3.11.10_8 -> 3.11.11_8 [pfSense]
	python311: 3.11.10 -> 3.11.11 [pfSense]
	radvd: 2.19_4 -> 2.20 [pfSense]
	rsync: 3.3.0 -> 3.4.0 [pfSense]
	socat: 1.8.0.1 -> 1.8.0.2 [pfSense]
	tailscale: 1.76.1 -> 1.82.5 [pfSense]
	unbound: 1.22.0 -> 1.23.0 [pfSense]
	zip: 3.0_2 -> 3.0_3 [pfSense]

Installed packages to be REINSTALLED:
	pkg-1.21.3_5 [pfSense]

Number of packages to be removed: 1
Number of packages to be installed: 2
Number of packages to be upgraded: 92
Number of packages to be reinstalled: 1

The operation will free 17 MiB.
[1/96] Upgrading protobuf from 28.2,1 to 28.3,1...
[1/96] Extracting protobuf-28.3,1: .......... done
[2/96] Upgrading libuv from 1.49.1 to 1.49.2...
[2/96] Extracting libuv-1.49.2: .......... done
[3/96] Upgrading libnghttp2 from 1.63.0 to 1.64.0...
[3/96] Extracting libnghttp2-1.64.0: .......... done
[4/96] Upgrading protobuf-c from 1.4.1_6 to 1.4.1_7...
[4/96] Extracting protobuf-c-1.4.1_7: .......... done
[5/96] Upgrading bind-tools from 9.20.2 to 9.20.6...
[5/96] Extracting bind-tools-9.20.6: .......... done
[6/96] Deinstalling bind918-9.18.30...
[6/96] Deleting files for bind918-9.18.30: .......... done
[7/96] Upgrading python311 from 3.11.10 to 3.11.11...
[7/96] Extracting python311-3.11.11: .......... done
Failed

It started true upgrade and failed a bit further (7/96) and immediately displayed notification:
"Notices
Upgrade

pkg: "out of space" returned error code 3 @ 2025-08-14 01:53:11

"
Screenshot 2025-08-14 at 02-00-59 router2.t24cc.com - System Boot Environments.png

As you see I removed some BootEnv prior this attempt and everything looked right, but upgrade failed AGAIN. Could WAN PPPoE be a factor here? It failed in different place though than last time.

Something must be BROKEN with upgrade process. Output of df -hi:

Filesystem                            Size    Used   Avail Capacity iused ifree %iused  Mounted on
pfSense/ROOT/default                   80G    1.6G     79G     2%     65k  165M    0%   /
devfs                                 1.0K      0B    1.0K     0%       0     0     -   /dev
pfSense/var                            79G    6.5M     79G     0%     169  165M    0%   /var
pfSense/tmp                            79G    3.0M     79G     0%     433  165M    0%   /tmp
pfSense/home                           79G     96K     79G     0%       7  165M    0%   /home
pfSense/var/log                        79G     19M     79G     0%    1.5k  165M    0%   /var/log
pfSense/var/cache                      79G    104K     79G     0%       9  165M    0%   /var/cache
pfSense/var/tmp                        79G    120K     79G     0%      20  165M    0%   /var/tmp
pfSense/var/db                         86G    7.4G     79G     9%    4.5k  165M    0%   /var/db
pfSense/ROOT/default/cf                80G    1.0G     79G     1%    5.6k  165M    0%   /cf
pfSense/ROOT/default/var_cache_pkg     79G    166M     79G     0%      27  165M    0%   /var/cache/pkg
pfSense/ROOT/default/var_db_pkg        79G     11M     79G     0%      28  165M    0%   /var/db/pkg
tmpfs                                 4.0M    192K    3.8M     5%      55   14k    0%   /var/run
devfs                                 1.0K      0B    1.0K     0%       0     0     -   /var/dhcpd/dev
devfs                                 1.0K      0B    1.0K     0%       0     0     -   /var/etc/named/dev

and zfs list -t snapshot:

NAME                                                       USED  AVAIL  REFER  MOUNTPOINT
pfSense/ROOT/default@2024-04-23-22:30:33-0                 544K      -  1.53G  -
pfSense/ROOT/default@2024-04-24-22:41:40-0                 520K      -  1.53G  -
pfSense/ROOT/default@2024-11-26-20:40:39-0                 968K      -  1.51G  -
pfSense/ROOT/default@2024-11-26-20:43:56-0                26.8M      -  1.51G  -
pfSense/ROOT/default@2024-12-17-22:06:27-0                45.1M      -  1.56G  -
pfSense/ROOT/default@2025-02-01-17:32:45-0                44.3M      -  1.56G  -
pfSense/ROOT/default@2025-04-05-21:03:09-0                50.4M      -  1.57G  -
pfSense/ROOT/default@2025-08-14-01:45:05-0                 616K      -  1.57G  -
pfSense/ROOT/default/cf@2024-04-23-22:30:33-0              948K      -  6.15M  -
pfSense/ROOT/default/cf@2024-04-24-22:41:40-0              476K      -  10.1M  -
pfSense/ROOT/default/cf@2024-11-26-20:40:39-0              536K      -  1.27G  -
pfSense/ROOT/default/cf@2024-11-26-20:43:56-0              508K      -  1.27G  -
pfSense/ROOT/default/cf@2024-12-17-22:06:27-0              122M      -  1.38G  -
pfSense/ROOT/default/cf@2025-02-01-17:32:45-0             1.55M      -   298M  -
pfSense/ROOT/default/cf@2025-04-05-21:03:09-0             2.17M      -   708M  -
pfSense/ROOT/default/cf@2025-08-14-01:45:05-0              520K      -  1.05G  -
pfSense/ROOT/default/var_cache_pkg@2024-04-23-22:30:33-0     0B      -   381M  -
pfSense/ROOT/default/var_cache_pkg@2024-04-24-22:41:40-0     0B      -   381M  -
pfSense/ROOT/default/var_cache_pkg@2024-11-26-20:40:39-0   144K      -   373M  -
pfSense/ROOT/default/var_cache_pkg@2024-11-26-20:43:56-0  11.7M      -   384M  -
pfSense/ROOT/default/var_cache_pkg@2024-12-17-22:06:27-0   272K      -   141M  -
pfSense/ROOT/default/var_cache_pkg@2025-02-01-17:32:45-0   256K      -   141M  -
pfSense/ROOT/default/var_cache_pkg@2025-04-05-21:03:09-0   260K      -   141M  -
pfSense/ROOT/default/var_cache_pkg@2025-08-14-01:45:05-0    96K      -   158M  -
pfSense/ROOT/default/var_db_pkg@2024-04-23-22:30:33-0     1.22M      -  9.98M  -
pfSense/ROOT/default/var_db_pkg@2024-04-24-22:41:40-0     1.23M      -  9.99M  -
pfSense/ROOT/default/var_db_pkg@2024-11-26-20:40:39-0     3.34M      -  10.0M  -
pfSense/ROOT/default/var_db_pkg@2024-11-26-20:43:56-0     3.46M      -  11.0M  -
pfSense/ROOT/default/var_db_pkg@2024-12-17-22:06:27-0     3.05M      -  11.1M  -
pfSense/ROOT/default/var_db_pkg@2025-02-01-17:32:45-0     2.74M      -  11.1M  -
pfSense/ROOT/default/var_db_pkg@2025-04-05-21:03:09-0     2.69M      -  11.0M  -
pfSense/ROOT/default/var_db_pkg@2025-08-14-01:45:05-0     2.58M      -  11.0M  -

Both output BEFORE 24.11 -> 25.07 upgrade. Diskspace is not a factor here?

The difference is I do not notice problems with SSH and missing/unreadable files, eg. /etc/pam.d/sshd. SSH is working - I can connect and see console even though "System upgrade failed"

Installer is broken or some packages are causing problems (bind?) or PPPoE WAN?

Guys is it possible that "Failed upgrade" corrupts latest/current snapshot somehow? (the one I take manually)

I will reboot box to see if it goes up after "Failed upgrade" and services work.

Note: box has been rebooted properly, but ACME certs that I just refreshed before upgrade are withdrawn back to expired ones and HSTS is enabled back. LOL :) Box is alive but has withdrawn my latest changes that I saved to BootEnv (?). I think I am not getting how BootEnvs work. Once creating backup snapshot manually I should activate it to avoid withdrawing changes?

PS. It seems now it is your turn - I can check/test something for you. But I am not able to upgrade this 6100 successfully.

JeGr

@sandie said in pfSense+ 24.11 to 25.07 upgrade failed - Netgate 6100 Max appliance:

pkg: "out of space" returned error code 3 @ 2025-08-14 01:53:11

As long as THAT is your error while upgrading...

@sandie said in pfSense+ 24.11 to 25.07 upgrade failed - Netgate 6100 Max appliance:

Both output BEFORE 24.11 -> 25.07 upgrade. Diskspace is not a factor here?

YES it is. All snapshots need space and you have A LOT of them. I don't know why you need that many and with 24.03 switching the behavior of updates, I don't understand why you manually do snapshots BEFORE upgrading, as the current BE running is NOT touched, but a new one created and that one is updated - then switched over. So the box can fail back when the update fails. So it's unnecessary to sit on that much older snaps when you have no real use for them. Before/After makes no sense, als the "old 24.03" and "new default" are literally the "before"/"after". I haven't had the need for a BE for "oh I need to have the exact situation right before/after the update" ever but perhaps you did? I don't know, just wanted to let you know you have a lot of IMHO obsolete snaps here that serve minimal to no purpose (at least for my limited knowledge of you setup).

Check the dashboard disk and ZFS widget! They show the really used space and if there's less then ~2.5GB of space free, the latest update to 25.07 won't run. I just came back from a call with a customer that had the exact problem with a 4100 and 5 old BEs. Deleted all up to the 24.03 one so only 2 were left and voilà the free space jumped up to around 8GB instead of ~2 before the removals. Hit upgrade again - box updated without a hitch.

Would have attached pics, but unfortunately that seems broken (at least for me right now).

Boot Environments - didn't look too bad?
Disk Widget - showed the problem - far less capacity then needed (8GB eMMC version)
ZFS widget showed ~2.66GB left free - that's not enough for the update.

Cheers :)

stephenw10

Mmm, that's a painful read. Yes the 'out of space' error is almost always caused by many old snapshots taking up space but that's usually on eMMC boot devices where space is far more limited. It's possible some other drive issue is triggering it.

Using a PPPoE WAN would not have any effect here.

If you need to show or modify the Boot Environment from the CLI you should use the bectl command. I've never seen any problems using that. Unfortunately it looks like you may have accidentally removed some ZFS mountpoint snapshot from more BEs than intended.

If you run just zfs list does it show available space for each mount point?