Upgrade from 24.11 to 25.07.1 fails extracting Python due to disk space

digininja99

This seems like a similar problem to other people having upgrades but at a different location.

The GUI upgrade failed so I tried from the command line, it got this far before failure:

[71/71] Fetching 7-zip-24.09.pkg: .......... done
Checking integrity... done (0 conflicting)
[1/71] Upgrading python311 from 3.11.10 to 3.11.11...
[1/71] Extracting python311-3.11.11: ....tee: /cf/conf/upgrade_log.txt: No space left on device

tee: /cf/conf/upgrade_log.txt: No space left on device
[1/71] Extracting python311-3.11.11... done

I don't know if this is reporting lack of disk space in /cf or wherever it is trying to extract Python to. Other reports talk of having /cf full due to it containing lots of config backup files, but I've deleted all but the last two files an have /cf at only 572K usage before the upgrade.

[24.11-RELEASE][root@firewall.krynn.int]/cf/conf/backup: df -h
Filesystem                            Size    Used   Avail Capacity  Mounted on
pfSense/ROOT/default                  1.5G    1.1G    385M    75%    /
devfs                                 1.0K      0B    1.0K     0%    /dev
pfSense/tmp                           388M    3.3M    385M     1%    /tmp
pfSense/var                           388M    3.2M    385M     1%    /var
pfSense/home                          385M    108K    385M     0%    /home
pfSense/var/log                       388M    3.2M    385M     1%    /var/log
pfSense/var/cache                     385M     96K    385M     0%    /var/cache
pfSense/var/tmp                       385M    112K    385M     0%    /var/tmp
pfSense/var/db                        388M    3.1M    385M     1%    /var/db
pfSense/ROOT/default/cf               386M    572K    385M     0%    /cf
pfSense/ROOT/default/var_cache_pkg    745M    360M    385M    48%    /var/cache/pkg
pfSense/ROOT/default/var_db_pkg       392M    7.4M    385M     2%    /var/db/pkg
tmpfs                                 4.0M    156K    3.8M     4%    /var/run
devfs                                 1.0K      0B    1.0K     0%    /var/dhcpd/dev
[24.11-RELEASE][root@firewall.krynn.int]/cf/conf/backup: ls
config-1754574438.xml config-1756474105.xml

I tried monitoring disk space during the upgrade and /cf got to about 7% usage before the upgrade failed, at that point it dropped back down to about 2%.

This is after the upgrade failed;

[24.11-RELEASE][root@firewall.krynn.int]/cf/conf/backup: ls -l
total 177
-rw-r--r--  1 root wheel 70128 Sep  2 09:09 config-1754574438.xml
-rw-r--r--  1 root wheel 70080 Sep  2 09:09 config-1756474105.xml
[24.11-RELEASE][root@firewall.krynn.int]/cf/conf/backup: cd ..
[24.11-RELEASE][root@firewall.krynn.int]/cf/conf: ls -l
total 202
drwxr-xr-x  2 root wheel     4 Sep  2 09:09 backup
-rw-r--r--  1 root wheel 70088 Sep  2 08:38 config.xml
-rw-r--r--  1 root wheel    14 Mar 18 11:44 copynotice_version
-rw-r--r--  1 root wheel  5337 Sep  2 08:38 copyright
-rw-r--r--  1 root wheel     0 Nov 27  2024 enableserial_force
-rw-r--r--  1 root wheel  1049 Jan  5  2023 pkg_log_pfSense-pkg-ipsec-profile-wizard.txt
-rw-r--r--  1 root wheel  1553 Jul 23 09:06 pkg_log_pfSense-pkg-openvpn-client-export.txt
-rw-r--r--  1 root wheel 12383 Aug  4 10:12 rules.debug.old
-rw-r--r--  1 root wheel     0 Jun 22  2022 syslog_default_uncompressed
-rw-r--r--  1 root wheel 77916 Mar 18 11:44 upgrade_log.latest.txt
-rw-r--r--  1 root wheel 11694 Sep  2 09:14 upgrade_log.txt

What do I need to free up to let the upgrade complete? Can I update some of the packages individually before running the main upgrade, would that help?

digininja99

In case it makes a difference, this is on a Netgate 1100.

slu

@digininja99

how many old boot environments are saved?
https://docs.netgate.com/pfsense/en/latest/backup/zfsbe/gui.html

I delete before update all expect of the last version.

digininja99

@slu Intersting, I've deleted all but the last two, interestingly, the amount of used space on / hasn't increased, but the size of the disk has.

Before:

[24.11-RELEASE][root@firewall.krynn.int]/cf/conf/backup: df -h
Filesystem                            Size    Used   Avail Capacity  Mounted on
pfSense/ROOT/default                  1.5G    1.1G    385M    75%    /

After:

[24.11-RELEASE][root@firewall.krynn.int]/cf/conf: df -h
Filesystem                            Size    Used   Avail Capacity  Mounted on
pfSense/ROOT/default                  4.4G    1.1G    3.3G    26%    /

I'm running another upgrade now to see what happens.

digininja99

That worked, the upgrade ran fine from the command line after deleting things.

Thanks.

stephenw10

Yup you really want to see at least 1G available there for the upgrade.

digininja99

@stephenw10 can you point me at something else explains why when I deleted files the total disk space got bigger rather than the free space going up? That is something I've not seen before.

To prevent this in the future, is it worth adding a check into the upgrade script to check for available space?

stephenw10

Yes the upgrade could check that. And the ZFS BE manager could also alert the user to limited space And report the actual used space for each BE. That's in progress.

But the values you see there from df show the used and available space inside the current BE only not for the full disk. So when you remove BEs the available total space for other BEs increases.

digininja99

@stephenw10 I've just had a look at boot environments. I know it's not the same but in my head it works like creating a new VM which pushes all the others aside so it can share the available space. Boot the new VM, check it works ok, if so all is good, if not, just boot the pervious.

It seems like a cool idea for making a system that is very stable when it comes to big system updates - as long as there is room for the new environment.

SteveITS

@digininja99 https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-shrink.html

digininja99

@SteveITS Thanks.