Upgrade to version 24.11 hangs because disk full

[alban4]

Hi,

I tried to upgrade my Netgate SG-1100 from version 23.09.1 to version 24.11, but the upgrade is stuck in step "Extracting new base tarball".

From the console, I can see the disk is full:

[23.09.1-RELEASE][root@pfSense.home.arpa]/root: df -h /
Filesystem                             Size    Used   Avail Capacity  Mounted on
pfSense/ROOT/default_20250118103817    1.1G    1.1G      0B   100%    /

[23.09.1-RELEASE][root@pfSense.home.arpa]/root: zfs list /
NAME                                  USED  AVAIL  REFER  MOUNTPOINT
pfSense/ROOT/default_20250118103817  4.99G     0B  1.08G  /

What can I do to fix this?

[patient0]

@alban4 you can check if you old ZFS snapshots. To do that got to 'SystemBoot Environments'

The one with the green tick is the current one. If you got others from previous version, you can delete them.

Also it's a good idea to save a backup of the config, just to be sure.

[alban4]

@patient0 I was able to backup my config, thanks for the suggestion.

Here's what my Boot Environments look like (sorry for the quality, my computer connected to the Netgate UI is offline):

Are any of these environments safe to delete?

[patient0]

@alban4 you can delete the auto-default-20230820111756 (22.05), auto-default-20230820114210 (23.01), auto-default-20231110201512 (23.05.1) ones for sure. These are old pfSense versions.

After that you can have another go at the update. The space column is not accurate for ZFS due to the nature of how ZFS works, it can free a lot more space then shown in the space column.

If not successful:

Usually pfSense boots into the 'default' one. I assume default_20250118103817 was created during the upgrade.

I'd boot into the 'default' one (see https://docs.netgate.com/pfsense/en/latest/backup/zfsbe/gui.html) using the |> icon. If it boots up and works as expected then, selectd default as the default boot env by clicking on the star. Then delete default_20250118103817 so that you are left with only the default and auto-default-20250118102905.

[alban4]

So I went ahead and deleted 2 of the oldest boot environments: 22.05 and 23.01. I figured it wouldn't make much of a difference since they were using so little space.

But as a result, the bottom boot environment "default 23.09.1" became the default boot environment, and default_20250118103817 disappeared. Here's the new state of things:

And the disk is no longer full:

[23.09.1-RELEASE][root@pfSense.home.arpa]/root: df -hi
Filesystem                            Size    Used   Avail Capacity iused ifree %iused  Mounted on
pfSense/ROOT/default                  3.8G    1.1G    2.8G    28%     52k  5.8M    1%   /
devfs                                 1.0K      0B    1.0K     0%       0     0     -   /dev
pfSense/var                           2.8G    3.1M    2.8G     0%     108  5.8M    0%   /var
pfSense/tmp                           2.8G    176K    2.8G     0%      41  5.8M    0%   /tmp
pfSense/var/log                       2.8G    3.7M    2.8G     0%      82  5.8M    0%   /var/log
pfSense/home                          2.8G     96K    2.8G     0%       7  5.8M    0%   /home
pfSense/var/cache                     2.8G     96K    2.8G     0%       8  5.8M    0%   /var/cache
pfSense/var/tmp                       2.8G    112K    2.8G     0%      14  5.8M    0%   /var/tmp
pfSense/var/db                        2.8G    2.6M    2.8G     0%      57  5.8M    0%   /var/db
pfSense/ROOT/default/cf               2.8G    1.2M    2.8G     0%      49  5.8M    0%   /cf
pfSense/ROOT/default/var_cache_pkg    2.9G    168M    2.8G     6%      37  5.8M    0%   /var/cache/pkg
pfSense/ROOT/default/var_db_pkg       2.8G    5.5M    2.8G     0%      14  5.8M    0%   /var/db/pkg
tmpfs                                 4.0M    116K    3.9M     3%      41   14k    0%   /var/run
devfs                                 1.0K      0B    1.0K     0%       0     0     -   /var/dhcpd/dev

[23.09.1-RELEASE][root@pfSense.home.arpa]/root: zfs list /
NAME                   USED  AVAIL  REFER  MOUNTPOINT
pfSense/ROOT/default  3.33G  2.76G  1.08G  /

But my upgrade is still stuck in step "Extracting new base tarball". Anything I can do to get out of this state?

[alban4]

@patient0 Thanks for these instructions. I'll go ahead and delete additional boot environments and retry the update.

[patient0]

@alban4 if you search the forum there are a few threads about SG-1100 and upgrade to 24.01. Especially if Snort or Suricata was used.

But first of all I'd give some time, like 20 oder 30 Minutes.

It seems one options is to run the update from the console (menu point 13).

[alban4]

@patient0 Thanks a lot for your help. Thinks are looking much brighter now. I was able to retry the update to version 24.11, and it completed successfully this time.

After a reboot, the dashboard shows the new version:

However, if I navigate to System Update, it appears to think the upgrade that was hanging earlier is still running:

Is it possible to kill that failed upgrade for good? Otherwise, I might be stuck at version 24.11 forever.

[patient0]

@alban4 said in Upgrade to version 24.11 hangs because disk full:

if I navigate to System Update, it appears to think the upgrade that was hanging earlier is still running

No I don't think so, that's just the upgrade check after a reboot which conflicts with your upgrade check. Give it a some time (don't know, 3 Minutes) and it will be ok.

[alban4]

@patient0 You're right! It did end up showing "Status Up to date." eventually. Looks like I'm back in business now.

In summary:

• Navigate to Diagnostics | Backup & Restore to create a backup configuration
• Navigate to System | Boot Environments and delete old boot environments, even if they look like they're barely using any space
• Return to System | Update and start a new update
• The upgrade now completes normally, and the system reboots successfully running version 24.11

Thanks again for all the help!

[patient0]

@alban4 I'm happy it worked :)