upgrade seems to simply stop

sgw

Trying to go from 23.09.1 to 24.03 today on a Netgate 1100 (just curious).

I started the upgrade 3 times already and it always seems to simply stop after a few packages.
There are 333 packages (correction: 333MB) to be downloaded ... (did that also 3 times) ... I also tried to do pkg bootstrap -f without success.

A reboot doesn't change anything. What can I try, what can I look at? thanks (starting it once more right now. Would be great to keep the downloaded packages ..)

EDIT: to pinpoint the snapshot: it currently pulls pfSense-base-24.03.b.20240315.0600.pkg

sgw

Doing pfSense-upgrade ends with:

Number of packages to be removed: 52
Number of packages to be installed: 56
Number of packages to be upgraded: 64
Number of packages to be reinstalled: 87

The operation will free 13 MiB.
[1/259] Reinstalling indexinfo-0.3.1...
[1/259] Extracting indexinfo-0.3.1: .... done
[2/259] Reinstalling mpdecimal-2.5.1...
[2/259] Extracting mpdecimal-2.5.1: .......... done
>>> Unlocking package pkg...done.

and that's it.

sgw

Just in case the BIOS version plays a role here:

Vendor: U-Boot
Version: 2018.03-devel-18.12.3-gc9aa92c-dirty
Release Date: Thu Oct 7 2021

Retried right now. Upgrade did ~8 packages, rebooted into 23.09.1 again. There are only BEs based on 23.09, 23.09.1 (and one old on 23.05).

stephenw10

Check the available drive space. If it's low you may need to remove some old BE snapshots if this is a ZFS install.

Steve

sgw

@stephenw10 there are 3.5G free, only 24% used (and 1.1G used).
Should not be an issue?

There are only a few BEs:

 bectl list
BE                                    Active Mountpoint Space Created
auto-default-20231208034714           -      -          553M  2023-12-08 03:47
default                               -      -          23.5M 2024-03-16 11:34
default_20240316113338                -      -          357M  2024-03-18 10:13
default_20240316113338_20240318101125 NR     /          2.02G 2023-05-22 17:52

stephenw10

That should be easily enough space.

Try upgrading at the command line with: pfSense-upgrade -d

That should show you more error output if it fails.

sgw

@stephenw10 tried that. Stays on 23.09.1

but I can't choose the beta in the dropdown "Update settings" ... maybe you guys are changing things right now?

In "System Update" I see: "MessagespfSense-repoc: no package 'name' pfSense-repoc: no pfSense packages installed"

But I can wait ... nothing to hurry.

stephenw10

Yes, there's an issue currently. But that's not what you were hitting.

Leave it at 23.09.1 for now until we resolve the backend problem.

sgw

@stephenw10 will do, thanks

sgw

I don't know if I am too impatient, tried it again right now.
There seem to be changes in place upstream, but the behavior is the same in the first run.

it simply stops after a few packages, again:

[5/259] Extracting libffi-3.4.4: .......... done
[6/259] Reinstalling expat-2.5.0...
[6/259] Extracting expat-2.5.0: .......... done
[7/259] Reinstalling libiconv-1.17...
[7/259] Extracting libiconv-1.17: .......... done
[8/259] Upgrading python311 from 3.11.4 to 3.11.7...
>>> Unlocking package pkg...
Unlocking pkg-1.20.8_2

stephenw10

That was running the upgrade at the CLI?

Does it return to the prompt or simply stop as though it's still trying?

sgw

@stephenw10

tried this today.

Retried pfSense-upgrade -d now, it returns to the prompt:

Number of packages to be removed: 52
Number of packages to be installed: 56
Number of packages to be upgraded: 64
Number of packages to be reinstalled: 87

The operation will free 13 MiB.
[1/259] Reinstalling indexinfo-0.3.1...
[1/259] Extracting indexinfo-0.3.1: .... done
[2/259] Reinstalling mpdecimal-2.5.1...
[2/259] Extracting mpdecimal-2.5.1: .......... done
[3/259] Upgrading gettext-runtime from 0.22_1 to 0.22.3...
[3/259] Extracting gettext-runtime-0.22.3: .......... done
[4/259] Upgrading readline from 8.2.1 to 8.2.7...
[4/259] Extracting readline-8.2.7: .......... done
[5/259] Reinstalling libffi-3.4.4...
[5/259] Extracting libffi-3.4.4: .......... done
[6/259] Reinstalling expat-2.5.0...
[6/259] Extracting expat-2.5.0: .......... done
[7/259] Reinstalling libiconv-1.17...
[7/259] Extracting libiconv-1.17: .......... done
[8/259] Upgrading python311 from 3.11.4 to 3.11.7...
[8/259] Extracting python311-3.11.7: .......... done
>>> Unlocking package pkg...
Unlocking pkg-1.20.8_2
[23.09.1-RELEASE][admin@some.fqdn]/root:

stephenw10

Hmm. Is there anything logged in the system log when that happens?

Try clearing the pkg cache first to rule out some bad download that may have happened: pkg-static clean -ay

sgw

@stephenw10 well, that was already done, as mentioned.

But OK:

# pkg-static clean -ay
Nothing to do.

it lists the same numbers again when I retry now:

Number of packages to be removed: 52
Number of packages to be installed: 56
Number of packages to be upgraded: 64
Number of packages to be reinstalled: 87

The operation will free 13 MiB.
333 MiB to be downloaded.

The download takes ~10 minutes or so (because I am in Europe?), I don't expect anything new now ;-)

had an idea: disabled pfBlockerNG for now ... and watching system.log now ...

EDIT: could the packages be somehow "cached" while I retry?

EDIT 2: it fails again, I see these lines in system.log:

<3>1 2024-03-23T18:02:08.974475+01:00 my.fqdn kernel - - - pid 15837 (pkg-static), jid 0, uid 0, was killed: failed to reclaim memory
<3>1 2024-03-23T18:02:09.137841+01:00 my.fqdn kernel - - - pid 48171 (netstat), jid 0, uid 0, was killed: failed to reclaim memory

The GUI tells me (after the upgrade failed ...): 63% of 958 MiB used.

I stop services now: telegraf, wireguard ...

EDIT 3: solved ... it was simply a lack of free RAM. Soved by disabling various services while upgrading.

feeling a bit stupid ;-) sorry for the noise.

sgw

Different issue, just adding:

[23-Mar-2024 17:38:56 UTC] PHP Warning:  PHP Startup: zlib: Unable to initialize module
Module compiled with module API=20230831
PHP    compiled with module API=20220829
These options need to match
 in Unknown on line 0
[23-Mar-2024 17:38:56 UTC] PHP Warning:  PHP Startup: pdo_sqlite: Unable to initialize module
Module compiled with module API=20230831
PHP    compiled with module API=20220829
These options need to match
 in Unknown on line 0
[23-Mar-2024 17:38:56 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'xmlreader.so' (tried: /usr/local/lib/php/20230831/xmlreader.so (/usr/local/lib/php/20230831/xmlreader.so: Undefined symbol "dom_node_class_entry"), /usr/local/lib/php/20230831/xmlreader.so.so (Cannot open "/usr/local/lib/php/20230831/xmlreader.so.so")) in Unknown on line 0
[23-Mar-2024 18:38:56 Europe/Vienna] PHP Fatal error:  Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:48
Stack trace:
#0 /etc/inc/functions.inc(34): require_once()
#1 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#2 /etc/inc/config.inc(37): require_once('/etc/inc/notice...')
#3 /usr/local/pkg/wireguard/includes/wg_service.inc(24): require_once('/etc/inc/config...')
#4 {main}
  thrown in /etc/inc/certs.inc on line 48



No FreeBSD crash data found.

there are many more errors in the log, I only paste the last lines ... seems to be a PHP mismatch between main package and modules or so. So wireguard does not start right now.

sgw

Reinstalling wireguard fixed wireguard. I don't know if there are other issues left, so far things work, thanks.

rtorres

@sgw said in upgrade seems to simply stop:

Reinstalling wireguard fixed wireguard. I don't know if there are other issues left, so far things work, thanks.

That error came up for me as well after upgrading to the Beta 24.03. Looks like Wireguard was causing that error because after I reinstalled Wireguard (like you did), Wireguard started working and no more errors.

stephenw10

Hmm, yeah that was a known issue but it should be fixed in the current beta. What version did you end up on?

sgw

@stephenw10

it shows: 24.03.b.20240322.1708

thanks for your help

stephenw10

OK that's correct. We'll dig into it.