23.01 Upgrade failed - dead in the water

bitslammer

Not having a good morning. Guessing there's no easy way out for this and USB recovery is the only option.

Fatal error: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:48
Stack trace:
#0 /etc/inc/functions.inc(34): require_once()
#1 /etc/rc.ecl(25): require_once('/etc/inc/functi...')
#2 {main}
thrown in /etc/inc/certs.inc on line 48
Launching the init system...Starting CRON... done.

Fatal error: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:48
Stack trace:
#0 /etc/inc/functions.inc(34): require_once()
#1 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#2 /etc/inc/config.inc(51): require_once('/etc/inc/notice...')
#3 /etc/rc.start_packages(25): require_once('/etc/inc/config...')
#4 {main}
thrown in /etc/inc/certs.inc on line 48
Netgate pfSense Plus 23.01-RELEASE arm Fri Feb 10 20:06:58 UTC 2023
Bootup complete

bitslammer

As a followup it seems I can't recover from USB as I get the Error: "recovery" not defined. This is the second time an update has essentially bricked my SG-3100 by breaking USB recovery. Not sure what the issue is but all I'm doing is selecting the upgrade from the GUI and giving it a good 15-20 minutes to run. I kept getting the yellow LED and did a power on/off reset and this is where I am.

SteveITS

@bitslammer said in 23.01 Upgrade failed - dead in the water:

giving it a good 15-20 minutes to run. I kept getting the yellow LED and did a power on/off reset

I have not updated any yet but I'd guess you probably didn't wait long enough. eMMC storage isn't fast and this update upgrades the OS to FreeBSD 14 too. I don't recall a yellow LED...though then again I'm generally not looking at any of our clients' 3100s while upgrading.

https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/reinstall-pfsense.html

FYI there is a bug on the 3100 that applies to upgrades not new installs: https://redmine.pfsense.org/issues/13963

bmeeks

I think I recall the release notes on the Netgate site saying the upgrade can take from 10 to 30 minutes. I would wait at least 30 minutes before giving up, and even then instead of powering off (which is highly likely to kill the install), I would connect with the USB serial cable to see what's happening.

In fact, if at all possible, it is always a good idea to have the USB serial connection up and running prior to kicking off the upgrade on these appliances. A cheap laptop and assortment of cables for connection to console ports is just about a "must have" for a network admin. You may need it as the emergency "way in" for firewalls and switches.

bitslammer

@bmeeks Support gave me a link to download the recovery image and I got it back up. It wasn't a matter of waiting it was erroring out in 2 min after booting.

dwighthenry

@bitslammer How did you request a copy of the image from support? My appliance is dead in the water as well.

SteveITS

@steveits said in 23.01 Upgrade failed - dead in the water:

https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/reinstall-pfsense.html

reggie14

@bitslammer
Did you get 23.01 working, or did you fall back?

I ran into this trying to do an upgrade from 2.6 CE. The 2.6 CE-> 22.01 upgrade seemed to go OK, but I got the same fatal error when doing the 23.01 update. It happened twice-- same problem when I restored a backup and tried again.

bitslammer

@reggie14 Got it running but had to recover from USB and restore a backup.

reggie14

@bitslammer I see you're running on Netgate hardware. When you did a USB recovery, did you recover straight to 23.01, or did you recover to 22.01/22.05 and redo the upgrade to 23.01?

I had to recover back to 2.6 CE and try the upgrade again, which failed again.

bitslammer

@reggie14 Went straight to 23.01.

tman222

@reggie14 said in 23.01 Upgrade failed - dead in the water:

@bitslammer
Did you get 23.01 working, or did you fall back?

I ran into this trying to do an upgrade from 2.6 CE. The 2.6 CE-> 22.01 upgrade seemed to go OK, but I got the same fatal error when doing the 23.01 update. It happened twice-- same problem when I restored a backup and tried again.

@reggie14 - does the error also occur if you uninstall packages first, upgrade, and then reinstall packages?

https://docs.netgate.com/pfsense/en/latest/install/upgrade-guide-prepare.html#packages

reggie14

@tman222
I haven't tried that yet. I'd be surprised if the packages are mucking with certs.inc.

I'm not sure what the mechanism would be, but I wonder if it has something to do with the CRL expiration bug system patch. That modifies certs.inc. I don't understand how those patches work, and whether pfSense tries to apply the same modifications to files in an update.

I haven't disabled it, though, since the WebGUI seems to warn you against disabling patches after updates.

SteveITS

@reggie14 The patches just directly modify the PHP file (it's a diff, if you look at the details you can see which lines are removed and added). An update replaces the file. Normally all Recommended patches are including in the subsequent version. After updating the System Patches package it shouldn't offer the old ones anymore, IIRC.

Per the above diff, line 48 of certs.inc isn't modifed by the patch.

Per various Netgate posts when upgrading to 23.01 there will be PHP errors shown while PHP is being upgraded. Those can be ignored if the upgrade completes.

reggie14

@steveits
Thanks. The problem is (apparently) specifically with the call to the gettext function. I can't figure out where that function gets included/declared, but I'm definitely not a PHP guy.

It doesn't seem to be just a warning. pfSense doesn't seem to ever boot properly. I can get to the pfSense menu in the console, but the only thing that works is opening a shell. none of the other options in the pfSense menu work, nor does the network work. There's no obvious problem in the certs.inc file, particularly on line 48, but I don't see where gettext ever gets defined.

Gertjan

@reggie14 said in 23.01 Upgrade failed - dead in the water:

particularly on line 48, but I don't see where gettext ever gets defined.

gettext is a standard, build in PHP function.

If it's 'missing', don't waste your time : re install pfSense.

reggie14

@gertjan

I did a reinstall, but the same problem happened again when I tried to update the software again.

Gertjan

@reggie14 said in 23.01 Upgrade failed - dead in the water:

@gertjan

I did a reinstall, but the same problem happened again when I tried to update the software again.

You mean : you install 23.01 - boot into it - no packages what so ever and you see this error ?

The gettext function is used :

grep -R 'gettext(' /etc/inc/* | wc -l

a ... lot !! (+2300 times) as it is used to translate the GUI.
It only fails on this file : /etc/inc/certs.inc on line 48 ?

Do you use the 'native' language - or some other GUI language ?

reggie14

@gertjan said in 23.01 Upgrade failed - dead in the water:

@reggie14 said in 23.01 Upgrade failed - dead in the water:

@gertjan

I did a reinstall, but the same problem happened again when I tried to update the software again.

You mean : you install 23.01 - boot into it - no packages what so ever and you see this error ?

The gettext function is used :
grep -R 'gettext(' /etc/inc/* | wc -l
a ... lot !! (+2300 times) as it is used to translate the GUI.
It only fails on this file : /etc/inc/certs.inc on line 48 ?

Do you use the 'native' language - or some other GUI language ?

The webGUI doesn't work at all after the 23.01 update. Even the network doesn't seem to work. The error is displayed in the console when directly attached to the fitlet2.

Jonb

I am having the same issue upgrading on a virtual appliance from 22.01. I have also done a complete reinstall (deleted virtual disk) and gone 2.6 > 22.01 >23.01

jimp

Something is causing it to fail partway through the upgrade. Is the disk full or nearly full?

What is in /conf/upgrade_log.txt and /conf/upgrade_log.latest.txt?

If you have a serial console on there, run the upgrade from the serial console and record the entire process to a file and post that.

reggie14

@jonb

Yes, that seems to be the exact same set of boot screen messages that I got after attempting the 23.01 update.

Darkk

@reggie14 said in 23.01 Upgrade failed - dead in the water:

@jonb

Yes, that seems to be the exact same set of boot screen messages that I got after attempting the 23.01 update.

This is happening to me as well. Lucky I've manually created a new boot environment before I did the upgrade so was able to recover from the bad upgrade.

My plan is to wait for 2.7 CE release and then install using that version. Then upgrade to Plus and restore the configuration file.

It's not ideal but at least with fresh install won't have these weird issues.

reggie14

@darkk

I still had it after doing a fresh reinstall/recovery, although that still means starting at 2.6 CE.

I'm going to wait for 2.7, too. I filed a ticket hoping they had been able to identify the problem from the other reports, but it sounds like they only have a solution for the Netgate hardware, which is to directly install a 23.01 image.

jimp

@reggie14 said in 23.01 Upgrade failed - dead in the water:

@darkk

I still had it after doing a fresh reinstall/recovery, although that still means starting at 2.6 CE.

I'm going to wait for 2.7, too. I filed a ticket hoping they had been able to identify the problem from the other reports, but it sounds like they only have a solution for the Netgate hardware, which is to directly install a 23.01 image.

That is a completely different problem and doesn't result in the kind of errors shown in this thread.

reggie14

@jimp said in 23.01 Upgrade failed - dead in the water:

@reggie14 said in 23.01 Upgrade failed - dead in the water:

@darkk

I still had it after doing a fresh reinstall/recovery, although that still means starting at 2.6 CE.

I'm going to wait for 2.7, too. I filed a ticket hoping they had been able to identify the problem from the other reports, but it sounds like they only have a solution for the Netgate hardware, which is to directly install a 23.01 image.

That is a completely different problem and doesn't result in the kind of errors shown in this thread.

Thanks. Any reason to think there would be a benefit to waiting until 2.7 CE comes out to try again?

jimp

@reggie14 said in 23.01 Upgrade failed - dead in the water:

Thanks. Any reason to think there would be a benefit to waiting until 2.7 CE comes out to try again?

Without knowing why these upgrade attempts failed, it's hard to speculate about possible solutions or workarounds. We need more information about what happened.

Jonb

@jimp
I will get the detail out of those logs today and post back.

Jonb

Bit later then hopped but I have a load of outputs. The key thing to say is I have no IP access to the machine only VMware console.

Looks like the PHP upgrade is corrupt.

Jonb

> Updating repositories metadata... 
Updating pfSense-core repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: . done
Processing entries: .. done
pfSense-core repository update completed. 15 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
pfSense repository update completed. 537 packages processed.
All repositories are up to date.
>>> Locking package pkg... done.
>>> Setting vital flag on pkg... done.
>>> Removing vital flag from php74... done.
>>> Unlocking package pkg... done.
>>> Downloading upgrade packages... 
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking for upgrades (247 candidates): .......... done
Processing candidates (247 candidates): ........ done
The following 316 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
	haproxy18: 1.8.30
	openldap24-client: 2.4.59_4
	php74: 7.4.26
	php74-bcmath: 7.4.26
	php74-bz2: 7.4.26
	php74-ctype: 7.4.26
	php74-curl: 7.4.26
	php74-dom: 7.4.26
	php74-filter: 7.4.26
	php74-ftp: 7.4.26
	php74-gettext: 7.4.26
	php74-intl: 7.4.26
	php74-json: 7.4.26
	php74-ldap: 7.4.26
	php74-mbstring: 7.4.26
	php74-opcache: 7.4.26
	php74-openssl: 7.4.26
	php74-openssl_x509_crl: 1.3
	php74-pcntl: 7.4.26
	php74-pdo: 7.4.26
	php74-pdo_sqlite: 7.4.26
	php74-pear: 1.10.12
	php74-pear-Auth_RADIUS: 1.1.0_4
	php74-pear-Cache_Lite: 1.8.3,1
	php74-pear-Crypt_CHAP: 1.5.0
	php74-pear-HTTP_Request2: 2.5.0,1
	php74-pear-Mail: 1.4.1,1
	php74-pear-Net_IPv6: 1.3.0.b2_2
	php74-pear-Net_SMTP: 1.10.0
	php74-pear-Net_Socket: 1.2.2
	php74-pear-Net_URL2: 2.2.1
	php74-pear-XML_RPC2: 1.1.4
	php74-pecl-mcrypt: 1.0.4
	php74-pecl-radius: 1.4.0b1_1
	php74-pecl-rrd: 2.0.1_1
	php74-pecl-ssh2: 1.3.1
	php74-pfSense-module: 0.76
	php74-phpseclib: 2.0.17
	php74-posix: 7.4.26
	php74-readline: 7.4.26
	php74-session: 7.4.26
	php74-shmop: 7.4.26
	php74-simplepie: 1.5.1_1
	php74-simplexml: 7.4.26
	php74-sockets: 7.4.26
	php74-sqlite3: 7.4.26
	php74-sysvmsg: 7.4.26
	php74-sysvsem: 7.4.26
	php74-sysvshm: 7.4.26
	php74-tokenizer: 7.4.26
	php74-xml: 7.4.26
	php74-xmlreader: 7.4.26
	php74-xmlwriter: 7.4.26
	php74-zlib: 7.4.26
	py38-libzfs: 1.1.2021100100
	py38-maxminddb: 2.0.3
	py38-ply: 3.11
	py38-setuptools: 57.0.0
	py38-sqlite3: 3.8.12_7
	python38: 3.8.12_1

New packages to be INSTALLED:
	7-zip: 21.07_2 [pfSense]
	gnugrep: 3.8 [pfSense]
	graphite2: 1.3.14 [pfSense]
	haproxy22: 2.2.25 [pfSense]
	harfbuzz: 5.3.1 [pfSense]
	hiredis: 1.0.2 [pfSense]
	libdeflate: 1.14 [pfSense]
	libimagequant: 2.17.0 [pfSense]
	libpsl: 0.21.1_6 [pfSense]
	libsysinfo: 0.0.3_2 [pfSense]
	lua54: 5.4.2 [pfSense]
	openldap26-client: 2.6.3 [pfSense]
	pfSense-boot: 23.01 [pfSense-core]
	php81: 8.1.11 [pfSense]
	php81-bcmath: 8.1.11 [pfSense]
	php81-bz2: 8.1.11 [pfSense]
	php81-ctype: 8.1.11 [pfSense]
	php81-curl: 8.1.11 [pfSense]
	php81-dom: 8.1.11 [pfSense]
	php81-filter: 8.1.11 [pfSense]
	php81-ftp: 8.1.11 [pfSense]
	php81-gettext: 8.1.11 [pfSense]
	php81-intl: 8.1.11_1 [pfSense]
	php81-ldap: 8.1.11 [pfSense]
	php81-libbe: 0.1.4.1 [pfSense]
	php81-mbstring: 8.1.11 [pfSense]
	php81-opcache: 8.1.11 [pfSense]
	php81-openssl_x509_crl: 1.3_1 [pfSense]
	php81-pcntl: 8.1.11 [pfSense]
	php81-pdo: 8.1.11 [pfSense]
	php81-pdo_sqlite: 8.1.11 [pfSense]
	php81-pear: 1.10.13 [pfSense]
	php81-pear-Auth_RADIUS: 1.1.0_4 [pfSense]
	php81-pear-Cache_Lite: 1.8.3,1 [pfSense]
	php81-pear-Crypt_CHAP: 1.5.0_2 [pfSense]
	php81-pear-HTTP_Request2: 2.5.1,1 [pfSense]
	php81-pear-Mail: 1.4.1,1 [pfSense]
	php81-pear-Net_IPv6: 1.3.0.b4_2 [pfSense]
	php81-pear-Net_SMTP: 1.10.1 [pfSense]
	php81-pear-Net_Socket: 1.2.2 [pfSense]
	php81-pear-Net_URL2: 2.2.1 [pfSense]
	php81-pear-XML_RPC2: 1.1.5 [pfSense]
	php81-pecl-mcrypt: 1.0.5 [pfSense]
	php81-pecl-radius: 1.4.0b1_2 [pfSense]
	php81-pecl-rrd: 2.0.3 [pfSense]
	php81-pecl-ssh2: 1.3.1 [pfSense]
	php81-pfSense-module: 0.91 [pfSense]
	php81-phpseclib: 2.0.17 [pfSense]
	php81-posix: 8.1.11 [pfSense]
	php81-readline: 8.1.11 [pfSense]
	php81-session: 8.1.11 [pfSense]
	php81-shmop: 8.1.11 [pfSense]
	php81-simplexml: 8.1.11 [pfSense]
	php81-sockets: 8.1.11 [pfSense]
	php81-sqlite3: 8.1.11 [pfSense]
	php81-sysvmsg: 8.1.11 [pfSense]
	php81-sysvsem: 8.1.11 [pfSense]
	php81-sysvshm: 8.1.11 [pfSense]
	php81-tokenizer: 8.1.11 [pfSense]
	php81-xml: 8.1.11 [pfSense]
	php81-xmlreader: 8.1.11 [pfSense]
	php81-xmlwriter: 8.1.11 [pfSense]
	php81-zlib: 8.1.11 [pfSense]
	py311-maxminddb: 2.2.0_2 [pfSense]
	py311-setuptools: 63.1.0 [pfSense]
	py311-sqlite3: 3.11.1_8 [pfSense]
	py39-libzfs: 1.1.2022081600 [pfSense]
	py39-setuptools: 63.1.0 [pfSense]
	python311: 3.11.1_1 [pfSense]
	python39: 3.9.16 [pfSense]

Installed packages to be UPGRADED:
	arj: 3.10.22_8 -> 3.10.22_9 [pfSense]
	bind-tools: 9.16.23 -> 9.18.8 [pfSense]
	ca_root_nss: 3.71 -> 3.83 [pfSense]
	ccid: 1.4.36 -> 1.5.0 [pfSense]
	check_reload_status: 0.0.11 -> 0.0.14 [pfSense]
	clamav: 0.104.3,1 -> 0.105.1_1,1 [pfSense]
	curl: 7.80.0 -> 7.85.0 [pfSense]
	cyrus-sasl: 2.1.27_2 -> 2.1.28 [pfSense]
	db5: 5.3.28_7 -> 5.3.28_8 [pfSense]
	dbus: 1.12.20_5 -> 1.14.4,1 [pfSense]
	devcpu-data: 20211109 -> 20221002 [pfSense]
	devcpu-data-amd: 20211115 -> 20221002 [pfSense]
	devcpu-data-intel: 20210608 -> 20220809 [pfSense]
	dmidecode: 3.3 -> 3.4_2 [pfSense]
	dnsmasq: 2.86,1 -> 2.87,1 [pfSense]
	dpinger: 3.0 -> 3.2 [pfSense]
	expat: 2.4.1 -> 2.4.9 [pfSense]
	filterdns: 2.0_5 -> 2.2 [pfSense]
	fontconfig: 2.13.94_1,1 -> 2.14.0,1 [pfSense]
	freetype2: 2.11.0 -> 2.12.1_2 [pfSense]
	gdbm: 1.22 -> 1.23 [pfSense]
	glib: 2.70.2,2 -> 2.74.0,2 [pfSense]
	graphviz: 2.44.1_17 -> 2.50.0_13 [pfSense]
	hostapd: 2.9_4 -> 2.10_5 [pfSense]
	icu: 70.1_1,1 -> 72.1,1 [pfSense]
	igmpproxy: 0.3,1 -> 0.3_1,1 [pfSense]
	isc-dhcp44-client: 4.4.2P1 -> 4.4.3P1 [pfSense]
	isc-dhcp44-relay: 4.4.2P1 -> 4.4.3P1 [pfSense]
	isc-dhcp44-server: 4.4.2P1_1 -> 4.4.3P1 [pfSense]
	jpeg-turbo: 2.1.1_1 -> 2.1.4 [pfSense]
	json-c: 0.15_1 -> 0.16 [pfSense]
	krb5: 1.19.2 -> 1.20 [pfSense]
	ldns: 1.7.1_2 -> 1.8.3 [pfSense]
	libedit: 3.1.20210216,1 -> 3.1.20210910,1 [pfSense]
	libffi: 3.3_1 -> 3.4.3 [pfSense]
	libgcrypt: 1.9.4 -> 1.9.4_1 [pfSense]
	libgd: 2.3.1,1 -> 2.3.3_2,1 [pfSense]
	libgpg-error: 1.43 -> 1.45 [pfSense]
	libiconv: 1.16 -> 1.17 [pfSense]
	libidn2: 2.3.2 -> 2.3.3_1 [pfSense]
	libinotify: 20180201_2 -> 20211018 [pfSense]
	libltdl: 2.4.6 -> 2.4.7 [pfSense]
	liblz4: 1.9.3,1 -> 1.9.4,1 [pfSense]
	libmaxminddb: 1.6.0 -> 1.7.1 [pfSense]
	libnghttp2: 1.46.0 -> 1.48.0 [pfSense]
	libssh2: 1.9.0_3,3 -> 1.10.0,3 [pfSense]
	libunistring: 0.9.10_1 -> 1.1 [pfSense]
	libuv: 1.42.0 -> 1.44.2 [pfSense]
	libxml2: 2.9.12 -> 2.10.3_1 [pfSense]
	libxslt: 1.1.34_2 -> 1.1.37 [pfSense]
	lighttpd: 1.4.63 -> 1.4.67 [pfSense]
	links: 2.20.2_1,1 -> 2.28,1 [pfSense]
	lua-resty-core: 0.1.22 -> 0.1.23 [pfSense]
	lua-resty-lrucache: 0.11 -> 0.13 [pfSense]
	luajit-openresty: 2.1.20210510 -> 2.1.20220915 [pfSense]
	mobile-broadband-provider-info: 20210805 -> 20220725 [pfSense]
	mpd5: 5.9_6 -> 5.9_12 [pfSense]
	mysql57-client: 5.7.36 -> 5.7.39_1 [pfSense]
	ndpi: 4.0.d20210818,1 -> 4.4.d20220714,1 [pfSense]
	nettle: 3.7.3 -> 3.8.1 [pfSense]
	nginx: 1.20.2_1,2 -> 1.22.0_9,3 [pfSense]
	norm: 1.5r6_1 -> 1.5r6_2 [pfSense]
	ntopng: 5.0.d20210923,1 -> 5.4.d20220812,1 [pfSense]
	ntp: 4.2.8p15_3 -> 4.2.8p15_5 [pfSense]
	oniguruma: 6.9.7.1 -> 6.9.8_1 [pfSense]
	open-vm-tools-nox11: 11.3.5_1,2 -> 12.1.0_1,2 [pfSense]
	openvpn: 2.5.4_1 -> 2.6.0_13 [pfSense]
	pcre: 8.45 -> 8.45_2 [pfSense]
	pcre2: 10.39 -> 10.40 [pfSense]
	pcsc-lite: 1.9.4,2 -> 1.9.5,2 [pfSense]
	perl5: 5.32.1_1 -> 5.32.1_3 [pfSense]
	pfSense: 22.01 -> 23.01 [pfSense]
	pfSense-Status_Monitoring: 1.7.11_4 -> 1.8 [pfSense]
	pfSense-base: 22.01 -> 23.01 [pfSense-core]
	pfSense-default-config: 22.01 -> 23.01 [pfSense-core]
	pfSense-kernel-pfSense: 22.01 -> 23.01 [pfSense-core]
	pfSense-pkg-LADVD: 1.2.2_2 -> 1.2.2_3 [pfSense]
	pfSense-pkg-Status_Traffic_Totals: 2.3.2_2 -> 2.3.2_3 [pfSense]
	pfSense-pkg-acme: 0.7.1_1 -> 0.7.3_1 [pfSense]
	pfSense-pkg-haproxy: 0.61_7 -> 0.61_9 [pfSense]
	pfSense-pkg-openvpn-client-export: 1.6_9 -> 1.8 [pfSense]
	pfSense-pkg-pfBlockerNG-devel: 3.1.0_4 -> 3.2.0_3 [pfSense]
	pfSense-pkg-snort: 4.1.6 -> 4.1.6_7 [pfSense]
	pfSense-pkg-squid: 0.4.45_9 -> 0.4.45_10 [pfSense]
	pfSense-rc: 22.01 -> 23.01 [pfSense-core]
	pftop: 0.7_9 -> 0.8 [pfSense]
	png: 1.6.37_1 -> 1.6.38 [pfSense]
	protobuf: 3.17.3,1 -> 3.21.7,1 [pfSense]
	readline: 8.1.1 -> 8.1.2 [pfSense]
	redis: 6.2.6 -> 7.0.5 [pfSense]
	rrdtool: 1.7.2_4 -> 1.8.0_1 [pfSense]
	rsync: 3.2.3_1 -> 3.2.7 [pfSense]
	smartmontools: 7.2_3 -> 7.3 [pfSense]
	snort: 2.9.20 -> 2.9.20_2 [pfSense]
	socat: 1.7.4.2 -> 1.7.4.3 [pfSense]
	sqlite3: 3.35.5_4,1 -> 3.39.3_1,1 [pfSense]
	squid: 4.15 -> 5.7 [pfSense]
	squidclamav: 7.1 -> 7.2 [pfSense]
	sshguard: 2.4.2_1,1 -> 2.4.2_2,1 [pfSense]
	strongswan: 5.9.4 -> 5.9.8 [pfSense]
	tiff: 4.3.0 -> 4.4.0 [pfSense]
	unbound: 1.13.2 -> 1.17.1_2 [pfSense]
	vnstat: 2.8 -> 2.9 [pfSense]
	webp: 1.2.1 -> 1.2.4 [pfSense]
	wpa_supplicant: 2.9_11 -> 2.10_6 [pfSense]
	wrapalixresetbutton: 0.0.11 -> 0.0.13 [pfSense]
	xxhash: 0.8.0 -> 0.8.1_2 [pfSense]
	zstd: 1.5.0 -> 1.5.2_1 [pfSense]

Installed packages to be REINSTALLED:
	arc-5.21p [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	beep-1.0_1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	brotli-1.0.9,1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	bsnmp-regex-0.6_2 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	bsnmp-ucd-0.4.5 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	bwi-firmware-kmod-3.130.20 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	c-icap-0.5.10,2 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	c-icap-modules-0.5.5 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	choparp-20150613 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	cpdup-1.22 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	cpustats-0.1_1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	daq-2.2.2_3 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	dhcp6-20080615.2_4 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	dhcpleases-0.5_1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	dhcpleases6-0.1_3 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	expiretable-0.6_2 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	filterlog-0.1_9 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	gettext-runtime-0.21 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	giflib-5.2.1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	gmp-6.2.1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	grepcidr-2.0 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	groff-1.22.4_4 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	iftop-1.0.p4 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	indexinfo-0.3.1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	ipmitool-1.8.18_3 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	iprange-1.0.4 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	jbigkit-2.1_1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	jq-1.6 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	ladvd-1.1.2_1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	libargon2-20190702 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	libdnet-1.13_3 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	libevent-2.1.12 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	libfontenc-1.1.4 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	libmcrypt-2.5.8_3 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	libmspack-0.10.1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	libpaper-1.1.28 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	libpcap-1.10.1_2 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	libsodium-1.0.18 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	libucl-0.8.1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	libzmq4-4.3.4 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	lua52-5.2.4 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	lua53-5.3.6 [pfSense]
	lzo2-2.10_1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	minicron-0.0.2 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	miniupnpd-2.2.1_1,1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	mkfontscale-1.2.1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	mpdecimal-2.5.1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	nss_ldap-1.265_14 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	openpgm-5.2.122_6 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	opensc-0.22.0 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	openvpn-auth-script-1.0.0.3 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	openvpn-client-export-2.5.8 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	pam_ldap-186_1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	pam_mkhomedir-0.2 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	pfSense-pkg-Open-VM-Tools-10.1.0_5,1 [pfSense]
	pfSense-pkg-ntopng-0.8.13_10 [pfSense]
	pfSense-pkg-squidGuard-1.16.18_20 [pfSense]
	pfSense-repo-23.01 [pfSense]
	pfSense-upgrade-1.0_58 [pfSense]
	pkg-1.18.4_4 [pfSense]
	psutils-1.17_5 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	qstats-0.2 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	radvd-2.19_2 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	rate-0.9_2 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	scponly-4.8.20110526_5 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	squidGuard-1.4_15 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	squid_radius_auth-1.10 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	ssh_tunnel_shell-0.2_1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	uchardet-0.0.7 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	uclcmd-0.1_3 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	unzoo-4.4_2 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	voucher-0.1_2 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	vstr-1.0.15_1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	webfonts-0.30_14 [pfSense] (ABI changed: 'freebsd:12:*' -> 'freebsd:14:*')
	whois-5.5.7 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	wol-0.7.1_4 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	xinetd-2.3.15_2 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')
	zip-3.0_1 [pfSense] (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:14:x86:64')

Number of packages to be removed: 60
Number of packages to be installed: 70
Number of packages to be upgraded: 108
Number of packages to be reinstalled: 78

jimp

@jonb said in 23.01 Upgrade failed - dead in the water:

Bit later then hopped but I have a load of outputs. The key thing to say is I have no IP access to the machine only VMware console.

The output you show isn't from pfSense-upgrade going from the old version to the new version, you're doing something directly with pkg and it's at least already partially on 23.01 there. You need to go back further and try the whole upgrade again.

@jonb said in 23.01 Upgrade failed - dead in the water:

Was that in the upgrade log latest file or console output? That's a start but there should be a lot more after that. That's just what shows up as it's preparing to do the upgrade, there should be more after for when it's doing the next part.

Gertjan

@jonb said in 23.01 Upgrade failed - dead in the water:

I have no IP access to the machine

Yep. pfSEnse says the same thing.
It can't go out, looking for updates etc.

Redo your VM setup : assign NICs etc.

When pfSense boots, and it find NICs, you'll be able to assign them, like 'em0' == WAN, em1 is LAN etc.

And yes, the gettext error means something is really bad, as the gettext function is part of PHP.

Jonb

@gertjan
I can't assign any nics through the usual menu. After upgrading it drops me to shell.

Jonb

@jimp
That output was after doing an update throught the web GUI. I though the same thing about the issue with the logs showing the old version.

When the system reboots it just presents me the shell and to login. The last SS was running pfSense-upgrade when it was broken.

I am trying to post the rest of the log but the forum has a limit. So here the file.upgrade log.txt

jimp

@jonb If /conf/upgrade_log.latest.txt is there it may also have more information. So far nothing in what you've posted looks out of the ordinary.

Jonb

@jimp said in 23.01 Upgrade failed - dead in the water:

/conf/upgrade_log.latest.txt

The second SS is /conf/upgrade_log.latest.txt

Jonb

I have just tried an update through console instead of GUI. After reboot I get:

Anything else I can provide to help find the issue?

jimp

Whatever happened would be up much higher in the boot output than that. I'm not sure if it works on your version of vmware, but try pressing the Scroll Lock key and then Page Up as far as you can and see if you can see the start of the errors.

So far the pfSense-upgrade output you've showed looked OK but incomplete, and nothing after that has been anything but errors that aren't suggesting any particular root cause.

djwopasadjlk

after upgrading to 23.01 and rebooting, good that I have cronjob saving my configs once a day

jimp

@djwopasadjlk said in 23.01 Upgrade failed - dead in the water:

after upgrading to 23.01 and rebooting, good that I have cronjob saving my configs once a day

That's unrelated to anything else going on in this thread. That looks like an issue some people have seen with Hyper-V failing to boot. There are already threads for that around.