23.01 Upgrade failed - dead in the water

reggie14

@bitslammer I see you're running on Netgate hardware. When you did a USB recovery, did you recover straight to 23.01, or did you recover to 22.01/22.05 and redo the upgrade to 23.01?

I had to recover back to 2.6 CE and try the upgrade again, which failed again.

bitslammer

@reggie14 Went straight to 23.01.

tman222

@reggie14 said in 23.01 Upgrade failed - dead in the water:

@bitslammer
Did you get 23.01 working, or did you fall back?

I ran into this trying to do an upgrade from 2.6 CE. The 2.6 CE-> 22.01 upgrade seemed to go OK, but I got the same fatal error when doing the 23.01 update. It happened twice-- same problem when I restored a backup and tried again.

@reggie14 - does the error also occur if you uninstall packages first, upgrade, and then reinstall packages?

https://docs.netgate.com/pfsense/en/latest/install/upgrade-guide-prepare.html#packages

reggie14

@tman222
I haven't tried that yet. I'd be surprised if the packages are mucking with certs.inc.

I'm not sure what the mechanism would be, but I wonder if it has something to do with the CRL expiration bug system patch. That modifies certs.inc. I don't understand how those patches work, and whether pfSense tries to apply the same modifications to files in an update.

I haven't disabled it, though, since the WebGUI seems to warn you against disabling patches after updates.

SteveITS

@reggie14 The patches just directly modify the PHP file (it's a diff, if you look at the details you can see which lines are removed and added). An update replaces the file. Normally all Recommended patches are including in the subsequent version. After updating the System Patches package it shouldn't offer the old ones anymore, IIRC.

Per the above diff, line 48 of certs.inc isn't modifed by the patch.

Per various Netgate posts when upgrading to 23.01 there will be PHP errors shown while PHP is being upgraded. Those can be ignored if the upgrade completes.

reggie14

@steveits
Thanks. The problem is (apparently) specifically with the call to the gettext function. I can't figure out where that function gets included/declared, but I'm definitely not a PHP guy.

It doesn't seem to be just a warning. pfSense doesn't seem to ever boot properly. I can get to the pfSense menu in the console, but the only thing that works is opening a shell. none of the other options in the pfSense menu work, nor does the network work. There's no obvious problem in the certs.inc file, particularly on line 48, but I don't see where gettext ever gets defined.

Gertjan

@reggie14 said in 23.01 Upgrade failed - dead in the water:

particularly on line 48, but I don't see where gettext ever gets defined.

gettext is a standard, build in PHP function.

If it's 'missing', don't waste your time : re install pfSense.

reggie14

@gertjan

I did a reinstall, but the same problem happened again when I tried to update the software again.

Gertjan

@reggie14 said in 23.01 Upgrade failed - dead in the water:

@gertjan

I did a reinstall, but the same problem happened again when I tried to update the software again.

You mean : you install 23.01 - boot into it - no packages what so ever and you see this error ?

The gettext function is used :

grep -R 'gettext(' /etc/inc/* | wc -l

a ... lot !! (+2300 times) as it is used to translate the GUI.
It only fails on this file : /etc/inc/certs.inc on line 48 ?

Do you use the 'native' language - or some other GUI language ?

reggie14

@gertjan said in 23.01 Upgrade failed - dead in the water:

@reggie14 said in 23.01 Upgrade failed - dead in the water:

@gertjan

I did a reinstall, but the same problem happened again when I tried to update the software again.

You mean : you install 23.01 - boot into it - no packages what so ever and you see this error ?

The gettext function is used :
grep -R 'gettext(' /etc/inc/* | wc -l
a ... lot !! (+2300 times) as it is used to translate the GUI.
It only fails on this file : /etc/inc/certs.inc on line 48 ?

Do you use the 'native' language - or some other GUI language ?

The webGUI doesn't work at all after the 23.01 update. Even the network doesn't seem to work. The error is displayed in the console when directly attached to the fitlet2.

Jonb

I am having the same issue upgrading on a virtual appliance from 22.01. I have also done a complete reinstall (deleted virtual disk) and gone 2.6 > 22.01 >23.01

jimp

Something is causing it to fail partway through the upgrade. Is the disk full or nearly full?

What is in /conf/upgrade_log.txt and /conf/upgrade_log.latest.txt?

If you have a serial console on there, run the upgrade from the serial console and record the entire process to a file and post that.

reggie14

@jonb

Yes, that seems to be the exact same set of boot screen messages that I got after attempting the 23.01 update.

Darkk

@reggie14 said in 23.01 Upgrade failed - dead in the water:

@jonb

Yes, that seems to be the exact same set of boot screen messages that I got after attempting the 23.01 update.

This is happening to me as well. Lucky I've manually created a new boot environment before I did the upgrade so was able to recover from the bad upgrade.

My plan is to wait for 2.7 CE release and then install using that version. Then upgrade to Plus and restore the configuration file.

It's not ideal but at least with fresh install won't have these weird issues.

reggie14

@darkk

I still had it after doing a fresh reinstall/recovery, although that still means starting at 2.6 CE.

I'm going to wait for 2.7, too. I filed a ticket hoping they had been able to identify the problem from the other reports, but it sounds like they only have a solution for the Netgate hardware, which is to directly install a 23.01 image.

jimp

@reggie14 said in 23.01 Upgrade failed - dead in the water:

@darkk

I still had it after doing a fresh reinstall/recovery, although that still means starting at 2.6 CE.

I'm going to wait for 2.7, too. I filed a ticket hoping they had been able to identify the problem from the other reports, but it sounds like they only have a solution for the Netgate hardware, which is to directly install a 23.01 image.

That is a completely different problem and doesn't result in the kind of errors shown in this thread.

reggie14

@jimp said in 23.01 Upgrade failed - dead in the water:

@reggie14 said in 23.01 Upgrade failed - dead in the water:

@darkk

I still had it after doing a fresh reinstall/recovery, although that still means starting at 2.6 CE.

I'm going to wait for 2.7, too. I filed a ticket hoping they had been able to identify the problem from the other reports, but it sounds like they only have a solution for the Netgate hardware, which is to directly install a 23.01 image.

That is a completely different problem and doesn't result in the kind of errors shown in this thread.

Thanks. Any reason to think there would be a benefit to waiting until 2.7 CE comes out to try again?

jimp

@reggie14 said in 23.01 Upgrade failed - dead in the water:

Thanks. Any reason to think there would be a benefit to waiting until 2.7 CE comes out to try again?

Without knowing why these upgrade attempts failed, it's hard to speculate about possible solutions or workarounds. We need more information about what happened.

Jonb

@jimp
I will get the detail out of those logs today and post back.

Jonb

Bit later then hopped but I have a load of outputs. The key thing to say is I have no IP access to the machine only VMware console.

Looks like the PHP upgrade is corrupt.