23.01 Upgrade failed - dead in the water
-
I think I recall the release notes on the Netgate site saying the upgrade can take from 10 to 30 minutes. I would wait at least 30 minutes before giving up, and even then instead of powering off (which is highly likely to kill the install), I would connect with the USB serial cable to see what's happening.
In fact, if at all possible, it is always a good idea to have the USB serial connection up and running prior to kicking off the upgrade on these appliances. A cheap laptop and assortment of cables for connection to console ports is just about a "must have" for a network admin. You may need it as the emergency "way in" for firewalls and switches.
-
@bmeeks Support gave me a link to download the recovery image and I got it back up. It wasn't a matter of waiting it was erroring out in 2 min after booting.
-
@bitslammer How did you request a copy of the image from support? My appliance is dead in the water as well.
-
-
@bitslammer
Did you get 23.01 working, or did you fall back?I ran into this trying to do an upgrade from 2.6 CE. The 2.6 CE-> 22.01 upgrade seemed to go OK, but I got the same fatal error when doing the 23.01 update. It happened twice-- same problem when I restored a backup and tried again.
-
@reggie14 Got it running but had to recover from USB and restore a backup.
-
@bitslammer I see you're running on Netgate hardware. When you did a USB recovery, did you recover straight to 23.01, or did you recover to 22.01/22.05 and redo the upgrade to 23.01?
I had to recover back to 2.6 CE and try the upgrade again, which failed again.
-
@reggie14 Went straight to 23.01.
-
@reggie14 said in 23.01 Upgrade failed - dead in the water:
@bitslammer
Did you get 23.01 working, or did you fall back?I ran into this trying to do an upgrade from 2.6 CE. The 2.6 CE-> 22.01 upgrade seemed to go OK, but I got the same fatal error when doing the 23.01 update. It happened twice-- same problem when I restored a backup and tried again.
@reggie14 - does the error also occur if you uninstall packages first, upgrade, and then reinstall packages?
https://docs.netgate.com/pfsense/en/latest/install/upgrade-guide-prepare.html#packages
-
@tman222
I haven't tried that yet. I'd be surprised if the packages are mucking with certs.inc.I'm not sure what the mechanism would be, but I wonder if it has something to do with the CRL expiration bug system patch. That modifies certs.inc. I don't understand how those patches work, and whether pfSense tries to apply the same modifications to files in an update.
I haven't disabled it, though, since the WebGUI seems to warn you against disabling patches after updates.
-
@reggie14 The patches just directly modify the PHP file (it's a diff, if you look at the details you can see which lines are removed and added). An update replaces the file. Normally all Recommended patches are including in the subsequent version. After updating the System Patches package it shouldn't offer the old ones anymore, IIRC.
Per the above diff, line 48 of certs.inc isn't modifed by the patch.
Per various Netgate posts when upgrading to 23.01 there will be PHP errors shown while PHP is being upgraded. Those can be ignored if the upgrade completes.
-
@steveits
Thanks. The problem is (apparently) specifically with the call to the gettext function. I can't figure out where that function gets included/declared, but I'm definitely not a PHP guy.It doesn't seem to be just a warning. pfSense doesn't seem to ever boot properly. I can get to the pfSense menu in the console, but the only thing that works is opening a shell. none of the other options in the pfSense menu work, nor does the network work. There's no obvious problem in the certs.inc file, particularly on line 48, but I don't see where gettext ever gets defined.
-
@reggie14 said in 23.01 Upgrade failed - dead in the water:
particularly on line 48, but I don't see where gettext ever gets defined.
gettext is a standard, build in PHP function.
If it's 'missing', don't waste your time : re install pfSense.
-
I did a reinstall, but the same problem happened again when I tried to update the software again.
-
@reggie14 said in 23.01 Upgrade failed - dead in the water:
I did a reinstall, but the same problem happened again when I tried to update the software again.
You mean : you install 23.01 - boot into it - no packages what so ever and you see this error ?
The gettext function is used :
grep -R 'gettext(' /etc/inc/* | wc -la ... lot !! (+2300 times) as it is used to translate the GUI.
It only fails on this file : /etc/inc/certs.inc on line 48 ?Do you use the 'native' language - or some other GUI language ?
-
@gertjan said in 23.01 Upgrade failed - dead in the water:
@reggie14 said in 23.01 Upgrade failed - dead in the water:
I did a reinstall, but the same problem happened again when I tried to update the software again.
You mean : you install 23.01 - boot into it - no packages what so ever and you see this error ?
The gettext function is used :
grep -R 'gettext(' /etc/inc/* | wc -la ... lot !! (+2300 times) as it is used to translate the GUI.
It only fails on this file : /etc/inc/certs.inc on line 48 ?Do you use the 'native' language - or some other GUI language ?
The webGUI doesn't work at all after the 23.01 update. Even the network doesn't seem to work. The error is displayed in the console when directly attached to the fitlet2.
-
I am having the same issue upgrading on a virtual appliance from 22.01. I have also done a complete reinstall (deleted virtual disk) and gone 2.6 > 22.01 >23.01
Hosted desktops and servers with support without complication.
www.blueskysystems.co.uk -
Something is causing it to fail partway through the upgrade. Is the disk full or nearly full?
What is in
/conf/upgrade_log.txtand/conf/upgrade_log.latest.txt?If you have a serial console on there, run the upgrade from the serial console and record the entire process to a file and post that.
-
Yes, that seems to be the exact same set of boot screen messages that I got after attempting the 23.01 update.
-
@reggie14 said in 23.01 Upgrade failed - dead in the water:
Yes, that seems to be the exact same set of boot screen messages that I got after attempting the 23.01 update.
This is happening to me as well. Lucky I've manually created a new boot environment before I did the upgrade so was able to recover from the bad upgrade.
My plan is to wait for 2.7 CE release and then install using that version. Then upgrade to Plus and restore the configuration file.
It's not ideal but at least with fresh install won't have these weird issues.
