23.01 Upgrade failed - dead in the water

Jonb

@jimp said in 23.01 Upgrade failed - dead in the water:

/conf/upgrade_log.latest.txt

The second SS is /conf/upgrade_log.latest.txt

Jonb

I have just tried an update through console instead of GUI. After reboot I get:

Anything else I can provide to help find the issue?

jimp

Whatever happened would be up much higher in the boot output than that. I'm not sure if it works on your version of vmware, but try pressing the Scroll Lock key and then Page Up as far as you can and see if you can see the start of the errors.

So far the pfSense-upgrade output you've showed looked OK but incomplete, and nothing after that has been anything but errors that aren't suggesting any particular root cause.

djwopasadjlk

after upgrading to 23.01 and rebooting, good that I have cronjob saving my configs once a day

jimp

@djwopasadjlk said in 23.01 Upgrade failed - dead in the water:

after upgrading to 23.01 and rebooting, good that I have cronjob saving my configs once a day

That's unrelated to anything else going on in this thread. That looks like an issue some people have seen with Hyper-V failing to boot. There are already threads for that around.

djwopasadjlk

@jimp it is happening on proxmox ve and I've tested it a couple of times, the vm is working on 2.6 common then upgraded to 22.01 and as soon as I upgrade to 23.01 and reboot I'm getting the kernel panic, so it should be update related.

jimp

@djwopasadjlk said in 23.01 Upgrade failed - dead in the water:

@jimp it is happening on proxmox ve and I've tested it a couple of times, the vm is working on 2.6 common then upgraded to 22.01 and as soon as I upgrade to 23.01 and reboot I'm getting the kernel panic, so it should be update related.

Then that's still going to need to be its own thread, it isn't even close to what's happening here.

Jonb

I have cloned the VM and upgraded it after a factory reset. The upgrade went fine. So I am concluding it is a package causing the failure.

I am wondering if it is acme but haven't tested as yet.

jimp

I have ACME on every VM in my lab and they've all upgraded fine, so it's doubtful that would be contributing to it. If it's a package, it's more likely something heavier like pfBlockerNG, snort, suricata, squid, that sort of thing.

Jonb

@jimp
Found it is squid or OpenVPN exporter.

Based on the above SS I would suggest squid.

morgenstern

@jonb

I have an almost identical issue running on a custom built physical hardware.

Upgrade paths:

pfsense CE 2.6 to pfsense+ 22.01 = success

pfsense+ 22.01 to 23.01 = fail

alt text

I followed by a clean install of CE 2.6 on an identical spare unit. Restored config, upgraded to +22.01 again, tried upgrading to 23.01 and failed with the same error.

Ended up doing another clean reinstall and restore (the Netgate support guys must have reactivated my activation token three times that night!) and tried upgrading only to 22.05.

This time limited success because the upgrade did not finish completely and left some packages (namely OpenVPN) in a limbo state. The updater on the dashboard kept saying there was a 22.05 update available even though I was already running it.

This luckily got fixed by running:

pfSense-upgrade -dy

To spare myself all the above misery I would have loved to do a clean install of pfsense+ 23.01 but the support guys said there is no memstick package available for whitebox hardware and the only way to get there is via the upgrades. So really, what's the point of migrating to pfsense+ on whitebox and virtual hardware if the upgrade path is broken and there isn't an option to install directly.

jimp

We're working on an installer for white box hardware and hope to have it ready with 22.05.

That said, there are quite a lot of people who have upgraded successfully. We still don't have information on vital pieces of the portion of the upgrade that failed here resulting the the errors that you and a small number of others have encountered. The posts so far have included the parts before the upgrade reboot and then these errors at the end, but what we need is the output in between where the first fatal error happened. Some errors from PHP are normal during the upgrade (they are unavoidable side effects of changing versions in-place) the errors we need to find in these failures are the errors that make the upgrade process terminate or get stuck in some other way.

Until we know what happened it's difficult to speculate about what will fix it. Unfortunately with only a video console it's very difficult to capture that part of the upgrade output, it would be simple to record it if you have a serial console available (and set as the primary console).

If you try again, I would definitely remove all packages first before performing the upgrade. You can reinstall them afterward.

morgenstern

@jimp

Hi, thanks for replying. I'm not sure I can repeat this on the production firewall anytime soon. I think I'll wait until the pfsense+ 22.05 installer is out. I don't won't to have to go through the re-registration process again.

Darkk

@morgenstern That is the exact same error screen I get on my Dell Optiplex PC. I guess the next time when I try the upgrade again I'll remove ALL the packages first and then give it a go. I did removed most of the packages and got that same screen.

Darkk

Here is mine. This is after I removed ALL packages. Rebooted a couple of times before upgrading to make sure any weird stuffs left behind are removed. Something to do with my openssl cert. It never get past this point.

So for now I've reverted back to the previous version.

Jonb

@darkk Where are you using that openssl cert. That is a different error to the one I posted which was caused by squid. I would be guessing openVPN.

j.sejo1

Hi,

22.05 to 23.01 FAIL.

install clean 2.6.0 CE
Upgrade clean to 22.05 OK
Upgrade clean to 23.01 OK
with 23.01 clean, run backup xml. OK.

euri

Seems I ran into the same/similar situation here. I get the same error as in this post by Jonb.

Device (whitebox) is already registered for pfSense+
Clean install CE 2.6.0, with restoring the previous configuration, went ok.
Upgrade to 23.01 failed; tried twice (clean install in between).

I reverted to CE 2.6.0 and now I'm thinking to remove some or all the packages and try the upgrade once more.

The packages I have are these:

For some reason I suspect pfBlockerNG-devel to be the culprit.

pfBlockerNG, ntopng and Wireguard have an explicit setting to retain their configuration.
Acme and HAproxy though do not have these options. Do you know if their configuration is retained upon re-installation?

Cheers!

JimBob Indiana

Dead here too. Followed steps. Upon boot 23.01 can't find the boot drive.

Back to square one, install 2.7 Feb release, now having this kind of error message during boot: "pfSense_plus-v23_01_amd64-pfsense_plus_v23_0//packagesite.pkg: Authentication error." Also I can no longer check for updates.

PatRyan

I have the same issue. Previously tried to upgrade (a few weeks ago) from a current 2.7 Dev version without uninstalling packages. Had mismatch issues with packages.

To remove the PHP v 8.1 vs 8.2 potential problem yesterday I reinstalled the 230215 build of 2.7. Then was able to see the upgrade choice to 23.01. Ran the upgrade which seemed to go fine until it rebooted and stopped at the same point as shown in the above posts. It stops at a login: prompt and won't go any further.

Reinstalled the current 2.7 Dev version from yesterday and tried the upgrade again. Same result.

Finally, reinstalled the current 2.7 Dev version while waiting for further progress.

Is there anyway to capture the complete startup log to post here?