pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading
-
Hmm, this seems familiar. It's similar to something we hit with a quirky UART implementation on specific hardware.
What does it show for detecting the UART(s) whenbooting verbose?
What happens if you set the console to EFI only?
-
In "efi" only, the server reboot after this screen :
set console="efi" boot -v
-
Hmm, well that's not helpful!
-
@stephenw10 Watching these other postings to see if any of them apply to my situation, nothing quite the same as far as I can tell.
Is there a boot option that will force the boot process to run a line at a time? -
Just tried to upgrade today and encountered the same problem as OP, freezing in nearly the identical location.
My pfsense instance is a proxmox vm, with the Intel Xeon D-1736NT host CPU passed through, 3 SR-IOV VFs passed through, q35 machine, SeaBIOS, VirtIO SCSI single controller, local ZFS volume.
Before I took this screenshot, it was freezing after the virtual IDE CD/DVD Drive was detected with no media, so I removed it from the hardware spec. Now it simply freezes at the preceding line.
Sorry I don't yet have time to start trouble shooting, I had to restore the snapshot to the previous 2.7.2, I just wanted to put it out there.
-
Supermicro just released new IPMI Firmware and a BIOS update for my X12SDV-4C-SPT4F motherboard. This seems like a good opportunity to see if that had any impact on the CE 2.8.0 upgrade and possibly the Intel firmware update package it has.
Sadly the CE 2.8.0 upgrade still did not succeed with the new IMPI and BIOS updates.
So, I tested a number of boot loader methods I see listed above.
set console="comconsole,efi" set console="efi" boot -vUnfortunately, none of those seemed to have an impact either.
I did try all console methods again just to make sure the boot process did not end up somewhere else. That also did not make a difference. Stuck loading at the same spot.
Not sure what else to do at this point. Back on the working CE 2.7.2.
-
Did you try option #6 to try booting from the previous kernel?
I just did that, and it booted the 2.7.2 kernel again, and at the very end, it said it was going to reboot to upgrade to 2.8. And miraculously it worked!
-
Judging by the number of views on this topic, it seems likely that a LOT of people are quietly having similar issues. Just adding my +1 here.
I've got a handful of Netgate branded appliances in operation as well as 20+ running as QEMU/libvirt virtual machines under Ubuntu Server 22.04 and 24.04 host servers. It's not Proxmox, but close enough. I've got similar symptoms, always stopping at "start_init: trying /sbin/init". Most of my VMs are High Availability, so with HA and VM snapshots, I've been able to make numerous attempts at upgrading the secondary firewalls without affecting normal network operation and can still revert back to a working version at the end of the day when none of the suggestions I've tried has been successful. (I definitely recommend a HA setup! It's not that hard to do and takes a lot of stress away.)
My pfSense VMs have always been BIOS and not EFI. The pfSense instance I've been testing on was a clean install using the new installer ISO, but installing the previous v2.7.2 (wasn't originally planning to upgrade to 2.8 at the moment, but eventually did...try.) The install was clean, but I did import an existing config afterward I reached GUI. As v2.7.2, it works fine.
I haven't yet tried to boot from the previous (2.7.2) kernel as @eracerxrs just mentioned. But I think I recall someone NOT having good luck with that earlier in the thread. Worth a try, though. What I also plan to try next is a clean install right to v2.8 and then import the existing config.
I've been using pfSense since the Electric Sheep Fencing days. This appears to be the worst widespread upgrade fail I've seen with pfSense. Kind of a bummer after so many years of being rock solid. But in the same vein and that track record, I'm inclined to give Netgate a little bit of a free pass here as long as a reasonable solution can be found in a reasonable amount of time!
-
The EFI console issue was known but was previously thought unrelated to the stalling at 'init'.
Are you also seeing both issues?
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
The EFI console issue was known but was previously thought unrelated to the stalling at 'init'.
Are you also seeing both issues?
If you're asking me, the screenshot above is showing the serial console (SOL) and the video console at the same time from the same boot session.
-
Well, same here.
After the update, it stops here:
Version 2.7.2, it starts normally.
-
Well I was asking anyone who's hitting it really. Ok, that's interesting.
So for anyone seeing this it's good to boot verbose to confirm it stalls at init on the serial console.
Also try booting verbose with the serial console as primary. It could still be waiting for something at the video console that you just aren't seeing.
-
@stephenw10 For what its worth, i see the same behavior under 2.7.2 with the Serial Console not displaying anything beyond the EFI information. even with it being the primary.
But 2.7.2 still reliably boots just fine.
-
Ooo that's a good data point. That screenshot seems confusing though. I don't expect to see that efi console output on a serial console at all. Do you have some bios level console redirect enabled?
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Ooo that's a good data point. That screenshot seems confusing though. I don't expect to see that efi console output on a serial console at all. Do you have some bios level console redirect enabled?
Yes, the motherboard is using SOL (Serial over LAN) to do the console redirect via the IPMI.
-
Right but that should be all in the IPMI. Separately thought the BIOS itself may have an option to redirect output to a serial console that can allow video only code to work on serial consoles. Hard to see what else would cause that EFI framebuffer output to show on a serail console.
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Right but that should be all in the IPMI. Separately thought the BIOS itself may have an option to redirect output to a serial console that can allow video only code to work on serial consoles. Hard to see what else would cause that EFI framebuffer output to show on a serail console.
By default, the COM1 redirection is not enabled on my MB, I did just enabled it though.
But I don't have a serial cable nor USB-Serial device to connect to the physical serial port anyway. I got rid of that stuff a decade ago. -
@stephenw10 so I've been looking for an excuse to switch teams from pf to opn but, it had the same issue as well.
I tried escaping to loader and setting the hints and running boot on both OSes and it made no difference.
I did re-enable the serial ports in BIOS and it seems to have fixed it. I don't know why that should matter. I usually prefer to disable ports that are unused and those ports have been disabled for years, through many updates without issues.
-
Hmm, that's on real hardware? We did see that some time ago in some VMs.
-
@stephenw10 yes real hardware, bare metal on a Dell R510
