pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading
-
Well, same here.
After the update, it stops here:
Version 2.7.2, it starts normally.
-
Well I was asking anyone who's hitting it really. Ok, that's interesting.
So for anyone seeing this it's good to boot verbose to confirm it stalls at init on the serial console.
Also try booting verbose with the serial console as primary. It could still be waiting for something at the video console that you just aren't seeing.
-
@stephenw10 For what its worth, i see the same behavior under 2.7.2 with the Serial Console not displaying anything beyond the EFI information. even with it being the primary.
But 2.7.2 still reliably boots just fine.
-
Ooo that's a good data point. That screenshot seems confusing though. I don't expect to see that efi console output on a serial console at all. Do you have some bios level console redirect enabled?
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Ooo that's a good data point. That screenshot seems confusing though. I don't expect to see that efi console output on a serial console at all. Do you have some bios level console redirect enabled?
Yes, the motherboard is using SOL (Serial over LAN) to do the console redirect via the IPMI.
-
Right but that should be all in the IPMI. Separately thought the BIOS itself may have an option to redirect output to a serial console that can allow video only code to work on serial consoles. Hard to see what else would cause that EFI framebuffer output to show on a serail console.
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Right but that should be all in the IPMI. Separately thought the BIOS itself may have an option to redirect output to a serial console that can allow video only code to work on serial consoles. Hard to see what else would cause that EFI framebuffer output to show on a serail console.
By default, the COM1 redirection is not enabled on my MB, I did just enabled it though.
But I don't have a serial cable nor USB-Serial device to connect to the physical serial port anyway. I got rid of that stuff a decade ago. -
@stephenw10 so I've been looking for an excuse to switch teams from pf to opn but, it had the same issue as well.
I tried escaping to loader and setting the hints and running boot on both OSes and it made no difference.
I did re-enable the serial ports in BIOS and it seems to have fixed it. I don't know why that should matter. I usually prefer to disable ports that are unused and those ports have been disabled for years, through many updates without issues.
-
Hmm, that's on real hardware? We did see that some time ago in some VMs.
-
@stephenw10 yes real hardware, bare metal on a Dell R510
-
@InstanceExtension said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
By default, the COM1 redirection is not enabled
FWIW, I recently factory reset the AMI BIOS on an older ASUS server/workstation-class MB (~2013, Haswell), and I was distinctly surprised to find all COM redirection options enabled by-default as I combed back through the settings. Same appearance, same options as your screencap.
But did that do it?! Might we finally have a resolution to this long and winding thread? Hard to tell at this point if something is actually going on here, or if it's a bunch of unrelated BIOS configuration options all loosely having to do with console redirection/display output settings.
@InstanceExtension said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
But I don't have a serial cable nor USB-Serial device to connect to the physical serial port anyway. I got rid of that stuff a decade ago.
What about a VGA cable?
-
@eracerxrs said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Did you try option #6 to try booting from the previous kernel?
I just did that, and it booted the 2.7.2 kernel again, and at the very end, it said it was going to reboot to upgrade to 2.8. And miraculously it worked!
Well I'll be damned. That worked for me too.
There must be something about having the the background upgrade process (renewing certs, etc) run under the FreeBSD 14 kernel (CE 2.7.2) that allows it to work successfully and fails miserably under FreeBSD 15 (CE 2.8.0).
Once the background upgrade process is complete it boots just fine under FreeBSD 15 (CE 2.8.0).
-
Hmm, that's very interesting. That feels like some driver mismatch perhaps. Something that hasn't been updated yet when upgrade moves to stage 2. I'd expect to see some errors though if it tried to load that. Hmm.
-
@eracerxrs, this worked for me too. Thanks!
-
Glad it worked out for you as well!
The last thing I saw in the Gui Terminal update process was:
Updating boot code... /usr/local/sbin/../libexec/install-boot.sh -b auto -f zfs -s gpt -u da0 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0 partcode written to da0p1 bootcode written to da0 No ESP partition found...skipping. Done. System is going to be upgraded. Rebooting in 10 seconds. SuccessMy FreeBSD experience is limited to when tinkering with pfSense, but whenever I experience an issue after an upgrade or systemd/grub modification on a Linux box, booting into the old kernel and then manually fixing the boot code is usually required.
Not sure if this was actually the problem in this case because it was so fast and thankfully auto-healed!
-
Hmm, if it was a bootloader problem I'd expect to see it fail to load the kernel. And it wouldn't effect clean installs to 2.8. But it could be a clue...
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Hmm, if it was a bootloader problem I'd expect to see it fail to load the kernel. And it wouldn't effect clean installs to 2.8. But it could be a clue...
What clean 2.8 installs? We don't have an ISO which is just ludicrous.
-
From the Net Installer which is also built on FreeBSD 15. So unlikely a bootloader issue.
-
Yes, I'm sure that would be the case for a fully corrupt bootloader, but I've encountered issues where it partially loads, but hangs where there are outdated or mismatched driver versions in the bootloader that required updates to initramfs with the expected versions.
To investigate a bit further:
Previously it would freeze after configuring the harddisk da0. Last line being:
da0: 16384MB (33554432 512 byte sectors)Now that is successfully boots, here are the lines between that and the "pfSense" ascii graphic:
Configuring crash dumps... Using /dev/da0p2 for dump device. Filesystems are clean, continuing... Mounting filesystems... Mounting ZFS boot environment... done.For kicks, I reloaded the 2.7.2 snapshot and redid the upgrade, again, it froze, so when I used the old kernel to reboot I scoured the output and this is the only thing that stood out soon after booting:
>>> Removing vital flag from php82...done. >>> Locking package pfSense-pkg-System_Patches...done. >>> Reinstalling pkg due to ABI change... Checking integrity... done (0 conflicting) The following 1 package(s) will be affected (of 0 checked) Installed packages to be REINSTALLED: pkg-1.21.3_4 [pfSense] Number of packages to be reinstalled: 1 [1/1] Reinstalling pkg-1.21.3_4... [1/1] Extracting pkg-1.21.3_4: .......... doneAnd it continued installing similar to before, but this time ending in success upon reboot.
-
If you're able to test further try setting
boot_verbose=YESin /boot/loader.conf.local.That should show you where it's actually hanging or not during the boot at each stage. In the normal boot you don't see any output after it mounts root until init starts.
