pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading

tinfoilmatt

@InstanceExtension Interesting troubleshoot. Gotta be someone else around here with a Xeon D.

stephenw10

@InstanceExtension said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:

Again this leads back to the cpu-microcode-intel-20250211.pkg as a possible culprit.

As far as I know it doesn't do that in single user mode. It doesn't run anything from pfSense_rc.

It really appears like it's just not the primary console. For example when booting into single user mode I see this at the serial console:

pass0: <QEMU HARDDISK 2.5+> ATA-7 SATA device
pass0: Serial Number QM00013
pass0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)
pass0: Command Queueing enabled
pass1 at ahcich1 bus 0 scbus1 target 0 lun 0
pass1: <QEMU HARDDISK 2.5+> ATA-7 SATA device
pass1: Serial Number QM00015
pass1: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)
pass1: Command Queueing enabled
pass2 at ahcich7 bus 0 scbus7 target 0 lun 0
pass2: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device
pass2: Serial Number QM00003
pass2: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
GEOM: new disk cd0
GEOM: new disk ada0
GEOM: new disk ada1
kvmclock0: providing initial system time
Dual Console: Video Primary, Serial Secondary
start_init: trying /sbin/init

And this at the video console:
Screenshot from 2025-06-04 17-36-23.png

Can you get the full boot log from the serial console? Did you try setting the serial console as primary?

InstanceExtension

@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:

It really appears like it's just not the primary console. For example when booting into single user mode I see this at the serial console:

But as you've already stated even if that was the issue, then the web interface would still come up. Its just stuck, nothing short of a reset does anything. Not CNTRL-T, not CNTRL-ALT-DEL, nothing.

I've set the CE 2.8 boot loader to:

Dual with video as primary and serial as secondary
Dual with serial as primary and video as secondary
Video only
Serial only

None of those has made any difference, same outcome even with a monitor attached.

stephenw10

Just to be clear you are viewing the serial console output via the BMC device?

If if were a primary console issue it could be waiting at some other console for some user input that never comes. I'm not sure what might be though!

oldhome7

I ran the update last night before going to bed and woke up to mine hung also. I did manage to get things to appear to be moving now by changing the kernel to kernel.old during the boot up and just hitting enter.

Edit: After a couple reboots and remembering to change it to kernel.old each time, the machine is up but, I've got no connection to the Internet. Says the link is up and I've got an IP address but no connectivity. It did throw a few errors, I'll have to sort through those later as I did have a few before upgrading also.

stephenw10

What hardware are you running?

oldhome7

@stephenw10 Dell R510 bare metal

stephenw10

And you are also seeing the same thing on both consoles?

drchanas

I get the same behavior upgrading pfSense 2.7.2 to 2.8 on Hyper-V on Windows 2022. After upgrade it gets stuck after displaying the disk info. And it is hung. No Ctrl-Anything, no Alt-Ctrl-Del.
Same thing in single user mode.
I did two installs:
One as Gen-2 VM and one as Gen-1. 4G RAM and fixed disk. I restored my config and tried the upgrade. Same result in both. CPU is Xeon Gold 6326 in case it matters. Even tried 1 core setup.

dropkick

HP DL360 Gen9, 2.8 upgrade no good in UEFI mode. Tried upgrades (2.7.2) and fresh installs, all matter of EFI BIOS switches, BIOS console settings, clearing EFI boot entries, 'boot'ing from the boot menu CLI, even fresh SSDs. Hard auto-reboot at the EFI framebuffer section after BSD boot selector screen each time. Finally switched to Legacy Mode and it worked. This is a FreeBSD issue, not Netgate's. Come on, BSD, it's 2025! Legacy mode? At least my NICs were still recognized. Other than that it's a bog standard DL360 with current firmware. Probably this issue: https://redmine.pfsense.org/issues/15074

Did several VM based installs at remote sites no problems. This was the first of the bare metal ones.

p.s. And not having an real install .iso is getting annoying. I could see where recovering from a cold start could be challenging in some environments.

stephenw10

@dropkick said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:

Probably this issue: https://redmine.pfsense.org/issues/15074

Hmm, could be related but that also affected 2.7.2.

oldhome7

@stephenw10 after having some time to actually look at mine, it seems to be hanging at the line

isa0: <ISA bus> on isab0

drchanas

@stephenw10
2.7.2 runs fine. Installer for 2.8 runs fine. Dead comes at first boot. I will try the serial thing but it would be weird.

stephenw10

@oldhome7 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:

it seems to be hanging at the line

isa0: <ISA bus> on isab0

It's possible you're hitting this:
https://docs.netgate.com/pfsense/en/latest/releases/2-8-0.html#legacy-serial-console

I didn't think anything with a UEFI BIOS was affected but maybe there are some edge cases. That is what it looks like on an affected system.

bnshe

Dead after upgrading to 2.8.0 from a fresh pfSense-CE-memstick-serial-2.7.2-RELEASE-amd64.img installed on pcengines APU5 board.

During booting there where hundreds of certificate missing and packets missing.

Log is attached.

pfsense_install_2_7_2_upgrade_to_2_8_0_console_log.txt

stephenw10

That's a completely different problem.

But nothing shown there is an error. There are no man pages included in pfSense which cause the missing file messages. The certctl output looks completely normal, it skips private CA certs.

The cert part can take some time, a few minutes on an APU, but it looks like you hit Ctl-C and aborted the upgrade: ^C3510

I would have expected that to complete correctly if you just wait.

oldhome7

@stephenw10 I tried the loader prompt with:

set hint.uart.0.at="isa"

And then running autoboot command.

But, that made no difference. Though I'm not entirely convinced I'm not dumb at troubleshooting unless I can copy and paste stuff so I probably need to change something in that I'm assuming.

stephenw10

Mmm, it seemed unlikely to apply to something that new but worth trying.

However the behaviour is so similar it has to be some console problem. Can you see how the uart devices are shown in the log?

oldhome7

@stephenw10 it's not "new", it's from 2012. I do have it set for legacy BIOS instead of UEFI.

Which logs should I be looking at for uart?

stephenw10

At the boot log. So on an affected device like the APU1:

uart0: <16550 or compatible> at port 0x3f8 irq 4 flags 0x10 on isa0
uart0: console (115200,n,8,1)