pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck in Stage 2
-
@stephenw10 Seems that way. How I came about this was. I did the GUI interface update from 2.7.2. First I backed up the system, then removed all packages, then rebooted. Next I did the GUI update. The update appeared to work smoothy. The system rebooted. I then forced pfblocker to do an update. I then got php errors saying I was running out of memory.
--------------------------> snip <--------------------------------
[03-Jun-2025 21:53:54 Canada/Pacific] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 4096 bytes) in /usr/local/bin/kea2unbound on line 524
--------------------------> snip <--------------------------------at this time I wasn't sure what was causing it. (I didn't realize I had to change Unbound to Python mode). I tried a few setting changes, with no luck. (I actually ended up making it so I could not longer use DNS, can't remember what I did to mess that up but I did). I then decided to set the system back to factory defaults, knowing I would be ok as I made my backup. As soon as I did that and the system rebooted, it hung for what felt like forever. (I run the system headless on a Protectli Vault that is in a network enclosure that is evil to get to and take apart). I then took everything apart, removed the Protectli Vault, then wired lan0 to internet and lan1 I connected my laptop with a monitor connected via hdmi and a usb keyboard and mouse. Booted up the system, everything looked normal, it got to the Y/N question about VLANs. I let it time out and autodetect. It found both wan and lan and then hung. Waited about 8 mins or so, finally I shut it down and rebooted. I got the exact same thing. Having no internet connection at this time I used my phone as a hotspot, downloaded the newest installer from the netgate site, created a new usb stick and proceeded to reinstall the entire system. Through the new installer (which is much nicer by the way) I set up the WAN and LAN settings via the install wizard. Everything installed as prescribed, and then it was time to reboot after the install. During the first boot it came to the Y/N question of setting up the VLAN's. Knowing I just did this in the wizard, I had no need to bother, I didn't choose either, just let it auto choose/continue on it's own, it auto found both WAN and LAN, then hung in the exact same spot again. This is when using a fresh install of 2.8.0. At this point I'm not sure what to do, I fired up the hotspot again, started looking on the forums and found this thread. I went to the bios, changed console port to off, no change, put it back on, I reinstalled again but in the wizard, I changed the console port setting under advanced settings to video, still no change, I put everything back in the bios and again reinstalled. No change. I then installed 2.7.2 again, then upgraded to 2.8.0, and got the exact same results. I'm thinking, there is someone out there much smarter than me that has this working and would not release this unless it was good to go. Grabbed a coffee, cause you know that helps.(Nod to Network Chuck). Reinstalled 2.8.0 again with all settings back to where I began, but this time when I was prompted to set up VLAN's during the reboot even though I had already set them up in the wizard I hit Y. As soon as I configured both WAN and LAN during the boot up, everything moved along they way it was suppose to be. I had my clean install of 2.8.0. I then rebooted numerous times to make sure that everything was working via Diagnostics>reboot it seems my hang issue is now gone. Confident with the new version now working I restored my backup and crossed my fingers hoping I didn't just break my fresh install as I didn't want to have to reset up everything if I didn't have to. After the restore, everything appeared ok (Whew). I had the system reboot again with all restored settings intact and braced for the outcome. Everything booted with no issues what so ever. I then forced the update to pfblocker to get the DNSBL running again, which gave me the memory errors again that started me on this journey. After changing Unbound to Python mode in both pfblocker and dns resolver, that memory issue has now disappeared.
Again sorry for the long story but I wanted to document all the steps on how to reproduce this issue. It looks to me like an issue with the wizard during the install not correctly writing the information to whichever config file needs it for the VLAN data. I don't think 2.8.0 is the problem here, think it's the installer being RC1.0 being the issue.
-
@InstanceExtension Interesting troubleshoot. Gotta be someone else around here with a Xeon D.
-
@InstanceExtension said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Again this leads back to the cpu-microcode-intel-20250211.pkg as a possible culprit.
As far as I know it doesn't do that in single user mode. It doesn't run anything from pfSense_rc.
It really appears like it's just not the primary console. For example when booting into single user mode I see this at the serial console:
pass0: <QEMU HARDDISK 2.5+> ATA-7 SATA device pass0: Serial Number QM00013 pass0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) pass0: Command Queueing enabled pass1 at ahcich1 bus 0 scbus1 target 0 lun 0 pass1: <QEMU HARDDISK 2.5+> ATA-7 SATA device pass1: Serial Number QM00015 pass1: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) pass1: Command Queueing enabled pass2 at ahcich7 bus 0 scbus7 target 0 lun 0 pass2: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device pass2: Serial Number QM00003 pass2: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) GEOM: new disk cd0 GEOM: new disk ada0 GEOM: new disk ada1 kvmclock0: providing initial system time Dual Console: Video Primary, Serial Secondary start_init: trying /sbin/initAnd this at the video console:
Can you get the full boot log from the serial console? Did you try setting the serial console as primary?
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
It really appears like it's just not the primary console. For example when booting into single user mode I see this at the serial console:
But as you've already stated even if that was the issue, then the web interface would still come up. Its just stuck, nothing short of a reset does anything. Not CNTRL-T, not CNTRL-ALT-DEL, nothing.
I've set the CE 2.8 boot loader to:
- Dual with video as primary and serial as secondary
- Dual with serial as primary and video as secondary
- Video only
- Serial only
None of those has made any difference, same outcome even with a monitor attached.
-
Just to be clear you are viewing the serial console output via the BMC device?
If if were a primary console issue it could be waiting at some other console for some user input that never comes. I'm not sure what might be though!
-
I ran the update last night before going to bed and woke up to mine hung also. I did manage to get things to appear to be moving now by changing the kernel to kernel.old during the boot up and just hitting enter.
Edit: After a couple reboots and remembering to change it to kernel.old each time, the machine is up but, I've got no connection to the Internet. Says the link is up and I've got an IP address but no connectivity. It did throw a few errors, I'll have to sort through those later as I did have a few before upgrading also.
-
What hardware are you running?
-
@stephenw10 Dell R510 bare metal
-
And you are also seeing the same thing on both consoles?
-
I get the same behavior upgrading pfSense 2.7.2 to 2.8 on Hyper-V on Windows 2022. After upgrade it gets stuck after displaying the disk info. And it is hung. No Ctrl-Anything, no Alt-Ctrl-Del.
Same thing in single user mode.
I did two installs:
One as Gen-2 VM and one as Gen-1. 4G RAM and fixed disk. I restored my config and tried the upgrade. Same result in both. CPU is Xeon Gold 6326 in case it matters. Even tried 1 core setup. -
HP DL360 Gen9, 2.8 upgrade no good in UEFI mode. Tried upgrades (2.7.2) and fresh installs, all matter of EFI BIOS switches, BIOS console settings, clearing EFI boot entries, 'boot'ing from the boot menu CLI, even fresh SSDs. Hard auto-reboot at the EFI framebuffer section after BSD boot selector screen each time. Finally switched to Legacy Mode and it worked. This is a FreeBSD issue, not Netgate's. Come on, BSD, it's 2025! Legacy mode? At least my NICs were still recognized. Other than that it's a bog standard DL360 with current firmware. Probably this issue: https://redmine.pfsense.org/issues/15074
Did several VM based installs at remote sites no problems. This was the first of the bare metal ones.
p.s. And not having an real install .iso is getting annoying. I could see where recovering from a cold start could be challenging in some environments.
-
@dropkick said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Probably this issue: https://redmine.pfsense.org/issues/15074
Hmm, could be related but that also affected 2.7.2.
-
@stephenw10 after having some time to actually look at mine, it seems to be hanging at the line
isa0: <ISA bus> on isab0
-
@stephenw10
2.7.2 runs fine. Installer for 2.8 runs fine. Dead comes at first boot. I will try the serial thing but it would be weird. -
@oldhome7 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
it seems to be hanging at the line
isa0: <ISA bus> on isab0
It's possible you're hitting this:
https://docs.netgate.com/pfsense/en/latest/releases/2-8-0.html#legacy-serial-consoleI didn't think anything with a UEFI BIOS was affected but maybe there are some edge cases. That is what it looks like on an affected system.
-
Dead after upgrading to 2.8.0 from a fresh pfSense-CE-memstick-serial-2.7.2-RELEASE-amd64.img installed on pcengines APU5 board.
During booting there where hundreds of certificate missing and packets missing.
Log is attached.
-
That's a completely different problem.
But nothing shown there is an error. There are no man pages included in pfSense which cause the missing file messages. The certctl output looks completely normal, it skips private CA certs.
The cert part can take some time, a few minutes on an APU, but it looks like you hit Ctl-C and aborted the upgrade:
^C3510I would have expected that to complete correctly if you just wait.
-
@stephenw10 I tried the loader prompt with:
set hint.uart.0.at="isa"
And then running autoboot command.
But, that made no difference. Though I'm not entirely convinced I'm not dumb at troubleshooting unless I can copy and paste stuff so I probably need to change something in that I'm assuming.
-
Mmm, it seemed unlikely to apply to something that new but worth trying.
However the behaviour is so similar it has to be some console problem. Can you see how the uart devices are shown in the log?
-
@stephenw10 it's not "new", it's from 2012. I do have it set for legacy BIOS instead of UEFI.
Which logs should I be looking at for uart?
