pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
@InstanceExtension Are you only installing via the BMC there? Have you tried a real console connection on the hardware directly?
Yes, as I said above to another member:
No other peripherals connected. Not even a keyboard, mouse or monitor although I did have them present to test and it made no difference.
-
Yes I read that but it was unclear to me if you had just connected them as a test or actually used a direct local console.
Does it really appear identically on local consoles and both video and serial console via the bmc?
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Does it really appear identically on local consoles and both video and serial console via the bmc?
Yes, identical. During my next test (going to attempt a very simple config on a fresh 2.7.2 install then upgrade to 2.8) I'll double check from an actual monitor and provide a pic.
The thing is my pfSense setup is really pretty basic already. Two interfaces, some aliases, NAT rules, WAN/LAN rules, NTP, UPnP and WOL services and that's about it. No DHCP, no DNS, no VPN, no Traffic shapers, No load balancing, No RAM disk, System Tunables are all default, nothing really elaborate at all. I am limiting the system to IPv4 though and I did turn off SSH access.
-
@InstanceExtension we upgrade a X11SDV-4C-TP8F today without an issue.
Maybe you find something in the specs what could be different?Update was only via BMC, nothing else connected.
-
@slu said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
@InstanceExtension we upgrade a X11SDV-4C-TP8F today without an issue.
Maybe you find something in the specs what could be different?Update was only via BMC, nothing else connected.
Thanks for the feedback.
-
Mmm, I would try setting the serial console as primary and making sure that works as expected in 2.7.2 when you access it via the BMC.
You could be seeing some issue with an updated driver in 2.8 that affects the video console.
-
I had a similar problem after upgrading to pfSense 2.8.0.
In my case, the solution was to switch from UEFI to Legacy BIOS mode.
After that, the system booted normally. -
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Mmm, I would try setting the serial console as primary and making sure that works as expected in 2.7.2 when you access it via the BMC.
You could be seeing some issue with an updated driver in 2.8 that affects the video console.
I really wish I could have a 2.8 ISO, that would make this entire process so much simpler to troubleshoot.
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
@tman222 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
could there be any issues related to the CPU microcode update?
It's possible I guess but I doubt it.
I'm starting to think that there is a very good chance that this is related to the updated CPU microcode in 2.8. This is what the 2.7.2 logs look like with verbose logging turned on.
Jun 3 17:01:12 kernel Origin="GenuineIntel" Id=0x606c1 Family=0x6 Model=0x6c Stepping=1 Jun 3 17:01:12 kernel CPU: Intel(R) Xeon(R) D-1718T CPU @ 2.60GHz (2600.00-MHz K8-class CPU) Jun 3 17:01:12 kernel No TSX change made Jun 3 17:01:12 kernel cpuctl: access to MSR registers/cpuid info. Jun 3 17:01:12 kernel start_init: trying /sbin/init Jun 3 17:01:12 kernel Dual Console: Video Primary, Serial SecondaryThe next logged action after "start_init: trying /sbin/init" is the "cpuctl" function.
Per the man page:
The special device /dev/cpuctl presents interface to the system CPU. It provides functionality to retrieve CPUID information, read/write machine specific registers (MSR) and perform CPU firmware updates.With CE 2.8 the system is stuck right after
start_init: trying /sbin/initprobably trying to do the "cpuctl" function and failing/stalling.
-
Does it boot to single user mode?
If so you can enable verbose RC there and reboot. That may get you more debug output.
touch /verbose_rc -
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Does it boot to single user mode?
If so you can enable verbose RC there and reboot. That may get you more debug output.
touch /verbose_rcI'll give that a try during the next maintenance window.
-
Not sure if this will help, I also had the same issue where after upgrading to 2.8.0 it would hang. Mine was hanging just after it detected the interfaces, right after you get the Y/N question of if you want to configure the VLANS. I wouldn't touch it, then it auto detects and then that's where I had it hang. I tried all of the above mentioned things, nothing worked. I erased the entire system and tried a fresh install of 2.8.0 and it hung in the exact same place again. After a few more changes and attempts in BIOS and install settings, I finally put everything back to the way it was before the upgrade and did one final install of 2.8.0 from scratch, this time I hit the Y when prompted to configure the VLANS and went through the set up of them. As soon as I did that, viola. The boot process continued on as the way it should. Not sure what is causing it to hang at the autodetect of the interfaces, but that seemed to fix it. Booted up right away, I restored my backup, and everything is working. Sorry for the long winded message. Hope this helps.
-
Hmm, so if you didn't agree to setting up vlans at the console immediately it stopped responding?
I've not seen that before.
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Does it boot to single user mode?
If so you can enable verbose RC there and reboot. That may get you more debug output.
touch /verbose_rcDid some more testing today.
-
Re-Ran the CE 2.8 upgrade from a working 2.7.2 - Stalled as usual
-
Used single user mode (w/verbose) with the 2.8 boot loader, stalled at the same exact spot so I could not run the debug output.
-
Since I had to revert back to 2.7.2 anyway decided to do a fresh install and config from 2.7.2 USB media. Just setup the interfaces nothing else, got online and then attempted the CE 2.8 upgrade again. - Stalled as usual in the same spot in the 2.8 load.
-
Reverted back to 2.7.2 and restored my config. Everything is fine again.
I'm 99% confident that this is not a hardware nor configuration issue on my part. Again this leads back to the cpu-microcode-intel-20250211.pkg as a possible culprit.
-
-
@stephenw10 Seems that way. How I came about this was. I did the GUI interface update from 2.7.2. First I backed up the system, then removed all packages, then rebooted. Next I did the GUI update. The update appeared to work smoothy. The system rebooted. I then forced pfblocker to do an update. I then got php errors saying I was running out of memory.
--------------------------> snip <--------------------------------
[03-Jun-2025 21:53:54 Canada/Pacific] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 4096 bytes) in /usr/local/bin/kea2unbound on line 524
--------------------------> snip <--------------------------------at this time I wasn't sure what was causing it. (I didn't realize I had to change Unbound to Python mode). I tried a few setting changes, with no luck. (I actually ended up making it so I could not longer use DNS, can't remember what I did to mess that up but I did). I then decided to set the system back to factory defaults, knowing I would be ok as I made my backup. As soon as I did that and the system rebooted, it hung for what felt like forever. (I run the system headless on a Protectli Vault that is in a network enclosure that is evil to get to and take apart). I then took everything apart, removed the Protectli Vault, then wired lan0 to internet and lan1 I connected my laptop with a monitor connected via hdmi and a usb keyboard and mouse. Booted up the system, everything looked normal, it got to the Y/N question about VLANs. I let it time out and autodetect. It found both wan and lan and then hung. Waited about 8 mins or so, finally I shut it down and rebooted. I got the exact same thing. Having no internet connection at this time I used my phone as a hotspot, downloaded the newest installer from the netgate site, created a new usb stick and proceeded to reinstall the entire system. Through the new installer (which is much nicer by the way) I set up the WAN and LAN settings via the install wizard. Everything installed as prescribed, and then it was time to reboot after the install. During the first boot it came to the Y/N question of setting up the VLAN's. Knowing I just did this in the wizard, I had no need to bother, I didn't choose either, just let it auto choose/continue on it's own, it auto found both WAN and LAN, then hung in the exact same spot again. This is when using a fresh install of 2.8.0. At this point I'm not sure what to do, I fired up the hotspot again, started looking on the forums and found this thread. I went to the bios, changed console port to off, no change, put it back on, I reinstalled again but in the wizard, I changed the console port setting under advanced settings to video, still no change, I put everything back in the bios and again reinstalled. No change. I then installed 2.7.2 again, then upgraded to 2.8.0, and got the exact same results. I'm thinking, there is someone out there much smarter than me that has this working and would not release this unless it was good to go. Grabbed a coffee, cause you know that helps.(Nod to Network Chuck). Reinstalled 2.8.0 again with all settings back to where I began, but this time when I was prompted to set up VLAN's during the reboot even though I had already set them up in the wizard I hit Y. As soon as I configured both WAN and LAN during the boot up, everything moved along they way it was suppose to be. I had my clean install of 2.8.0. I then rebooted numerous times to make sure that everything was working via Diagnostics>reboot it seems my hang issue is now gone. Confident with the new version now working I restored my backup and crossed my fingers hoping I didn't just break my fresh install as I didn't want to have to reset up everything if I didn't have to. After the restore, everything appeared ok (Whew). I had the system reboot again with all restored settings intact and braced for the outcome. Everything booted with no issues what so ever. I then forced the update to pfblocker to get the DNSBL running again, which gave me the memory errors again that started me on this journey. After changing Unbound to Python mode in both pfblocker and dns resolver, that memory issue has now disappeared.
Again sorry for the long story but I wanted to document all the steps on how to reproduce this issue. It looks to me like an issue with the wizard during the install not correctly writing the information to whichever config file needs it for the VLAN data. I don't think 2.8.0 is the problem here, think it's the installer being RC1.0 being the issue.
-
@InstanceExtension Interesting troubleshoot. Gotta be someone else around here with a Xeon D.
-
@InstanceExtension said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Again this leads back to the cpu-microcode-intel-20250211.pkg as a possible culprit.
As far as I know it doesn't do that in single user mode. It doesn't run anything from pfSense_rc.
It really appears like it's just not the primary console. For example when booting into single user mode I see this at the serial console:
pass0: <QEMU HARDDISK 2.5+> ATA-7 SATA device pass0: Serial Number QM00013 pass0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) pass0: Command Queueing enabled pass1 at ahcich1 bus 0 scbus1 target 0 lun 0 pass1: <QEMU HARDDISK 2.5+> ATA-7 SATA device pass1: Serial Number QM00015 pass1: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) pass1: Command Queueing enabled pass2 at ahcich7 bus 0 scbus7 target 0 lun 0 pass2: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device pass2: Serial Number QM00003 pass2: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) GEOM: new disk cd0 GEOM: new disk ada0 GEOM: new disk ada1 kvmclock0: providing initial system time Dual Console: Video Primary, Serial Secondary start_init: trying /sbin/initAnd this at the video console:
Can you get the full boot log from the serial console? Did you try setting the serial console as primary?
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
It really appears like it's just not the primary console. For example when booting into single user mode I see this at the serial console:
But as you've already stated even if that was the issue, then the web interface would still come up. Its just stuck, nothing short of a reset does anything. Not CNTRL-T, not CNTRL-ALT-DEL, nothing.
I've set the CE 2.8 boot loader to:
- Dual with video as primary and serial as secondary
- Dual with serial as primary and video as secondary
- Video only
- Serial only
None of those has made any difference, same outcome even with a monitor attached.
-
Just to be clear you are viewing the serial console output via the BMC device?
If if were a primary console issue it could be waiting at some other console for some user input that never comes. I'm not sure what might be though!
-
I ran the update last night before going to bed and woke up to mine hung also. I did manage to get things to appear to be moving now by changing the kernel to kernel.old during the boot up and just hitting enter.
Edit: After a couple reboots and remembering to change it to kernel.old each time, the machine is up but, I've got no connection to the Internet. Says the link is up and I've got an IP address but no connectivity. It did throw a few errors, I'll have to sort through those later as I did have a few before upgrading also.
