pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck in Stage 2
-
@InstanceExtension I wonder if you could get the upgrade to complete if you wiped 2.7.2 first, reinstalled, performed upgrade prior to restoring config, etc.—just to see if the hardware will complete the 2.8 upgrade.
-
@tinfoilmatt said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
@InstanceExtension I wonder if you could get the upgrade to complete if you wiped 2.7.2 first, reinstalled, performed upgrade prior to restoring config, etc.—just to see if the hardware will complete the 2.8 upgrade.
Not possible. There is no 2.8 ISO and without the router configured no internet.
-
@InstanceExtension That's why I'm saying to wipe 2.7.2 first to remove your config. 'Factory reset' essentially just to see if you can get the update to complete the only way currently supported (i.e., upgrade from a 2.7.2 instance) on your hardware.
It will connect to the internet during the 2.7.2 install, minimal config (without restoring your working 2.7.2 config), and then attempt 2.8 update from a 'fresh' 2.7.2 install.
-
@InstanceExtension You don't lose anything trying that. You can always restore working 2.7.2 config to fresh 2.7.2 install.
-
I'm not sure how helpful this is, but I have been running pfSense Plus 24.11 on a Supermicro SYS-510D-4C-FN6P mini 1U server without any issues, which is also based on FreeBSD 15 and has a similar hardware setup as what the OP is using. This raises the question then, what is different between pfSense CE 2.8.0 and pfSense Plus 24.11? @stephenw10 - any thoughts from your side?
I also looked at my OS boot log and it looks very similar to what is in the first post screenshot. The next entry after
Dual Console: Video Primary, Serial SecondaryI see is theCPU:followed by the CPU string. Just to throw another idea out there, could there be any issues related to the CPU microcode update? -
@tman222 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
could there be any issues related to the CPU microcode update?
It's possible I guess but I doubt it.
@InstanceExtension Are you only installing via the BMC there? Have you tried a real console connection on the hardware directly?
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
@InstanceExtension Are you only installing via the BMC there? Have you tried a real console connection on the hardware directly?
Yes, as I said above to another member:
No other peripherals connected. Not even a keyboard, mouse or monitor although I did have them present to test and it made no difference.
-
Yes I read that but it was unclear to me if you had just connected them as a test or actually used a direct local console.
Does it really appear identically on local consoles and both video and serial console via the bmc?
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Does it really appear identically on local consoles and both video and serial console via the bmc?
Yes, identical. During my next test (going to attempt a very simple config on a fresh 2.7.2 install then upgrade to 2.8) I'll double check from an actual monitor and provide a pic.
The thing is my pfSense setup is really pretty basic already. Two interfaces, some aliases, NAT rules, WAN/LAN rules, NTP, UPnP and WOL services and that's about it. No DHCP, no DNS, no VPN, no Traffic shapers, No load balancing, No RAM disk, System Tunables are all default, nothing really elaborate at all. I am limiting the system to IPv4 though and I did turn off SSH access.
-
@InstanceExtension we upgrade a X11SDV-4C-TP8F today without an issue.
Maybe you find something in the specs what could be different?Update was only via BMC, nothing else connected.
-
@slu said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
@InstanceExtension we upgrade a X11SDV-4C-TP8F today without an issue.
Maybe you find something in the specs what could be different?Update was only via BMC, nothing else connected.
Thanks for the feedback.
-
Mmm, I would try setting the serial console as primary and making sure that works as expected in 2.7.2 when you access it via the BMC.
You could be seeing some issue with an updated driver in 2.8 that affects the video console.
-
I had a similar problem after upgrading to pfSense 2.8.0.
In my case, the solution was to switch from UEFI to Legacy BIOS mode.
After that, the system booted normally. -
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Mmm, I would try setting the serial console as primary and making sure that works as expected in 2.7.2 when you access it via the BMC.
You could be seeing some issue with an updated driver in 2.8 that affects the video console.
I really wish I could have a 2.8 ISO, that would make this entire process so much simpler to troubleshoot.
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
@tman222 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
could there be any issues related to the CPU microcode update?
It's possible I guess but I doubt it.
I'm starting to think that there is a very good chance that this is related to the updated CPU microcode in 2.8. This is what the 2.7.2 logs look like with verbose logging turned on.
Jun 3 17:01:12 kernel Origin="GenuineIntel" Id=0x606c1 Family=0x6 Model=0x6c Stepping=1 Jun 3 17:01:12 kernel CPU: Intel(R) Xeon(R) D-1718T CPU @ 2.60GHz (2600.00-MHz K8-class CPU) Jun 3 17:01:12 kernel No TSX change made Jun 3 17:01:12 kernel cpuctl: access to MSR registers/cpuid info. Jun 3 17:01:12 kernel start_init: trying /sbin/init Jun 3 17:01:12 kernel Dual Console: Video Primary, Serial SecondaryThe next logged action after "start_init: trying /sbin/init" is the "cpuctl" function.
Per the man page:
The special device /dev/cpuctl presents interface to the system CPU. It provides functionality to retrieve CPUID information, read/write machine specific registers (MSR) and perform CPU firmware updates.With CE 2.8 the system is stuck right after
start_init: trying /sbin/initprobably trying to do the "cpuctl" function and failing/stalling.
-
Does it boot to single user mode?
If so you can enable verbose RC there and reboot. That may get you more debug output.
touch /verbose_rc -
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Does it boot to single user mode?
If so you can enable verbose RC there and reboot. That may get you more debug output.
touch /verbose_rcI'll give that a try during the next maintenance window.
-
Not sure if this will help, I also had the same issue where after upgrading to 2.8.0 it would hang. Mine was hanging just after it detected the interfaces, right after you get the Y/N question of if you want to configure the VLANS. I wouldn't touch it, then it auto detects and then that's where I had it hang. I tried all of the above mentioned things, nothing worked. I erased the entire system and tried a fresh install of 2.8.0 and it hung in the exact same place again. After a few more changes and attempts in BIOS and install settings, I finally put everything back to the way it was before the upgrade and did one final install of 2.8.0 from scratch, this time I hit the Y when prompted to configure the VLANS and went through the set up of them. As soon as I did that, viola. The boot process continued on as the way it should. Not sure what is causing it to hang at the autodetect of the interfaces, but that seemed to fix it. Booted up right away, I restored my backup, and everything is working. Sorry for the long winded message. Hope this helps.
-
Hmm, so if you didn't agree to setting up vlans at the console immediately it stopped responding?
I've not seen that before.
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Does it boot to single user mode?
If so you can enable verbose RC there and reboot. That may get you more debug output.
touch /verbose_rcDid some more testing today.
-
Re-Ran the CE 2.8 upgrade from a working 2.7.2 - Stalled as usual
-
Used single user mode (w/verbose) with the 2.8 boot loader, stalled at the same exact spot so I could not run the debug output.
-
Since I had to revert back to 2.7.2 anyway decided to do a fresh install and config from 2.7.2 USB media. Just setup the interfaces nothing else, got online and then attempted the CE 2.8 upgrade again. - Stalled as usual in the same spot in the 2.8 load.
-
Reverted back to 2.7.2 and restored my config. Everything is fine again.
I'm 99% confident that this is not a hardware nor configuration issue on my part. Again this leads back to the cpu-microcode-intel-20250211.pkg as a possible culprit.
-
