pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading
-
I spent a couple of hours on this today. No resolution yet.
Tried:- Disabled BMC "Host Interface"
- Verbose logging
- Safe Mode
- Disabled Serial ports and console in BIOS
- Letting it sit trying to load for an hour
- Manually set Console to Video and Serial at pfSense boot screen
- Reset BIOS to optimized defaults. I did have a slight change in there to support Intel Speed Shift. Setting that to back to default did not help. Was set to : Hardware P-States: Native Mode (to support Speed Shift)
I just can't get it to load pfSense. This was the console output with Verbose logging turn on and using the Serial console output. (was the same when using just Video console)
I'm not sure what to do next.
-
@InstanceExtension This is a bare metal host, correct? What other storage is present aside from the Samsung NVMe? No other peripherals connected?
-
@tinfoilmatt said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
@InstanceExtension This is a bare metal host, correct? What other storage is present aside from the Samsung NVMe? No other peripherals connected?
Bare metal, correct. No other storage is present during the upgrade.
No other peripherals connected. Not even a keyboard, mouse or monitor although I did have them present to test and it made no difference.
Just the MB itself and its default components.
-
@InstanceExtension I wonder if you could get the upgrade to complete if you wiped 2.7.2 first, reinstalled, performed upgrade prior to restoring config, etc.—just to see if the hardware will complete the 2.8 upgrade.
-
@tinfoilmatt said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
@InstanceExtension I wonder if you could get the upgrade to complete if you wiped 2.7.2 first, reinstalled, performed upgrade prior to restoring config, etc.—just to see if the hardware will complete the 2.8 upgrade.
Not possible. There is no 2.8 ISO and without the router configured no internet.
-
@InstanceExtension That's why I'm saying to wipe 2.7.2 first to remove your config. 'Factory reset' essentially just to see if you can get the update to complete the only way currently supported (i.e., upgrade from a 2.7.2 instance) on your hardware.
It will connect to the internet during the 2.7.2 install, minimal config (without restoring your working 2.7.2 config), and then attempt 2.8 update from a 'fresh' 2.7.2 install.
-
@InstanceExtension You don't lose anything trying that. You can always restore working 2.7.2 config to fresh 2.7.2 install.
-
I'm not sure how helpful this is, but I have been running pfSense Plus 24.11 on a Supermicro SYS-510D-4C-FN6P mini 1U server without any issues, which is also based on FreeBSD 15 and has a similar hardware setup as what the OP is using. This raises the question then, what is different between pfSense CE 2.8.0 and pfSense Plus 24.11? @stephenw10 - any thoughts from your side?
I also looked at my OS boot log and it looks very similar to what is in the first post screenshot. The next entry after
Dual Console: Video Primary, Serial SecondaryI see is theCPU:followed by the CPU string. Just to throw another idea out there, could there be any issues related to the CPU microcode update? -
@tman222 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
could there be any issues related to the CPU microcode update?
It's possible I guess but I doubt it.
@InstanceExtension Are you only installing via the BMC there? Have you tried a real console connection on the hardware directly?
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
@InstanceExtension Are you only installing via the BMC there? Have you tried a real console connection on the hardware directly?
Yes, as I said above to another member:
No other peripherals connected. Not even a keyboard, mouse or monitor although I did have them present to test and it made no difference.
-
Yes I read that but it was unclear to me if you had just connected them as a test or actually used a direct local console.
Does it really appear identically on local consoles and both video and serial console via the bmc?
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Does it really appear identically on local consoles and both video and serial console via the bmc?
Yes, identical. During my next test (going to attempt a very simple config on a fresh 2.7.2 install then upgrade to 2.8) I'll double check from an actual monitor and provide a pic.
The thing is my pfSense setup is really pretty basic already. Two interfaces, some aliases, NAT rules, WAN/LAN rules, NTP, UPnP and WOL services and that's about it. No DHCP, no DNS, no VPN, no Traffic shapers, No load balancing, No RAM disk, System Tunables are all default, nothing really elaborate at all. I am limiting the system to IPv4 though and I did turn off SSH access.
-
@InstanceExtension we upgrade a X11SDV-4C-TP8F today without an issue.
Maybe you find something in the specs what could be different?Update was only via BMC, nothing else connected.
-
@slu said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
@InstanceExtension we upgrade a X11SDV-4C-TP8F today without an issue.
Maybe you find something in the specs what could be different?Update was only via BMC, nothing else connected.
Thanks for the feedback.
-
Mmm, I would try setting the serial console as primary and making sure that works as expected in 2.7.2 when you access it via the BMC.
You could be seeing some issue with an updated driver in 2.8 that affects the video console.
-
I had a similar problem after upgrading to pfSense 2.8.0.
In my case, the solution was to switch from UEFI to Legacy BIOS mode.
After that, the system booted normally. -
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Mmm, I would try setting the serial console as primary and making sure that works as expected in 2.7.2 when you access it via the BMC.
You could be seeing some issue with an updated driver in 2.8 that affects the video console.
I really wish I could have a 2.8 ISO, that would make this entire process so much simpler to troubleshoot.
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
@tman222 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
could there be any issues related to the CPU microcode update?
It's possible I guess but I doubt it.
I'm starting to think that there is a very good chance that this is related to the updated CPU microcode in 2.8. This is what the 2.7.2 logs look like with verbose logging turned on.
Jun 3 17:01:12 kernel Origin="GenuineIntel" Id=0x606c1 Family=0x6 Model=0x6c Stepping=1 Jun 3 17:01:12 kernel CPU: Intel(R) Xeon(R) D-1718T CPU @ 2.60GHz (2600.00-MHz K8-class CPU) Jun 3 17:01:12 kernel No TSX change made Jun 3 17:01:12 kernel cpuctl: access to MSR registers/cpuid info. Jun 3 17:01:12 kernel start_init: trying /sbin/init Jun 3 17:01:12 kernel Dual Console: Video Primary, Serial SecondaryThe next logged action after "start_init: trying /sbin/init" is the "cpuctl" function.
Per the man page:
The special device /dev/cpuctl presents interface to the system CPU. It provides functionality to retrieve CPUID information, read/write machine specific registers (MSR) and perform CPU firmware updates.With CE 2.8 the system is stuck right after
start_init: trying /sbin/initprobably trying to do the "cpuctl" function and failing/stalling.
-
Does it boot to single user mode?
If so you can enable verbose RC there and reboot. That may get you more debug output.
touch /verbose_rc -
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Does it boot to single user mode?
If so you can enable verbose RC there and reboot. That may get you more debug output.
touch /verbose_rcI'll give that a try during the next maintenance window.
