Consistently stuck at "Updating CPU Microcode" during bootup
-
Version: PFSense CE 2.7.2 (Fresh install not upgraded)
Hardware" Protectli Vault 6 port (VP4670 – 6 Port Intel
i7 (VP4670))
Message at point of failure: Launching the init system...Updating CPU Microcode...After successful installation the device reboots normal and will possibly run into getting stuck at "Updating CPU Microcode" during bootup. It's not every time. After a power cycle of the device, it will usually boot up fine without issue and will run perfectly until its rebooted again. Once rebooted it runs into the issue again. The only way to get past this is to power cycle the device and it will usually boot up without the issue but not always sometimes multi power cycles have to happen before it boots up fine again. I have tried installing PFSense CE 2.7.2 on multiple other devices both from different manufacturers and also multiple other Protectli Vault 6 port devices.
I have already opened a support case with Protecli and so far, the claim is this is a PFSense issue. So figured I would open a forum post as well. If anyone else has ran into the same issue or similar would appreciate any assistance, you can provide.
Attached photo of the issue.
-
Try booting in verbose mode. At the loader prompt:
boot -v -
escaped normal boot process and at grub> did boot -v
Gets stuck at the same area without any additional output on the screen. Picture attached.
Also had serial console connected and output from that is attached as well. pfsense output.txt
-
That doesn't look like verbose output. pfSense doesn't use grub. Booting verbose should be entered at the bsd loader prompt like:
╔════ Welcome to Netgate pfSense Plus ════╗ __________________________ ║ ║ / ___\ ║ 1. Boot Multi user [Enter] ║ | /` ║ 2. Boot Single user ║ | / :-| ║ 3. Escape to loader prompt ║ | _________ ___/ /_ | ║ 4. Reboot ║ | /` ____ / /__ ___/ | ║ 5. Cons: Serial ║ | / / / / / / | ║ ║ | / /___/ / / / | ║ Options: ║ | / ______/ / / _ | ║ 6. Kernel: default/kernel (1 of 2) ║ |/ / / / _| |_ | ║ 7. Boot Options ║ / /___/ |_ _| | ║ 8. Boot Environments ║ / |_| | ║ ║ /_________________________/ ╚═════════════════════════════════════════╝ \ Exiting menu! Type '?' for a list of commands, 'help' for more detailed help. OK boot -v -
I did option 3 this time "escape to loader prompt" and it's the same as doing esc on the keyboard at least visually. I do not know why I thought and said I saw grub > as I definitely don't. I just see OK and type boot -v and hit enter.
I get the same output every time. Does not appear to be adding anything new to the output when doing boot -v.
Picture attached of screen before hitting enter for boot -v
I can upload a video to youtube and provide a link to it if that would help with seeing what its doing.
Thanks
-
@EULABreaker remove the flash drive and try again?
-
Flash drive has been removed and boot -v completed again. Stuck at the same Updating CPU microcode. also does not appear to be any additional output from the last time.
Thanks
-
You could comment out L446-447 here
boot into single user mode (there is an option for this at the loader menu)
zfs set readonly=off pfSense/ROOT/default vi /etc/pfSense-rc # make the above edits and save shutdown -r nowThis disables microcode updating.
See if it boots now
-
After making those changes it does boot without issue which makes since as we just disabled the microcode update.
What information do you need to be able to find the reason the microcode update is causing freezing.
As I assume this is not a valid permanent fix as I am assuming any future updates to pfSense that modifies that file will wipe out that change and cause the issue to reappear.
Thanks
-
Wanted to give a update on this. I got Protecli troubleshooting as well and they believe they may have found an issue with coreboot on their side. If they flash the bios to ami the issue stops. I will add their response below. As we have some of these devices in the field we are going to use the current work around of disabling microcode update while we wait to see if Protectli can identify the exact issue on their side. I thank you for your assistance and will provide another update as protectli works it out on their side with coreboot.
Is it possible to do these steps while the units are already booted up and working. We have about 4 that are in the field and working as long as they are not rebooted. I would like to do the modification to them before they get rebooted and someone has to go out with a monitor and keyboard to fix them.
zfs set readonly=off pfSense/ROOT/default
vi /etc/pfSense-rcmake the above edits and save
shutdown -r now
I found out what is going on… I got the same lock up you see but only after my third reboot. It took another 30+ reboots to figure out what is happening. The OS is changing the primary output, “cons”, option #5 on the boot option menu. You can see it when the very first pfSense menu pops up (with the ASCII art). If you hit “6” you can stop the boot up and change the primary output from: video > serial > dual (serial primary) > dual (video primary). I changed it to “video” and it still locked up. Upon rebooting it I saw the option changed itself to “dual (serial primary)”. So I tried everything I could to try and get it to stay at “video” but the OS changed it every time to “dual (serial primary).
Next I flashed the BIOS to AMI and it booted correctly 10+ times in a row regardless of the primary output as long as it wasn’t “serial” only while I was using a monitor and an HDMI or DP cable. So the OS behavior is the same as with coreboot, it always changed it to “dual (serial primary), but it doesn’t seem to matter with AMI. It still boots with a monitor attached.
I have to do a lot more testing to do on this so I will be working on logging the behavior to see if it happens on other Vaults or on older versions of Vaults or pfSense. I also searched every ticket and there was one issue about a month ago where a customer saw the same behavior and we couldn’t troubleshoot it so we sent them a replacement Vault, got the “faulty” one back and it passed every test we threw at it. So we chalked it up to software error but didn’t have a solid cause.
Is using our Flashli tool (link) to flash your Vaults to AMI a viable option for you to get pfSense booting properly every time? At least while we try and figure out if the issue is caused by the microcode with the new pfSense kernel?
-
Hmm, I was going to recommend disabling the audio hardware in the BIOS:
hdacc0: <Realtek ALC897 HDA CODEC> at cad 0 on hdac0 hdaa0: <Realtek ALC897 Audio Function Group> at nid 1 on hdacc0 pcm0: <Realtek ALC897 (Right Analog)> at nid 20 and 24 on hdaa0 hdacc1: <Intel Kaby Lake HDA CODEC> at cad 2 on hdac0 hdaa1: <Intel Kaby Lake Audio Function Group> at nid 1 on hdacc1 pcm1: <Intel Kaby Lake (HDMI/DP 8ch)> at nid 3 on hdaa1But you probably can't do that in Coreboot.
You can see in your output though that it is booting with Video as the primary console:
Dual Console: Video Primary, Serial SecondaryIf you have a serial connection I recommend setting serial as the primary console if only because it's much easier to log and copy and output from a serial terminal.
