pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading
-
@stephenw10 Thanks for the reply. The system is completely non-responsive at that point. Hard reset (via IPMI console or Reset switch) is the only thing that gets it back.
I don't see any of this behavior in 2.7.2, the boot process is just quick and clean.
-
Hmm can you enable a serial console and check that?
I would definitely compare the boot log with 2.7.2 if you can though. I'd bet there is some new or changed driver at play here.
-
@stephenw10 This is what occurs in 2.7.2 after the Dual Console log message. Taken from the System logs so bottom up is the order.
May 29 12:50:11 kernel coretemp0: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu0
May 29 12:50:11 kernel aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
May 29 12:50:11 kernel ue0: Ethernet address: be:3a:f2:b6:05:9f
May 29 12:50:11 kernel ue0: <USB Ethernet> on urndis0
May 29 12:50:11 kernel urndis0: <RNDIS Communications Control> on usbus0
May 29 12:50:11 kernel urndis0 numa-domain 0 on uhub1
May 29 12:50:11 kernel TSC: P-state invariant, performance statistics
May 29 12:50:11 kernel VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
May 29 12:50:11 kernel AMD Extended Feature......CPU features that won't post here.......
May 29 12:50:11 kernel Origin="GenuineIntel" Id=0x606c1 Family=0x6 Model=0x6c Stepping=1
May 29 12:50:11 kernel CPU: Intel(R) Xeon(R) D-1718T CPU @ 2.60GHz (2600.00-MHz K8-class CPU)
May 29 12:50:11 kernel g_vfs_done():da0p1[READ(offset=65536, length=8192)]error = 5
May 29 12:50:11 kernel Dual Console: Video Primary, Serial Secondary -
Hmm. Well seeing ue0 is always a concern but shouldn't hang entirely.
Nothing before that in the boot is different?
-
@stephenw10 I'll need to get some free time to get details on the boot process for 2.8.
It means I'll be down for some time when attempting the upgrade again. I don't have a redundant system to test on. It may not be until this weekend.First thing I will try is to turn off the "IPMI Host Interface" and let that fall back to KCS for communication between host and BMC.
-
@InstanceExtension said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
May 29 12:50:11 kernel g_vfs_done():da0p1[READ(offset=65536, length=8192)]error = 5
Interesting, especially since that's about where the upgrade hangs...
Doing some reading and shot in the dark—did/do you have a USB drive plugged-in while trying to perform the upgrade?
EDIT: Is
da0the same drive asnda0(the Samsung 1 TB SSD)?DOUBLE EDIT: Answer my own question—no. Different drivers.
So then the update has a problem about where a read error is detected on device
da0p1during successful 2.7.2 boot—which may or may not be relevant to the 2.8 upgrade hang.I think I agree that either this...
@InstanceExtension said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
I have to wonder if somehow the ASPEED AST2600 BMC is somehow causing this issue. Looks like the OS is waiting on it before it will mount the disk.
...or some other driver/device (removable or not) is causing the upgrade hang.
-
I had a long pause at the same boot point as you (5-10 minutes) on both of my servers but they successfully completed the upgrade afterwards.
Both servers are bare metal Dell PowerEdge R220s.
-
Mmm, if it does eventually show the console menu that's a sign that it just isn't using the current console as primary. The menu is shown on all consoles.
-
I spent a couple of hours on this today. No resolution yet.
Tried:- Disabled BMC "Host Interface"
- Verbose logging
- Safe Mode
- Disabled Serial ports and console in BIOS
- Letting it sit trying to load for an hour
- Manually set Console to Video and Serial at pfSense boot screen
- Reset BIOS to optimized defaults. I did have a slight change in there to support Intel Speed Shift. Setting that to back to default did not help. Was set to : Hardware P-States: Native Mode (to support Speed Shift)
I just can't get it to load pfSense. This was the console output with Verbose logging turn on and using the Serial console output. (was the same when using just Video console)
I'm not sure what to do next.
-
@InstanceExtension This is a bare metal host, correct? What other storage is present aside from the Samsung NVMe? No other peripherals connected?
-
@tinfoilmatt said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
@InstanceExtension This is a bare metal host, correct? What other storage is present aside from the Samsung NVMe? No other peripherals connected?
Bare metal, correct. No other storage is present during the upgrade.
No other peripherals connected. Not even a keyboard, mouse or monitor although I did have them present to test and it made no difference.
Just the MB itself and its default components.
-
@InstanceExtension I wonder if you could get the upgrade to complete if you wiped 2.7.2 first, reinstalled, performed upgrade prior to restoring config, etc.—just to see if the hardware will complete the 2.8 upgrade.
-
@tinfoilmatt said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
@InstanceExtension I wonder if you could get the upgrade to complete if you wiped 2.7.2 first, reinstalled, performed upgrade prior to restoring config, etc.—just to see if the hardware will complete the 2.8 upgrade.
Not possible. There is no 2.8 ISO and without the router configured no internet.
-
@InstanceExtension That's why I'm saying to wipe 2.7.2 first to remove your config. 'Factory reset' essentially just to see if you can get the update to complete the only way currently supported (i.e., upgrade from a 2.7.2 instance) on your hardware.
It will connect to the internet during the 2.7.2 install, minimal config (without restoring your working 2.7.2 config), and then attempt 2.8 update from a 'fresh' 2.7.2 install.
-
@InstanceExtension You don't lose anything trying that. You can always restore working 2.7.2 config to fresh 2.7.2 install.
-
I'm not sure how helpful this is, but I have been running pfSense Plus 24.11 on a Supermicro SYS-510D-4C-FN6P mini 1U server without any issues, which is also based on FreeBSD 15 and has a similar hardware setup as what the OP is using. This raises the question then, what is different between pfSense CE 2.8.0 and pfSense Plus 24.11? @stephenw10 - any thoughts from your side?
I also looked at my OS boot log and it looks very similar to what is in the first post screenshot. The next entry after
Dual Console: Video Primary, Serial SecondaryI see is theCPU:followed by the CPU string. Just to throw another idea out there, could there be any issues related to the CPU microcode update? -
@tman222 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
could there be any issues related to the CPU microcode update?
It's possible I guess but I doubt it.
@InstanceExtension Are you only installing via the BMC there? Have you tried a real console connection on the hardware directly?
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
@InstanceExtension Are you only installing via the BMC there? Have you tried a real console connection on the hardware directly?
Yes, as I said above to another member:
No other peripherals connected. Not even a keyboard, mouse or monitor although I did have them present to test and it made no difference.
-
Yes I read that but it was unclear to me if you had just connected them as a test or actually used a direct local console.
Does it really appear identically on local consoles and both video and serial console via the bmc?
-
@stephenw10 said in pfSense CE 2.8.0 upgrade stalls after reboot and gets stuck when loading:
Does it really appear identically on local consoles and both video and serial console via the bmc?
Yes, identical. During my next test (going to attempt a very simple config on a fresh 2.7.2 install then upgrade to 2.8) I'll double check from an actual monitor and provide a pic.
The thing is my pfSense setup is really pretty basic already. Two interfaces, some aliases, NAT rules, WAN/LAN rules, NTP, UPnP and WOL services and that's about it. No DHCP, no DNS, no VPN, no Traffic shapers, No load balancing, No RAM disk, System Tunables are all default, nothing really elaborate at all. I am limiting the system to IPv4 though and I did turn off SSH access.
