Kernel Panic when Upgrading to 2.8.0 beta
-
For the past year, PFSense has been crashing once every 2 weeks. No log messages, no console messages, nothing, so I figured it was something w/ the hardware. I've replaced every single piece of hardware except for the CPU (AMD Ryzen 5 2400G) and the motherboard (ASRock AB350 micro-ITX), and my 10G NIC (10Gtek Dual 10Gb Intel 82599ES Controller).
I figured why not try 2.8.0 to see if the crashes will finally stop.
As soon as PFSense tries to boot to FreeBSD 15.0, I get a kernel panic..
)I see in the kernel panic message, right before the crash it reports the on-board Wifi AC adapter (iwm0 / iwm3168fw).
So, I disabled the Wifi adapter in the BIOS, but I still see in there in the output, upon boot. Tried several times, even resetting bios back to defaults, then changing Wifi adapter to disabled.
This motherboard is pretty old ~2017, so I'm thinking it's probably time to just go buy a "new" AM4 motherboard and see if I have any better luck.
I tried the 2.8.0 installation several times-- upgrading from 2.72, and several times formatting the SSD and using a USB installer to go straight to 2.8.0-- still get the same kernel panic each time.
Had to re-install 2.7.2 to be able to boot PFSense.
-
@CCNewb said in Kernel Panic when Upgrading to 2.8.0 beta:
For the past year, PFSense has been crashing once every 2 weeks. No log messages, no console messages, nothing, so I figured it was something w/ the hardware. I've replaced every single piece of hardware except ...
You have more patience with old hardware than I do. Hardware often last longer than 6 years but not always.
-
@Patch man, my whole day was ruined. I went to Microcenter at 3pm and got a "new" AM4 micro-ITX board, they only had 1 choice. And it's a bad board. Can tell it was used but they sold it to me as new, and the thing wouldn't post. Spent all afternoon messing around w/ it, family upset internet was down, ugh. Ended up reverting back to the old mobo and gonna return all the crap I bought tomorrow. So disappointing.
-
@CCNewb what you can try is disable the wifi using a device hint, to see if it was the reason for the kernel panic.
At the loader prompt before pfSense boots (press '3' when the boot menu shows) you enter:
set hint.iwm.0.disabled="1"or since you upgrade from 2.7.2 you can add/set it in the
/boot/loader.conf.localfile (it may not exist yet):hint.iwm.0.disabled="1"The age of your hardware should not be an issue, there is generally good support for it. Wifi cards being the exception in general for FreeBSD.
Addition: if you get a kernel panic, run
bt(backtrace) at the debug prompt, that will give more information about the module involved. -
@patient0 thanks! I will try one night this week and report back
-
Or physically remove the wifi device? Disabling it in the BIOS doesn't seem to be working as it should. A jumper maybe?
But it also might be something else, the panic string there is unhelpful. Try running
btat the db> prompt there and see what the backtrace is. That should where it's panicking. -
@patient0 Dude, you rock!!! That totally fixed the issue. 2.8.0 upgrade went super smooth after adding that
/boot/loader.conf.localfileThanks man
-
@stephenw10 thank you, didn't know about
bt, patient0's tip about disabling the hardware via theloader.conf.localfile did the trick. I was able to upgrade, it went super fast. -
The crashes on 2.8.0 only got worse. Typical state level espionage, I hate the attention. No crash data, no console messages, nothing, just compete system lock. Every single day on 2.8.0 it’s locking up. Replaced the riser card and 10g nic the other day. Didn’t help. Just ordered another replacement mobo and newer CPU, thus completing replacing every single piece of hardware.
-
No crash data, no console messages, nothing, just compete system lock
It's frustrating, I understand.
Have you done the classic, memory testing for a few hours? pfSense is default configured to create core dump and for me it's rare to have no logs or core dumps at all.If your setup allows, switching to another router OS (temporarily, OS: Untangle, VyOS, OPNsense) would be an option to see if other OS run into the same issue. Boot/Install from another disk and run it for a few days.
-
How locked is locked? Does the caps lock button still work on an attached keyboard for example?
A hard lock with no output is almost always hardware though.
-
I have the same problem but i have no wifi boards in the server and no usb devices attached apart from a keyboard
can anyone help me! -
Run
btat the db> prompt there to see the backtrace from the panic.Though it looks like a problem with the firmware for the isp_2400 device.
Are you actually using that? If not I'd remove or disable it. -
@stephenw10 hi thanks for the reply i can't use bt as its frozen and no key inputs work also i have no idea what that device even is as there is nothing plugged into the server apart from the usb keyboard.
-
On-board fibrechannel adapter I'd guess.
I assume the keyboard works before that? Capslock LED still work after the crash?
-
@stephenw10 yes the keyboard works right up to to the error then freezes and i have to hard reset im now back on ver 2.7.2 witch works fine.
-
Do you see the isp_2400 device in 2.7.2? Are you using it to boot from? A SCSI disk?
-
@stephenw10 sorry i have no idea its running on a dl 360 g5 server tho if that helps i have had no problems with any updates in years this is the first one that is giving the problem
-
Check the bootlog in 2.7.2. If it's failing to load there too, but obviously not crashing, then you can probably just disable that device in the BIOS.
-
yep i see it there
