Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7
-
Yup that's identical to the second crash reported initially. Not much to go on unfortunately.
-
Got it!!! Quite a bit larger in size than the others and I can't seem to upload it here due to sizing limit. Somwhere else on here I can upload it?
So to recap here, just want to speak a bit to my test bed. Using a Cisco 3750 switch as my 'inside' switch carved up into various VLANs to simulate my actual prod setup and to get OSPF routes into the FWs to make sure the routes get populated properly. The inside interfaces of both FWs are in 'VLAN 10'. No tagging is done in the FW configuration. Ports are just using 'switchport access vlan 10' in the switch and no VLANs are configured in the FWs. WAN ports are plugged into just an old Netgear steel case unmanaged switch. Switchports on there are FE for WAN and the LAN ports are gigabit. I don't think any of this matters but did want to include it here in case it does.
Thanks for looking here and for being patient with this.
-
Aha, nice. Yup I'd expect it to be large. How big is it?
You can probably upload it here: https://nc.netgate.com/nextcloud/s/3zcPmr5JE694eDn
Though I think there is a size limit there.
-
@stephenw10 Its about 2.6GB. I tried uploading to your link but the thing never sems to complete. I uploaded it to GDrive. Maybe can see it with this link:
https://drive.google.com/file/d/1ePOeUzoFD911MFNodwCZLY17gZdTpn6k/view?usp=drive_link
Let me know if that doesn't work. Thanks for looking.
-
Great I see that. Let's see if it reveals anything...
-
So today I got this to happen by restarting both FWs and the switch they connect to all at once. I think mostly when this happens its when I fire these up in the AM at which time everything gets started all together. After one of these things panics they seem to be pretty stable. I guess its somewhat rare anymore this happens just randomly throughout the day. Used to but not much anymore. Maybe this is something to do with the switch starting up doing something at startup. Guessing again but if true this might not be seen in my prod environment. But good to know whats going on here. Thanks again for looking.
-
@rfranzke said in Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7:
Panic Dump Link
That's the same core? The link looks the same.
-
@stephenw10 Its the same dump. I just made it into an actual link. I couldn't edit original post.
-
Ok the core dump yielded useful info but not enough to solve it unfortunately.
Are you able to load a debug kernel on this and get a core with that running?
If so grab the pkg here. Install it. Then:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/debug-kernel.html#booting-the-debug-kernel -
@stephenw10 Yes I can install this and run it but not sure the proper way to actually install the pkg file. Enable SSH, copy the file over, and then run it or is there some other method to do this via WebUI?
-
@rfranzke
pkg install -U pfSense-kernel-debug-pfSense-2.8.0.b.20250814.0928.pkg -
@kprovost So I think I have this loaded. I just enabled SSH on the box, connected to it, and transferred the download debug kernel file. Then ran the installer:
[2.8.0-RELEASE][admin@fw2.mdaemon.int]/root: pkg install -y pfSense-kernel-debug-pfSense-2.8.0.b.20250814.0928.pkg
Updating pfSense-core repository catalogue...
Fetching meta.conf: 0%
Fetching data.pkg: 0%
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
Fetching meta.conf: 0%
Fetching data.pkg: 0%
pfSense repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):New packages to be INSTALLED:
pfSense-kernel-debug-pfSense: 2.8.0.b.20250814.0928 [unknown-repository]Number of packages to be installed: 1
The process will require 254 MiB more space.
[1/1] Installing pfSense-kernel-debug-pfSense-2.8.0.b.20250814.0928...
Extracting pfSense-kernel-debug-pfSense-2.8.0.b.20250814.0928: 100% 193 B 0.1kB/s 00:02
[2.8.0-RELEASE][admin@fw2.mdaemon.int]/root: rebootThen added this to the /boot/loader.conf.local file I created:
kernel="kernel.debug"
so it boots this kernel all the time. I made the file change before the reboot. Box booted up and I can access it still so assuming its running the debug kernel now. Slick way to tell if its running that new kernel?
Edit: Maybe this tells us what we want to know?
[2.8.0-RELEASE][admin@fw2.mdaemon.int]/root: uname -a
FreeBSD fw2.mdaemon.int 15.0-CURRENT FreeBSD 15.0-CURRENT #1 RELENG_2_8_0-n256082-57273ac5fb19-dirty: Thu Aug 14 09:32:59 CEST 2025 root@nut:/usr/home/kp/netgate/crossbuild-2.8.0/obj/amd64/OdB78hjz/usr/home/kp/netgate/crossbuild-2.8.0/sources/FreeBSD-src-RELENG_2_8_0/amd64.amd64/sys/pfSense-DEBUG amd64
[2.8.0-RELEASE][admin@fw2.mdaemon.int]/root: -
@rfranzke said in Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7:
[2.8.0-RELEASE][admin@fw2.mdaemon.int]/root: uname -a
FreeBSD fw2.mdaemon.int 15.0-CURRENT FreeBSD 15.0-CURRENT #1 RELENG_2_8_0-n256082-57273ac5fb19-dirty: Thu Aug 14 09:32:59 CEST 2025 root@nut:/usr/home/kp/netgate/crossbuild-2.8.0/obj/amd64/OdB78hjz/usr/home/kp/netgate/crossbuild-2.8.0/sources/FreeBSD-src-RELENG_2_8_0/amd64.amd64/sys/pfSense-DEBUG amd64Yes, that's the kernel we want to be running now.
It has one extra assertion on top of the default debug kernel. Hopefully that will give us more clues about why the panic happens. -
So got the box to crash again but no dump file was created at all this time:
[2.8.0-RELEASE][admin@fw2.mdaemon.int]/root: cd /var/crash
[2.8.0-RELEASE][admin@fw2.mdaemon.int]/var/crash: ls
[2.8.0-RELEASE][admin@fw2.mdaemon.int]/var/crash:No file. Any reason this debug version wouldn't create a debug file on panic? Should I have to reconfigure the debug configs as a result of loading the new kernel?
-
Hmm, does
sysctl debug.ddb.scripting.scriptsstill show the modified script? -
@stephenw10 I believe so. Here in case I am missing something:
[2.8.0-RELEASE][admin@fw2.mdaemon.int]/root: sysctl debug.ddb.scripting.scripts
debug.ddb.scripting.scripts: lockinfo=show locks; show alllocks; show lockedvnods
pfs=bt ; show registers ; show pcpu ; run lockinfo ; acttrace ; ps ; alltrace
kdb.enter.default=bt ; show registers ; dump ; reset
kdb.enter.witness=run lockinfo[2.8.0-RELEASE][admin@fw2.mdaemon.int]/root:
I forced a manual panic to test it again and it created a dump file as one would expect, so not sure what happened here:
[2.8.0-RELEASE][admin@fw2.mdaemon.int]/var/crash: ls
bounds info.0 info.last vmcore.0 vmcore.lastLet me know if something is missing from above. Thanks for looking.
-
Huh that's odd. Did it actually panic before with the arp crash? Did it reboot automatically afterwards?
Might need a new script line there if it's doing some special. In which case we might need wisdom from @kprovost
-
@stephenw10 Off the top of my head I don't know of any reason why a panic wouldn't leave a core dump.
What makes you say the box crashed if there's no core dump? Did it reboot? Is there anything in the logs?
If it did crash it really should have left a core dump. The only thing that I can think of that might (and I stress might, I've not confirmed this) cause it to not produce a core dump is if the dump area is too small. How much memory does the machine have, and how big is the swap?
-
@stephenw10 I heard the fans spin up like it does and checked if I could access the device on the network. I couldn't so it definitely seemed to crash. Trying to get this to crash on its own now. The box definitely restarted for some reason. I assumed it was due to the same crash that I've been chasing. I suppose its possible that it just restarted for some reason, but I would say fairly unlikely that would be the case. Plenty of swap space available:
[2.8.0-RELEASE][admin@fw2.mdaemon.int]/root: swapinfo -h
Device Size Used Avail Capacity
/dev/gpt/swap1 128G 0B 128G 0%
[2.8.0-RELEASE][admin@fw2.mdaemon.int]/root: -
Hmm. I guess try and trigger it again and see if you can confirm it.
