Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7

rfranzke

OK I think I have the swap configured now:

Anything else required to get this to get the info we need?

stephenw10

Ok cool. So to enable full core dumps you need to edit the file /etc/pfSense-ddb.conf.

Change the script kdb.enter.default line to:
script kdb.enter.default=bt ; show registers ; dump ; reset

Reboot then check the output of: sysctl debug.ddb.scripting.scripts

Make sure it shows the changed line.

Then you can test it by manually triggering a panic by running: sysctl sysctl debug.kdb.panic=1
You should see the core file after it reboots.

After that just wait for the next crash or somehow trigger it if you can.

rfranzke

@stephenw10 said in Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7:

Ok cool. So to enable full core dumps you need to edit the file /etc/pfSense-ddb.conf.

Change the script kdb.enter.default line to:
script kdb.enter.default=bt ; show registers ; dump ; reset

Reboot then check the output of: sysctl debug.ddb.scripting.scripts

Make sure it shows the changed line.

Then you can test it by manually triggering a panic by running: sysctl sysctl debug.kdb.panic=1
You should see the core file after it reboots.

After that just wait for the next crash or somehow trigger it if you can.

OK I think I have this done:

# $FreeBSD$
#
# This file is read when going to multi-user and its contents piped thru
# ddb'' to define debugging scripts. \# \# see man 4 ddb'' and ``man 8 ddb'' for details.
#

script lockinfo=show locks; show alllocks; show lockedvnods
script pfs=bt ; show registers ; show pcpu ; run lockinfo ; acttrace ; ps ; alltrace

# kdb.enter.panic panic(9) was called.
# script kdb.enter.default=textdump set; capture on; run pfs ; capture off; textdump dump; reset
script kdb.enter.default=bt ; show registers ; dump ; reset

# kdb.enter.witness witness(4) detected a locking error.
script kdb.enter.witness=run lockinfo

sysctl debug.ddb.scripting.scripts

debug.ddb.scripting.scripts: lockinfo=show locks; show alllocks; show lockedvnods
pfs=bt ; show registers ; show pcpu ; run lockinfo ; acttrace ; ps ; alltrace
kdb.enter.default=bt ; show registers ; dump ; reset
kdb.enter.witness=run lockinfo

I cannot seem to have this thing crash anymore. I'll see if I can mess with it to get it to panic again. Let me know if this setting looks right. Thanks again here for all the help. Really appreciate the time.

stephenw10

Yup that looks good. You can try the forced manual panic just to make sure it create the core file but I'm pretty confident it will.

Otherwise just wait for the next crash.

rfranzke

@stephenw10 said in Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7:

You can try the forced manual panic just to make sure it create the core file but I'm pretty confident it will.

Yeah, I forgot to do that. Did it just now and it did restart. Created a file called 'VMCore.0' thats like 2.5GB in size. That sound about right?

Mikesco3

I don't know if it helps anyone but I was having a kernel panic issue on the first boot after trying to install 2.8 and in my case it was:

iwm7265Dfw: could not load firmware image, error 6

I was able to fix it by dropping into the shell of the installer after the installation process and before the final reboot, and adding this line:

hint.iwm.0.disabled="1"

to the end of /mnt/boot/loader.conf

stephenw10

No that's an unrelated bug. This one looks more difficult to fix unfortunately!

rfranzke

So unfortunately, I have been monkeying with this all day and have not been able to get this to panic. I'm not sure what's changed other than the panic dump config changes and re-installing the software via the NetGate installer. These things must know we are on to them.

I've tried ever combination of restarting, shutting off switches, unplugging ports, restart one, keep one running, blah, blah. They won't panic now.

I did notice some of my FRR OSPF configuration did not come over in the re-install process, namely the interface authentication config. It's quite possible I had removed it at some point in my testing, but I don't think so. No issues anyone is aware of with FRR configs not importing correctly on 2.8? I would doubt it, and its not important to this issue, but thought I'd ask while we wait for these things to panic again.

stephenw10

Well at least you're setup to catch it now if/when it does.

rfranzke

So still no panics overnight and most of yesterday. Not sure what to think here. The only real thing that's changed here (other than the swap/dump configurations) is the way I did the upgrade: Reinstall versus GUI update. I did have one panic just after the update was done via NetGate installer, but otherwise it's been rock solid.

Is there any reason to think that the original upgrade process contributed here and now that I've done a fresh install with a fresh set of package installs (which I never did originally) perhaps whatever issue was causing this has gone away? I would say no as I did have that one panic just after the first re-install but just throwing it out there in case its possible. Seems unlikely that would be it to me but grasping at straws to explain this one.

I let these run all night which I normally don't do, so I'll shut these down tonight and fire them up in the AM and see if I can get either one to panic. Not sure what to say on this. Thanks again all for all the contributions here.

netblues

@rfranzke Its waaaay too difficult to blame faulty installation for random crashes.
If something like that happens (say, a faulty drive) then crashes are immediate and repeatable.

The bsd bug that Steven has found is a better candidate.
Obviously its rare, if it wasn't there would be plenty of reports here about it.

Now you are able to catch full crash dumps. A debug kernel is the next thing.
This is deep waters and you know it.

Give it some time.