Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7
-
@stephenw10 said in Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7:
Mmm, looks like that first crash could be this: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285813
Hrmm......would that sort of issue be seen with CARP enabled? And seems that's still an open bug if true. Wonder how soon something like that gets fixed in BSD.
Strangely, I cannot get this to crash now. Sine the reinstall and the one panic I posted, these have been rock-solid with no panics. Good and bad.
-
OK I think I have the swap configured now:
Anything else required to get this to get the info we need?
-
Ok cool. So to enable full core dumps you need to edit the file
/etc/pfSense-ddb.conf.Change the script kdb.enter.default line to:
script kdb.enter.default=bt ; show registers ; dump ; resetReboot then check the output of:
sysctl debug.ddb.scripting.scriptsMake sure it shows the changed line.
Then you can test it by manually triggering a panic by running:
sysctl sysctl debug.kdb.panic=1
You should see the core file after it reboots.After that just wait for the next crash or somehow trigger it if you can.
-
@stephenw10 said in Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7:
Ok cool. So to enable full core dumps you need to edit the file
/etc/pfSense-ddb.conf.Change the script kdb.enter.default line to:
script kdb.enter.default=bt ; show registers ; dump ; resetReboot then check the output of:
sysctl debug.ddb.scripting.scriptsMake sure it shows the changed line.
Then you can test it by manually triggering a panic by running:
sysctl sysctl debug.kdb.panic=1
You should see the core file after it reboots.After that just wait for the next crash or somehow trigger it if you can.
OK I think I have this done:
# $FreeBSD$
#
# This file is read when going to multi-user and its contents piped thru
#ddb'' to define debugging scripts. \# \# seeman 4 ddb'' and ``man 8 ddb'' for details.
#script lockinfo=show locks; show alllocks; show lockedvnods
script pfs=bt ; show registers ; show pcpu ; run lockinfo ; acttrace ; ps ; alltrace# kdb.enter.panic panic(9) was called.
# script kdb.enter.default=textdump set; capture on; run pfs ; capture off; textdump dump; reset
script kdb.enter.default=bt ; show registers ; dump ; reset# kdb.enter.witness witness(4) detected a locking error.
script kdb.enter.witness=run lockinfosysctl debug.ddb.scripting.scripts
debug.ddb.scripting.scripts: lockinfo=show locks; show alllocks; show lockedvnods
pfs=bt ; show registers ; show pcpu ; run lockinfo ; acttrace ; ps ; alltrace
kdb.enter.default=bt ; show registers ; dump ; reset
kdb.enter.witness=run lockinfoI cannot seem to have this thing crash anymore. I'll see if I can mess with it to get it to panic again. Let me know if this setting looks right. Thanks again here for all the help. Really appreciate the time.
-
Yup that looks good. You can try the forced manual panic just to make sure it create the core file but I'm pretty confident it will.
Otherwise just wait for the next crash.
-
@stephenw10 said in Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7:
You can try the forced manual panic just to make sure it create the core file but I'm pretty confident it will.
Yeah, I forgot to do that. Did it just now and it did restart. Created a file called 'VMCore.0' thats like 2.5GB in size. That sound about right?
-
I don't know if it helps anyone but I was having a kernel panic issue on the first boot after trying to install 2.8 and in my case it was:
iwm7265Dfw: could not load firmware image, error 6I was able to fix it by dropping into the shell of the installer after the installation process and before the final reboot, and adding this line:
hint.iwm.0.disabled="1"to the end of
/mnt/boot/loader.conf -
No that's an unrelated bug. This one looks more difficult to fix unfortunately!
-
So unfortunately, I have been monkeying with this all day and have not been able to get this to panic. I'm not sure what's changed other than the panic dump config changes and re-installing the software via the NetGate installer. These things must know we are on to them.
I've tried ever combination of restarting, shutting off switches, unplugging ports, restart one, keep one running, blah, blah. They won't panic now.
I did notice some of my FRR OSPF configuration did not come over in the re-install process, namely the interface authentication config. It's quite possible I had removed it at some point in my testing, but I don't think so. No issues anyone is aware of with FRR configs not importing correctly on 2.8? I would doubt it, and its not important to this issue, but thought I'd ask while we wait for these things to panic again.
-
Well at least you're setup to catch it now if/when it does.
-
So still no panics overnight and most of yesterday. Not sure what to think here. The only real thing that's changed here (other than the swap/dump configurations) is the way I did the upgrade: Reinstall versus GUI update. I did have one panic just after the update was done via NetGate installer, but otherwise it's been rock solid.
Is there any reason to think that the original upgrade process contributed here and now that I've done a fresh install with a fresh set of package installs (which I never did originally) perhaps whatever issue was causing this has gone away? I would say no as I did have that one panic just after the first re-install but just throwing it out there in case its possible. Seems unlikely that would be it to me but grasping at straws to explain this one.
I let these run all night which I normally don't do, so I'll shut these down tonight and fire them up in the AM and see if I can get either one to panic. Not sure what to say on this. Thanks again all for all the contributions here.
-
@rfranzke Its waaaay too difficult to blame faulty installation for random crashes.
If something like that happens (say, a faulty drive) then crashes are immediate and repeatable.The bsd bug that Steven has found is a better candidate.
Obviously its rare, if it wasn't there would be plenty of reports here about it.Now you are able to catch full crash dumps. A debug kernel is the next thing.
This is deep waters and you know it.Give it some time.
