Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7
-
Yup that looks good. You can try the forced manual panic just to make sure it create the core file but I'm pretty confident it will.
Otherwise just wait for the next crash.
-
@stephenw10 said in Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7:
You can try the forced manual panic just to make sure it create the core file but I'm pretty confident it will.
Yeah, I forgot to do that. Did it just now and it did restart. Created a file called 'VMCore.0' thats like 2.5GB in size. That sound about right?
-
I don't know if it helps anyone but I was having a kernel panic issue on the first boot after trying to install 2.8 and in my case it was:
iwm7265Dfw: could not load firmware image, error 6I was able to fix it by dropping into the shell of the installer after the installation process and before the final reboot, and adding this line:
hint.iwm.0.disabled="1"to the end of
/mnt/boot/loader.conf -
No that's an unrelated bug. This one looks more difficult to fix unfortunately!
-
So unfortunately, I have been monkeying with this all day and have not been able to get this to panic. I'm not sure what's changed other than the panic dump config changes and re-installing the software via the NetGate installer. These things must know we are on to them.
I've tried ever combination of restarting, shutting off switches, unplugging ports, restart one, keep one running, blah, blah. They won't panic now.
I did notice some of my FRR OSPF configuration did not come over in the re-install process, namely the interface authentication config. It's quite possible I had removed it at some point in my testing, but I don't think so. No issues anyone is aware of with FRR configs not importing correctly on 2.8? I would doubt it, and its not important to this issue, but thought I'd ask while we wait for these things to panic again.
-
Well at least you're setup to catch it now if/when it does.
-
So still no panics overnight and most of yesterday. Not sure what to think here. The only real thing that's changed here (other than the swap/dump configurations) is the way I did the upgrade: Reinstall versus GUI update. I did have one panic just after the update was done via NetGate installer, but otherwise it's been rock solid.
Is there any reason to think that the original upgrade process contributed here and now that I've done a fresh install with a fresh set of package installs (which I never did originally) perhaps whatever issue was causing this has gone away? I would say no as I did have that one panic just after the first re-install but just throwing it out there in case its possible. Seems unlikely that would be it to me but grasping at straws to explain this one.
I let these run all night which I normally don't do, so I'll shut these down tonight and fire them up in the AM and see if I can get either one to panic. Not sure what to say on this. Thanks again all for all the contributions here.
-
@rfranzke Its waaaay too difficult to blame faulty installation for random crashes.
If something like that happens (say, a faulty drive) then crashes are immediate and repeatable.The bsd bug that Steven has found is a better candidate.
Obviously its rare, if it wasn't there would be plenty of reports here about it.Now you are able to catch full crash dumps. A debug kernel is the next thing.
This is deep waters and you know it.Give it some time.
-
This post is deleted! -
So these have been fairly stable. I finally got one of them to panic this morning, but it was not the FW that I have the SWAP/debug stuff set up on. I set that up on the backup FW, as it was the main one that was having the issue, and of course now it won't panic. I'll add the dump file here but its likely not very useful. Stay tuned.
-
Yup that's identical to the second crash reported initially. Not much to go on unfortunately.
-
Got it!!! Quite a bit larger in size than the others and I can't seem to upload it here due to sizing limit. Somwhere else on here I can upload it?
So to recap here, just want to speak a bit to my test bed. Using a Cisco 3750 switch as my 'inside' switch carved up into various VLANs to simulate my actual prod setup and to get OSPF routes into the FWs to make sure the routes get populated properly. The inside interfaces of both FWs are in 'VLAN 10'. No tagging is done in the FW configuration. Ports are just using 'switchport access vlan 10' in the switch and no VLANs are configured in the FWs. WAN ports are plugged into just an old Netgear steel case unmanaged switch. Switchports on there are FE for WAN and the LAN ports are gigabit. I don't think any of this matters but did want to include it here in case it does.
Thanks for looking here and for being patient with this.
-
Aha, nice. Yup I'd expect it to be large. How big is it?
You can probably upload it here: https://nc.netgate.com/nextcloud/s/3zcPmr5JE694eDn
Though I think there is a size limit there.
-
@stephenw10 Its about 2.6GB. I tried uploading to your link but the thing never sems to complete. I uploaded it to GDrive. Maybe can see it with this link:
https://drive.google.com/file/d/1ePOeUzoFD911MFNodwCZLY17gZdTpn6k/view?usp=drive_link
Let me know if that doesn't work. Thanks for looking.
-
Great I see that. Let's see if it reveals anything...
-
So today I got this to happen by restarting both FWs and the switch they connect to all at once. I think mostly when this happens its when I fire these up in the AM at which time everything gets started all together. After one of these things panics they seem to be pretty stable. I guess its somewhat rare anymore this happens just randomly throughout the day. Used to but not much anymore. Maybe this is something to do with the switch starting up doing something at startup. Guessing again but if true this might not be seen in my prod environment. But good to know whats going on here. Thanks again for looking.
-
@rfranzke said in Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7:
Panic Dump Link
That's the same core? The link looks the same.
-
@stephenw10 Its the same dump. I just made it into an actual link. I couldn't edit original post.
-
Ok the core dump yielded useful info but not enough to solve it unfortunately.
Are you able to load a debug kernel on this and get a core with that running?
If so grab the pkg here. Install it. Then:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/debug-kernel.html#booting-the-debug-kernel -
@stephenw10 Yes I can install this and run it but not sure the proper way to actually install the pkg file. Enable SSH, copy the file over, and then run it or is there some other method to do this via WebUI?
