Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7
-
@netblues said in Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7:
@SteveITS said in Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7:
Generally, yes, but per the docs pf may not sync states correctly between FreeBSD versions:
pfsync tries very hard to be compatible between versions too, but bugs do happen.
I agree, however 2.7.2 and 2.8 are on the same freebsd version, so it's safe to do so.
They are not.
They may both say "15", but they're not the same "15". -
@kprovost Still, minor versions/differences.
No one would do that long term, but considering the situation above, that's the least of problems too.
-
Hmm, well that's a completely different backtrace.
First Panic:
db:1:pfs> bt Tracing pid 2 tid 100058 td 0xfffff8006c1a5740 kdb_enter() at kdb_enter+0x33/frame 0xfffffe015852db10 panic() at panic+0x43/frame 0xfffffe015852db70 trap_fatal() at trap_fatal+0x40b/frame 0xfffffe015852dbd0 trap_pfault() at trap_pfault+0x46/frame 0xfffffe015852dc20 calltrap() at calltrap+0x8/frame 0xfffffe015852dc20 --- trap 0xc, rip = 0xffffffff80cf2042, rsp = 0xfffffe015852dcf0, rbp = 0xfffffe015852dd90 --- __rw_wlock_hard() at __rw_wlock_hard+0x152/frame 0xfffffe015852dd90 arptimer() at arptimer+0x252/frame 0xfffffe015852de10 softclock_call_cc() at softclock_call_cc+0x16d/frame 0xfffffe015852dec0 softclock_thread() at softclock_thread+0xe5/frame 0xfffffe015852def0 fork_exit() at fork_exit+0x7b/frame 0xfffffe015852df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe015852df30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf ---2nd Panic:
db:1:pfs> bt Tracing pid 11 tid 100003 td 0xfffff8006c179740 kdb_enter() at kdb_enter+0x33/frame 0xfffffe015840b9c0 panic() at panic+0x43/frame 0xfffffe015840ba20 trap_fatal() at trap_fatal+0x40b/frame 0xfffffe015840ba80 trap_pfault() at trap_pfault+0x46/frame 0xfffffe015840bad0 calltrap() at calltrap+0x8/frame 0xfffffe015840bad0 --- trap 0xc, rip = 0xffffffff80d15bdd, rsp = 0xfffffe015840bba0, rbp = 0xfffffe015840bc00 --- callout_process() at callout_process+0x1ad/frame 0xfffffe015840bc00 handleevents() at handleevents+0x186/frame 0xfffffe015840bc40 timercb() at timercb+0x236/frame 0xfffffe015840bc90 lapic_handle_timer() at lapic_handle_timer+0xab/frame 0xfffffe015840bcb0 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe015840bcb0 --- interrupt, rip = 0xffffffff804eb162, rsp = 0xfffffe015840bd80, rbp = 0xfffffe015840bdb0 --- acpi_cpu_idle() at acpi_cpu_idle+0x2e2/frame 0xfffffe015840bdb0 cpu_idle_acpi() at cpu_idle_acpi+0x46/frame 0xfffffe015840bdd0 cpu_idle() at cpu_idle+0x9d/frame 0xfffffe015840bdf0 sched_idletd() at sched_idletd+0x546/frame 0xfffffe015840bef0 fork_exit() at fork_exit+0x7b/frame 0xfffffe015840bf30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe015840bf30 --- trap 0xafafafaf, rip = 0xafafafafafafafaf, rsp = 0xafafafafafafafaf, rbp = 0xafafafafafafafaf ---I would try to compare more crashes if you can. Different and random backtraces like that usually points to a hardware issue. But that's from different version so it may simply be different there.
-
OK quick update here. I ran through the reinstall process on the backup firewall using the USB stick for install as well as using the USB config import on first boot. Seems to have worked mostly. None of the packages got re-installed however using this method so I re-installed them manually. Not sure why that did not work or happen.....maybe it just doesn't do that using this method. No matter I know the process now. Very cool it works this way and good to know so thanks for that heads up here. For those of you keeping track at home, I have re-installed the backup FW with a fresh USB stick using the 2.8.0 option. So I at the moment have a 2.8.1 install on the primary firewall and 2.8.0 running on the backup firewall. I'll see how this goes in terms of crash dumps for a bit and then upgrade the primary if all goes well. I wanted to do this to see if I can get on a proper latest stable build of CE. If the crashes continue I will have new dumps to post of the same OS. 2.8.0 is where I think I want to be as the changes in the new .1 beta did not help me here. But if nothing else its good for this N00B to run through the backup/restore process. Will seehowit goes. Thanks for the help all.
-
Hmm, the package should get reinstalled when importing a config. The only reason it wouldn't is if there was no access to the repo after the boot when the install is triggered. But you should see something logged if that happens.
I sometimes see that if the interface config is very different and that's still in progress. Restoring the config again after boot will correctly reinstall packages if that's the case. -
@stephenw10 Yes, I think this is exactly what happened here. After my last post I realized that the gateway I currently use to get to the Internet was not configured. I have a third link I use to get off net in my test environment. This config is for our data center environment and has IP address that do not exist here. So, I created a third DHCP interface to tie this into the actual LAN the boxes are currently on. I switch to using this interface as the gateway when I need them to be able to download PFBlockerNG updates, access Netgate Servers, etc. For some reason in my config I imported the GW was set to the normal GW which will work in my DC setup. Just doesn't work here. So, I had to manually switch to using the secondary gateway to download and install the missing packages.
I got impatient and went ahead and reinstalled the primary with 2.8.0 and restored the config there. This time I saw a message on first login that said it was re-installing the packages in the background. I switched to the opt1 interface gateway and all the packages were installed perfectly. Not sure why I had to much trouble with previous backup/restores but this works slick today.
So now I am running the HA pair both on fresh installs of 2.8.0 (not upgraded from 2.7.0). Will let this bake for today and see what we get. Will post any additional dumps I get here.
Thanks all for the help.
-
Well almost as soon as I hit 'Submit' on my last post, the backup firewall panicked again. ITs been running solid for several hours since the re-install. As soon as I get the primary up and running, something happened and all of a sudden crashed. Was just posting here so doing nothing on the FW. Just crash. Not sure I made this clear here but these are in my test environment. No traffic is going through them currently. Likely known but wanted to make sure folks know. They are sitting next to me on a test rack I have and can hear when the server restarts due to the fans spinning up.
Anyway here is the latest dump on 2.8.0:
-
Any idea if FreeBSD/PFSense 15 has issues with nvme drives. Thats what I am running on these systems. Drives I am running in these boxes:
Crucial P3 2TB PCIe Gen3 3D NAND NVMe M.2 SSD CT2000P3SSD8
Memory:
Samsung 64GB DDR4 PC4-21300 2666MHz LRDIMM Quad Ranked Registered ECC Memory (M386A8K40BM2-CTD)
-
Hmm, OK that's identical to the first crash. Which was also in 2.8.0. Was that actually the same device?
No there's no known issue with NVMe drives. We use them in our hardware.
-
@stephenw10 said in Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7:
Hmm, OK that's identical to the first crash. Which was also in 2.8.0. Was that actually the same device?
No there's no known issue with NVMe drives. We use them in our hardware.
Yes, same code, same device as the first dump post. It's the backup device in the CARP HA pair if that makes any difference. Both devices have the same exact hardware (CPU, MB, Disk, Mem, etc.).
Does the dump tell you anything specific as to what is happening, or just that it is the same as before. Should we be able to glean anything from this type of dump as to a specific cause, or do they more tell you that a crash occurred. Like a marker that something happened in case you are not around to witness it firsthand. These are sitting right next to me, so I am lucky enough to know when panics happen before even seeing the dump file. Are these dumps not capturing enough data to tell WHY the crash happened? Something I can tweak in the config to get additional info as to the cause?
Incidentally I got the one panic this morning and tried to get another today but this thing never panicked again. Go figure.
-
It doesn't mean anything to me. I can see it doesn't have much that's non-generic but I'll run it past some devs tomorrow.
Next step is either to enable a full core dump to analyse or try running the debug kernel.
https://docs.netgate.com/pfsense/en/latest/troubleshooting/debug-kernel.htmlDo you have SWAP enabled on those? How big is it?
To get a full core dump usually requires SWAP at least as large as the RAM to dump it to. -
@stephenw10 said in Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7:
It doesn't mean anything to me. I can see it doesn't have much that's non-generic but I'll run it past some devs tomorrow.
Next step is either to enable a full core dump to analyse or try running the debug kernel.
https://docs.netgate.com/pfsense/en/latest/troubleshooting/debug-kernel.htmlDo you have SWAP enabled on those? How big is it?
To get a full core dump usually requires SWAP at least as large as the RAM to dump it to.I'm not sure how one would go about "enabling swap". I didn't do anything to specifically enable it anywhere that I am aware of. Installed using the installer, imported my config from before. If it's not enabled by default, then I likely don't have it enabled. Both boxes have 64GB of RAM installed in them and 2 TB Nvme drives.
Thanks for the link. I'll see about loading up the debug kernel to see it reveals anything useful.
Thanks for checking with the devs here. Again really appreciate the help on this. -
It would be enabled by default but probably not at >64GB so dumping a full core to it may or may not be possible depending on how much RAM is actually in use. But first check how much SWAP there is. It's shown on the dashboard.
-
Looks like maybe SWAP is enabled but only to 1024MB.
Seem correct? Dark is primary, light is secondary and the one that's crashed the most.
-
Hmm. Unfortunately I think you'd almost certainly need to reinstall that with a more SWAP to be able to dump the full core.
-
@stephenw10 said in Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7:
Hmm. Unfortunately I think you'd almost certainly need to reinstall that with a more SWAP to be able to dump the full core.
Would the running debug kernel you propose avoid the need to reinstall to change swap to get what we would need, or would I need to change swap to get the dumps even with debug kernel in place? There is no way to change the amount of swap without reinstalling?
-
If you have any additional backtraces to compare that would be useful. Particularly from 2.8.1 to confirm you get the same thing there repeatedly.
-
The debug kernel doesn't require SWAP, so no reinstall, but it may not tell us more. It's worth trying though.
-
Mmm, looks like that first crash could be this: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285813
-
@stephenw10 said in Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7:
The debug kernel doesn't require SWAP, so no reinstall, but it may not tell us more. It's worth trying though.
If having proper swap in place to catch this is the sure-fire way to capture the relevant information needed to determine what this is, I'll work on that. I have the process of reinstalling down pretty good now.....not sure I know how to properly make swap adjustments needed to get this right but am willing to give it a go if it reveals something useful here.
Any guidance on what the swap size should be here? It doesn't look like I'm using a ton of memory and I think general FreeBSD guidance is twice the amount of memory in the system. Does that sound like a reasonable set up for this test?
