CRASH Report: My Netgate 7100 crashes every ten to fifteen days.
-
Backtrace:
db:1:pfs> bt Tracing pid 6 tid 100085 td 0xfffffe00f1fd8560 kdb_enter() at kdb_enter+0x32/frame 0xfffffe00d4f8eb40 vpanic() at vpanic+0x163/frame 0xfffffe00d4f8ec70 panic() at panic+0x43/frame 0xfffffe00d4f8ecd0 trap_fatal() at trap_fatal+0x40c/frame 0xfffffe00d4f8ed30 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00d4f8ed90 calltrap() at calltrap+0x8/frame 0xfffffe00d4f8ed90 --- trap 0xc, rip = 0xffffffff80faa59b, rsp = 0xfffffe00d4f8ee60, rbp = 0xfffffe00d4f8ee80 --- pf_state_expires() at pf_state_expires+0xb/frame 0xfffffe00d4f8ee80 pf_purge_expired_states() at pf_purge_expired_states+0xd5/frame 0xfffffe00d4f8eec0 pf_purge_thread() at pf_purge_thread+0x13b/frame 0xfffffe00d4f8eef0 fork_exit() at fork_exit+0x7f/frame 0xfffffe00d4f8ef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d4f8ef30 --- trap 0, rip = 0, rsp = 0, rbp = 0 ---Do you have the message buffer output from the crash?
It looks like this: https://redmine.pfsense.org/issues/13417
That should be fixed in current versions though. What pfSense version are you running?Steve
-
-
Ok, panic string:
Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 10 fault virtual address = 0x412 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80faa59b stack pointer = 0x28:0xfffffe00d4f8ee60 frame pointer = 0x28:0xfffffe00d4f8ee80 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 6 (pf purge) rdi: 0000000000000400 rsi: ffffffff83a10000 rdx: 0000000000000180 rcx: fffffe00f1fd8560 r8: 0000000000000004 r9: ffffffff83a13000 rax: 0000000000000000 rbx: 0000000000061a31 rbp: fffffe00d4f8ee80 r10: 000000000000000f r11: 00000000a33c0ad8 r12: fffffe00f84837c8 r13: 0000000000000400 r14: fffffe00f84837a8 r15: 0000000000004c2b trap number = 12 panic: page fault cpuid = 2 time = 1704229531 KDB: enter: panicLet me see....
-
Do you see the same backtrace and panic every time?
-
Yes
-
S stephenw10 forked this topic on
-
I'm wondering what my next step should be. Have you had a chance to look deeper into this? I saw that you had another crash that was not like mine so you forked the topic.
-
One of our devs is looking at it. We'll probably need to run some debug code to get more info. Are you able to do that?
Did it just start doing this? After an upgrade?
-
Yes, everything began as soon as I updated to the 23.09.1 version. I've been operating pfsense for at least 12 to 15 years without experiencing any crashes. Up until this point, pfsense has been flawless.
I'm willing to give it a shot with some debugging, so I'll wait to hear from a member of your development team. Should I keep the crash on my system or remove it. I have already downloaded the debugging data files.
-
So, it's not getting any better and it's definitely not going away.
Dumptime: 2024-01-02 16:05:31
Dumptime: 2024-01-07 17:14:39
Dumptime: 2024-01-08 22:11:09
Dumptime: 2024-01-09 07:40:00What do I need to do to elevate this so that I can get it figured out?
-
Are you able to boot the debug kernel?
https://docs.netgate.com/pfsense/en/latest/troubleshooting/debug-kernel.html -
Yes and standing by.
Just got home and my system was completely crashed. Had to press the pwr button to get it to reboot.
-
You already have it loaded and running that?
-
Yes, the debug kernel was installed and I just experienced another crash.
-
OK great, do you have a new crash report from that?
-
-
That isn't the debug kernel, it should show:
---<<BOOT>>--- Copyright (c) 1992-2023 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 14.0-CURRENT amd64 1400094 #1 plus-RELENG_23_09_1-n256200-3de1e293f3a: Wed Dec 6 21:01:42 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_09_1-main/obj/amd64/Obhu6gXB/var/jenkins/workspace/pfSense-Plus-snapshots-23_09_1-main/sources/FreeBSD-src-plus-RELENG_23_09_1/amd64.amd64/sys/pfSense-DEBUG amd64Did you add the loader value to make it boot the debug kernel every time?
-
No, I didn't at first, but it is now and I haven't experienced a crash since I have been on the debug kernel. It's been running for five days so far without a crash.
-
---<<BOOT>>---
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-CURRENT amd64 1400094 #1 plus-RELENG_23_09_1-n256200-3de1e293f3a: Wed Dec 6 21:01:42 UTC 2023
root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_09_1-main/obj/amd64/Obhu6gXB/var/jenkins/workspace/pfSense-Plus-snapshots-23_09_1-main/sources/FreeBSD-src-plus-RELENG_23_09_1/amd64.amd64/sys/pfSense-DEBUG amd64 -
Ok, cool so we're just waiting for a crash then?
-
It's been 31 days, and I have not experienced a crash since I went to the debug kernal.
