Help with crashdump
-
Hello,
today my pfSense crashed after months without problems, but I can't find the reason why. I hope someone can help me with the crashdump. The pfSense is running in a cluster on a Proxmox Hypervisor.
Thanks in advance,
Martin -
2.4.4p2 is quite badly out of date.
Should be addressed and has previously caused panics:
[zone: pf frag entries] PF frag entries limit reachedActual backtrace:
db:0:kdb.enter.default> show pcpu cpuid = 15 dynamic pcpu = 0xfffffe08bb68d480 curthread = 0xfffff801ec96b620: pid 17352 "dpinger" curpcb = 0xfffffe0861585cc0 fpcurthread = 0xfffff801ec96b620: pid 17352 "dpinger" idlethread = 0xfffff8000b3f5620: tid 100018 "idle: cpu15" curpmap = 0xfffff8004a0fb138 tssp = 0xffffffff82bb4d28 commontssp = 0xffffffff82bb4d28 rsp0 = 0xfffffe0861585cc0 gs32p = 0xffffffff82bbb580 ldt = 0xffffffff82bbb5c0 tss = 0xffffffff82bbb5b0 db:0:kdb.enter.default> bt Tracing pid 17352 tid 101295 td 0xfffff801ec96b620 ip_output() at ip_output+0x1433/frame 0xfffffe0861585820 rip_output() at rip_output+0x2c1/frame 0xfffffe08615858c0 sosend_generic() at sosend_generic+0x556/frame 0xfffffe0861585970 kern_sendit() at kern_sendit+0x1f9/frame 0xfffffe0861585a20 sendit() at sendit+0x19e/frame 0xfffffe0861585a70 sys_sendto() at sys_sendto+0x4d/frame 0xfffffe0861585ac0 amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe0861585bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0861585bf0 --- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x800b4317a, rsp = 0x7fffdfdfcf38, rbp = 0x7fffdfdfcf80 ---Something in ip_output like that is probably this if you have IPSec configured.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246951That is fixed in 2.5.X. Or rather FreeBSD 12 is not affected.
Steve
-
First ever pfSense crash today for me, also on Proxmox.
From info.0:
Dump header from device: /dev/label/swap0 Architecture: amd64 Architecture Version: 4 Dump Length: 88576 Blocksize: 512 Compression: none Dumptime: Tue Jun 22 13:08:02 2021 Hostname: firewall.<redacted>.com Magic: FreeBSD Text Dump Version String: FreeBSD 12.2-STABLE 1b709158e581(RELENG_2_5_0) pfSense Panic String: page fault Dump Parity: 2268264720 Bounds: 0 Dump Status: goodEnd of textdump.tar.0:
Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x12d fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff8109273b stack pointer = 0x28:0xfffffe00005c6c60 frame pointer = 0x28:0xfffffe00005c6c80 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 18 (pf purge) trap number = 12 panic: page fault cpuid = 0 time = 1624381682 KDB: enter: panic panic.txt 0600 0 0 12 14064414362 7135 ustar root wheel page fault version.txt 0600 0 0 67 14064414362 7542 ustar root wheel FreeBSD 12.2-STABLE 1b709158e581(RELENG_2_5_0) pfSenseWhen I went to shell it was at
db>. I restarted and it booted up without issues. -
We need the pcpu and bt data from ddb.txt to know more.
-
@stephenw10 said in Help with crashdump:
ddb.txt
Here are the pcpu and bt sections:
db:0:kdb.enter.default> show pcpu cpuid = 0 dynamic pcpu = 0x993380 curthread = 0xfffff800045fb000: pid 18 tid 100085 "pf purge" curpcb = 0xfffff800045fb5a0 fpcurthread = none idlethread = 0xfffff80004169000: tid 100003 "idle: cpu0" curpmap = 0xffffffff8368d5a8 tssp = 0xffffffff83717620 commontssp = 0xffffffff83717620 rsp0 = 0xfffffe00005c6e00 kcr3 = 0xffffffffffffffff ucr3 = 0xffffffffffffffff scr3 = 0x0 gs32p = 0xffffffff8371de38 ldt = 0xffffffff8371de78 tss = 0xffffffff8371de68 tlb gen = 1306259 curvnet = 0xfffff8000406de80 db:0:kdb.enter.default> bt Tracing pid 18 tid 100085 td 0xfffff800045fb000 kdb_enter() at kdb_enter+0x37/frame 0xfffffe00005c6920 vpanic() at vpanic+0x197/frame 0xfffffe00005c6970 panic() at panic+0x43/frame 0xfffffe00005c69d0 trap_fatal() at trap_fatal+0x391/frame 0xfffffe00005c6a30 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00005c6a80 trap() at trap+0x286/frame 0xfffffe00005c6b90 calltrap() at calltrap+0x8/frame 0xfffffe00005c6b90 --- trap 0xc, rip = 0xffffffff8109273b, rsp = 0xfffffe00005c6c60, rbp = 0xfffffe00005c6c80 --- pf_state_expires() at pf_state_expires+0xb/frame 0xfffffe00005c6c80 pf_purge_expired_states() at pf_purge_expired_states+0xd5/frame 0xfffffe00005c6cc0 pf_purge_thread() at pf_purge_thread+0x10f/frame 0xfffffe00005c6cf0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00005c6d30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005c6d30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- -
That's in 2.5.1?
-
@stephenw10 Correct, 2.5.1
-
There were significant changes in pf after 2.5.1 which will be in 2.5.2, especially in state handling. That panic is unlikely to still happen on 2.5.2, or at least would have a different backtrace which would be more helpful.
