Random Crashes
-
Hello, I keep having random crashes. No idea where to start on the dump file but it seems to happen random times of day. Sometimes I have multiple in a day somedays none. Also updated to 2.7.2, no change, and reinstalled all packages.
Thanks!
-
@MolagBalls take a look at https://docs.netgate.com/pfsense/en/latest/troubleshooting/unexpected-reboots.html
-
@SteveITS I think i read this one last night. So if I cant find "mbuf, uma_zalloc_arg" like mentioned in the article in my crash report, I can assume its hardware at that point?
-
Backtrace:
db:0:kdb.enter.default> bt Tracing pid 43177 tid 100372 td 0xfffffe00c7876900 kdb_enter() at kdb_enter+0x32/frame 0xfffffe0010baedb0 vpanic() at vpanic+0x183/frame 0xfffffe0010baee00 panic() at panic+0x43/frame 0xfffffe0010baee60 dblfault_handler() at dblfault_handler+0x1ce/frame 0xfffffe0010baef20 Xdblfault() at Xdblfault+0xd7/frame 0xfffffe0010baef20 --- trap 0x17, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53c8ff0, rbp = 0xfffffe00c53c9100 --- trap() at trap+0x17/frame 0xfffffe00c53c9100 calltrap() at calltrap+0x8/frame 0xfffffe00c53c9100 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53c91d0, rbp = 0xfffffe00c53c92e0 --- trap() at trap+0x17/frame 0xfffffe00c53c92e0 calltrap() at calltrap+0x8/frame 0xfffffe00c53c92e0 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53c93b0, rbp = 0xfffffe00c53c94c0 --- trap() at trap+0x17/frame 0xfffffe00c53c94c0 calltrap() at calltrap+0x8/frame 0xfffffe00c53c94c0 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53c9590, rbp = 0xfffffe00c53c96a0 --- trap() at trap+0x17/frame 0xfffffe00c53c96a0 calltrap() at calltrap+0x8/frame 0xfffffe00c53c96a0 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53c9770, rbp = 0xfffffe00c53c9880 --- trap() at trap+0x17/frame 0xfffffe00c53c9880 calltrap() at calltrap+0x8/frame 0xfffffe00c53c9880 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53c9950, rbp = 0xfffffe00c53c9a60 --- trap() at trap+0x17/frame 0xfffffe00c53c9a60 calltrap() at calltrap+0x8/frame 0xfffffe00c53c9a60 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53c9b30, rbp = 0xfffffe00c53c9c40 --- trap() at trap+0x17/frame 0xfffffe00c53c9c40 calltrap() at calltrap+0x8/frame 0xfffffe00c53c9c40 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53c9d10, rbp = 0xfffffe00c53c9e20 --- trap() at trap+0x17/frame 0xfffffe00c53c9e20 calltrap() at calltrap+0x8/frame 0xfffffe00c53c9e20 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53c9ef0, rbp = 0xfffffe00c53ca000 --- trap() at trap+0x17/frame 0xfffffe00c53ca000 calltrap() at calltrap+0x8/frame 0xfffffe00c53ca000 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53ca0d0, rbp = 0xfffffe00c53ca1e0 --- trap() at trap+0x17/frame 0xfffffe00c53ca1e0 calltrap() at calltrap+0x8/frame 0xfffffe00c53ca1e0 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53ca2b0, rbp = 0xfffffe00c53ca3c0 --- trap() at trap+0x17/frame 0xfffffe00c53ca3c0 calltrap() at calltrap+0x8/frame 0xfffffe00c53ca3c0 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53ca490, rbp = 0xfffffe00c53ca5a0 --- trap() at trap+0x17/frame 0xfffffe00c53ca5a0 calltrap() at calltrap+0x8/frame 0xfffffe00c53ca5a0 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53ca670, rbp = 0xfffffe00c53ca780 --- trap() at trap+0x17/frame 0xfffffe00c53ca780 calltrap() at calltrap+0x8/frame 0xfffffe00c53ca780 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53ca850, rbp = 0xfffffe00c53ca960 --- trap() at trap+0x17/frame 0xfffffe00c53ca960 calltrap() at calltrap+0x8/frame 0xfffffe00c53ca960 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53caa30, rbp = 0xfffffe00c53cab40 --- trap() at trap+0x17/frame 0xfffffe00c53cab40 calltrap() at calltrap+0x8/frame 0xfffffe00c53cab40 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53cac10, rbp = 0xfffffe00c53cad20 --- trap() at trap+0x17/frame 0xfffffe00c53cad20 calltrap() at calltrap+0x8/frame 0xfffffe00c53cad20 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53cadf0, rbp = 0xfffffe00c53caf00 --- trap() at trap+0x17/frame 0xfffffe00c53caf00 calltrap() at calltrap+0x8/frame 0xfffffe00c53caf00 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53cafd0, rbp = 0xfffffe00c53cb0e0 --- trap() at trap+0x17/frame 0xfffffe00c53cb0e0 calltrap() at calltrap+0x8/frame 0xfffffe00c53cb0e0 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53cb1b0, rbp = 0xfffffe00c53cb2c0 --- trap() at trap+0x17/frame 0xfffffe00c53cb2c0 calltrap() at calltrap+0x8/frame 0xfffffe00c53cb2c0 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53cb390, rbp = 0xfffffe00c53cb4a0 --- trap() at trap+0x17/frame 0xfffffe00c53cb4a0 calltrap() at calltrap+0x8/frame 0xfffffe00c53cb4a0 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53cb570, rbp = 0xfffffe00c53cb680 --- trap() at trap+0x17/frame 0xfffffe00c53cb680 calltrap() at calltrap+0x8/frame 0xfffffe00c53cb680 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53cb750, rbp = 0xfffffe00c53cb860 --- trap() at trap+0x17/frame 0xfffffe00c53cb860 calltrap() at calltrap+0x8/frame 0xfffffe00c53cb860 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53cb930, rbp = 0xfffffe00c53cba40 --- trap() at trap+0x17/frame 0xfffffe00c53cba40 calltrap() at calltrap+0x8/frame 0xfffffe00c53cba40 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53cbb10, rbp = 0xfffffe00c53cbc20 --- trap() at trap+0x17/frame 0xfffffe00c53cbc20 calltrap() at calltrap+0x8/frame 0xfffffe00c53cbc20 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53cbcf0, rbp = 0xfffffe00c53cbe00 --- trap() at trap+0x17/frame 0xfffffe00c53cbe00 calltrap() at calltrap+0x8/frame 0xfffffe00c53cbe00 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53cbed0, rbp = 0xfffffe00c53cbfe0 --- trap() at trap+0x17/frame 0xfffffe00c53cbfe0 calltrap() at calltrap+0x8/frame 0xfffffe00c53cbfe0 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53cc0b0, rbp = 0xfffffe00c53cc1c0 --- trap() at trap+0x17/frame 0xfffffe00c53cc1c0 calltrap() at calltrap+0x8/frame 0xfffffe00c53cc1c0 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53cc290, rbp = 0xfffffe00c53cc3a0 --- trap() at trap+0x17/frame 0xfffffe00c53cc3a0 calltrap() at calltrap+0x8/frame 0xfffffe00c53cc3a0 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53cc470, rbp = 0xfffffe00c53cc580 --- trap() at trap+0x17/frame 0xfffffe00c53cc580 calltrap() at calltrap+0x8/frame 0xfffffe00c53cc580 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53cc650, rbp = 0xfffffe00c53cc760 --- trap() at trap+0x17/frame 0xfffffe00c53cc760 calltrap() at calltrap+0x8/frame 0xfffffe00c53cc760 --- trap 0xc, rip = 0xffffffff81299c17, rsp = 0xfffffe00c53cc830, rbp = 0xfffffe00c53cc940 --- trap() at trap+0x17/frame 0xfffffe00c53cc940 calltrap() at calltrap+0x8/frame 0xfffffe00c53cc940 --- trap 0xc, rip = 0xffffffff81271a53, rsp = 0xfffffe00c53cca10, rbp = 0xfffffe00c53ccbc0 --- Xfast_syscall_pti() at Xfast_syscall_pti+0x3/frame 0xfffffe00c53ccbc0 sosend_generic() at sosend_generic+0x609/frame 0xfffffe00c53ccc70 sousrsend() at sousrsend+0x5f/frame 0xfffffe00c53cccd0 kern_sendit() at kern_sendit+0x132/frame 0xfffffe00c53ccd60 sendit() at sendit+0xb7/frame 0xfffffe00c53ccdb0 sys_sendto() at sys_sendto+0x4d/frame 0xfffffe00c53cce00 amd64_syscall() at amd64_syscall+0x109/frame 0xfffffe00c53ccf30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00c53ccf30 --- syscall (133, FreeBSD ELF64, sendto), rip = 0x274a208f1e3a, rsp = 0x274a26c6bf28, rbp = 0x274a26c6bf70 ---
Panic 1:
<118>pfSense 2.7.0-RELEASE amd64 Wed Jun 28 03:53:34 UTC 2023 <118>Bootup complete Fatal trap 1: privileged instruction fault while in kernel mode cpuid = 3; apic id = 06 instruction pointer = 0x20:0xffffffff80a89a39 stack pointer = 0x28:0xfffffe0083d8ae50 frame pointer = 0x28:0xfffffe0083d8ae90 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 16 (usbus0) rdi: fffffe008f018428 rsi: 7e rdx: fffffe008f018bb8 rcx: ffffffff r8: fffffe0083e0b700 r9: fffffe0083d8b000 rax: 7f rbx: 2 rbp: fffffe0083d8ae90 r10: 0 r11: 84a982c5 r12: fffffe008f018548 r13: 80 r14: fffff8000474f000 r15: fffffe008f018428 trap number = 1 panic: privileged instruction fault cpuid = 3 time = 1705193616 KDB: enter: panic
Panic 2:
<6>ovpnc2: link state changed to DOWN <6>ovpnc2: link state changed to UP Fatal trap 1: privileged instruction fault while in kernel mode cpuid = 3; apic id = 06 instruction pointer = 0x20:0xffffffff80a89a39 stack pointer = 0x28:0xfffffe0083d8ae50 frame pointer = 0x28:0xfffffe0083d8ae90 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 16 (usbus0) rdi: fffffe008f018428 rsi: 7e rdx: fffffe008f018bb8 rcx: ffffffff r8: fffffe0083e0b700 r9: fffffe0083d8b000 rax: 7f rbx: 2 rbp: fffffe0083d8ae90 r10: 0 r11: 83b34a67 r12: fffffe008f018548 r13: 80 r14: fffff8000473c000 r15: fffffe008f018428 trap number = 1 panic: privileged instruction fault cpuid = 3 time = 1705256344 KDB: enter: panic
Panic 3:
<118>Bootup complete <6>arp: 192.168.69.17 moved from 8e:cb:1c:f6:96:02 to 1c:91:80:34:7a:d9 on igb1.69 Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 06 fault virtual address = 0x8011 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80fae50b stack pointer = 0x28:0xfffffe008fcb9e50 frame pointer = 0x28:0xfffffe008fcb9e70 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 7 (pf purge) rdi: 8000 rsi: 4 rdx: 180 rcx: fffffe0090d71c80 r8: ffffffff83e10000 r9: 3 rax: 0 rbx: 8000 rbp: fffffe008fcb9e70 r10: 3 r11: 819a8437 r12: fffffe00916b75d8 r13: ddbf r14: fffffe00916b75d8 r15: fffffe00916b75f8 trap number = 12 panic: page fault cpuid = 2 time = 1705283893 KDB: enter: panic
Panic 4:
<118>Bootup complete config_aqm Unable to configure flowset, flowset busy! config_aqm Unable to configure flowset, flowset busy! Fatal trap 1: privileged instruction fault while in kernel mode cpuid = 2; apic id = 04 instruction pointer = 0x20:0xffffffff80a89a39 stack pointer = 0x28:0xfffffe0083d8ae50 frame pointer = 0x28:0xfffffe0083d8ae90 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 16 (usbus0) rdi: fffffe008f018428 rsi: 7e rdx: fffffe008f018bb8 rcx: ffffffff r8: fffffe0083e0b700 r9: fffffe0083d8b000 rax: 7f rbx: 2 rbp: fffffe0083d8ae90 r10: 0 r11: 80fed958 r12: fffffe008f018548 r13: 80 r14: fffff80004755000 r15: fffffe008f018428 trap number = 1 panic: privileged instruction fault cpuid = 2 time = 1705301241 KDB: enter: panic
Panic 5:
<118>Bootup complete (ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 28 10 73 67 40 06 00 00 00 00 00 (ada0:ahcich0:0:0:0): CAM status: Uncorrectable parity/CRC error (ada0:ahcich0:0:0:0): Retrying command, 3 more tries remain (ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 50 e0 b5 ec 40 0e 00 00 00 00 00 (ada0:ahcich0:0:0:0): CAM status: Uncorrectable parity/CRC error (ada0:ahcich0:0:0:0): Retrying command, 3 more tries remain config_aqm Unable to configure flowset, flowset busy! config_aqm Unable to configure flowset, flowset busy! <6>pid 89445 (php), jid 0, uid 0: exited on signal 11 (core dumped) config_aqm Unable to configure flowset, flowset busy! config_aqm Unable to configure flowset, flowset busy! Fatal double fault rip 0xffffffff81299c17 rsp 0xfffffe00c53c8ff0 rbp 0xfffffe00c53c9100 rax 0 rdx 0xfffff801d47c6e00 rbx 0xfffffe00c53c9110 rcx 0xffffffff80f7351a rsi 0xfffffe00c53c9110 rdi 0 r8 0 r9 0xfffffe00c7876900 r10 0x141e69bfe r11 0x246 r12 0x9 r13 0xfffff801d3912aa0 r14 0xfffff80040592b40 r15 0x2f rflags 0x10086 cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b fsbase 0x274a259c5120 gsbase 0xffffffff83e11000 kgsbase 0xffffffff83e11000 cpuid = 1; apic id = 02 panic: double fault cpuid = 1 time = 1705398983 KDB: enter: panic
That last panic, which is where the backtrace is from, looks like a drive error.
Steve
-
@stephenw10 thank you! I went ahead and put a new device in service for now. Going to try formatting and reinstalling the original one and let it sit and run to see if it does it again. If it does then it will get a new drive and put in place as the backup.