pfsense crashes lately - how can i analyze logs?
-
If there is a crash report generated after a kernel panic it should be presented as an alert on the pfSense dashboard. Or it can be accessed in /var/crash.
-
@stephenw10
again i have some crashes.
And the logFatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x30000000028
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff81164fa3
stack pointer = 0x0:0xfffffe012f290c40
frame pointer = 0x0:0xfffffe012f290c50
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 50766 (sysctl)
rdi: fffff8051bc69478 rsi: 000000000000000c rdx: 000000000000000c
rcx: 0000000000000000 r8: 0000000000183335 r9: 000063a4e52b8000
rax: fffffe0115e9f740 rbx: 0000030000000000 rbp: fffffe012f290c50
r10: 00001238c9cd7000 r11: 00001238c9cd7000 r12: 0000045e03d10000
r13: fffff8024ea639a0 r14: 0000000000000002 r15: 0000000000000000
trap number = 12
panic: page fault
cpuid = 0
time = 1734127277
KDB: enter: paniccan you help me?
-
Do you have the full crash report?
You can upload it here so I can check it: https://nc.netgate.com/nextcloud/s/fpRokRoTPfjoHKN
-
@stephenw10 i have uploaded them. thanks in advance
-
Hmm, two completely different crashes there:
db:0:kdb.enter.default> show pcpu cpuid = 0 dynamic pcpu = 0x1170f80 curthread = 0xfffffe0115e9f740: pid 50766 tid 100296 critnest 2 "sysctl" curpcb = 0xfffffe0115e9fc60 fpcurthread = 0xfffffe0115e9f740: pid 50766 "sysctl" idlethread = 0xfffffe0038bb13a0: tid 100003 "idle: cpu0" self = 0xffffffff84010000 curpmap = 0xfffff8024ea63ad0 tssp = 0xffffffff84010384 rsp0 = 0xfffffe012f291000 kcr3 = 0xffffffffffffffff ucr3 = 0xffffffffffffffff scr3 = 0x0 gs32p = 0xffffffff84010404 ldt = 0xffffffff84010444 tss = 0xffffffff84010434 curvnet = 0 db:0:kdb.enter.default> bt Tracing pid 50766 tid 100296 td 0xfffffe0115e9f740 kdb_enter() at kdb_enter+0x32/frame 0xfffffe012f290920 vpanic() at vpanic+0x163/frame 0xfffffe012f290a50 panic() at panic+0x43/frame 0xfffffe012f290ab0 trap_fatal() at trap_fatal+0x40c/frame 0xfffffe012f290b10 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe012f290b70 calltrap() at calltrap+0x8/frame 0xfffffe012f290b70 --- trap 0xc, rip = 0xffffffff81164fa3, rsp = 0xfffffe012f290c40, rbp = 0xfffffe012f290c50 --- vm_radix_lookup_unlocked() at vm_radix_lookup_unlocked+0x63/frame 0xfffffe012f290c50 vm_fault() at vm_fault+0x8ba/frame 0xfffffe012f290d60 vm_fault_trap() at vm_fault_trap+0x6b/frame 0xfffffe012f290db0 trap_pfault() at trap_pfault+0x1d9/frame 0xfffffe012f290e10 trap() at trap+0x442/frame 0xfffffe012f290f30 calltrap() at calltrap+0x8/frame 0xfffffe012f290f30 --- trap 0xc, rip = 0x45e03c85e3e, rsp = 0x45e02b76350, rbp = 0x45e02b763e0 ---and
db:0:kdb.enter.default> show pcpu cpuid = 1 dynamic pcpu = 0xfffffe00b5be6f80 curthread = 0xfffffe01344e73a0: pid 55230 tid 101286 critnest 1 "snort" curpcb = 0xfffffe01344e78c0 fpcurthread = 0xfffffe01344e73a0: pid 55230 "snort" idlethread = 0xfffffe0038bb0c80: tid 100004 "idle: cpu1" self = 0xffffffff84011000 curpmap = 0xfffff8002037f868 tssp = 0xffffffff84011384 rsp0 = 0xfffffe012f3e7000 kcr3 = 0xffffffffffffffff ucr3 = 0xffffffffffffffff scr3 = 0x0 gs32p = 0xffffffff84011404 ldt = 0xffffffff84011444 tss = 0xffffffff84011434 curvnet = 0 db:0:kdb.enter.default> bt Tracing pid 55230 tid 101286 td 0xfffffe01344e73a0 kdb_enter() at kdb_enter+0x32/frame 0xfffffe012f3e62b0 vpanic() at vpanic+0x163/frame 0xfffffe012f3e63e0 panic() at panic+0x43/frame 0xfffffe012f3e6440 trap_fatal() at trap_fatal+0x40c/frame 0xfffffe012f3e64a0 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe012f3e6500 calltrap() at calltrap+0x8/frame 0xfffffe012f3e6500 --- trap 0xc, rip = 0xffffffff81280d34, rsp = 0xfffffe012f3e65d0, rbp = 0xfffffe012f3e65d0 --- pmap_pvh_remove() at pmap_pvh_remove+0x4/frame 0xfffffe012f3e65d0 pmap_enter() at pmap_enter+0xc84/frame 0xfffffe012f3e66a0 vm_fault() at vm_fault+0xbf4/frame 0xfffffe012f3e67b0 core_output() at core_output+0xf0/frame 0xfffffe012f3e6820 elf64_coredump() at elf64_coredump+0x576/frame 0xfffffe012f3e68f0 sigexit() at sigexit+0xbd5/frame 0xfffffe012f3e6d60 postsig() at postsig+0x237/frame 0xfffffe012f3e6e20 ast_sig() at ast_sig+0x1d7/frame 0xfffffe012f3e6ed0 ast_handler() at ast_handler+0x88/frame 0xfffffe012f3e6f10 ast() at ast+0x20/frame 0xfffffe012f3e6f30 doreti_ast() at doreti_ast+0x1c/frame 0x82134def0That second one is associated with a Snort coredump. Do you have the current Snort package installed?
Have you seen more crashes? Are they also different? Numerous different crashes are usually a hardware issue.
That aside it looks like you have Snort, Suricata and Zeek installed and you should only ever use one of those.
You have some invalid sysctl settings:
<118>Setting up extended sysctls...sysctl: oid 'net.isr.maxthreads' is a read only tunable <118>sysctl: Tunable values are set in /boot/loader.conf <118>sysctl: oid 'net.isr.numthreads' is read only <118>sysctl: oid 'net.isr.maxthreads' is a read only tunable <118>sysctl: Tunable values are set in /boot/loader.conf <118>sysctl: oid 'net.isr.numthreads' is read onlyThose are loader tunables as it shows there.
-
@stephenw10 i have all of them installed (snort, zeek, suricata) but none of them activated simultaneously with each other. Just for testing. I will remove them and keep only one.
For the tunables i have them fir wireguard tweaking - found somewhere.
Thanks for the analysis of my crash logs. -
@stephenw10 while removing packages - suricata system crashed again...
-
Same crash or a new different one?
If it's different again I would run a ram test.
-
@stephenw10 said in pfsense crashes lately - how can i analyze logs?:
Same crash or a new different one?
If it's different again I would run a ram test.
can i upload them?
-
Yes, same link should still work.
-
@stephenw10 thanks again. uploaded them
-
Yup, two completely different crashes again. I would definitely do a memory test here as a next step. A software bug would not present such widely varying crashes.
