pfsense crashes lately - how can i analyze logs?

AlexanderK

@stephenw10
again i have some crashes.
And the log

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x30000000028
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff81164fa3
stack pointer = 0x0:0xfffffe012f290c40
frame pointer = 0x0:0xfffffe012f290c50
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 50766 (sysctl)
rdi: fffff8051bc69478 rsi: 000000000000000c rdx: 000000000000000c
rcx: 0000000000000000 r8: 0000000000183335 r9: 000063a4e52b8000
rax: fffffe0115e9f740 rbx: 0000030000000000 rbp: fffffe012f290c50
r10: 00001238c9cd7000 r11: 00001238c9cd7000 r12: 0000045e03d10000
r13: fffff8024ea639a0 r14: 0000000000000002 r15: 0000000000000000
trap number = 12
panic: page fault
cpuid = 0
time = 1734127277
KDB: enter: panic

can you help me?

stephenw10

Do you have the full crash report?

You can upload it here so I can check it: https://nc.netgate.com/nextcloud/s/fpRokRoTPfjoHKN

AlexanderK

@stephenw10 i have uploaded them. thanks in advance

stephenw10

Hmm, two completely different crashes there:

db:0:kdb.enter.default>  show pcpu
cpuid        = 0
dynamic pcpu = 0x1170f80
curthread    = 0xfffffe0115e9f740: pid 50766 tid 100296 critnest 2 "sysctl"
curpcb       = 0xfffffe0115e9fc60
fpcurthread  = 0xfffffe0115e9f740: pid 50766 "sysctl"
idlethread   = 0xfffffe0038bb13a0: tid 100003 "idle: cpu0"
self         = 0xffffffff84010000
curpmap      = 0xfffff8024ea63ad0
tssp         = 0xffffffff84010384
rsp0         = 0xfffffe012f291000
kcr3         = 0xffffffffffffffff
ucr3         = 0xffffffffffffffff
scr3         = 0x0
gs32p        = 0xffffffff84010404
ldt          = 0xffffffff84010444
tss          = 0xffffffff84010434
curvnet      = 0
db:0:kdb.enter.default>  bt
Tracing pid 50766 tid 100296 td 0xfffffe0115e9f740
kdb_enter() at kdb_enter+0x32/frame 0xfffffe012f290920
vpanic() at vpanic+0x163/frame 0xfffffe012f290a50
panic() at panic+0x43/frame 0xfffffe012f290ab0
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe012f290b10
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe012f290b70
calltrap() at calltrap+0x8/frame 0xfffffe012f290b70
--- trap 0xc, rip = 0xffffffff81164fa3, rsp = 0xfffffe012f290c40, rbp = 0xfffffe012f290c50 ---
vm_radix_lookup_unlocked() at vm_radix_lookup_unlocked+0x63/frame 0xfffffe012f290c50
vm_fault() at vm_fault+0x8ba/frame 0xfffffe012f290d60
vm_fault_trap() at vm_fault_trap+0x6b/frame 0xfffffe012f290db0
trap_pfault() at trap_pfault+0x1d9/frame 0xfffffe012f290e10
trap() at trap+0x442/frame 0xfffffe012f290f30
calltrap() at calltrap+0x8/frame 0xfffffe012f290f30
--- trap 0xc, rip = 0x45e03c85e3e, rsp = 0x45e02b76350, rbp = 0x45e02b763e0 ---

and

db:0:kdb.enter.default>  show pcpu
cpuid        = 1
dynamic pcpu = 0xfffffe00b5be6f80
curthread    = 0xfffffe01344e73a0: pid 55230 tid 101286 critnest 1 "snort"
curpcb       = 0xfffffe01344e78c0
fpcurthread  = 0xfffffe01344e73a0: pid 55230 "snort"
idlethread   = 0xfffffe0038bb0c80: tid 100004 "idle: cpu1"
self         = 0xffffffff84011000
curpmap      = 0xfffff8002037f868
tssp         = 0xffffffff84011384
rsp0         = 0xfffffe012f3e7000
kcr3         = 0xffffffffffffffff
ucr3         = 0xffffffffffffffff
scr3         = 0x0
gs32p        = 0xffffffff84011404
ldt          = 0xffffffff84011444
tss          = 0xffffffff84011434
curvnet      = 0
db:0:kdb.enter.default>  bt
Tracing pid 55230 tid 101286 td 0xfffffe01344e73a0
kdb_enter() at kdb_enter+0x32/frame 0xfffffe012f3e62b0
vpanic() at vpanic+0x163/frame 0xfffffe012f3e63e0
panic() at panic+0x43/frame 0xfffffe012f3e6440
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe012f3e64a0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe012f3e6500
calltrap() at calltrap+0x8/frame 0xfffffe012f3e6500
--- trap 0xc, rip = 0xffffffff81280d34, rsp = 0xfffffe012f3e65d0, rbp = 0xfffffe012f3e65d0 ---
pmap_pvh_remove() at pmap_pvh_remove+0x4/frame 0xfffffe012f3e65d0
pmap_enter() at pmap_enter+0xc84/frame 0xfffffe012f3e66a0
vm_fault() at vm_fault+0xbf4/frame 0xfffffe012f3e67b0
core_output() at core_output+0xf0/frame 0xfffffe012f3e6820
elf64_coredump() at elf64_coredump+0x576/frame 0xfffffe012f3e68f0
sigexit() at sigexit+0xbd5/frame 0xfffffe012f3e6d60
postsig() at postsig+0x237/frame 0xfffffe012f3e6e20
ast_sig() at ast_sig+0x1d7/frame 0xfffffe012f3e6ed0
ast_handler() at ast_handler+0x88/frame 0xfffffe012f3e6f10
ast() at ast+0x20/frame 0xfffffe012f3e6f30
doreti_ast() at doreti_ast+0x1c/frame 0x82134def0

That second one is associated with a Snort coredump. Do you have the current Snort package installed?

Have you seen more crashes? Are they also different? Numerous different crashes are usually a hardware issue.

That aside it looks like you have Snort, Suricata and Zeek installed and you should only ever use one of those.

You have some invalid sysctl settings:

<118>Setting up extended sysctls...sysctl: oid 'net.isr.maxthreads' is a read only tunable
<118>sysctl: Tunable values are set in /boot/loader.conf
<118>sysctl: oid 'net.isr.numthreads' is read only
<118>sysctl: oid 'net.isr.maxthreads' is a read only tunable
<118>sysctl: Tunable values are set in /boot/loader.conf
<118>sysctl: oid 'net.isr.numthreads' is read only

Those are loader tunables as it shows there.

AlexanderK

@stephenw10 i have all of them installed (snort, zeek, suricata) but none of them activated simultaneously with each other. Just for testing. I will remove them and keep only one.

For the tunables i have them fir wireguard tweaking - found somewhere.
Thanks for the analysis of my crash logs.

AlexanderK

@stephenw10 while removing packages - suricata system crashed again...

stephenw10

Same crash or a new different one?

If it's different again I would run a ram test.

AlexanderK

@stephenw10 said in pfsense crashes lately - how can i analyze logs?:

Same crash or a new different one?

If it's different again I would run a ram test.

can i upload them?

stephenw10

Yes, same link should still work.

AlexanderK

@stephenw10 thanks again. uploaded them

stephenw10

Yup, two completely different crashes again. I would definitely do a memory test here as a next step. A software bug would not present such widely varying crashes.