pfsense crashes lately - how can i analyze logs?

AlexanderK

My pfsense lately it crashes. I have all crash logs. Is there any guide on how i can analyse the crash report?

stephenw10

Check the panic and the backtrace.

Check redmine for existing reports.

Or paste them here so others can check-over it.

Steve

lewistorphy

@AlexanderK

@AlexanderK said in geometry lite pfsense crashes lately - how can i analyze logs?:

My pfsense lately it crashes. I have all crash logs. Is there any guide on how i can analyse the crash report?

Locate the Crash Report:
Crash reports are typically stored in the /var/log/messages file. You can access this file using a console or SSH connection to your pfSense firewall.
If your pfSense is configured to generate crash dumps, these will be stored in the /var/crash directory.

Gather Additional Information:
Review other system logs like /var/log/kern.log and /var/log/messages for related error messages or warnings.
Monitor network traffic to identify any unusual patterns or spikes that might have contributed to the crash.
Check hardware monitoring tools to ensure that your system's CPU, memory, and storage are operating within normal parameters.

stephenw10

If there is a crash report generated after a kernel panic it should be presented as an alert on the pfSense dashboard. Or it can be accessed in /var/crash.

AlexanderK

@stephenw10
again i have some crashes.
And the log

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x30000000028
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff81164fa3
stack pointer = 0x0:0xfffffe012f290c40
frame pointer = 0x0:0xfffffe012f290c50
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 50766 (sysctl)
rdi: fffff8051bc69478 rsi: 000000000000000c rdx: 000000000000000c
rcx: 0000000000000000 r8: 0000000000183335 r9: 000063a4e52b8000
rax: fffffe0115e9f740 rbx: 0000030000000000 rbp: fffffe012f290c50
r10: 00001238c9cd7000 r11: 00001238c9cd7000 r12: 0000045e03d10000
r13: fffff8024ea639a0 r14: 0000000000000002 r15: 0000000000000000
trap number = 12
panic: page fault
cpuid = 0
time = 1734127277
KDB: enter: panic

can you help me?

stephenw10

Do you have the full crash report?

You can upload it here so I can check it: https://nc.netgate.com/nextcloud/s/fpRokRoTPfjoHKN

AlexanderK

@stephenw10 i have uploaded them. thanks in advance

stephenw10

Hmm, two completely different crashes there:

db:0:kdb.enter.default>  show pcpu
cpuid        = 0
dynamic pcpu = 0x1170f80
curthread    = 0xfffffe0115e9f740: pid 50766 tid 100296 critnest 2 "sysctl"
curpcb       = 0xfffffe0115e9fc60
fpcurthread  = 0xfffffe0115e9f740: pid 50766 "sysctl"
idlethread   = 0xfffffe0038bb13a0: tid 100003 "idle: cpu0"
self         = 0xffffffff84010000
curpmap      = 0xfffff8024ea63ad0
tssp         = 0xffffffff84010384
rsp0         = 0xfffffe012f291000
kcr3         = 0xffffffffffffffff
ucr3         = 0xffffffffffffffff
scr3         = 0x0
gs32p        = 0xffffffff84010404
ldt          = 0xffffffff84010444
tss          = 0xffffffff84010434
curvnet      = 0
db:0:kdb.enter.default>  bt
Tracing pid 50766 tid 100296 td 0xfffffe0115e9f740
kdb_enter() at kdb_enter+0x32/frame 0xfffffe012f290920
vpanic() at vpanic+0x163/frame 0xfffffe012f290a50
panic() at panic+0x43/frame 0xfffffe012f290ab0
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe012f290b10
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe012f290b70
calltrap() at calltrap+0x8/frame 0xfffffe012f290b70
--- trap 0xc, rip = 0xffffffff81164fa3, rsp = 0xfffffe012f290c40, rbp = 0xfffffe012f290c50 ---
vm_radix_lookup_unlocked() at vm_radix_lookup_unlocked+0x63/frame 0xfffffe012f290c50
vm_fault() at vm_fault+0x8ba/frame 0xfffffe012f290d60
vm_fault_trap() at vm_fault_trap+0x6b/frame 0xfffffe012f290db0
trap_pfault() at trap_pfault+0x1d9/frame 0xfffffe012f290e10
trap() at trap+0x442/frame 0xfffffe012f290f30
calltrap() at calltrap+0x8/frame 0xfffffe012f290f30
--- trap 0xc, rip = 0x45e03c85e3e, rsp = 0x45e02b76350, rbp = 0x45e02b763e0 ---

and

db:0:kdb.enter.default>  show pcpu
cpuid        = 1
dynamic pcpu = 0xfffffe00b5be6f80
curthread    = 0xfffffe01344e73a0: pid 55230 tid 101286 critnest 1 "snort"
curpcb       = 0xfffffe01344e78c0
fpcurthread  = 0xfffffe01344e73a0: pid 55230 "snort"
idlethread   = 0xfffffe0038bb0c80: tid 100004 "idle: cpu1"
self         = 0xffffffff84011000
curpmap      = 0xfffff8002037f868
tssp         = 0xffffffff84011384
rsp0         = 0xfffffe012f3e7000
kcr3         = 0xffffffffffffffff
ucr3         = 0xffffffffffffffff
scr3         = 0x0
gs32p        = 0xffffffff84011404
ldt          = 0xffffffff84011444
tss          = 0xffffffff84011434
curvnet      = 0
db:0:kdb.enter.default>  bt
Tracing pid 55230 tid 101286 td 0xfffffe01344e73a0
kdb_enter() at kdb_enter+0x32/frame 0xfffffe012f3e62b0
vpanic() at vpanic+0x163/frame 0xfffffe012f3e63e0
panic() at panic+0x43/frame 0xfffffe012f3e6440
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe012f3e64a0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe012f3e6500
calltrap() at calltrap+0x8/frame 0xfffffe012f3e6500
--- trap 0xc, rip = 0xffffffff81280d34, rsp = 0xfffffe012f3e65d0, rbp = 0xfffffe012f3e65d0 ---
pmap_pvh_remove() at pmap_pvh_remove+0x4/frame 0xfffffe012f3e65d0
pmap_enter() at pmap_enter+0xc84/frame 0xfffffe012f3e66a0
vm_fault() at vm_fault+0xbf4/frame 0xfffffe012f3e67b0
core_output() at core_output+0xf0/frame 0xfffffe012f3e6820
elf64_coredump() at elf64_coredump+0x576/frame 0xfffffe012f3e68f0
sigexit() at sigexit+0xbd5/frame 0xfffffe012f3e6d60
postsig() at postsig+0x237/frame 0xfffffe012f3e6e20
ast_sig() at ast_sig+0x1d7/frame 0xfffffe012f3e6ed0
ast_handler() at ast_handler+0x88/frame 0xfffffe012f3e6f10
ast() at ast+0x20/frame 0xfffffe012f3e6f30
doreti_ast() at doreti_ast+0x1c/frame 0x82134def0

That second one is associated with a Snort coredump. Do you have the current Snort package installed?

Have you seen more crashes? Are they also different? Numerous different crashes are usually a hardware issue.

That aside it looks like you have Snort, Suricata and Zeek installed and you should only ever use one of those.

You have some invalid sysctl settings:

<118>Setting up extended sysctls...sysctl: oid 'net.isr.maxthreads' is a read only tunable
<118>sysctl: Tunable values are set in /boot/loader.conf
<118>sysctl: oid 'net.isr.numthreads' is read only
<118>sysctl: oid 'net.isr.maxthreads' is a read only tunable
<118>sysctl: Tunable values are set in /boot/loader.conf
<118>sysctl: oid 'net.isr.numthreads' is read only

Those are loader tunables as it shows there.

AlexanderK

@stephenw10 i have all of them installed (snort, zeek, suricata) but none of them activated simultaneously with each other. Just for testing. I will remove them and keep only one.

For the tunables i have them fir wireguard tweaking - found somewhere.
Thanks for the analysis of my crash logs.

AlexanderK

@stephenw10 while removing packages - suricata system crashed again...

stephenw10

Same crash or a new different one?

If it's different again I would run a ram test.

AlexanderK

@stephenw10 said in pfsense crashes lately - how can i analyze logs?:

Same crash or a new different one?

If it's different again I would run a ram test.

can i upload them?

stephenw10

Yes, same link should still work.

AlexanderK

@stephenw10 thanks again. uploaded them

stephenw10

Yup, two completely different crashes again. I would definitely do a memory test here as a next step. A software bug would not present such widely varying crashes.