Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfsense crashes lately - how can i analyze logs?

    Scheduled Pinned Locked Moved General pfSense Questions
    15 Posts 3 Posters 1.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator
      last edited by

      Check the panic and the backtrace.

      Check redmine for existing reports.

      Or paste them here so others can check-over it.

      Steve

      1 Reply Last reply Reply Quote 1
      • L
        lewistorphy @AlexanderK
        last edited by

        @AlexanderK

        @AlexanderK said in geometry lite pfsense crashes lately - how can i analyze logs?:

        My pfsense lately it crashes. I have all crash logs. Is there any guide on how i can analyse the crash report?

        Locate the Crash Report:
        Crash reports are typically stored in the /var/log/messages file. You can access this file using a console or SSH connection to your pfSense firewall.
        If your pfSense is configured to generate crash dumps, these will be stored in the /var/crash directory.

        Gather Additional Information:
        Review other system logs like /var/log/kern.log and /var/log/messages for related error messages or warnings.
        Monitor network traffic to identify any unusual patterns or spikes that might have contributed to the crash.
        Check hardware monitoring tools to ensure that your system's CPU, memory, and storage are operating within normal parameters.

        1 Reply Last reply Reply Quote 1
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          If there is a crash report generated after a kernel panic it should be presented as an alert on the pfSense dashboard. Or it can be accessed in /var/crash.

          A 1 Reply Last reply Reply Quote 0
          • A
            AlexanderK @stephenw10
            last edited by

            @stephenw10
            again i have some crashes.
            And the log

            Fatal trap 12: page fault while in kernel mode
            cpuid = 0; apic id = 00
            fault virtual address = 0x30000000028
            fault code = supervisor read data, page not present
            instruction pointer = 0x20:0xffffffff81164fa3
            stack pointer = 0x0:0xfffffe012f290c40
            frame pointer = 0x0:0xfffffe012f290c50
            code segment = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, long 1, def32 0, gran 1
            processor eflags = interrupt enabled, resume, IOPL = 0
            current process = 50766 (sysctl)
            rdi: fffff8051bc69478 rsi: 000000000000000c rdx: 000000000000000c
            rcx: 0000000000000000 r8: 0000000000183335 r9: 000063a4e52b8000
            rax: fffffe0115e9f740 rbx: 0000030000000000 rbp: fffffe012f290c50
            r10: 00001238c9cd7000 r11: 00001238c9cd7000 r12: 0000045e03d10000
            r13: fffff8024ea639a0 r14: 0000000000000002 r15: 0000000000000000
            trap number = 12
            panic: page fault
            cpuid = 0
            time = 1734127277
            KDB: enter: panic

            can you help me?

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              Do you have the full crash report?

              You can upload it here so I can check it: https://nc.netgate.com/nextcloud/s/fpRokRoTPfjoHKN

              A 1 Reply Last reply Reply Quote 1
              • A
                AlexanderK @stephenw10
                last edited by

                @stephenw10 i have uploaded them. thanks in advance

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  Hmm, two completely different crashes there:

                  db:0:kdb.enter.default>  show pcpu
cpuid        = 0
dynamic pcpu = 0x1170f80
curthread    = 0xfffffe0115e9f740: pid 50766 tid 100296 critnest 2 "sysctl"
curpcb       = 0xfffffe0115e9fc60
fpcurthread  = 0xfffffe0115e9f740: pid 50766 "sysctl"
idlethread   = 0xfffffe0038bb13a0: tid 100003 "idle: cpu0"
self         = 0xffffffff84010000
curpmap      = 0xfffff8024ea63ad0
tssp         = 0xffffffff84010384
rsp0         = 0xfffffe012f291000
kcr3         = 0xffffffffffffffff
ucr3         = 0xffffffffffffffff
scr3         = 0x0
gs32p        = 0xffffffff84010404
ldt          = 0xffffffff84010444
tss          = 0xffffffff84010434
curvnet      = 0
db:0:kdb.enter.default>  bt
Tracing pid 50766 tid 100296 td 0xfffffe0115e9f740
kdb_enter() at kdb_enter+0x32/frame 0xfffffe012f290920
vpanic() at vpanic+0x163/frame 0xfffffe012f290a50
panic() at panic+0x43/frame 0xfffffe012f290ab0
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe012f290b10
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe012f290b70
calltrap() at calltrap+0x8/frame 0xfffffe012f290b70
--- trap 0xc, rip = 0xffffffff81164fa3, rsp = 0xfffffe012f290c40, rbp = 0xfffffe012f290c50 ---
vm_radix_lookup_unlocked() at vm_radix_lookup_unlocked+0x63/frame 0xfffffe012f290c50
vm_fault() at vm_fault+0x8ba/frame 0xfffffe012f290d60
vm_fault_trap() at vm_fault_trap+0x6b/frame 0xfffffe012f290db0
trap_pfault() at trap_pfault+0x1d9/frame 0xfffffe012f290e10
trap() at trap+0x442/frame 0xfffffe012f290f30
calltrap() at calltrap+0x8/frame 0xfffffe012f290f30
--- trap 0xc, rip = 0x45e03c85e3e, rsp = 0x45e02b76350, rbp = 0x45e02b763e0 ---

                  and

                  db:0:kdb.enter.default>  show pcpu
cpuid        = 1
dynamic pcpu = 0xfffffe00b5be6f80
curthread    = 0xfffffe01344e73a0: pid 55230 tid 101286 critnest 1 "snort"
curpcb       = 0xfffffe01344e78c0
fpcurthread  = 0xfffffe01344e73a0: pid 55230 "snort"
idlethread   = 0xfffffe0038bb0c80: tid 100004 "idle: cpu1"
self         = 0xffffffff84011000
curpmap      = 0xfffff8002037f868
tssp         = 0xffffffff84011384
rsp0         = 0xfffffe012f3e7000
kcr3         = 0xffffffffffffffff
ucr3         = 0xffffffffffffffff
scr3         = 0x0
gs32p        = 0xffffffff84011404
ldt          = 0xffffffff84011444
tss          = 0xffffffff84011434
curvnet      = 0
db:0:kdb.enter.default>  bt
Tracing pid 55230 tid 101286 td 0xfffffe01344e73a0
kdb_enter() at kdb_enter+0x32/frame 0xfffffe012f3e62b0
vpanic() at vpanic+0x163/frame 0xfffffe012f3e63e0
panic() at panic+0x43/frame 0xfffffe012f3e6440
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe012f3e64a0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe012f3e6500
calltrap() at calltrap+0x8/frame 0xfffffe012f3e6500
--- trap 0xc, rip = 0xffffffff81280d34, rsp = 0xfffffe012f3e65d0, rbp = 0xfffffe012f3e65d0 ---
pmap_pvh_remove() at pmap_pvh_remove+0x4/frame 0xfffffe012f3e65d0
pmap_enter() at pmap_enter+0xc84/frame 0xfffffe012f3e66a0
vm_fault() at vm_fault+0xbf4/frame 0xfffffe012f3e67b0
core_output() at core_output+0xf0/frame 0xfffffe012f3e6820
elf64_coredump() at elf64_coredump+0x576/frame 0xfffffe012f3e68f0
sigexit() at sigexit+0xbd5/frame 0xfffffe012f3e6d60
postsig() at postsig+0x237/frame 0xfffffe012f3e6e20
ast_sig() at ast_sig+0x1d7/frame 0xfffffe012f3e6ed0
ast_handler() at ast_handler+0x88/frame 0xfffffe012f3e6f10
ast() at ast+0x20/frame 0xfffffe012f3e6f30
doreti_ast() at doreti_ast+0x1c/frame 0x82134def0

                  That second one is associated with a Snort coredump. Do you have the current Snort package installed?

                  Have you seen more crashes? Are they also different? Numerous different crashes are usually a hardware issue.

                  That aside it looks like you have Snort, Suricata and Zeek installed and you should only ever use one of those.

                  You have some invalid sysctl settings:

                  <118>Setting up extended sysctls...sysctl: oid 'net.isr.maxthreads' is a read only tunable
<118>sysctl: Tunable values are set in /boot/loader.conf
<118>sysctl: oid 'net.isr.numthreads' is read only
<118>sysctl: oid 'net.isr.maxthreads' is a read only tunable
<118>sysctl: Tunable values are set in /boot/loader.conf
<118>sysctl: oid 'net.isr.numthreads' is read only

                  Those are loader tunables as it shows there.

                  A 2 Replies Last reply Reply Quote 1
                  • A
                    AlexanderK @stephenw10
                    last edited by

                    @stephenw10 i have all of them installed (snort, zeek, suricata) but none of them activated simultaneously with each other. Just for testing. I will remove them and keep only one.

                    For the tunables i have them fir wireguard tweaking - found somewhere.
                    Thanks for the analysis of my crash logs.

                    1 Reply Last reply Reply Quote 0
                    • A
                      AlexanderK @stephenw10
                      last edited by

                      @stephenw10 while removing packages - suricata system crashed again...

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        Same crash or a new different one?

                        If it's different again I would run a ram test.

                        A 1 Reply Last reply Reply Quote 0
                        • A
                          AlexanderK @stephenw10
                          last edited by

                          @stephenw10 said in pfsense crashes lately - how can i analyze logs?:

                          Same crash or a new different one?

                          If it's different again I would run a ram test.

                          can i upload them?

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by

                            Yes, same link should still work.

                            A 1 Reply Last reply Reply Quote 0
                            • A
                              AlexanderK @stephenw10
                              last edited by

                              @stephenw10 thanks again. uploaded them

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                Yup, two completely different crashes again. I would definitely do a memory test here as a next step. A software bug would not present such widely varying crashes.

                                1 Reply Last reply Reply Quote 2
                                • First post
                                  Last post
                                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.