Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfsense crashes lately - how can i analyze logs?

    Scheduled Pinned Locked Moved
    General pfSense Questions
    3
    15
    338
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      AlexanderK
      last edited by

      My pfsense lately it crashes. I have all crash logs. Is there any guide on how i can analyse the crash report?

      L 1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Check the panic and the backtrace.

        Check redmine for existing reports.

        Or paste them here so others can check-over it.

        Steve

        1 Reply Last reply Reply Quote 1
        • L
          lewistorphy @AlexanderK
          last edited by

          @AlexanderK

          @AlexanderK said in geometry lite pfsense crashes lately - how can i analyze logs?:

          My pfsense lately it crashes. I have all crash logs. Is there any guide on how i can analyse the crash report?

          Locate the Crash Report:
          Crash reports are typically stored in the /var/log/messages file. You can access this file using a console or SSH connection to your pfSense firewall.
          If your pfSense is configured to generate crash dumps, these will be stored in the /var/crash directory.

          Gather Additional Information:
          Review other system logs like /var/log/kern.log and /var/log/messages for related error messages or warnings.
          Monitor network traffic to identify any unusual patterns or spikes that might have contributed to the crash.
          Check hardware monitoring tools to ensure that your system's CPU, memory, and storage are operating within normal parameters.

          1 Reply Last reply Reply Quote 1
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            If there is a crash report generated after a kernel panic it should be presented as an alert on the pfSense dashboard. Or it can be accessed in /var/crash.

            A 1 Reply Last reply Reply Quote 0
            • A
              AlexanderK @stephenw10
              last edited by

              @stephenw10
              again i have some crashes.
              And the log

              Fatal trap 12: page fault while in kernel mode
              cpuid = 0; apic id = 00
              fault virtual address = 0x30000000028
              fault code = supervisor read data, page not present
              instruction pointer = 0x20:0xffffffff81164fa3
              stack pointer = 0x0:0xfffffe012f290c40
              frame pointer = 0x0:0xfffffe012f290c50
              code segment = base 0x0, limit 0xfffff, type 0x1b
              = DPL 0, pres 1, long 1, def32 0, gran 1
              processor eflags = interrupt enabled, resume, IOPL = 0
              current process = 50766 (sysctl)
              rdi: fffff8051bc69478 rsi: 000000000000000c rdx: 000000000000000c
              rcx: 0000000000000000 r8: 0000000000183335 r9: 000063a4e52b8000
              rax: fffffe0115e9f740 rbx: 0000030000000000 rbp: fffffe012f290c50
              r10: 00001238c9cd7000 r11: 00001238c9cd7000 r12: 0000045e03d10000
              r13: fffff8024ea639a0 r14: 0000000000000002 r15: 0000000000000000
              trap number = 12
              panic: page fault
              cpuid = 0
              time = 1734127277
              KDB: enter: panic

              can you help me?

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                Do you have the full crash report?

                You can upload it here so I can check it: https://nc.netgate.com/nextcloud/s/fpRokRoTPfjoHKN

                A 1 Reply Last reply Reply Quote 1
                • A
                  AlexanderK @stephenw10
                  last edited by

                  @stephenw10 i have uploaded them. thanks in advance

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Hmm, two completely different crashes there:

                    db:0:kdb.enter.default>  show pcpu
cpuid        = 0
dynamic pcpu = 0x1170f80
curthread    = 0xfffffe0115e9f740: pid 50766 tid 100296 critnest 2 "sysctl"
curpcb       = 0xfffffe0115e9fc60
fpcurthread  = 0xfffffe0115e9f740: pid 50766 "sysctl"
idlethread   = 0xfffffe0038bb13a0: tid 100003 "idle: cpu0"
self         = 0xffffffff84010000
curpmap      = 0xfffff8024ea63ad0
tssp         = 0xffffffff84010384
rsp0         = 0xfffffe012f291000
kcr3         = 0xffffffffffffffff
ucr3         = 0xffffffffffffffff
scr3         = 0x0
gs32p        = 0xffffffff84010404
ldt          = 0xffffffff84010444
tss          = 0xffffffff84010434
curvnet      = 0
db:0:kdb.enter.default>  bt
Tracing pid 50766 tid 100296 td 0xfffffe0115e9f740
kdb_enter() at kdb_enter+0x32/frame 0xfffffe012f290920
vpanic() at vpanic+0x163/frame 0xfffffe012f290a50
panic() at panic+0x43/frame 0xfffffe012f290ab0
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe012f290b10
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe012f290b70
calltrap() at calltrap+0x8/frame 0xfffffe012f290b70
--- trap 0xc, rip = 0xffffffff81164fa3, rsp = 0xfffffe012f290c40, rbp = 0xfffffe012f290c50 ---
vm_radix_lookup_unlocked() at vm_radix_lookup_unlocked+0x63/frame 0xfffffe012f290c50
vm_fault() at vm_fault+0x8ba/frame 0xfffffe012f290d60
vm_fault_trap() at vm_fault_trap+0x6b/frame 0xfffffe012f290db0
trap_pfault() at trap_pfault+0x1d9/frame 0xfffffe012f290e10
trap() at trap+0x442/frame 0xfffffe012f290f30
calltrap() at calltrap+0x8/frame 0xfffffe012f290f30
--- trap 0xc, rip = 0x45e03c85e3e, rsp = 0x45e02b76350, rbp = 0x45e02b763e0 ---

                    and

                    db:0:kdb.enter.default>  show pcpu
cpuid        = 1
dynamic pcpu = 0xfffffe00b5be6f80
curthread    = 0xfffffe01344e73a0: pid 55230 tid 101286 critnest 1 "snort"
curpcb       = 0xfffffe01344e78c0
fpcurthread  = 0xfffffe01344e73a0: pid 55230 "snort"
idlethread   = 0xfffffe0038bb0c80: tid 100004 "idle: cpu1"
self         = 0xffffffff84011000
curpmap      = 0xfffff8002037f868
tssp         = 0xffffffff84011384
rsp0         = 0xfffffe012f3e7000
kcr3         = 0xffffffffffffffff
ucr3         = 0xffffffffffffffff
scr3         = 0x0
gs32p        = 0xffffffff84011404
ldt          = 0xffffffff84011444
tss          = 0xffffffff84011434
curvnet      = 0
db:0:kdb.enter.default>  bt
Tracing pid 55230 tid 101286 td 0xfffffe01344e73a0
kdb_enter() at kdb_enter+0x32/frame 0xfffffe012f3e62b0
vpanic() at vpanic+0x163/frame 0xfffffe012f3e63e0
panic() at panic+0x43/frame 0xfffffe012f3e6440
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe012f3e64a0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe012f3e6500
calltrap() at calltrap+0x8/frame 0xfffffe012f3e6500
--- trap 0xc, rip = 0xffffffff81280d34, rsp = 0xfffffe012f3e65d0, rbp = 0xfffffe012f3e65d0 ---
pmap_pvh_remove() at pmap_pvh_remove+0x4/frame 0xfffffe012f3e65d0
pmap_enter() at pmap_enter+0xc84/frame 0xfffffe012f3e66a0
vm_fault() at vm_fault+0xbf4/frame 0xfffffe012f3e67b0
core_output() at core_output+0xf0/frame 0xfffffe012f3e6820
elf64_coredump() at elf64_coredump+0x576/frame 0xfffffe012f3e68f0
sigexit() at sigexit+0xbd5/frame 0xfffffe012f3e6d60
postsig() at postsig+0x237/frame 0xfffffe012f3e6e20
ast_sig() at ast_sig+0x1d7/frame 0xfffffe012f3e6ed0
ast_handler() at ast_handler+0x88/frame 0xfffffe012f3e6f10
ast() at ast+0x20/frame 0xfffffe012f3e6f30
doreti_ast() at doreti_ast+0x1c/frame 0x82134def0

                    That second one is associated with a Snort coredump. Do you have the current Snort package installed?

                    Have you seen more crashes? Are they also different? Numerous different crashes are usually a hardware issue.

                    That aside it looks like you have Snort, Suricata and Zeek installed and you should only ever use one of those.

                    You have some invalid sysctl settings:

                    <118>Setting up extended sysctls...sysctl: oid 'net.isr.maxthreads' is a read only tunable
<118>sysctl: Tunable values are set in /boot/loader.conf
<118>sysctl: oid 'net.isr.numthreads' is read only
<118>sysctl: oid 'net.isr.maxthreads' is a read only tunable
<118>sysctl: Tunable values are set in /boot/loader.conf
<118>sysctl: oid 'net.isr.numthreads' is read only

                    Those are loader tunables as it shows there.

                    A 2 Replies Last reply Reply Quote 1
                    • A
                      AlexanderK @stephenw10
                      last edited by

                      @stephenw10 i have all of them installed (snort, zeek, suricata) but none of them activated simultaneously with each other. Just for testing. I will remove them and keep only one.

                      For the tunables i have them fir wireguard tweaking - found somewhere.
                      Thanks for the analysis of my crash logs.

                      1 Reply Last reply Reply Quote 0
                      • A
                        AlexanderK @stephenw10
                        last edited by

                        @stephenw10 while removing packages - suricata system crashed again...

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          Same crash or a new different one?

                          If it's different again I would run a ram test.

                          A 1 Reply Last reply Reply Quote 0
                          • A
                            AlexanderK @stephenw10
                            last edited by

                            @stephenw10 said in pfsense crashes lately - how can i analyze logs?:

                            Same crash or a new different one?

                            If it's different again I would run a ram test.

                            can i upload them?

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              Yes, same link should still work.

                              A 1 Reply Last reply Reply Quote 0
                              • A
                                AlexanderK @stephenw10
                                last edited by

                                @stephenw10 thanks again. uploaded them

                                1 Reply Last reply Reply Quote 0
                                • stephenw10S
                                  stephenw10 Netgate Administrator
                                  last edited by

                                  Yup, two completely different crashes again. I would definitely do a memory test here as a next step. A software bug would not present such widely varying crashes.

                                  1 Reply Last reply Reply Quote 2
                                  • First post
                                    Last post