Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfSense Crash diagnose

    Scheduled Pinned Locked Moved General pfSense Questions
    22 Posts 2 Posters 2.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      aschmi
      last edited by

      Hi this is Adrian, and I installed many pfSense box (netgate)
      Now I have a little problem at home and is driving my nuts. My pfSense (free version) started to randomly reboot. I already changed the disk because previous one (solid state) was giving errors, but now even with that, it does randomly reboots.
      Here is my crashdump
      Help please!
      Thank you so much in advance

      A 1 Reply Last reply Reply Quote 0
      • A
        aschmi @aschmi
        last edited by

        @aschmi

        Fatal trap 12: page fault while in kernel mode
        cpuid = 3; apic id = 06
        fault virtual address = 0x28
        fault code = supervisor read data, page not present
        instruction pointer = 0x20:0xffffffff80ec01fe
        stack pointer = 0x28:0xfffffe00747f2950
        frame pointer = 0x28:0xfffffe00747f2980
        code segment = base 0x0, limit 0xfffff, type 0x1b
        = DPL 0, pres 1, long 1, def32 0, gran 1
        processor eflags = interrupt enabled, resume, IOPL = 0
        current process = 12 (swi4: clock (0))
        trap number = 12
        panic: page fault
        cpuid = 3
        time = 1629423140
        KDB: enter: panic
        panic.txt0600001214107603044 7130 ustarrootwheelpage faultversion.txt0600006714107603044 7535 ustarrootwheelFreeBSD 12.2-STABLE fd0f54f44b5c(RELENG_2_5_0) pfSense

        1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          Can we see the full crash report? We need to see at least the backtrace. The console output leading up to it is usually helpful too.

          Steve

          A 1 Reply Last reply Reply Quote 0
          • A
            aschmi @stephenw10
            last edited by

            @stephenw10 thank you for your willingness to help!
            I have these files

            drwxr-xr-x 2 aschmi users 4.0K Aug 20 09:09 .
            drwxr-xr-x 8 aschmi users 12K Aug 19 19:42 ..
            -rw------- 1 aschmi users 6.2K Aug 19 18:47 config.txt
            -rw------- 1 aschmi users 48K Aug 19 18:47 ddb.txt
            -rw------- 1 aschmi users 83K Aug 19 18:47 msgbuf.txt
            -rw------- 1 aschmi users 10 Aug 19 18:47 panic.txt
            -rw-r--r-- 1 aschmi users 141K Aug 20 09:08 textdump.tar
            -rw------- 1 aschmi users 55 Aug 19 18:47 version.txt

            Wich one can it help? How do I upload them here?
            Thanks!!!

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              The msgbuf and ddb files are where I would normally check. That is normally in the compressed crash file though which you should be able to upload.
              It does contain a lot of details from you network though, WAN IP address etc.

              Steve

              1 Reply Last reply Reply Quote 0
              • A
                aschmi
                last edited by

                ddb.txt msgbuf.txt
                Thank you so much!

                stephenw10S 1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator @aschmi
                  last edited by

                  OK, so the important part of that is this:

                  db:0:kdb.enter.default>  show pcpu
                  cpuid        = 2
                  dynamic pcpu = 0xfffffe0080d71380
                  curthread    = 0xfffff80005740000: pid 12 tid 100035 "swi4: clock (0)"
                  curpcb       = 0xfffff800057405a0
                  fpcurthread  = none
                  idlethread   = 0xfffff80005622000: tid 100005 "idle: cpu2"
                  curpmap      = 0xffffffff8368d5a8
                  tssp         = 0xffffffff837176f0
                  commontssp   = 0xffffffff837176f0
                  rsp0         = 0xfffffe00747f2cc0
                  kcr3         = 0x80000000040cb002
                  ucr3         = 0xffffffffffffffff
                  scr3         = 0x10b690d52
                  gs32p        = 0xffffffff8371df08
                  ldt          = 0xffffffff8371df48
                  tss          = 0xffffffff8371df38
                  tlb gen      = 130953
                  curvnet      = 0
                  db:0:kdb.enter.default>  bt
                  Tracing pid 12 tid 100035 td 0xfffff80005740000
                  kdb_enter() at kdb_enter+0x37/frame 0xfffffe00747f2610
                  vpanic() at vpanic+0x197/frame 0xfffffe00747f2660
                  panic() at panic+0x43/frame 0xfffffe00747f26c0
                  trap_fatal() at trap_fatal+0x391/frame 0xfffffe00747f2720
                  trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00747f2770
                  trap() at trap+0x286/frame 0xfffffe00747f2880
                  calltrap() at calltrap+0x8/frame 0xfffffe00747f2880
                  --- trap 0xc, rip = 0xffffffff80ec01fe, rsp = 0xfffffe00747f2950, rbp = 0xfffffe00747f2980 ---
                  ether_8021q_frame() at ether_8021q_frame+0x2e/frame 0xfffffe00747f2980
                  vlan_transmit() at vlan_transmit+0xc8/frame 0xfffffe00747f29f0
                  vlan_altq_start() at vlan_altq_start+0xb4/frame 0xfffffe00747f2a20
                  cbqrestart() at cbqrestart+0x64/frame 0xfffffe00747f2a50
                  rmc_restart() at rmc_restart+0x6f/frame 0xfffffe00747f2a80
                  softclock_call_cc() at softclock_call_cc+0x141/frame 0xfffffe00747f2b30
                  softclock() at softclock+0x79/frame 0xfffffe00747f2b50
                  ithread_loop() at ithread_loop+0x23c/frame 0xfffffe00747f2bb0
                  fork_exit() at fork_exit+0x7e/frame 0xfffffe00747f2bf0
                  fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00747f2bf0
                  --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
                  

                  It looks like on that last boot where us then crashed you has added a new VLAN interface:
                  vlan3: changing name to 'igb1.2'
                  Is there traffic shaping on that VLAN? Using CBQ?
                  Is it different to the other VLANs?

                  This looks pretty much exactly like this:
                  https://forum.netgate.com/topic/165827/random-crash
                  That was resolved removing the shaping from the VLAN.

                  Steve

                  A 1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Yeah, looks to be this: https://redmine.pfsense.org/issues/11470

                    1 Reply Last reply Reply Quote 0
                    • A
                      aschmi @stephenw10
                      last edited by

                      @stephenw10 Hi yes I do have CBQ traffic, in all the VLAN interfaces. The vlan3 is for "guest" so has a lower speed than the others.
                      This seems to be crashing when I'm heavily using that VLAN (on some guest machines that I was running Windows updates on)
                      Should I remove completely the traffic shapping, or is the problem with just CBQ?
                      Good eye I can still not see how you discovered the issue with CBQ and VLANS but great catch!
                      Adrian

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        The last lines in the backtrace before it panics point to it, read in reverse order:

                        ether_8021q_frame() at ether_8021q_frame+0x2e/frame 0xfffffe00747f2980
                        vlan_transmit() at vlan_transmit+0xc8/frame 0xfffffe00747f29f0
                        vlan_altq_start() at vlan_altq_start+0xb4/frame 0xfffffe00747f2a20
                        cbqrestart() at cbqrestart+0x64/frame 0xfffffe00747f2a50
                        

                        That bug report confirms it.

                        Can you share any details of how you have it configured?
                        I'm trying to replicate it here but failing. It's stubbornly not crashing!

                        Steve

                        A 1 Reply Last reply Reply Quote 0
                        • A
                          aschmi @stephenw10
                          last edited by

                          @stephenw10 Hi Stephen how do I send you the config file in a safe way? What do I need to sanitize? I do have radius on my server, should I remove that part of the xml? And what about the admin password is that stored on the xml backup?
                          I have one WAN interface, and one intel with 4 nics for the internal part, and is an old Dell desktop (those mini ones) that I can give you more specs, 16Gb of RAM. I had a ssd that failed before, but now it was working with a normal drive (but again failing)
                          The interesting thing is that this was only failing when that network that was being shaped was under heavy usage (probably reaching the limit of CBQ?)
                          Thanks

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by

                            We probably don't need the full config but I can PM you something.

                            Just the details of VLAN in question and the shaper queues that were defined on it.

                            Steve

                            A 1 Reply Last reply Reply Quote 0
                            • A
                              aschmi @stephenw10
                              last edited by

                              @stephenw10 Sorry I was sure I sent you the config! Just to let you know that I reenabled QOS on that interface, and had a crash again... here is the config (what I think should be relevant)
                              (attached tried to post here but got rejected by anti spam)config.xml

                              Thank you so much for your time!!! And let me know If I need to send you anything else

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                igb1.2 is not in that config. Which interface did you enable it on to cause the crash?

                                Using the same queues that are on the other interfaces?

                                Steve

                                A 1 Reply Last reply Reply Quote 0
                                • A
                                  aschmi @stephenw10
                                  last edited by

                                  @stephenw10 Sorry I tried to copy/paste but didn't work too well. Here is my actual config

                                  c2ef1c5a-fda3-4b86-97a2-0250b750e692-image.png

                                  I have QOS disabled now so it doesn't crash

                                  Thanks again!

                                  stephenw10S 1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator @aschmi
                                    last edited by

                                    I was able to get the config OK it's just that it doesn't have the VLAN igb1.2 in it and it looks like that is what caused the crash.
                                    Can I assume you had that configured the same as the other VLANs? Same traffic shaping queues on it?

                                    Steve

                                    A 1 Reply Last reply Reply Quote 0
                                    • A
                                      aschmi @stephenw10
                                      last edited by

                                      @stephenw10 Good you noticed and yes I was using the VLAN2 on igb1 (second interface on a 4 port 1Gbps interface) but not on my normal interfaces

                                      b48cca15-9dfb-4a16-afbc-4f81fbe3c541-image.png

                                      I can send you the whole .XML if you tell me how to sanitize not to send usernames/passwords if that will help

                                      1 Reply Last reply Reply Quote 0
                                      • stephenw10S
                                        stephenw10 Netgate Administrator
                                        last edited by

                                        What I have should be enough if the queues were the same.

                                        However you can get the sanitised config as part of the status file by going directly to:
                                        [your firewall IP]/status.php

                                        Nothing special about igb1? igb0 is also a port on the same card?

                                        Steve

                                        A 1 Reply Last reply Reply Quote 0
                                        • A
                                          aschmi @stephenw10
                                          last edited by

                                          @stephenw10 status_output.tgz
                                          There you have it. Yes igbo, 1, 2 and 3 is just one card (supposedly intel but bought on ebay)
                                          I was trying to use igb1.2 as a bridge with igb0.2, but crashes happened before I started playing with that configuration

                                          1 Reply Last reply Reply Quote 0
                                          • stephenw10S
                                            stephenw10 Netgate Administrator
                                            last edited by

                                            OK cool, so you had not added the bridge when it crashed?

                                            A 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.