Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Kernel Panic

    Scheduled Pinned Locked Moved 2.0-RC Snapshot Feedback and Problems - RETIRED
    325 Posts 35 Posters 248.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jimpJ
      jimp Rebel Alliance Developer Netgate
      last edited by

      @LostInIgnorance:

      EDIT: Did you try it with squid havp, etc. loaded on it?

      I haven't tried it with squid, I'll load that on next. Do you have the OpenVPN server assigned as an interface and squid listening there? Otherwise, squid wouldn't be doing anything to the traffic coming from OpenVPN

      Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

      Need help fast? Netgate Global Support!

      Do not Chat/PM for help!

      1 Reply Last reply Reply Quote 0
      • L
        LostInIgnorance
        last edited by

        No I am not having squid listen to anything but the 3 different vr networks.  I am just wondering if the setup may have something to do with it.  Doesn't hurt to recreate it completely.

        1 Reply Last reply Reply Quote 0
        • jimpJ
          jimp Rebel Alliance Developer Netgate
          last edited by

          fair enough. I'll give it a go.

          Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

          Need help fast? Netgate Global Support!

          Do not Chat/PM for help!

          1 Reply Last reply Reply Quote 0
          • V
            vito
            last edited by

            @jimp:

            @LostInIgnorance:

            EDIT: Did you try it with squid havp, etc. loaded on it?

            I haven't tried it with squid, I'll load that on next. Do you have the OpenVPN server assigned as an interface and squid listening there? Otherwise, squid wouldn't be doing anything to the traffic coming from OpenVPN

            Jimp,
            it was most likely missed in this thread, but in my original thread with this problem from 12/13
            had the packages i am running.  http://forum.pfsense.org/index.php/topic,31031.msg165512.html#msg165512

            Packages i am running:
            Squid
            Squid Guard
            Snort
            OpenVPN Export
            Light Squid
            Anyterm

            If need be, i can test by stopping Squid and such….

            1 Reply Last reply Reply Quote 0
            • jimpJ
              jimp Rebel Alliance Developer Netgate
              last edited by

              It might be worth trying without any packages. (the only one I have is OpenVPN client export) to see if it makes a difference.

              One thing - for everyone seeing the panic on em(4) devices, the output from

              pciconf -lvb
              

              For the em card seen in the crash would be helpful.

              Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

              Need help fast? Netgate Global Support!

              Do not Chat/PM for help!

              1 Reply Last reply Reply Quote 0
              • L
                LostInIgnorance
                last edited by

                ??? JimP, I am a little confused, do you want me to type that in after the crash happens and the db comes up? If not, what would you like from the crash?

                1 Reply Last reply Reply Quote 0
                • C
                  clarknova
                  last edited by

                  I have a net5501-70, a Pro 1000GT (pci), and a spare pppoe DSL line. Jim, I can set this up with openvpn if you want to test against it. I have no experience with openvpn though, so would need some guidance setting that up.

                  Also note that my DSL line has pretty poor sync rates right now, and occasional disconnects, which is why it's not in use :/

                  db

                  1 Reply Last reply Reply Quote 0
                  • jimpJ
                    jimp Rebel Alliance Developer Netgate
                    last edited by

                    @LostInIgnorance:

                    ??? JimP, I am a little confused, do you want me to type that in after the crash happens and the db comes up? If not, what would you like from the crash?

                    Just at a normal command prompt. It will show the chip id of the em card.

                    Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                    Need help fast? Netgate Global Support!

                    Do not Chat/PM for help!

                    1 Reply Last reply Reply Quote 0
                    • jimpJ
                      jimp Rebel Alliance Developer Netgate
                      last edited by

                      @clarknova:

                      I have a net5501-70, a Pro 1000GT (pci), and a spare pppoe DSL line. Jim, I can set this up with openvpn if you want to test against it. I have no experience with openvpn though, so would need some guidance setting that up.

                      Also note that my DSL line has pretty poor sync rates right now, and occasional disconnects, which is why it's not in use :/

                      I'll keep that in mind, though I was hoping to do it locally since that makes it a lot easier to track down…

                      Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                      Need help fast? Netgate Global Support!

                      Do not Chat/PM for help!

                      1 Reply Last reply Reply Quote 0
                      • V
                        vito
                        last edited by

                        @jimp:

                        It might be worth trying without any packages. (the only one I have is OpenVPN client export) to see if it makes a difference.

                        One thing - for everyone seeing the panic on em(4) devices, the output from

                        pciconf -lvb
                        

                        For the em card seen in the crash would be helpful.

                        Will try later tonight. :)

                        from my system
                        em1@pci0:3:0:0: class=0x020000 card=0x060a15d9 chip=0x10d38086 rev=0x00 hdr=0x00
                            class      = network
                            subclass  = ethernet
                            bar  [10] = type Memory, range 32, base 0xfeae0000, size 131072, enabled
                            bar  [18] = type I/O Port, range 32, base 0xec00, size 32, enabled
                            bar  [1c] = type Memory, range 32, base 0xfeadc000, size 16384, enabled

                        1 Reply Last reply Reply Quote 0
                        • V
                          vito
                          last edited by

                          from another system having problems. (i can not get to the console to see what card)

                          em0@pci0:2:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
                              class      = network
                              subclass  = ethernet
                              bar  [10] = type Memory, range 32, base 0xfe6e0000, size 131072, enabled
                              bar  [18] = type I/O Port, range 32, base 0x9c00, size 32, enabled
                              bar  [1c] = type Memory, range 32, base 0xfe6dc000, size 16384, enabled
                          em1@pci0:3:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
                              class      = network
                              subclass  = ethernet
                              bar  [10] = type Memory, range 32, base 0xfe7e0000, size 131072, enabled
                              bar  [18] = type I/O Port, range 32, base 0xac00, size 32, enabled
                              bar  [1c] = type Memory, range 32, base 0xfe7dc000, size 16384, enabled
                          em2@pci0:4:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
                              class      = network
                              subclass  = ethernet
                              bar  [10] = type Memory, range 32, base 0xfe8e0000, size 131072, enabled
                              bar  [18] = type I/O Port, range 32, base 0xbc00, size 32, enabled
                              bar  [1c] = type Memory, range 32, base 0xfe8dc000, size 16384, enabled
                          em3@pci0:5:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
                              class      = network
                              subclass  = ethernet
                              bar  [10] = type Memory, range 32, base 0xfe9e0000, size 131072, enabled
                              bar  [18] = type I/O Port, range 32, base 0xcc00, size 32, enabled
                              bar  [1c] = type Memory, range 32, base 0xfe9dc000, size 16384, enabled
                          em4@pci0:6:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
                              class      = network
                              subclass  = ethernet
                              bar  [10] = type Memory, range 32, base 0xfeae0000, size 131072, enabled
                              bar  [18] = type I/O Port, range 32, base 0xdc00, size 32, enabled
                              bar  [1c] = type Memory, range 32, base 0xfeadc000, size 16384, enabled
                          em5@pci0:7:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
                              class      = network
                              subclass  = ethernet
                              bar  [10] = type Memory, range 32, base 0xfebe0000, size 131072, enabled
                              bar  [18] = type I/O Port, range 32, base 0xec00, size 32, enabled
                              bar  [1c] = type Memory, range 32, base 0xfebdc000, size 16384, enabled

                          1 Reply Last reply Reply Quote 0
                          • jimpJ
                            jimp Rebel Alliance Developer Netgate
                            last edited by

                            Chip ID is the same on both of those, and on another one I'm talking to that crashes… Mine that run fine are a different chip.

                            Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                            Need help fast? Netgate Global Support!

                            Do not Chat/PM for help!

                            1 Reply Last reply Reply Quote 0
                            • C
                              clarknova
                              last edited by

                              em0@pci0:2:0:0: class=0x020000 card=0x060a15d9 chip=0x10d38086 rev=0x00 hdr=0x00
                                  class      = network
                                  subclass  = ethernet
                                  bar  [10] = type Memory, range 32, base 0xfeae0000, size 131072, enabled
                                  bar  [18] = type I/O Port, range 32, base 0xdc00, size 32, enabled
                                  bar  [1c] = type Memory, range 32, base 0xfeadc000, size 16384, enabled
                              em1@pci0:3:0:0: class=0x020000 card=0x060a15d9 chip=0x10d38086 rev=0x00 hdr=0x00
                                  class      = network
                                  subclass  = ethernet
                                  bar  [10] = type Memory, range 32, base 0xfebe0000, size 131072, enabled
                                  bar  [18] = type I/O Port, range 32, base 0xec00, size 32, enabled
                                  bar  [1c] = type Memory, range 32, base 0xfebdc000, size 16384, enabled

                              Interesting, my SM board has the same chip ID. My panics come around at 6-8 day intervals with no vpn use, no known factor other than time elapsed.

                              db

                              1 Reply Last reply Reply Quote 0
                              • jimpJ
                                jimp Rebel Alliance Developer Netgate
                                last edited by

                                LostInIgnorance - one more test for you since you seem to hit it faster than anyone else - can you try switching your OpenVPN instance to TCP instead of UDP and see if it makes any difference? Just curious.

                                Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                Need help fast? Netgate Global Support!

                                Do not Chat/PM for help!

                                1 Reply Last reply Reply Quote 0
                                • jnorellJ
                                  jnorell
                                  last edited by

                                  Just another "me too" .. panics in devd on the carp backup host.  Just got them while adding firewall aliases on the primary (no recent changes to VIP addrs).  I'll report if tomorrow's snapshot doesn't stop them.

                                  1 Reply Last reply Reply Quote 0
                                  • jimpJ
                                    jimp Rebel Alliance Developer Netgate
                                    last edited by

                                    vito and others with em failing, can you look at the output of this:

                                    sysctl -a | grep 'dev.em.*fail'
                                    

                                    And report any non-zero values. Specifically with dev.em.0.mbuf_alloc_fail and dev.em.0.tx_dma_fail

                                    Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                    Need help fast? Netgate Global Support!

                                    Do not Chat/PM for help!

                                    1 Reply Last reply Reply Quote 0
                                    • V
                                      vito
                                      last edited by

                                      No non zero's

                                      here is the output
                                      dev.em.0.mbuf_alloc_fail: 0
                                      dev.em.0.cluster_alloc_fail: 0
                                      dev.em.0.tx_dma_fail: 0
                                      dev.em.0.mac_stats.tso_ctx_fail: 0
                                      dev.em.1.mbuf_alloc_fail: 0
                                      dev.em.1.cluster_alloc_fail: 0
                                      dev.em.1.tx_dma_fail: 0
                                      dev.em.1.mac_stats.tso_ctx_fail: 0

                                      1 Reply Last reply Reply Quote 0
                                      • L
                                        LostInIgnorance
                                        last edited by

                                        Nope, over TCP still did this:

                                        Kernel page fault with the following non-sleepable locks held:
                                        exclusive sleep mutex vr1 (network driver) r = 0 (0xc3640aec) locked @ /usr/pfSensesrc/src/sys/dev/vr/if_vr.c:1675
                                        KDB: stack backtrace:
                                        X_db_sym_numargs(c0c4e35f,d5341a88,c092ce85,68b,0,...) at X_db_sym_numargs+0x146
                                        kdb_backtrace(68b,0,ffffffff,c11b7554,d5341ac0,...) at kdb_backtrace+0x29
                                        witness_display_spinlock(c0c50877,d5341ad4,4,1,0,...) at witness_display_spinlock+0x75
                                        witness_warn(5,0,c0c823d7,c1981a94,c3556aa0,...) at witness_warn+0x20d
                                        trap(d5341b60) at trap+0x172
                                        alltraps(c39bf700,dedeadc0,c39bf700,c39bf700,d5341be8,...) at alltraps+0x1b
                                        m_tag_delete_chain(c39bf700,0,c092cc2b,0,0,...) at m_tag_delete_chain+0x3f
                                        m_pkthdr_init(c39bf700,100,0,c092cc2b,c0c3acf7,...) at m_pkthdr_init+0x8b5
                                        uma_zfree_arg(c1981a80,c39bf700,0,c3640000,d5341c70,...) at uma_zfree_arg+0x29
                                        m_freem(c39bf700,4,c0c3acf7,5a3,0,...) at m_freem+0x43
                                        ucom_attach(c3640aec,0,c0c3acf7,68b,c3640aec,...) at ucom_attach+0x88f5
                                        ucom_attach(c3640000,d5341cc8,c08d8a54,c107b5c0,c3554238,...) at ucom_attach+0xaa17
                                        intr_event_execute_handlers(c3556aa0,c3554200,c0c4616c,533,c3554270,...) at intr_event_execute_handlers+0x125
                                        intr_event_add_handler(c3644b60,d5341d38,c0c45ecc,344,c3556aa0,...) at intr_event_add_handler+0x42f
                                        fork_exit(c08c1b70,c3644b60,d5341d38) at fork_exit+0xb8
                                        fork_trampoline() at fork_trampoline+0x8
                                        --- trap 0, eip = 0, esp = 0xd5341d70, ebp = 0 ---
                                        
                                        Fatal trap 12: page fault while in kernel mode
                                        cpuid = 0; apic id = 00
                                        fault virtual address   = 0xdedeadc0
                                        fault code              = supervisor read, page not present
                                        instruction pointer     = 0x20:0xc094b038
                                        stack pointer           = 0x28:0xd5341ba0
                                        frame pointer           = 0x28:0xd5341bb0
                                        code segment            = base 0x0, limit 0xfffff, type 0x1b
                                                                = DPL 0, pres 1, def32 1, gran 1
                                        processor eflags        = interrupt enabled, resume, IOPL = 0
                                        current process         = 11 (irq5: vr1)
                                        [thread]
                                        Stopped at      m_tag_delete+0x48:      movl    0(%ecx),%eax
                                        db> bt
                                        Tracing pid 11 tid 64025 td 0xc358d780
                                        m_tag_delete(c39bf700,dedeadc0,c39bf700,c39bf700,d5341be8,...) at m_tag_delete+0x48
                                        m_tag_delete_chain(c39bf700,0,c092cc2b,0,0,...) at m_tag_delete_chain+0x3f
                                        m_pkthdr_init(c39bf700,100,0,c092cc2b,c0c3acf7,...) at m_pkthdr_init+0x8b5
                                        uma_zfree_arg(c1981a80,c39bf700,0,c3640000,d5341c70,...) at uma_zfree_arg+0x29
                                        m_freem(c39bf700,4,c0c3acf7,5a3,0,...) at m_freem+0x43
                                        ucom_attach(c3640aec,0,c0c3acf7,68b,c3640aec,...) at ucom_attach+0x88f5
                                        ucom_attach(c3640000,d5341cc8,c08d8a54,c107b5c0,c3554238,...) at ucom_attach+0xaa17
                                        intr_event_execute_handlers(c3556aa0,c3554200,c0c4616c,533,c3554270,...) at intr_event_execute_handlers+0x125
                                        intr_event_add_handler(c3644b60,d5341d38,c0c45ecc,344,c3556aa0,...) at intr_event_add_handler+0x42f
                                        fork_exit(c08c1b70,c3644b60,d5341d38) at fork_exit+0xb8
                                        fork_trampoline() at fork_trampoline+0x8
                                        --- trap 0, eip = 0, esp = 0xd5341d70, ebp = 0 ---
                                        db>
                                        [/thread]
                                        
                                        1 Reply Last reply Reply Quote 0
                                        • jimpJ
                                          jimp Rebel Alliance Developer Netgate
                                          last edited by

                                          It was worth trying/checking…

                                          Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                          Need help fast? Netgate Global Support!

                                          Do not Chat/PM for help!

                                          1 Reply Last reply Reply Quote 0
                                          • E
                                            eri--
                                            last edited by

                                            Can anyone of you please test with this kernel http://files.pfsense.org/kernel.gz
                                            Just copy it under /boot/kernel/kernel.gz on pfsense and reboot.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.