Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Kernel Panic

    Scheduled Pinned Locked Moved 2.0-RC Snapshot Feedback and Problems - RETIRED
    325 Posts 35 Posters 253.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • E
      eri--
      last edited by

      Wrong snapshot sorry.
      This is the commit that fixes the error https://rcs.pfsense.org/projects/pfsense-tools/repos/mainline/commits/08f1322c7d5d9fae8ef52dc356c75a59d2483263

      1 Reply Last reply Reply Quote 0
      • F
        FisherKing
        last edited by

        Running 2.0-BETA5 (i386) built on Fri Feb 4 02:36:03 EST 2011 on both Master & Slave.

        It ran longer this time before the panic.  I changed my configuration a little before it happened this time.

        Steps:
        1 - Pulled the plug on the master WAN
          (Slave became the new CARP master for all carp VIPs)
        1a - Verified LAN clients were able to surf the web.
        2 - Cleared XMLRPC Sync settings from the old master
        3 - Set XMLRPC Sync settings on the new master (old slave)
        4 - Changed DHCP settings on new master (old slave)
        5 - Old master had a panic.

        ![](http://Carp Panic1.png)

        ![](http://carp bt1.png)

        ![Carp Panic1.png](/public/imported_attachments/1/Carp Panic1.png)
        ![Carp Panic1.png_thumb](/public/imported_attachments/1/Carp Panic1.png_thumb)
        ![carp bt1.png](/public/imported_attachments/1/carp bt1.png)
        ![carp bt1.png_thumb](/public/imported_attachments/1/carp bt1.png_thumb)

        1 Reply Last reply Reply Quote 0
        • jimpJ
          jimp Rebel Alliance Developer Netgate
          last edited by

          @PJ2:

          2 - Cleared XMLRPC Sync settings from the old master
          3 - Set XMLRPC Sync settings on the new master (old slave)

          Why on earth would you do that? That isn't necessary at all, and could lead to other issues.

          Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

          Need help fast? Netgate Global Support!

          Do not Chat/PM for help!

          1 Reply Last reply Reply Quote 0
          • V
            vito
            last edited by

            Jimp/Ermal
            just an update. I installed an updated firmware on another system about 5 days ago.
            All seem good!
            Thanks for all your help!!!!

            vito

            1 Reply Last reply Reply Quote 0
            • F
              FisherKing
              last edited by

              As I was typing I wondered if that might be the response I would get. It's always amazing what crazy and unexpected things an end user can come up with. :)

              This is a test environment, so I test things.
              Comp1 = initial CARP master, XMLRPC master
              Comp2 = initial CARP slave, XMLRPC slave

              I wanted to know what would happen if Comp1 had an issue and Comp2 took over.  Could I also make Comp2 become the XMLRPC master since it was acting as the CARP master?  I did clear the XMLRPC settings on Comp1 first.  I wouldn't have thought that reversing the roles would cause a panic.

              1 Reply Last reply Reply Quote 0
              • jimpJ
                jimp Rebel Alliance Developer Netgate
                last edited by

                When a master syncs the config to a slave it makes several alterations in the process.

                Just flipping a slave to a master in the XMLRPC settings it going to cause you problems because of this.

                If you really want to swap their roles, restore the master's config to the slave box and vice versa.

                When a CARP slave takes over automatically - if you have configured everything right - it can function that way indefinitely until you bring the master back up, you just can't make any changes to the slave's config you want to keep. But that is all a topic for another thread. It's not the source of the panic - it's just a Bad Thing™.

                Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                Need help fast? Netgate Global Support!

                Do not Chat/PM for help!

                1 Reply Last reply Reply Quote 0
                • F
                  FisherKing
                  last edited by

                  Good to know - I'll avoid doing that in the future.

                  1 Reply Last reply Reply Quote 0
                  • P
                    pwnell
                    last edited by

                    Ok so I waited until I thought this snapshot would be okaish… Within 10 minutes of installing this my machine crashed:

                    2.0-BETA5 (i386)
                    built on Fri Feb 4 15:47:28 EST 2011

                    Pentium(R) Dual-Core CPU E5400 @ 2.70GHz

                    [2.0-BETA5][root@fw.home]/root(1): uname -a
                    FreeBSD fw.home 8.1-RELEASE-p2 FreeBSD 8.1-RELEASE-p2 #1: Fri Feb  4 15:45:20 EST 2011    sullrich@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8  i386

                    –---------

                    Did not do anything special on the firewall when it died.

                    IMG_0233.JPG
                    IMG_0233.JPG_thumb

                    1 Reply Last reply Reply Quote 0
                    • P
                      pwnell
                      last edited by

                      Bt backtrace.

                      IMG_0234.JPG
                      IMG_0234.JPG_thumb

                      1 Reply Last reply Reply Quote 0
                      • L
                        LostInIgnorance
                        last edited by

                        Sorry, I haven't had time to exactly test the new snapshots till now.  I loaded it up with 2.0-BETA5 (i386) built on Sun Feb 6 23:54:00 EST 2011.  Crashed when trying to use openVPN.  This is the P4 Dell machine that was described prior.

                        Kernel page fault with the following non-sleepable locks held:
                        exclusive sleep mutex em0 (EM TX Lock) r = 0 (0xc2f52580) locked @ /usr/pfSensesrc/src/sys/dev/e1000/if_lem.c:1350
                        KDB: stack backtrace:
                        db_trace_self_wrapper(c0eb72bd,ccc3ca90,c0a41f05,546,0,...) at db_trace_self_wrapper+0x26
                        kdb_backtrace(546,0,ffffffff,c145d504,ccc3cac8,...) at kdb_backtrace+0x29
                        _witness_debugger(c0eb97d5,ccc3cadc,4,1,0,...) at _witness_debugger+0x25
                        witness_warn(5,0,c0ef7b9a,14,c131b440,...) at witness_warn+0x20d
                        trap(ccc3cb68) at trap+0x19e
                        calltrap() at calltrap+0x6
                        --- trap 0xc, eip = 0xc0a611ca, esp = 0xccc3cba8, ebp = 0xccc3cbb8 ---
                        m_tag_delete(c341cd00,dedeadc0,c341cd00,c341cd00,ccc3cbf0,...) at m_tag_delete+0x5a
                        m_tag_delete_chain(c341cd00,0,c0e6e71f,0,c2ed91e0,...) at m_tag_delete_chain+0x3f
                        mb_dtor_mbuf(c341cd00,100,0,c0a42958,df,...) at mb_dtor_mbuf+0x35
                        uma_zfree_arg(c1d7e380,c341cd00,0,1e,ccc3cc84,...) at uma_zfree_arg+0x29
                        m_freem(c341cd00,4,c0e6e71f,b87,c2f4e000,...) at m_freem+0x43
                        lem_txeof(c2f52580,0,c0e6e71f,546,c2f525bc,...) at lem_txeof+0x158
                        lem_handle_rxtx(c2f4e000,1,c0eb8b8e,4f,c2edb8d8,...) at lem_handle_rxtx+0x60
                        taskqueue_run(c2edb8c0,c2edb8d8,c0ea5f47,0,c0eb21ed,...) at taskqueue_run+0x103
                        taskqueue_thread_loop(c2f525ec,ccc3cd38,c0eaed5c,344,c131b440,...) at taskqueue_thread_loop+0x68
                        fork_exit(c0a3b180,c2f525ec,ccc3cd38) at fork_exit+0xb8
                        fork_trampoline() at fork_trampoline+0x8
                        --- trap 0, eip = 0, esp = 0xccc3cd70, ebp = 0 ---
                        
                        Fatal trap 12: page fault while in kernel mode
                        cpuid = 0; apic id = 00
                        fault virtual address= 0xdedeadc0
                        fault code= supervisor read, page not present
                        instruction pointer= 0x20:0xc0a611ca
                        stack pointer        = 0x28:0xccc3cba8
                        frame pointer        = 0x28:0xccc3cbb8
                        code segment= base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
                        processor eflags= interrupt enabled, resume, IOPL = 0
                        current process= 0 (em0 taskq)
                        [thread]
                        Stopped at      m_tag_delete+0x5a:      movl    0(%ecx),%eax
                        db> bt
                        Tracing pid 0 tid 64050 td 0xc2f4bc80
                        m_tag_delete(c341cd00,dedeadc0,c341cd00,c341cd00,ccc3cbf0,...) at m_tag_delete+0x5a
                        m_tag_delete_chain(c341cd00,0,c0e6e71f,0,c2ed91e0,...) at m_tag_delete_chain+0x3f
                        mb_dtor_mbuf(c341cd00,100,0,c0a42958,df,...) at mb_dtor_mbuf+0x35
                        uma_zfree_arg(c1d7e380,c341cd00,0,1e,ccc3cc84,...) at uma_zfree_arg+0x29
                        m_freem(c341cd00,4,c0e6e71f,b87,c2f4e000,...) at m_freem+0x43
                        lem_txeof(c2f52580,0,c0e6e71f,546,c2f525bc,...) at lem_txeof+0x158
                        lem_handle_rxtx(c2f4e000,1,c0eb8b8e,4f,c2edb8d8,...) at lem_handle_rxtx+0x60
                        taskqueue_run(c2edb8c0,c2edb8d8,c0ea5f47,0,c0eb21ed,...) at taskqueue_run+0x103
                        taskqueue_thread_loop(c2f525ec,ccc3cd38,c0eaed5c,344,c131b440,...) at taskqueue_thread_loop+0x68
                        fork_exit(c0a3b180,c2f525ec,ccc3cd38) at fork_exit+0xb8
                        fork_trampoline() at fork_trampoline+0x8
                        --- trap 0, eip = 0, esp = 0xccc3cd70, ebp = 0 ---
                        db>
                        
                        It is kinda nice though that the pfSense now knows it crashed and asks me if I would like to send the error info to the developers.
                        [quote]We have detected a crash report. Click here for more information.[/quote]
                        though the only thing the tab shows on the error report is this
                        [code]Crash report begins.  Anonymous machine information:
                        
                        i386
                        8.1-RELEASE-p2
                        FreeBSD 8.1-RELEASE-p2 #1: Sun Feb  6 23:47:16 EST 2011     sullrich@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_Dev.8
                        
                        Crash report details:
                        
                        2048
                        [/code][/thread]
                        
                        1 Reply Last reply Reply Quote 0
                        • F
                          fasteddy
                          last edited by

                          I don't have access to my pfSense box(s) to get crash info at the moment, but I can say I am in the same boat as LostInConfusion - still getting panics when I try to access a pfSense box through the VPN (though in my case it's IPSec, not OpenVPN).  This is using the most recent snapshot.  For what it's worth, the previously mentioned carp lock issue probably wasn't at the root of the issue I am having, since I am not using any carp functionality whatsoever with my setup…

                          1 Reply Last reply Reply Quote 0
                          • jimpJ
                            jimp Rebel Alliance Developer Netgate
                            last edited by

                            @LostInIgnorance:

                            It is kinda nice though that the pfSense now knows it crashed and asks me if I would like to send the error info to the developers.

                            We have detected a crash report. Click here for more information.

                            though the only thing the tab shows on the error report is this

                            Crash report begins.  Anonymous machine information:
                            
                            i386
                            8.1-RELEASE-p2
                            FreeBSD 8.1-RELEASE-p2 #1: Sun Feb  6 23:47:16 EST 2011     sullrich@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_Dev.8
                            
                            Crash report details:
                            
                            2048
                            
                            

                            That one is a false positive.

                            The actual report would have done a lot more.

                            If it stops at a db> prompt, it isn't collecting the crash info automatically for whatever reason.

                            Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                            Need help fast? Netgate Global Support!

                            Do not Chat/PM for help!

                            1 Reply Last reply Reply Quote 0
                            • P
                              pwnell
                              last edited by

                              @jimp - any ideas about my crash?  I assume it is related to this thread…

                              1 Reply Last reply Reply Quote 0
                              • F
                                fasteddy
                                last edited by

                                I'm in the same situation there as well - my pfSense box stops at the 'db>' prompt after a panic, generating similar information-free reports.  Also, in contrast to what I believe other folks have reported, my pfSense box doesn't automatically reboot after a panic, but rather sits happily at the db> prompt, patiently - and indefinitely - awaiting my input.  (Which is why I am loathe to do any testing unless I'm within arms reach of the particular box in question!)  Is there a setting I've missed somewhere that enables this automatic rebooting, or is my pfSense box just that unhappy?

                                1 Reply Last reply Reply Quote 0
                                • jimpJ
                                  jimp Rebel Alliance Developer Netgate
                                  last edited by

                                  pwnell,

                                  I don't know, it looks similar to another panic that was already reported over the weekend. Have you tried a new snapshot?

                                  fasteddy,

                                  What type of install are you on? full/nano? i386 or amd64? How much RAM and swap space?

                                  If you can get a capture of the panic (even a picture of it) and the output of 'bt' at the debug prompt it would help.

                                  Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                  Need help fast? Netgate Global Support!

                                  Do not Chat/PM for help!

                                  1 Reply Last reply Reply Quote 0
                                  • F
                                    fasteddy
                                    last edited by

                                    I am on a full install, i386, 2GB RAM, 80GB Hard drive, 4GB swap space.  The panic information, backtrace info, and general situation remain the same as my original post (on page 15 of this thread).  http://forum.pfsense.org/index.php/topic,31721.msg170362.html#msg170362

                                    1 Reply Last reply Reply Quote 0
                                    • jimpJ
                                      jimp Rebel Alliance Developer Netgate
                                      last edited by

                                      @fasteddy:

                                      I am on a full install, i386, 2GB RAM, 80GB Hard drive, 4GB swap space.  The panic information, backtrace info, and general situation remain the same as my original post (on page 15 of this thread).  http://forum.pfsense.org/index.php/topic,31721.msg170362.html#msg170362

                                      Show me the output of:

                                      sysctl debug.ddb
                                      

                                      Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                      Need help fast? Netgate Global Support!

                                      Do not Chat/PM for help!

                                      1 Reply Last reply Reply Quote 0
                                      • F
                                        fasteddy
                                        last edited by

                                        Here goes:

                                        # sysctl debug.ddb
                                        debug.ddb.capture.data:
                                        debug.ddb.capture.bufsize: 49152
                                        debug.ddb.capture.inprogress: 0
                                        debug.ddb.capture.maxbufsize: 5242880
                                        debug.ddb.capture.bufoff: 0
                                        debug.ddb.scripting.unscript:
                                        debug.ddb.scripting.scripts: lockinfo=show locks; show alllocks; show lockedvnods
                                        kdb.enter.panic=textdump set; capture on; run lockinfo; show pcpu; bt; ps; alltrace; capture off; call doadump; reset
                                        kdb.enter.witness=run lockinfo
                                        
                                        debug.ddb.textdump.do_version: 1
                                        debug.ddb.textdump.do_panic: 1
                                        debug.ddb.textdump.do_msgbuf: 1
                                        debug.ddb.textdump.do_ddb: 1
                                        debug.ddb.textdump.do_config: 1
                                        debug.ddb.textdump.pending: 0
                                        
                                        
                                        1 Reply Last reply Reply Quote 0
                                        • jimpJ
                                          jimp Rebel Alliance Developer Netgate
                                          last edited by

                                          That all looks normal, like it should be doing the textdumps…

                                          What about:

                                          cat /boot/kernel/pfsense_kernel.txt
                                          

                                          Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                          Need help fast? Netgate Global Support!

                                          Do not Chat/PM for help!

                                          1 Reply Last reply Reply Quote 0
                                          • F
                                            fasteddy
                                            last edited by

                                            Doesn't have a whole lot to say:

                                            # cat /boot/kernel/pfsense_kernel.txt
                                            UP
                                            #
                                            
                                            
                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.