Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Need assist to examine crash log

    Scheduled Pinned Locked Moved General pfSense Questions
    6 Posts 2 Posters 2.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      charlie1970
      last edited by

      Hello All Advance Guys.
      I met system crash problem almost 1 month. And search forum or freebsd related topics. But sill not able to figure the direction.
      And no idea after upload crash report via Pfsense WebUI.

      Try to post here if could get further information will very appreciate. (Attached full crash log)

      Machine information
        Hardware: Axiomtek NA550. with AX93306-4GIL (4-Port Gigabit with LAN bypass)
        CPU: Intel Atom i3-2120@3.3 GHz, 2 Cores with 2 SMT Threads
        Memory: 2GB
        HD: SSD 60GB
        LAN Chipset: Intel 82580DB

      Purpose: Traffic shaping (LAN Bandwidth Control)
      Maximum Throughput: 357Mb/s (Not crash at peak loading)

      Part of crash log

      Crash report begins.  Anonymous machine information:

      i386
      8.3-RELEASE-p16
      FreeBSD 8.3-RELEASE-p16 #0: Mon Aug 25 08:25:41 EDT 2014    root@pf2_1_1_i386.pfsense.org:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.8

      Crash report details:

      Filename: /var/crash/bounds
      1

      Filename: /var/crash/info.0
      Dump header from device /dev/ad4s1b
        Architecture: i386
        Architecture Version: 1
        Dump Length: 78336B (0 MB)
        Blocksize: 512
        Dumptime: Fri Dec  5 09:34:45 2014
        Hostname: tpepfs02
        Magic: FreeBSD Text Dump
        Version String: FreeBSD 8.3-RELEASE-p16 #0: Mon Aug 25 08:25:41 EDT 2014
          root@pf2_1_1_i386.pfsense.org:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.8
        Panic String: double fault
        Dump Parity: 2406528355
        Bounds: 0
        Dump Status: good

      Filename: /var/crash/textdump.tar.0
      ddb.txt06000014000012440205665  7076 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
      db:1:lockinfo> show locks
      No such command
      db:1:locks>  show alllocks
      No such command
      db:1:alllocks>  show lockedvnods
      Locked vnodes
      db:0:kdb.enter.default>  show pcpu
      cpuid        = 2
      dynamic pcpu = 0x42a8700
      curthread    = 0xc59945c0: pid 11 "idle: cpu2"
      curpcb      = 0xc57e7d80
      fpcurthread  = none
      idlethread  = 0xc59945c0: tid 100004 "idle: cpu2"
      APIC ID      = 2
      currentldt  = 0x50
      db:0:kdb.enter.default>  bt
      Tracing pid 11 tid 100004 td 0xc59945c0
      kdb_enter(c0f98d73,c0f98d73,c0fccc1e,c15d02f0,2,…) at kdb_enter+0x3b
      panic(c0fccc1e,2,2,0,0,...) at panic+0x102
      dblfault_handler() at dblfault_handler+0x9b
      --- trap 0x17, eip = 0xc0e750d8, esp = 0xc57e7000, ebp = 0xc57e7c7c ---
      Xpage(c57e7c98,c151f8b0,2,0,ffffffff,...) at Xpage
      acpi_cpu_idle(1,c57e7ccc,c0f99f18,a08,c59945c0,...) at acpi_cpu_idle+0x1e1
      sched_idletd(0,c57e7d28,245c8948,0,0,...) at sched_idletd+0x116
      fork_exit(c0ac8fb0,0,c57e7d28) at fork_exit+0x87
      fork_trampoline() at fork_trampoline+0x8
      --- trap 0, eip = 0, esp = 0xc57e7d60, ebp = 0 ---
      db:0:kdb.enter.default>  ps
      :
      :

      Fatal double fault:
      eip = 0xc0e750d8
      esp = 0xc57e7000
      ebp = 0xc57e7c7c
      cpuid = 2; apic id = 02
      panic: double fault
      cpuid = 2
      KDB: enter: panic
      panic.txt0600001412440205665  7137 ustarrootwheeldouble faultversion.txt06000021612440205665  7616 ustarrootwheelFreeBSD 8.3-RELEASE-p16 #0: Mon Aug 25 08:25:41 EDT 2014
          root@pf2_1_1_i386.pfsense.org:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.8

      TPEPFS02-1205-0934-crash.txt
      TPEPFS02-1204-1104-crash.txt

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        You have 16 igb interfaces?! And 2 em interfaces?
        I would certainly try the suggestions here in that case:
        https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#Intel_igb.284.29_and_em.284.29_Cards

        Steve

        1 Reply Last reply Reply Quote 0
        • C
          charlie1970
          last edited by

          Hi Steve:
          Thanks for your input at first.
          Yes, we put 12 Gigabit Ethernet and 4 Fiber for prioritize traffic. And use 12 GBE for now.

          I did add kern.ipc.nmbclusters="2304000" into /boot/loader.conf. not /boot/loader.conf.local.
          Then I will try put into "loader.conf.local" file and feedback result here.

          Any other clue or config need turn on for more detail when crash occured ?
          Examine the RRD graph could to link crashed caused by out of MBUF.

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            Yes if it is MBUF exhaustion you should be able to see it recorded in the RRD data. Assuming it wasn't lost in the crash.

            The Intel NICs will record detailed stats in the sysctls that may hold some clue. Otherwise I think this in one for the devs.  ;)

            Steve

            1 Reply Last reply Reply Quote 0
            • C
              charlie1970
              last edited by

              Hi Steve:

              It's look work well now. But question is MBUF usage not figure out on RRD graph. (See attached.)
              RRD graph show not exhaust but stable, hardly to link caused system crash.  :-
              And found the MBUF usage goes down when apply change. (Seed 1Week graph)

              Anyway, will keep track 1 week to see this issue solved by change the "boot.conf.local"  :) :) :)

              Appreciate your information again.

              Charlie

              PS: Sorry, not able to upload in reply.

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                This sort of system is outside my experience to honest. There were some issues a while ago. For example I believe this no longer applies but because none of the boxes I use igb NICs I can't be sure  :-\ :
                https://forum.pfsense.org/index.php?topic=69486.0
                If it happens again you might consider going to 2.2 which has newer drivers as well as many other improvements.

                Steve

                1 Reply Last reply Reply Quote 0
                • First post
                  Last post
                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.