Need assist to examine crash log



  • Hello All Advance Guys.
    I met system crash problem almost 1 month. And search forum or freebsd related topics. But sill not able to figure the direction.
    And no idea after upload crash report via Pfsense WebUI.

    Try to post here if could get further information will very appreciate. (Attached full crash log)

    Machine information
      Hardware: Axiomtek NA550. with AX93306-4GIL (4-Port Gigabit with LAN bypass)
      CPU: Intel Atom i3-2120@3.3 GHz, 2 Cores with 2 SMT Threads
      Memory: 2GB
      HD: SSD 60GB
      LAN Chipset: Intel 82580DB

    Purpose: Traffic shaping (LAN Bandwidth Control)
    Maximum Throughput: 357Mb/s (Not crash at peak loading)

    Part of crash log

    Crash report begins.  Anonymous machine information:

    i386
    8.3-RELEASE-p16
    FreeBSD 8.3-RELEASE-p16 #0: Mon Aug 25 08:25:41 EDT 2014    root@pf2_1_1_i386.pfsense.org:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.8

    Crash report details:

    Filename: /var/crash/bounds
    1

    Filename: /var/crash/info.0
    Dump header from device /dev/ad4s1b
      Architecture: i386
      Architecture Version: 1
      Dump Length: 78336B (0 MB)
      Blocksize: 512
      Dumptime: Fri Dec  5 09:34:45 2014
      Hostname: tpepfs02
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 8.3-RELEASE-p16 #0: Mon Aug 25 08:25:41 EDT 2014
        root@pf2_1_1_i386.pfsense.org:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.8
      Panic String: double fault
      Dump Parity: 2406528355
      Bounds: 0
      Dump Status: good

    Filename: /var/crash/textdump.tar.0
    ddb.txt06000014000012440205665  7076 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
    db:1:lockinfo> show locks
    No such command
    db:1:locks>  show alllocks
    No such command
    db:1:alllocks>  show lockedvnods
    Locked vnodes
    db:0:kdb.enter.default>  show pcpu
    cpuid        = 2
    dynamic pcpu = 0x42a8700
    curthread    = 0xc59945c0: pid 11 "idle: cpu2"
    curpcb      = 0xc57e7d80
    fpcurthread  = none
    idlethread  = 0xc59945c0: tid 100004 "idle: cpu2"
    APIC ID      = 2
    currentldt  = 0x50
    db:0:kdb.enter.default>  bt
    Tracing pid 11 tid 100004 td 0xc59945c0
    kdb_enter(c0f98d73,c0f98d73,c0fccc1e,c15d02f0,2,…) at kdb_enter+0x3b
    panic(c0fccc1e,2,2,0,0,...) at panic+0x102
    dblfault_handler() at dblfault_handler+0x9b
    --- trap 0x17, eip = 0xc0e750d8, esp = 0xc57e7000, ebp = 0xc57e7c7c ---
    Xpage(c57e7c98,c151f8b0,2,0,ffffffff,...) at Xpage
    acpi_cpu_idle(1,c57e7ccc,c0f99f18,a08,c59945c0,...) at acpi_cpu_idle+0x1e1
    sched_idletd(0,c57e7d28,245c8948,0,0,...) at sched_idletd+0x116
    fork_exit(c0ac8fb0,0,c57e7d28) at fork_exit+0x87
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc57e7d60, ebp = 0 ---
    db:0:kdb.enter.default>  ps
    :
    :

    Fatal double fault:
    eip = 0xc0e750d8
    esp = 0xc57e7000
    ebp = 0xc57e7c7c
    cpuid = 2; apic id = 02
    panic: double fault
    cpuid = 2
    KDB: enter: panic
    panic.txt0600001412440205665  7137 ustarrootwheeldouble faultversion.txt06000021612440205665  7616 ustarrootwheelFreeBSD 8.3-RELEASE-p16 #0: Mon Aug 25 08:25:41 EDT 2014
        root@pf2_1_1_i386.pfsense.org:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.8

    TPEPFS02-1205-0934-crash.txt
    TPEPFS02-1204-1104-crash.txt


  • Netgate Administrator

    You have 16 igb interfaces?! And 2 em interfaces?
    I would certainly try the suggestions here in that case:
    https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#Intel_igb.284.29_and_em.284.29_Cards

    Steve



  • Hi Steve:
    Thanks for your input at first.
    Yes, we put 12 Gigabit Ethernet and 4 Fiber for prioritize traffic. And use 12 GBE for now.

    I did add kern.ipc.nmbclusters="2304000" into /boot/loader.conf. not /boot/loader.conf.local.
    Then I will try put into "loader.conf.local" file and feedback result here.

    Any other clue or config need turn on for more detail when crash occured ?
    Examine the RRD graph could to link crashed caused by out of MBUF.


  • Netgate Administrator

    Yes if it is MBUF exhaustion you should be able to see it recorded in the RRD data. Assuming it wasn't lost in the crash.

    The Intel NICs will record detailed stats in the sysctls that may hold some clue. Otherwise I think this in one for the devs.  ;)

    Steve



  • Hi Steve:

    It's look work well now. But question is MBUF usage not figure out on RRD graph. (See attached.)
    RRD graph show not exhaust but stable, hardly to link caused system crash.  :-
    And found the MBUF usage goes down when apply change. (Seed 1Week graph)

    Anyway, will keep track 1 week to see this issue solved by change the "boot.conf.local"  :) :) :)

    Appreciate your information again.

    Charlie

    PS: Sorry, not able to upload in reply.


  • Netgate Administrator

    This sort of system is outside my experience to honest. There were some issues a while ago. For example I believe this no longer applies but because none of the boxes I use igb NICs I can't be sure  :-\ :
    https://forum.pfsense.org/index.php?topic=69486.0
    If it happens again you might consider going to 2.2 which has newer drivers as well as many other improvements.

    Steve


Log in to reply