Need assist to examine crash log

charlie1970

Hello All Advance Guys.
I met system crash problem almost 1 month. And search forum or freebsd related topics. But sill not able to figure the direction.
And no idea after upload crash report via Pfsense WebUI.

Try to post here if could get further information will very appreciate. (Attached full crash log)

Machine information
Hardware: Axiomtek NA550. with AX93306-4GIL (4-Port Gigabit with LAN bypass)
CPU: Intel Atom i3-2120@3.3 GHz, 2 Cores with 2 SMT Threads
Memory: 2GB
HD: SSD 60GB
LAN Chipset: Intel 82580DB

Purpose: Traffic shaping (LAN Bandwidth Control)
Maximum Throughput: 357Mb/s (Not crash at peak loading)

Part of crash log

Crash report begins. Anonymous machine information:

i386
8.3-RELEASE-p16
FreeBSD 8.3-RELEASE-p16 #0: Mon Aug 25 08:25:41 EDT 2014 root@pf2_1_1_i386.pfsense.org:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.8

Crash report details:

Filename: /var/crash/bounds
1

Filename: /var/crash/info.0
Dump header from device /dev/ad4s1b
Architecture: i386
Architecture Version: 1
Dump Length: 78336B (0 MB)
Blocksize: 512
Dumptime: Fri Dec 5 09:34:45 2014
Hostname: tpepfs02
Magic: FreeBSD Text Dump
Version String: FreeBSD 8.3-RELEASE-p16 #0: Mon Aug 25 08:25:41 EDT 2014
root@pf2_1_1_i386.pfsense.org:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.8
Panic String: double fault
Dump Parity: 2406528355
Bounds: 0
Dump Status: good

Filename: /var/crash/textdump.tar.0
ddb.txt06000014000012440205665 7076 ustarrootwheeldb:0:kdb.enter.default> run lockinfo
db:1:lockinfo> show locks
No such command
db:1:locks> show alllocks
No such command
db:1:alllocks> show lockedvnods
Locked vnodes
db:0:kdb.enter.default> show pcpu
cpuid = 2
dynamic pcpu = 0x42a8700
curthread = 0xc59945c0: pid 11 "idle: cpu2"
curpcb = 0xc57e7d80
fpcurthread = none
idlethread = 0xc59945c0: tid 100004 "idle: cpu2"
APIC ID = 2
currentldt = 0x50
db:0:kdb.enter.default> bt
Tracing pid 11 tid 100004 td 0xc59945c0
kdb_enter(c0f98d73,c0f98d73,c0fccc1e,c15d02f0,2,…) at kdb_enter+0x3b
panic(c0fccc1e,2,2,0,0,...) at panic+0x102
dblfault_handler() at dblfault_handler+0x9b
--- trap 0x17, eip = 0xc0e750d8, esp = 0xc57e7000, ebp = 0xc57e7c7c ---
Xpage(c57e7c98,c151f8b0,2,0,ffffffff,...) at Xpage
acpi_cpu_idle(1,c57e7ccc,c0f99f18,a08,c59945c0,...) at acpi_cpu_idle+0x1e1
sched_idletd(0,c57e7d28,245c8948,0,0,...) at sched_idletd+0x116
fork_exit(c0ac8fb0,0,c57e7d28) at fork_exit+0x87
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc57e7d60, ebp = 0 ---
db:0:kdb.enter.default> ps
:
:

Fatal double fault:
eip = 0xc0e750d8
esp = 0xc57e7000
ebp = 0xc57e7c7c
cpuid = 2; apic id = 02
panic: double fault
cpuid = 2
KDB: enter: panic
panic.txt0600001412440205665 7137 ustarrootwheeldouble faultversion.txt06000021612440205665 7616 ustarrootwheelFreeBSD 8.3-RELEASE-p16 #0: Mon Aug 25 08:25:41 EDT 2014
root@pf2_1_1_i386.pfsense.org:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.8

TPEPFS02-1205-0934-crash.txt
TPEPFS02-1204-1104-crash.txt

stephenw10

You have 16 igb interfaces?! And 2 em interfaces?
I would certainly try the suggestions here in that case:
https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#Intel_igb.284.29_and_em.284.29_Cards

Steve

charlie1970

Hi Steve:
Thanks for your input at first.
Yes, we put 12 Gigabit Ethernet and 4 Fiber for prioritize traffic. And use 12 GBE for now.

I did add kern.ipc.nmbclusters="2304000" into /boot/loader.conf. not /boot/loader.conf.local.
Then I will try put into "loader.conf.local" file and feedback result here.

Any other clue or config need turn on for more detail when crash occured ?
Examine the RRD graph could to link crashed caused by out of MBUF.

stephenw10

Yes if it is MBUF exhaustion you should be able to see it recorded in the RRD data. Assuming it wasn't lost in the crash.

The Intel NICs will record detailed stats in the sysctls that may hold some clue. Otherwise I think this in one for the devs. ;)

Steve

charlie1970

Hi Steve:

It's look work well now. But question is MBUF usage not figure out on RRD graph. (See attached.)
RRD graph show not exhaust but stable, hardly to link caused system crash. :-
And found the MBUF usage goes down when apply change. (Seed 1Week graph)

Anyway, will keep track 1 week to see this issue solved by change the "boot.conf.local" :) :) :)

Appreciate your information again.

Charlie

PS: Sorry, not able to upload in reply.

stephenw10

This sort of system is outside my experience to honest. There were some issues a while ago. For example I believe this no longer applies but because none of the boxes I use igb NICs I can't be sure :-\ :
https://forum.pfsense.org/index.php?topic=69486.0
If it happens again you might consider going to 2.2 which has newer drivers as well as many other improvements.

Steve