Need help with crash

travisbrackett · Aug 21, 2015, 6:57 PM

Hi. I recently had my pfsense box crash. The wiki https://doc.pfsense.org/index.php/Obtaining_Panic_Information_for_Developers says to ask the forums for help with analyzing. Here's the relevant information:

IP address: 216.57.195.98
Submitted @ 18:49 UTC/GMT on Friday, August 21, 2015

Any help would be greatly appreciated!

jimp · Aug 24, 2015, 1:16 PM

There was no clear known cause I could see in the crash report, but everything centered around your bce0 NIC. Hardware issue perhaps? Or maybe some tuning changes need made.

db:0:kdb.enter.default>  show pcpu
cpuid        = 9
dynamic pcpu = 0xfffffe010daed800
curthread    = 0xfffff800037f0000: pid 12 "irq256: bce0"
curpcb       = 0xfffffe00003a9cc0
fpcurthread  = none
idlethread   = 0xfffff800035f5000: tid 100012 "idle: cpu9"
curpmap      = 0xffffffff82181fd8
tssp         = 0xffffffff8219d338
commontssp   = 0xffffffff8219d338
rsp0         = 0xfffffe00003a9cc0
gs32p        = 0xffffffff8219ed90
ldt          = 0xffffffff8219edd0
tss          = 0xffffffff8219edc0
db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100072 td 0xfffff800037f0000
bce_intr() at bce_intr+0x4e1/frame 0xfffffe00003a9b20
intr_event_execute_handlers() at intr_event_execute_handlers+0xab/frame 0xfffffe00003a9b60
ithread_loop() at ithread_loop+0x96/frame 0xfffffe00003a9bb0
fork_exit() at fork_exit+0x9a/frame 0xfffffe00003a9bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00003a9bf0

Fatal trap 12: page fault while in kernel mode
cpuid = 9; apic id = 01
fault virtual address	= 0x18
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff804ad801
stack pointer	        = 0x28:0xfffffe00003a9a90
frame pointer	        = 0x28:0xfffffe00003a9b20
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 12 (irq256: bce0)

There were several link state transitions for bce0 along with watchdog timeouts before the crash, but the message buffer isn't timestamped so there is no way to tell if they were immediately before the crash or some time ago:

bce0: /usr/pfSensesrc/src/sys/dev/bce/if_bce.c(7869): Watchdog timeout occurred, resetting!
<5>bce0: link state changed to DOWN
bce0: 
<5>bce0: link state changed to UP
Gigabit link up!
bce0: Gigabit link up!
<5>bce0: link state changed to DOWN
bce0: 
<5>bce0: link state changed to UP
Gigabit link up!
bce0: Gigabit link up!
bce0: /usr/pfSensesrc/src/sys/dev/bce/if_bce.c(7869): Watchdog timeout occurred, resetting!
<5>bce0: link state changed to DOWN
bce0: 
<5>bce0: link state changed to UP
Gigabit link up!
bce0: Gigabit link up!
bce0: /usr/pfSensesrc/src/sys/dev/bce/if_bce.c(7869): Watchdog timeout occurred, resetting!
<5>bce0: link state changed to DOWN
bce0: 
<5>bce0: link state changed to UP
Gigabit link up!
bce0: Gigabit link up!
<5>ovpns3: link state changed to DOWN
<5>ovpns3: link state changed to UP
<5>ovpns3: link state changed to DOWN
<5>ovpns3: link state changed to UP
bce0: /usr/pfSensesrc/src/sys/dev/bce/if_bce.c(7869): Watchdog timeout occurred, resetting!
<5>bce0: link state changed to DOWN
bce0: 
<5>bce0: link state changed to UP
Gigabit link up!
bce0: Gigabit link up!
bce0: /usr/pfSensesrc/src/sys/dev/bce/if_bce.c(7869): Watchdog timeout occurred, resetting!
<5>bce0: link state changed to DOWN
bce0: 
<5>bce0: link state changed to UP
Gigabit link up!
bce0: Gigabit link up!

I'd be suspicious of that NIC if it were me.