System Crash, Crash Report Submitted

pfarbi

Hi,

We had a system crash today, firewall became unresponsive and then it rebooted.

pfsense version details are as follows

2.3.4-RELEASE-p1 (i386)
built on Fri Jul 14 14:53:03 CDT 2017
FreeBSD 10.3-RELEASE-p19

Once we got back into the GUI we uploaded the crash report. https://doc.pfsense.org/index.php/Obtaining_Panic_Information_for_Developers

Now I would like to call attention to that crash report in the hope that a developer would like to spend few moments to analyze it and give us some feedback on a cause and any steps we need to take to prevent the issue from happening again.

jimp

I see a crash report submitted today from your IP address, at least the one you posted on the forum from.

Fatal double fault:
eip = 0xc12b7f08
esp = 0xc82c5008
ebp = 0xc82c6768
cpuid = 2; apic id = 02
panic: double fault
cpuid = 2
KDB: enter: panic

db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100042 td 0xc8954c80
kdb_enter(c145f7d6,c145f7d6,c1626a97,c1f98574,2,...) at kdb_enter+0x3d/frame 0xc1f98520
vpanic(c1626a97,c1f98574,c1f98574,c1f9858c,c12ce1bb,...) at vpanic+0x13b/frame 0xc1f98554
panic(c1626a97,2,2,2,c82c6768,...) at panic+0x1b/frame 0xc1f98568
dblfault_handler() at dblfault_handler+0xab/frame 0xc1f98568
--- trap 0x17, eip = 0xc12b7f08, esp = 0xc82c5008, ebp = 0xc82c6768 ---
Xpage(8,28,28,0,ca247728,...) at Xpage/frame 0xc82c6768
Xinvlrng(c82c6940,e,0,0,c82c6960,...) at Xinvlrng+0x2d/frame 0xc82c6920
ipfw_check_frame(0,c82c6a94,c8937800,1,0,...) at ipfw_check_frame+0x130/frame 0xc82c6a70
pfil_run_hooks(c1fdf9a0,c82c6ae0,c8937800,1,0,...) at pfil_run_hooks+0x88/frame 0xc82c6ac4
ether_demux(c8937800,cc7b3c00,6,c1f9ba00,c82c6b80,...) at ether_demux+0x54/frame 0xc82c6af0
ether_nh_input(cc7b3c00,c82c6b7c,c895a000,37,c97b3600,...) at ether_nh_input+0x36b/frame 0xc82c6b3c
netisr_dispatch_src(9,0,cc7b3c00) at netisr_dispatch_src+0x8b/frame 0xc82c6b84
netisr_dispatch(9,cc7b3c00) at netisr_dispatch+0x20/frame 0xc82c6b98
ether_input(c8937800,cc7b3c00,c0d258e0,c1f6892c,2,...) at ether_input+0x19/frame 0xc82c6ba8
bge_rxeof(1,c8712960,2,2,2,...) at bge_rxeof+0x4f5/frame 0xc82c6c08
bge_intr(c895a000,0,246,0,2f761479,...) at bge_intr+0x175/frame 0xc82c6c3c
intr_event_execute_handlers(109,c8710280,c145a5db,55b,0,...) at intr_event_execute_handlers+0xaa/frame 0xc82c6c68
ithread_loop(c8935140,c82c6ce8,0,0,0,...) at ithread_loop+0x80/frame 0xc82c6ca4
fork_exit(c0cbe860,c8935140,c82c6ce8) at fork_exit+0xa3/frame 0xc82c6cd4
fork_trampoline() at fork_trampoline+0x8/frame 0xc82c6cd4
--- trap 0, eip = 0, esp = 0xc82c6d20, ebp = 0 ---

That doesn't look familiar, though. A "double fault" is usually, but not always, in drivers or hardware somewhere. The presence of "ipfw" in the backtrace could mean a couple different things though.

Are you using captive portal? With Per-user bandwidth limits?
Are you using Limiters?
Is this part of an HA cluster? Do you have pfsync enabled?

pfarbi

Thanks for your reply

Yes we are using captive portal with per user bandwidth limits
No we are not using limiters
No we don't have pfsync enabled and neither is it part of HA cluster

jimp

If there is no ha or pfsync then it's probably not anything we've seen before, or at least anything I recognize.

The double fault makes me lean toward hardware, if pfsync isn't a factor. Is that hardware capable of running a 64-bit version? Or is it only 32-bit?