System Crash, Crash Report Submitted

[pfarbi]

Hi,

We had a system crash today, firewall became unresponsive and then it rebooted.

pfsense version details are as follows

2.3.4-RELEASE-p1 (i386)
built on Fri Jul 14 14:53:03 CDT 2017
FreeBSD 10.3-RELEASE-p19

Once we got back into the GUI we uploaded the crash report. https://doc.pfsense.org/index.php/Obtaining_Panic_Information_for_Developers

Now I would like to call attention to that crash report in the hope that a developer would like to spend few moments to analyze it and give us some feedback on a cause and any steps we need to take to prevent the issue from happening again.

[jimp]

I see a crash report submitted today from your IP address, at least the one you posted on the forum from.

Fatal double fault:
eip = 0xc12b7f08
esp = 0xc82c5008
ebp = 0xc82c6768
cpuid = 2; apic id = 02
panic: double fault
cpuid = 2
KDB: enter: panic

db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100042 td 0xc8954c80
kdb_enter(c145f7d6,c145f7d6,c1626a97,c1f98574,2,...) at kdb_enter+0x3d/frame 0xc1f98520
vpanic(c1626a97,c1f98574,c1f98574,c1f9858c,c12ce1bb,...) at vpanic+0x13b/frame 0xc1f98554
panic(c1626a97,2,2,2,c82c6768,...) at panic+0x1b/frame 0xc1f98568
dblfault_handler() at dblfault_handler+0xab/frame 0xc1f98568
--- trap 0x17, eip = 0xc12b7f08, esp = 0xc82c5008, ebp = 0xc82c6768 ---
Xpage(8,28,28,0,ca247728,...) at Xpage/frame 0xc82c6768
Xinvlrng(c82c6940,e,0,0,c82c6960,...) at Xinvlrng+0x2d/frame 0xc82c6920
ipfw_check_frame(0,c82c6a94,c8937800,1,0,...) at ipfw_check_frame+0x130/frame 0xc82c6a70
pfil_run_hooks(c1fdf9a0,c82c6ae0,c8937800,1,0,...) at pfil_run_hooks+0x88/frame 0xc82c6ac4
ether_demux(c8937800,cc7b3c00,6,c1f9ba00,c82c6b80,...) at ether_demux+0x54/frame 0xc82c6af0
ether_nh_input(cc7b3c00,c82c6b7c,c895a000,37,c97b3600,...) at ether_nh_input+0x36b/frame 0xc82c6b3c
netisr_dispatch_src(9,0,cc7b3c00) at netisr_dispatch_src+0x8b/frame 0xc82c6b84
netisr_dispatch(9,cc7b3c00) at netisr_dispatch+0x20/frame 0xc82c6b98
ether_input(c8937800,cc7b3c00,c0d258e0,c1f6892c,2,...) at ether_input+0x19/frame 0xc82c6ba8
bge_rxeof(1,c8712960,2,2,2,...) at bge_rxeof+0x4f5/frame 0xc82c6c08
bge_intr(c895a000,0,246,0,2f761479,...) at bge_intr+0x175/frame 0xc82c6c3c
intr_event_execute_handlers(109,c8710280,c145a5db,55b,0,...) at intr_event_execute_handlers+0xaa/frame 0xc82c6c68
ithread_loop(c8935140,c82c6ce8,0,0,0,...) at ithread_loop+0x80/frame 0xc82c6ca4
fork_exit(c0cbe860,c8935140,c82c6ce8) at fork_exit+0xa3/frame 0xc82c6cd4
fork_trampoline() at fork_trampoline+0x8/frame 0xc82c6cd4
--- trap 0, eip = 0, esp = 0xc82c6d20, ebp = 0 ---

That doesn't look familiar, though. A "double fault" is usually, but not always, in drivers or hardware somewhere. The presence of "ipfw" in the backtrace could mean a couple different things though.

Are you using captive portal? With Per-user bandwidth limits?
Are you using Limiters?
Is this part of an HA cluster? Do you have pfsync enabled?

[pfarbi]

Thanks for your reply

• Yes we are using captive portal with per user bandwidth limits

• No we are not using limiters

• No we don't have pfsync enabled and neither is it part of HA cluster

[jimp]

If there is no ha or pfsync then it's probably not anything we've seen before, or at least anything I recognize.

The double fault makes me lean toward hardware, if pfsync isn't a factor. Is that hardware capable of running a 64-bit version? Or is it only 32-bit?