Firewall has crashed several times over last week

sand7000

Our firewall crashed 3 times on Thursday of last week and twice yesterday, Tuesday May 10th. We submitted the most recent crash report at around 9:40AM today Central time from IP: 38.92.135.226.

We are on:

2.3-RELEASE (amd64)
built on Mon Apr 11 18:10:34 CDT 2016
FreeBSD 10.3-RELEASE

I found this page:

https://doc.pfsense.org/index.php/Obtaining_Panic_Information_for_Developers

that advised calling attention to the crash report submission for analysis on the forum. Advice on what is happening would be much appreciated!

jimp

Something appears to be very, very unhappy in the bce card/driver. Was there anything different/special about those days, traffic-wise? Or anything new that was turned on/configured?

If this is an HA pair, do you perhaps have limiters and state synchronization both enabled?

[zone: pf states] PF states limit reached
bce0: discard frame w/o packet header
bce0: discard frame w/o leading ethernet header (len 0 pkt len 0)

Fatal trap 9: general protection fault while in kernel mode
cpuid = 2; apic id = 02
instruction pointer	= 0x20:0xffffffff80b7e8fa
stack pointer	        = 0x28:0xfffffe003442a910
frame pointer	        = 0x28:0xfffffe003442a930
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 12 (irq284: bce0)

db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100076 td 0xfffff80003510960
mb_free_ext() at mb_free_ext+0x8a/frame 0xfffffe003442a930
m_freem() at m_freem+0x18/frame 0xfffffe003442a950
bce_intr() at bce_intr+0x646/frame 0xfffffe003442a9e0
intr_event_execute_handlers() at intr_event_execute_handlers+0xab/frame 0xfffffe003442aa20
ithread_loop() at ithread_loop+0x96/frame 0xfffffe003442aa70
fork_exit() at fork_exit+0x9a/frame 0xfffffe003442aab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003442aab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Fatal trap 9: general protection fault while in kernel mode

cpuid = 2; 
apic id = 02
Fatal trap 9: general protection fault while in kernel mode
instruction pointer	= 0x20:0xffffffff80fa109a
cpuid = 3; stack pointer	        = 0x28:0xfffffe003442a910
apic id = 03
frame pointer	        = 0x28:0xfffffe003442a920
instruction pointer	= 0x20:0xffffffff80be5854
code segment		= base 0x0, limit 0xfffff, type 0x1b
stack pointer	        = 0x28:0xfffffe0031ff7810
			= DPL 0, pres 1, long 1, def32 0, gran 1
frame pointer	        = 0x28:0xfffffe0031ff7870
processor eflags	= code segment		= base 0x0, limit 0xfffff, type 0x1b
interrupt enabled, 			= DPL 0, pres 1, long 1, def32 0, gran 1
resume, processor eflags	= IOPL = 0
interrupt enabled, current process		= 12 (irq284: bce0)

db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100076 td 0xfffff80003510960
bcopy() at bcopy+0x1a/frame 0xfffffe003442a920
m_cat() at m_cat+0xbf/frame 0xfffffe003442a950
bce_intr() at bce_intr+0x4f9/frame 0xfffffe003442a9e0
intr_event_execute_handlers() at intr_event_execute_handlers+0xab/frame 0xfffffe003442aa20
ithread_loop() at ithread_loop+0x96/frame 0xfffffe003442aa70
fork_exit() at fork_exit+0x9a/frame 0xfffffe003442aab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003442aab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

bce0: discard frame w/o packet header
bce0: discard frame w/o packet header
bce0: discard frame w/o packet header

Fatal trap 9: general protection fault while in kernel mode
cpuid = 2; apic id = 02
instruction pointer	= 0x20:0xffffffff80b7e8fa
stack pointer	        = 0x28:0xfffffe003442a910
frame pointer	        = 0x28:0xfffffe003442a930
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 12 (irq284: bce0)

db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100076 td 0xfffff80003510960
mb_free_ext() at mb_free_ext+0x8a/frame 0xfffffe003442a930
m_freem() at m_freem+0x18/frame 0xfffffe003442a950
bce_intr() at bce_intr+0x646/frame 0xfffffe003442a9e0
intr_event_execute_handlers() at intr_event_execute_handlers+0xab/frame 0xfffffe003442aa20
ithread_loop() at ithread_loop+0x96/frame 0xfffffe003442aa70
fork_exit() at fork_exit+0x9a/frame 0xfffffe003442aab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003442aab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

sand7000

Thanks for looking. No major changes to the network, the load or the firewall config recently. We don't have a HA pair. We upgraded to 2.3-RELEASE on the 17th of last month.

We do have a cold backup which I swapped over to but we had another crash afterwards, I uploaded the crash report at about 08:24AM central time. To be extra safe I did not transfer the config from the running firewall to the backup so it is still running the config from the 17th of last month when we upgraded both firewalls.

The new crash looks pretty different from the last one I uploaded I am unsure if it still points towards the bce card/driver.