Firewall has crashed several times over last week



  • Our firewall crashed 3 times on Thursday of last week and twice yesterday, Tuesday May 10th.  We submitted the most recent crash report  at around 9:40AM today Central time from IP: 38.92.135.226.

    We are on:

    2.3-RELEASE (amd64)
    built on Mon Apr 11 18:10:34 CDT 2016
    FreeBSD 10.3-RELEASE

    I found this page:

    https://doc.pfsense.org/index.php/Obtaining_Panic_Information_for_Developers

    that advised calling attention to the crash report submission for analysis on the forum.  Advice on what is happening would be much appreciated!


  • Rebel Alliance Developer Netgate

    Something appears to be very, very unhappy in the bce card/driver. Was there anything different/special about those days, traffic-wise? Or anything new that was turned on/configured?

    If this is an HA pair, do you perhaps have limiters and state synchronization both enabled?

    [zone: pf states] PF states limit reached
    bce0: discard frame w/o packet header
    bce0: discard frame w/o leading ethernet header (len 0 pkt len 0)
    
    Fatal trap 9: general protection fault while in kernel mode
    cpuid = 2; apic id = 02
    instruction pointer	= 0x20:0xffffffff80b7e8fa
    stack pointer	        = 0x28:0xfffffe003442a910
    frame pointer	        = 0x28:0xfffffe003442a930
    code segment		= base 0x0, limit 0xfffff, type 0x1b
    			= DPL 0, pres 1, long 1, def32 0, gran 1
    processor eflags	= interrupt enabled, resume, IOPL = 0
    current process		= 12 (irq284: bce0)
    
    
    db:0:kdb.enter.default>  bt
    Tracing pid 12 tid 100076 td 0xfffff80003510960
    mb_free_ext() at mb_free_ext+0x8a/frame 0xfffffe003442a930
    m_freem() at m_freem+0x18/frame 0xfffffe003442a950
    bce_intr() at bce_intr+0x646/frame 0xfffffe003442a9e0
    intr_event_execute_handlers() at intr_event_execute_handlers+0xab/frame 0xfffffe003442aa20
    ithread_loop() at ithread_loop+0x96/frame 0xfffffe003442aa70
    fork_exit() at fork_exit+0x9a/frame 0xfffffe003442aab0
    fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003442aab0
    --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
    
    
    Fatal trap 9: general protection fault while in kernel mode
    
    cpuid = 2; 
    apic id = 02
    Fatal trap 9: general protection fault while in kernel mode
    instruction pointer	= 0x20:0xffffffff80fa109a
    cpuid = 3; stack pointer	        = 0x28:0xfffffe003442a910
    apic id = 03
    frame pointer	        = 0x28:0xfffffe003442a920
    instruction pointer	= 0x20:0xffffffff80be5854
    code segment		= base 0x0, limit 0xfffff, type 0x1b
    stack pointer	        = 0x28:0xfffffe0031ff7810
    			= DPL 0, pres 1, long 1, def32 0, gran 1
    frame pointer	        = 0x28:0xfffffe0031ff7870
    processor eflags	= code segment		= base 0x0, limit 0xfffff, type 0x1b
    interrupt enabled, 			= DPL 0, pres 1, long 1, def32 0, gran 1
    resume, processor eflags	= IOPL = 0
    interrupt enabled, current process		= 12 (irq284: bce0)
    
    
    db:0:kdb.enter.default>  bt
    Tracing pid 12 tid 100076 td 0xfffff80003510960
    bcopy() at bcopy+0x1a/frame 0xfffffe003442a920
    m_cat() at m_cat+0xbf/frame 0xfffffe003442a950
    bce_intr() at bce_intr+0x4f9/frame 0xfffffe003442a9e0
    intr_event_execute_handlers() at intr_event_execute_handlers+0xab/frame 0xfffffe003442aa20
    ithread_loop() at ithread_loop+0x96/frame 0xfffffe003442aa70
    fork_exit() at fork_exit+0x9a/frame 0xfffffe003442aab0
    fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003442aab0
    --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
    
    
    bce0: discard frame w/o packet header
    bce0: discard frame w/o packet header
    bce0: discard frame w/o packet header
    
    Fatal trap 9: general protection fault while in kernel mode
    cpuid = 2; apic id = 02
    instruction pointer	= 0x20:0xffffffff80b7e8fa
    stack pointer	        = 0x28:0xfffffe003442a910
    frame pointer	        = 0x28:0xfffffe003442a930
    code segment		= base 0x0, limit 0xfffff, type 0x1b
    			= DPL 0, pres 1, long 1, def32 0, gran 1
    processor eflags	= interrupt enabled, resume, IOPL = 0
    current process		= 12 (irq284: bce0)
    
    
    db:0:kdb.enter.default>  bt
    Tracing pid 12 tid 100076 td 0xfffff80003510960
    mb_free_ext() at mb_free_ext+0x8a/frame 0xfffffe003442a930
    m_freem() at m_freem+0x18/frame 0xfffffe003442a950
    bce_intr() at bce_intr+0x646/frame 0xfffffe003442a9e0
    intr_event_execute_handlers() at intr_event_execute_handlers+0xab/frame 0xfffffe003442aa20
    ithread_loop() at ithread_loop+0x96/frame 0xfffffe003442aa70
    fork_exit() at fork_exit+0x9a/frame 0xfffffe003442aab0
    fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003442aab0
    --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
    
    


  • Thanks for looking.  No major changes to the network, the load or the firewall config recently.  We don't have a HA pair.  We upgraded to 2.3-RELEASE on the 17th of last month.

    We do have a cold backup which I swapped over to but we had another crash afterwards,  I uploaded the crash report at about 08:24AM central time.  To be extra safe I did not transfer the config from the running firewall to the backup so it is still running the config from the 17th of last month when we upgraded both firewalls.

    The new crash looks pretty different from the last one I uploaded I am unsure if it still points towards the bce card/driver.


Log in to reply