PfSense page faults on IPS mode, crashes on netmap_default_mbuf_destructor()



  • Hello!

    • I'm running pfSense 2.4.0-BETA, latest snapshot, on a PCEngines apu2c4 with a 30GB Intel 525 SSD.

    • All hardware offloads are disabled, and PowerD is set to Maximum.

    • All Suricata options are set to default, besides the rules loaded (mainly ET rules).

    • Some Suricata rules are set to DROP via dropsid.conf

    • Suricata is listening on the WAN interface, which is a VLAN on a i210AT port.

    • The other two i210AT are LAGG'd and expose a few other VLANs. I think this might be problematic but I've found no information about it, nor do I know if this should have a connection as they are independent PCI devices as far as I know.

    I'm having a problem where, if I enable IPS mode, netmap seems to work properly for some time, even blocking matching packets, but after a while, maybe 3 min or maybe 2 days, it will crash with a page fault when calling netmap_default_mbuf_destructor(). pfSense will then sometimes reboot gracefully and re-up all the services, or it will reboot in a glitched state where maybe Suricata is down or maybe all the other services are down.

    It seems to be very similar to this bug where the user was advised to ask on the forums for more information.

    I'll gladly provide more information as needed if there's anything I'm missing.

    Thanks for any help in advance.
    log.txt



  • I have same issue when I active suricata inline mode it's work for awhile then it's crash with infinity text error on console so i have to turn off power and turn on again. if i use legacy mode it's work fine.

    I tried below tune without chance to solve issue:

    net.inet.tcp.tso=0
    hw.igb.num_queues=1
    hw.pci.enable_msix=0

    error message header :

    
    Fatal trap 19: non-maskable interrupt trap while in kernel mode
    
    Fatal trap 19: non-maskable interrupt trap while in kernel mode
    cpuid = 0; cpuid = 2; Fatal trap 19: non-maskable interrupt trap while in kernel mode
    Fatal trap 19: non-maskable interrupt trap while in kernel mode
    apic id = 04
    cpuid = 1; cpuid = 3; instruction pointer       = 0x20:0xffffffff813071e6
    apic id = 00
    apic id = 02
    apic id = 06
    stack pointer           = 0x28:0xfffffe0226bc4fe0
    instruction pointer     = 0x20:0xffffffff813071e6
    frame pointer           = 0x28:0xfffffe0226be88f0
    instruction pointer     = 0x20:0xffffffff813071e6
    code segment            = base 0x0, limit 0xfffff, type 0x1b
    stack pointer           = 0x28:0xfffffe01e9df8fe0
    stack pointer           = 0x28:0xfffffe0226bccfe0
                            = DPL 0, pres 1, long 1, def32 0, gran 1
    instruction pointer     = 0x20:0xffffffff813071e6
    frame pointer           = 0x28:0xfffffe0226bed8f0
    processor eflags        = stack pointer         = 0x28:0xffffffff82978820
    interrupt enabled, frame pointer                = 0x28:0xfffffe0226be38f0
    IOPL = 0
    code segment            = base 0x0, limit 0xfffff, type 0x1b
    current process         = 11 (idle: cpu2)
    code segment            = base 0x0, limit 0xfffff, type 0x1b
    frame pointer           = 0x28:0xfffffe0226bde8f0
                            = DPL 0, pres 1, long 1, def32 0, gran 1
                            = DPL 0, pres 1, long 1, def32 0, gran 1
    code segment            = base 0x0, limit 0xfffff, type 0x1b
    processor eflags        = processor eflags      = interrupt enabled,                    = DPL 0, pres 1, long 1, def32 0, gran 1
    IOPL = 0
    processor eflags        = current process               = 11 (idle: cpu3)
    interrupt enabled, interrupt enabled, IOPL = 0
    IOPL = 0
    current process         = 11 (idle: cpu0)
    current process         = 11 (idle: cpu1)
    timeout stopping cpus
    [ thread pid 11 tid 100005 ]
    Stopped at      acpi_cpu_c1+0x6:        popq    %rbp
    db:0:kdb.enter.default> textdump set
    textdump set
    db:0:kdb.enter.default>  capture on
    db:0:kdb.enter.default>  run lockinfo
    db:1:lockinfo> show locks
    No such command
    db:1:locks>  show alllocks
    No such command
    db:1:alllocks>  show lockedvnods
    Locked vnodes
    db:0:kdb.enter.default>  show pcpu
    cpuid        = 2
    dynamic pcpu = 0xfffffe02a45b9200
    curthread    = 0xfffff80005202500: pid 11 "idle: cpu2"
    curpcb       = 0xfffffe0226be8b80
    fpcurthread  = none
    idlethread   = 0xfffff80005202500: tid 100005 "idle: cpu2"
    curpmap      = 0xffffffff829e6300
    tssp         = 0xffffffff82a1ebe0
    commontssp   = 0xffffffff82a1ebe0
    rsp0         = 0xfffffe0226be8b80
    gs32p        = 0xffffffff82a25438
    ldt          = 0xffffffff82a25478
    tss          = 0xffffffff82a25468
    db:0:kdb.enter.default>  bt
    Tracing pid 11 tid 100005 td 0xfffff80005202500
    acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0226be88f0
    acpi_cpu_idle() at acpi_cpu_idle+0x2e2/frame 0xfffffe0226be8940
    cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0226be8960
    cpu_idle() at cpu_idle+0x95/frame 0xfffffe0226be8980
    sched_idletd() at sched_idletd+0x3d3/frame 0xfffffe0226be8a70
    fork_exit() at fork_exit+0x85/frame 0xfffffe0226be8ab0
    fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0226be8ab0
    --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
    db:0:kdb.enter.default>  ps
      pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
    52152 89222   298     0  S       nanslp   0xffffffff82866b31 sleep
    24906   298   298     0  S       accept   0xfffff8000c61306c php-fpm
    60283 59946 60283     0  S+      ttyin    0xfffff800080060a8 sh
    59946 59726 59946     0  S+      wait     0xfffff8006e8c3528 sh
    59726     1 59726     0  Ss+     wait     0xfffff8000c7ad528 login
    89222     1   298     0  S       wait     0xfffff8000c967528 sh
    88607     1 88607   136  Ss      select   0xfffff8000c298040 dhcpd
    77035     1 77035    59  Ss      (threaded)                  unbound
    100654                   S       kqread   0xfffff8000c9cae00 unbound
    100691                   S       kqread   0xfffff8000c9c5b00 unbound
    100692                   S       kqread   0xfffff8000c489000 unbound
    100693                   S       kqread   0xfffff8000c9b0600 unbound
    64289     1 64289     0  Ss      (threaded)                  dpinger
    
    

    anyone can help us on this matter . Thanks


Log in to reply