PfSense page faults on IPS mode, crashes on netmap_default_mbuf_destructor()

jvelez · Jul 12, 2017, 10:46 AM

Hello!

I'm running pfSense 2.4.0-BETA, latest snapshot, on a PCEngines apu2c4 with a 30GB Intel 525 SSD.
All hardware offloads are disabled, and PowerD is set to Maximum.
All Suricata options are set to default, besides the rules loaded (mainly ET rules).
Some Suricata rules are set to DROP via dropsid.conf
Suricata is listening on the WAN interface, which is a VLAN on a i210AT port.
The other two i210AT are LAGG'd and expose a few other VLANs. I think this might be problematic but I've found no information about it, nor do I know if this should have a connection as they are independent PCI devices as far as I know.

I'm having a problem where, if I enable IPS mode, netmap seems to work properly for some time, even blocking matching packets, but after a while, maybe 3 min or maybe 2 days, it will crash with a page fault when calling netmap_default_mbuf_destructor(). pfSense will then sometimes reboot gracefully and re-up all the services, or it will reboot in a glitched state where maybe Suricata is down or maybe all the other services are down.

It seems to be very similar to this bug where the user was advised to ask on the forums for more information.

I'll gladly provide more information as needed if there's anything I'm missing.

Thanks for any help in advance.
log.txt

barakah · Jul 17, 2017, 3:01 PM

I have same issue when I active suricata inline mode it's work for awhile then it's crash with infinity text error on console so i have to turn off power and turn on again. if i use legacy mode it's work fine.

I tried below tune without chance to solve issue:

net.inet.tcp.tso=0
hw.igb.num_queues=1
hw.pci.enable_msix=0

error message header :


Fatal trap 19: non-maskable interrupt trap while in kernel mode

Fatal trap 19: non-maskable interrupt trap while in kernel mode
cpuid = 0; cpuid = 2; Fatal trap 19: non-maskable interrupt trap while in kernel mode
Fatal trap 19: non-maskable interrupt trap while in kernel mode
apic id = 04
cpuid = 1; cpuid = 3; instruction pointer       = 0x20:0xffffffff813071e6
apic id = 00
apic id = 02
apic id = 06
stack pointer           = 0x28:0xfffffe0226bc4fe0
instruction pointer     = 0x20:0xffffffff813071e6
frame pointer           = 0x28:0xfffffe0226be88f0
instruction pointer     = 0x20:0xffffffff813071e6
code segment            = base 0x0, limit 0xfffff, type 0x1b
stack pointer           = 0x28:0xfffffe01e9df8fe0
stack pointer           = 0x28:0xfffffe0226bccfe0
                        = DPL 0, pres 1, long 1, def32 0, gran 1
instruction pointer     = 0x20:0xffffffff813071e6
frame pointer           = 0x28:0xfffffe0226bed8f0
processor eflags        = stack pointer         = 0x28:0xffffffff82978820
interrupt enabled, frame pointer                = 0x28:0xfffffe0226be38f0
IOPL = 0
code segment            = base 0x0, limit 0xfffff, type 0x1b
current process         = 11 (idle: cpu2)
code segment            = base 0x0, limit 0xfffff, type 0x1b
frame pointer           = 0x28:0xfffffe0226bde8f0
                        = DPL 0, pres 1, long 1, def32 0, gran 1
                        = DPL 0, pres 1, long 1, def32 0, gran 1
code segment            = base 0x0, limit 0xfffff, type 0x1b
processor eflags        = processor eflags      = interrupt enabled,                    = DPL 0, pres 1, long 1, def32 0, gran 1
IOPL = 0
processor eflags        = current process               = 11 (idle: cpu3)
interrupt enabled, interrupt enabled, IOPL = 0
IOPL = 0
current process         = 11 (idle: cpu0)
current process         = 11 (idle: cpu1)
timeout stopping cpus
[ thread pid 11 tid 100005 ]
Stopped at      acpi_cpu_c1+0x6:        popq    %rbp
db:0:kdb.enter.default> textdump set
textdump set
db:0:kdb.enter.default>  capture on
db:0:kdb.enter.default>  run lockinfo
db:1:lockinfo> show locks
No such command
db:1:locks>  show alllocks
No such command
db:1:alllocks>  show lockedvnods
Locked vnodes
db:0:kdb.enter.default>  show pcpu
cpuid        = 2
dynamic pcpu = 0xfffffe02a45b9200
curthread    = 0xfffff80005202500: pid 11 "idle: cpu2"
curpcb       = 0xfffffe0226be8b80
fpcurthread  = none
idlethread   = 0xfffff80005202500: tid 100005 "idle: cpu2"
curpmap      = 0xffffffff829e6300
tssp         = 0xffffffff82a1ebe0
commontssp   = 0xffffffff82a1ebe0
rsp0         = 0xfffffe0226be8b80
gs32p        = 0xffffffff82a25438
ldt          = 0xffffffff82a25478
tss          = 0xffffffff82a25468
db:0:kdb.enter.default>  bt
Tracing pid 11 tid 100005 td 0xfffff80005202500
acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0226be88f0
acpi_cpu_idle() at acpi_cpu_idle+0x2e2/frame 0xfffffe0226be8940
cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0226be8960
cpu_idle() at cpu_idle+0x95/frame 0xfffffe0226be8980
sched_idletd() at sched_idletd+0x3d3/frame 0xfffffe0226be8a70
fork_exit() at fork_exit+0x85/frame 0xfffffe0226be8ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0226be8ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
db:0:kdb.enter.default>  ps
  pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
52152 89222   298     0  S       nanslp   0xffffffff82866b31 sleep
24906   298   298     0  S       accept   0xfffff8000c61306c php-fpm
60283 59946 60283     0  S+      ttyin    0xfffff800080060a8 sh
59946 59726 59946     0  S+      wait     0xfffff8006e8c3528 sh
59726     1 59726     0  Ss+     wait     0xfffff8000c7ad528 login
89222     1   298     0  S       wait     0xfffff8000c967528 sh
88607     1 88607   136  Ss      select   0xfffff8000c298040 dhcpd
77035     1 77035    59  Ss      (threaded)                  unbound
100654                   S       kqread   0xfffff8000c9cae00 unbound
100691                   S       kqread   0xfffff8000c9c5b00 unbound
100692                   S       kqread   0xfffff8000c489000 unbound
100693                   S       kqread   0xfffff8000c9b0600 unbound
64289     1 64289     0  Ss      (threaded)                  dpinger

anyone can help us on this matter . Thanks