Suricata inline mode and kernal error message

ntct · Apr 20, 2016, 5:09 AM

I use suricata inline mode and bridge mode, pfsense installed in esxi 6.0

It works for eight hour. But system show "em0: watchdog timeout – resetting" suddenly.

Apr 19 22:29:24 fw119 kernel: 164.027197 [ 143] lem_netmap_txsync         bad addr/len ring 0 slot 240 idx 242 len
 4113
Apr 19 22:29:24 fw119 kernel: 164.034653 [ 143] lem_netmap_txsync         bad addr/len ring 0 slot 6 idx 8 len 411
3
Apr 19 22:29:24 fw119 kernel: 164.041177 [ 143] lem_netmap_txsync         bad addr/len ring 0 slot 12 idx 14 len 4
113
Apr 19 22:29:24 fw119 kernel: 164.058985 [2860] netmap_transmit           em0 full hwcur 197 hwtail 73 qlen 123 le
n 66 m 0xfffff80291600000
Apr 19 22:29:24 fw119 kernel: 164.065552 [2860] netmap_transmit           em0 full hwcur 197 hwtail 73 qlen 123 le
n 1144 m 0xfffff8015197d000
Apr 19 22:29:24 fw119 kernel: 164.072057 [2860] netmap_transmit           em0 full hwcur 197 hwtail 73 qlen 123 le
n 66 m 0xfffff801bd702700
Apr 19 22:29:24 fw119 kernel: 164.076033 [ 143] lem_netmap_txsync         bad addr/len ring 0 slot 50 idx 52 len 4
113
Apr 19 22:29:24 fw119 kernel: 164.078760 [2860] netmap_transmit           em0 full hwcur 211 hwtail 196 qlen 14 le
n 1223 m 0xfffff8055058f900
Apr 19 22:29:26 fw119 kernel: 166.796907 [ 143] lem_netmap_txsync         bad addr/len ring 0 slot 166 idx 168 len

I delete suricata interface settings and add new. It works again. When I reboot it. It show the error messages again. I power off it and power on next day(No traffic in brdige). It works. :-. I do not know how to explain this situation. Maybe esxi 6.0?

bmeeks · Apr 20, 2016, 2:38 AM

I suspect a problem with the ESXi emulation of the Intel NIC driver and its interaction with Netmap. Netmap is highly dependent on support from the NIC driver. I have run Suricata with inline mode for short periods on a VMware Workstation VM, but not on ESXi. However, my virtual machine Suricata is where I do the development testing and it gets frequent reboots and otherwise is toyed with. It never gets a ton of run time. That being said, I have let it run overnight several times without issues being noted. I specifically configured my "hardware settings" for the virtual machine to use the e1000 NIC.

There may be some adjustments you can do with various NIC parameters that might help. Also, did you remember to disable all the NIC offloading options under System > Advanced > Networking in pfSense?

Bill

ntct · Apr 20, 2016, 8:42 AM

I disable all the NIC offloading options under System > Advanced > Networking in pfSense before. I test it today, it also occurs on esxi 5.5. When no traffic through bridge interface or few traffic, It works properly. But I move pfSense to production environment ( about 700 Mbps and 500000 states ). It show kernel errors. Maybe I should use on-virtualized hardware.

the esxi are consisting of

Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
24 CPUs: 1 package(s) x 16 core(s)
96GB RAM

ntct · Apr 20, 2016, 8:50 AM

So Sad…..

I reinstall pfsnese in HP DL380 G9 (not esxi environment) and enable inline mode for 10G NIC (ix0). When I use in production environment. System still show error messages and crash finally. :'( . If I purchase a support incident pack, pfSense team can help me??

bmeeks · Apr 20, 2016, 12:52 PM

I can't speak for pfSense support. I don't know if they support virtualized installs or not.

You might want to do some research here and on Google for mbuf settings and other optimizations for some network drivers under pfSense 2.3 (which is FreeBSD 10.3). Find the archived thread here on the forum for the 2.3-BETA program and search through it. You can also try posting in the Virtualization sub-forum.

Netmap support is kind of new everywhere, and there may indeed be some weird bugs with it. Do you by chance any physical hardware you could temporarily dedicate as a pfSense 2.3 box for testing? Sounds like you have a moderately busy network and your testing could help uncover hidden issues. I do not have the facilities to test high traffic loads in my environment.

Bill

ntct · Apr 21, 2016, 7:35 AM

Hi Bill.

I enable inline mode , then disable inline mode and start suricata again. I found suricata.log show below

21/4/2016 – 15:28:34 - <info>-- Netmap IPS mode activated em0->em0+
21/4/2016 -- 15:28:34 - <info>-- preallocated 1024 packets. Total memory 3557376
21/4/2016 -- 15:28:34 - <info>-- Using 1 threads for interface em0+
21/4/2016 -- 15:28:34 - <info>-- Netmap IPS mode activated em0+->em0

I have disable block, I don't know why it still show "IPS mode activated"??

Thx!</info></info></info></info>

ntct · Apr 22, 2016, 1:07 AM

Update:

When I disable block mode, IPS config is still in suricata.yaml.

netmap:
 - interface: default
   threads: auto
   copy-mode: ips
   disable-promisc: no
   checksum-checks: auto
 - interface: ix0
   copy-iface: ix0+
 - interface: ix0+
   copy-iface: ix0

I edit it manually to below

pcap:
  - interface: ix0
    checksum-checks: auto
    promisc: yes

But after start suricata, it restore to block mode. :o

bmeeks · Apr 22, 2016, 3:41 AM

@ntct:

Update:

When I disable block mode, IPS config is still in suricata.yaml.

netmap:
 - interface: default
   threads: auto
   copy-mode: ips
   disable-promisc: no
   checksum-checks: auto
 - interface: ix0
   copy-iface: ix0+
 - interface: ix0+
   copy-iface: ix0

I edit it manually to below

pcap:
  - interface: ix0
    checksum-checks: auto
    promisc: yes

But after start suricata, it restore to block mode. :o

When you disable the block mode or change it from legacy to inline, are you remembering to click the SAVE button down at the bottom of the page? Also, once you make the change and save it, you need to restart Suricata on the affected interface.

Manually editing the config files is pointless. Each time you click the START icon on the INTERFACES tab, the suricata configuration file (suricata.yaml) is rebuilt with the saved settings.

Bill

ntct · Apr 22, 2016, 4:29 AM

Yes, I disable the block mode and save then restart suricata, it 's the same.

bmeeks · Apr 22, 2016, 1:34 PM

I tested this last night in a pfSense virtual machine. I set both legacy mode and inline mode repeatedly on the WAN interface. Suricata properly swapped modes and updated the suricata.yaml file correctly. In short, I am unable to reproduce this problem. My VM was running pfSense 2.3-RELEASE and Suricata 3.0_6.

Bill

ntct · Apr 22, 2016, 2:10 PM

Hmm…

I don't use legacy mode in my test.

The step I test is below.

1. Check block offenders, set inline mode and save then restart suricata.

2. Uncheck block offenders, save then restart suricata directly.

3. view suricata.log, it show "Netmap IPS mode activated".

Thx!

bmeeks · Apr 25, 2016, 3:41 AM

@ntct:

Hmm…

I don't use legacy mode in my test.

The step I test is below.

1. Check block offenders, set inline mode and save then restart suricata.

2. Uncheck block offenders, save then restart suricata directly.

3. view suricata.log, it show "Netmap IPS mode activated".

Thx!

Oh… let me think about that a minute and review the code. You may have hit upon a sequence of events I did not adequately address in the code. I will test that process out. I was simply switching modes.

In the interim, while I am testing, you can "disable blocks" by switching to Legacy Mode, saving that, then un-checking the "Block Offenders" checkbox and saving that.

Edit Update: I verified that the GUI code was not disabling Netmap when disabling block offenders. So if you still had DROP rules, then it would still block. To fix this I pushed an update into a currently pending Pull Request that switches Suricata back to pcap legacy mode when "block offenders" is disabled. This way it will not block, but can still alert.

Bill

ntct · Apr 24, 2016, 9:24 AM

Maybe similar problems about netmap.

https://github.com/luigirizzo/netmap/issues/156

https://github.com/luigirizzo/netmap/issues/134

bmeeks · Apr 25, 2016, 3:38 AM

@ntct:

Maybe similar problems about netmap.

https://github.com/luigirizzo/netmap/issues/156

https://github.com/luigirizzo/netmap/issues/134

Hmm…might be some Netmap problems that are not directly related to Suricata. pfSense 2.3 now compiles Netmap support into the kernel by default.

Bill