Suricata inline mode and kernal error message



  • I use suricata inline mode and bridge mode, pfsense installed in esxi 6.0

    It works for eight hour. But system show "em0: watchdog timeout – resetting" suddenly.

    Apr 19 22:29:24 fw119 kernel: 164.027197 [ 143] lem_netmap_txsync         bad addr/len ring 0 slot 240 idx 242 len
     4113
    Apr 19 22:29:24 fw119 kernel: 164.034653 [ 143] lem_netmap_txsync         bad addr/len ring 0 slot 6 idx 8 len 411
    3
    Apr 19 22:29:24 fw119 kernel: 164.041177 [ 143] lem_netmap_txsync         bad addr/len ring 0 slot 12 idx 14 len 4
    113
    Apr 19 22:29:24 fw119 kernel: 164.058985 [2860] netmap_transmit           em0 full hwcur 197 hwtail 73 qlen 123 le
    n 66 m 0xfffff80291600000
    Apr 19 22:29:24 fw119 kernel: 164.065552 [2860] netmap_transmit           em0 full hwcur 197 hwtail 73 qlen 123 le
    n 1144 m 0xfffff8015197d000
    Apr 19 22:29:24 fw119 kernel: 164.072057 [2860] netmap_transmit           em0 full hwcur 197 hwtail 73 qlen 123 le
    n 66 m 0xfffff801bd702700
    Apr 19 22:29:24 fw119 kernel: 164.076033 [ 143] lem_netmap_txsync         bad addr/len ring 0 slot 50 idx 52 len 4
    113
    Apr 19 22:29:24 fw119 kernel: 164.078760 [2860] netmap_transmit           em0 full hwcur 211 hwtail 196 qlen 14 le
    n 1223 m 0xfffff8055058f900
    Apr 19 22:29:26 fw119 kernel: 166.796907 [ 143] lem_netmap_txsync         bad addr/len ring 0 slot 166 idx 168 len
    

    I delete suricata interface settings and add new. It works again. When I reboot it. It show the error messages again. I power off it and power on next day(No traffic in brdige). It works. :-. I do not know how to explain this situation. Maybe esxi 6.0?



  • I suspect a problem with the ESXi emulation of the Intel NIC driver and its interaction with Netmap.  Netmap is highly dependent on support from the NIC driver.  I have run Suricata with inline mode for short periods on a VMware Workstation VM, but not on ESXi.  However, my virtual machine Suricata is where I do the development testing and it gets frequent reboots and otherwise is toyed with.  It never gets a ton of run time.  That being said, I have let it run overnight several times without issues being noted.  I specifically configured my "hardware settings" for the virtual machine to use the e1000 NIC.

    There may be some adjustments you can do with various NIC parameters that might help.  Also, did you remember to disable all the NIC offloading options under System > Advanced > Networking in pfSense?

    Bill



  • I disable all the NIC offloading options under System > Advanced > Networking in pfSense before.  I test it today,  it also occurs on esxi 5.5.  When no traffic through bridge interface or few traffic, It works properly.  But I move pfSense to production environment  (  about 700 Mbps and 500000 states ).  It show kernel errors.  Maybe I should use on-virtualized hardware.

    the esxi are consisting of

    Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
    24 CPUs: 1 package(s) x 16 core(s)
    96GB RAM



  • So Sad…..

    I reinstall pfsnese in HP DL380 G9 (not esxi environment) and enable inline mode for 10G NIC (ix0).  When I use in  production environment.  System still show error messages and crash finally. :'( . If I purchase a support incident pack, pfSense team can help me??



  • I can't speak for pfSense support.  I don't know if they support virtualized installs or not.

    You might want to do some research here and on Google for mbuf settings and other optimizations for some network drivers under pfSense 2.3 (which is FreeBSD 10.3).  Find the archived thread here on the forum for the 2.3-BETA program and search through it.  You can also try posting in the Virtualization sub-forum.

    Netmap support is kind of new everywhere, and there may indeed be some weird bugs with it.  Do you by chance any physical hardware you could temporarily dedicate as a pfSense 2.3 box for testing?  Sounds like you have a moderately busy network and your testing could help uncover hidden issues.  I do not have the facilities to test high traffic loads in my environment.

    Bill



  • Hi Bill.

    I enable inline mode ,  then disable inline mode and start suricata again. I found suricata.log show below

    21/4/2016 – 15:28:34 - <info>-- Netmap IPS mode activated em0->em0+
    21/4/2016 -- 15:28:34 - <info>-- preallocated 1024 packets. Total memory 3557376
    21/4/2016 -- 15:28:34 - <info>-- Using 1 threads for interface em0+
    21/4/2016 -- 15:28:34 - <info>-- Netmap IPS mode activated em0+->em0

    I have disable block,  I don't know why it still show "IPS mode activated"??

    Thx!</info></info></info></info>



  • Update:

    When I disable block mode,  IPS config is still in suricata.yaml.

    netmap:
     - interface: default
       threads: auto
       copy-mode: ips
       disable-promisc: no
       checksum-checks: auto
     - interface: ix0
       copy-iface: ix0+
     - interface: ix0+
       copy-iface: ix0
    

    I edit it manually to below

    pcap:
      - interface: ix0
        checksum-checks: auto
        promisc: yes
    

    But after start suricata,  it restore to block mode. :o



  • @ntct:

    Update:

    When I disable block mode,  IPS config is still in suricata.yaml.

    netmap:
     - interface: default
       threads: auto
       copy-mode: ips
       disable-promisc: no
       checksum-checks: auto
     - interface: ix0
       copy-iface: ix0+
     - interface: ix0+
       copy-iface: ix0
    

    I edit it manually to below

    pcap:
      - interface: ix0
        checksum-checks: auto
        promisc: yes
    

    But after start suricata,  it restore to block mode. :o

    When you disable the block mode or change it from legacy to inline, are you remembering to click the SAVE button down at the bottom of the page?  Also, once you make the change and save it, you need to restart Suricata on the affected interface.

    Manually editing the config files is pointless.  Each time you click the START icon on the INTERFACES tab, the suricata configuration file (suricata.yaml) is rebuilt with the saved settings.

    Bill



  • Yes, I disable the block mode and save then restart suricata, it 's the same.




  • I tested this last night in a pfSense virtual machine.  I set both legacy mode and inline mode repeatedly on the WAN interface.  Suricata properly swapped modes and updated the suricata.yaml file correctly.  In short, I am unable to reproduce this problem.  My VM was running pfSense 2.3-RELEASE and Suricata 3.0_6.

    Bill



  • Hmm…

    I don't use legacy mode in my test.

    The step I test is below.

    1. Check block offenders, set inline mode and save then restart suricata.

    2. Uncheck block offenders, save then restart suricata directly.

    3. view suricata.log, it show "Netmap IPS mode activated".

    Thx!



  • @ntct:

    Hmm…

    I don't use legacy mode in my test.

    The step I test is below.

    1. Check block offenders, set inline mode and save then restart suricata.

    2. Uncheck block offenders, save then restart suricata directly.

    3. view suricata.log, it show "Netmap IPS mode activated".

    Thx!

    Oh… let me think about that a minute and review the code.  You may have hit upon a sequence of events I did not adequately address in the code.  I will test that process out.  I was simply switching modes.

    In the interim, while I am testing, you can "disable blocks" by switching to Legacy Mode, saving that, then un-checking the "Block Offenders" checkbox and saving that.

    Edit Update:  I verified that the GUI code was not disabling Netmap when disabling block offenders.  So if you still had DROP rules, then it would still block.  To fix this I pushed an update into a currently pending Pull Request that switches Suricata back to pcap legacy mode when "block offenders" is disabled.  This way it will not block, but can still alert.

    Bill





  • @ntct:

    Maybe similar problems about netmap.

    https://github.com/luigirizzo/netmap/issues/156

    https://github.com/luigirizzo/netmap/issues/134

    Hmm…might be some Netmap problems that are not directly related to Suricata.  pfSense 2.3 now compiles Netmap support into the kernel by default.

    Bill


Log in to reply