Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Suricata inline mode and kernal error message

    Scheduled Pinned Locked Moved IDS/IPS
    14 Posts 2 Posters 4.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      ntct
      last edited by

      I use suricata inline mode and bridge mode, pfsense installed in esxi 6.0

      It works for eight hour. But system show "em0: watchdog timeout – resetting" suddenly.

      Apr 19 22:29:24 fw119 kernel: 164.027197 [ 143] lem_netmap_txsync         bad addr/len ring 0 slot 240 idx 242 len
 4113
Apr 19 22:29:24 fw119 kernel: 164.034653 [ 143] lem_netmap_txsync         bad addr/len ring 0 slot 6 idx 8 len 411
3
Apr 19 22:29:24 fw119 kernel: 164.041177 [ 143] lem_netmap_txsync         bad addr/len ring 0 slot 12 idx 14 len 4
113
Apr 19 22:29:24 fw119 kernel: 164.058985 [2860] netmap_transmit           em0 full hwcur 197 hwtail 73 qlen 123 le
n 66 m 0xfffff80291600000
Apr 19 22:29:24 fw119 kernel: 164.065552 [2860] netmap_transmit           em0 full hwcur 197 hwtail 73 qlen 123 le
n 1144 m 0xfffff8015197d000
Apr 19 22:29:24 fw119 kernel: 164.072057 [2860] netmap_transmit           em0 full hwcur 197 hwtail 73 qlen 123 le
n 66 m 0xfffff801bd702700
Apr 19 22:29:24 fw119 kernel: 164.076033 [ 143] lem_netmap_txsync         bad addr/len ring 0 slot 50 idx 52 len 4
113
Apr 19 22:29:24 fw119 kernel: 164.078760 [2860] netmap_transmit           em0 full hwcur 211 hwtail 196 qlen 14 le
n 1223 m 0xfffff8055058f900
Apr 19 22:29:26 fw119 kernel: 166.796907 [ 143] lem_netmap_txsync         bad addr/len ring 0 slot 166 idx 168 len

      I delete suricata interface settings and add new. It works again. When I reboot it. It show the error messages again. I power off it and power on next day(No traffic in brdige). It works. :-. I do not know how to explain this situation. Maybe esxi 6.0?

      1 Reply Last reply Reply Quote 0
      • bmeeksB
        bmeeks
        last edited by

        I suspect a problem with the ESXi emulation of the Intel NIC driver and its interaction with Netmap.  Netmap is highly dependent on support from the NIC driver.  I have run Suricata with inline mode for short periods on a VMware Workstation VM, but not on ESXi.  However, my virtual machine Suricata is where I do the development testing and it gets frequent reboots and otherwise is toyed with.  It never gets a ton of run time.  That being said, I have let it run overnight several times without issues being noted.  I specifically configured my "hardware settings" for the virtual machine to use the e1000 NIC.

        There may be some adjustments you can do with various NIC parameters that might help.  Also, did you remember to disable all the NIC offloading options under System > Advanced > Networking in pfSense?

        Bill

        1 Reply Last reply Reply Quote 0
        • N
          ntct
          last edited by

          I disable all the NIC offloading options under System > Advanced > Networking in pfSense before.  I test it today,  it also occurs on esxi 5.5.  When no traffic through bridge interface or few traffic, It works properly.  But I move pfSense to production environment  (  about 700 Mbps and 500000 states ).  It show kernel errors.  Maybe I should use on-virtualized hardware.

          the esxi are consisting of

          Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
          24 CPUs: 1 package(s) x 16 core(s)
          96GB RAM

          1 Reply Last reply Reply Quote 0
          • N
            ntct
            last edited by

            So Sad…..

            I reinstall pfsnese in HP DL380 G9 (not esxi environment) and enable inline mode for 10G NIC (ix0).  When I use in  production environment.  System still show error messages and crash finally. :'( . If I purchase a support incident pack, pfSense team can help me??

            1 Reply Last reply Reply Quote 0
            • bmeeksB
              bmeeks
              last edited by

              I can't speak for pfSense support.  I don't know if they support virtualized installs or not.

              You might want to do some research here and on Google for mbuf settings and other optimizations for some network drivers under pfSense 2.3 (which is FreeBSD 10.3).  Find the archived thread here on the forum for the 2.3-BETA program and search through it.  You can also try posting in the Virtualization sub-forum.

              Netmap support is kind of new everywhere, and there may indeed be some weird bugs with it.  Do you by chance any physical hardware you could temporarily dedicate as a pfSense 2.3 box for testing?  Sounds like you have a moderately busy network and your testing could help uncover hidden issues.  I do not have the facilities to test high traffic loads in my environment.

              Bill

              1 Reply Last reply Reply Quote 0
              • N
                ntct
                last edited by

                Hi Bill.

                I enable inline mode ,  then disable inline mode and start suricata again. I found suricata.log show below

                21/4/2016 – 15:28:34 - <info>-- Netmap IPS mode activated em0->em0+
                21/4/2016 -- 15:28:34 - <info>-- preallocated 1024 packets. Total memory 3557376
                21/4/2016 -- 15:28:34 - <info>-- Using 1 threads for interface em0+
                21/4/2016 -- 15:28:34 - <info>-- Netmap IPS mode activated em0+->em0

                I have disable block,  I don't know why it still show "IPS mode activated"??

                Thx!</info></info></info></info>

                1 Reply Last reply Reply Quote 0
                • N
                  ntct
                  last edited by

                  Update:

                  When I disable block mode,  IPS config is still in suricata.yaml.

                  netmap:
 - interface: default
   threads: auto
   copy-mode: ips
   disable-promisc: no
   checksum-checks: auto
 - interface: ix0
   copy-iface: ix0+
 - interface: ix0+
   copy-iface: ix0

                  I edit it manually to below

                  pcap:
  - interface: ix0
    checksum-checks: auto
    promisc: yes

                  But after start suricata,  it restore to block mode. :o

                  1 Reply Last reply Reply Quote 0
                  • bmeeksB
                    bmeeks
                    last edited by

                    @ntct:

                    Update:

                    When I disable block mode,  IPS config is still in suricata.yaml.

                    netmap:
 - interface: default
   threads: auto
   copy-mode: ips
   disable-promisc: no
   checksum-checks: auto
 - interface: ix0
   copy-iface: ix0+
 - interface: ix0+
   copy-iface: ix0

                    I edit it manually to below

                    pcap:
  - interface: ix0
    checksum-checks: auto
    promisc: yes

                    But after start suricata,  it restore to block mode. :o

                    When you disable the block mode or change it from legacy to inline, are you remembering to click the SAVE button down at the bottom of the page?  Also, once you make the change and save it, you need to restart Suricata on the affected interface.

                    Manually editing the config files is pointless.  Each time you click the START icon on the INTERFACES tab, the suricata configuration file (suricata.yaml) is rebuilt with the saved settings.

                    Bill

                    1 Reply Last reply Reply Quote 0
                    • N
                      ntct
                      last edited by

                      Yes, I disable the block mode and save then restart suricata, it 's the same.

                      1.jpg
                      1.jpg_thumb

                      1 Reply Last reply Reply Quote 0
                      • bmeeksB
                        bmeeks
                        last edited by

                        I tested this last night in a pfSense virtual machine.  I set both legacy mode and inline mode repeatedly on the WAN interface.  Suricata properly swapped modes and updated the suricata.yaml file correctly.  In short, I am unable to reproduce this problem.  My VM was running pfSense 2.3-RELEASE and Suricata 3.0_6.

                        Bill

                        1 Reply Last reply Reply Quote 0
                        • N
                          ntct
                          last edited by

                          Hmm…

                          I don't use legacy mode in my test.

                          The step I test is below.

                          1. Check block offenders, set inline mode and save then restart suricata.

                          2. Uncheck block offenders, save then restart suricata directly.

                          3. view suricata.log, it show "Netmap IPS mode activated".

                          Thx!

                          1 Reply Last reply Reply Quote 0
                          • bmeeksB
                            bmeeks
                            last edited by

                            @ntct:

                            Hmm…

                            I don't use legacy mode in my test.

                            The step I test is below.

                            1. Check block offenders, set inline mode and save then restart suricata.

                            2. Uncheck block offenders, save then restart suricata directly.

                            3. view suricata.log, it show "Netmap IPS mode activated".

                            Thx!

                            Oh… let me think about that a minute and review the code.  You may have hit upon a sequence of events I did not adequately address in the code.  I will test that process out.  I was simply switching modes.

                            In the interim, while I am testing, you can "disable blocks" by switching to Legacy Mode, saving that, then un-checking the "Block Offenders" checkbox and saving that.

                            Edit Update:  I verified that the GUI code was not disabling Netmap when disabling block offenders.  So if you still had DROP rules, then it would still block.  To fix this I pushed an update into a currently pending Pull Request that switches Suricata back to pcap legacy mode when "block offenders" is disabled.  This way it will not block, but can still alert.

                            Bill

                            1 Reply Last reply Reply Quote 0
                            • N
                              ntct
                              last edited by

                              Maybe similar problems about netmap.

                              https://github.com/luigirizzo/netmap/issues/156

                              https://github.com/luigirizzo/netmap/issues/134

                              ixgbe_netmap_txsync_1.jpg_thumb
                              ixgbe_netmap_txsync_1.jpg

                              1 Reply Last reply Reply Quote 0
                              • bmeeksB
                                bmeeks
                                last edited by

                                @ntct:

                                Maybe similar problems about netmap.

                                https://github.com/luigirizzo/netmap/issues/156

                                https://github.com/luigirizzo/netmap/issues/134

                                Hmm…might be some Netmap problems that are not directly related to Suricata.  pfSense 2.3 now compiles Netmap support into the kernel by default.

                                Bill

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post
                                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.