Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Suricata InLine with igb NICs

    Scheduled Pinned Locked Moved IDS/IPS
    77 Posts 6 Posters 10.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • NollipfSenseN
      NollipfSense @boobletins
      last edited by stephenw10

      @boobletins said in Suricata InLine with igb NICs:

      ifconfig igb0 | grep CSUM

      Shell Output - ifconfig igb0 | grep CSUM
      options=5400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,NETMAP,TXCSUM_IPV6>

      Shell Output - sysctl -a | grep igb

      device	igb
      hw.igb.tx_process_limit: -1
      hw.igb.rx_process_limit: 100
      hw.igb.num_queues: 0
      hw.igb.header_split: 0
      hw.igb.max_interrupt_rate: 8000
      hw.igb.enable_msix: 1
      hw.igb.enable_aim: 1
      hw.igb.txd: 1024
      hw.igb.rxd: 1024
      dev.igb.1.host.header_redir_missed: 0
      dev.igb.1.host.serdes_violation_pkt: 0
      dev.igb.1.host.length_errors: 0
      dev.igb.1.host.tx_good_bytes: 5014864175
      dev.igb.1.host.rx_good_bytes: 344809214
      dev.igb.1.host.breaker_tx_pkt_drop: 0
      dev.igb.1.host.tx_good_pkt: 63
      dev.igb.1.host.breaker_rx_pkt_drop: 0
      dev.igb.1.host.breaker_rx_pkts: 0
      dev.igb.1.host.rx_pkt: 77
      dev.igb.1.host.host_tx_pkt_discard: 0
      dev.igb.1.host.breaker_tx_pkt: 0
      dev.igb.1.interrupts.rx_overrun: 0
      dev.igb.1.interrupts.rx_desc_min_thresh: 0
      dev.igb.1.interrupts.tx_queue_min_thresh: 0
      dev.igb.1.interrupts.tx_queue_empty: 4315841
      dev.igb.1.interrupts.tx_abs_timer: 0
      dev.igb.1.interrupts.tx_pkt_timer: 4315904
      dev.igb.1.interrupts.rx_abs_timer: 2921232
      dev.igb.1.interrupts.rx_pkt_timer: 2921155
      dev.igb.1.interrupts.asserts: 8803973
      dev.igb.1.mac_stats.tso_ctx_fail: 0
      dev.igb.1.mac_stats.tso_txd: 0
      dev.igb.1.mac_stats.tx_frames_1024_1522: 3233544
      dev.igb.1.mac_stats.tx_frames_512_1023: 62481
      dev.igb.1.mac_stats.tx_frames_256_511: 72052
      dev.igb.1.mac_stats.tx_frames_128_255: 119162
      dev.igb.1.mac_stats.tx_frames_65_127: 781667
      dev.igb.1.mac_stats.tx_frames_64: 46998
      dev.igb.1.mac_stats.mcast_pkts_txd: 269918
      dev.igb.1.mac_stats.bcast_pkts_txd: 118
      dev.igb.1.mac_stats.good_pkts_txd: 4315904
      dev.igb.1.mac_stats.total_pkts_txd: 4315904
      dev.igb.1.mac_stats.total_octets_txd: 5014886629
      dev.igb.1.mac_stats.good_octets_txd: 5014885349
      dev.igb.1.mac_stats.total_octets_recvd: 344809463
      dev.igb.1.mac_stats.good_octets_recvd: 344808248
      dev.igb.1.mac_stats.rx_frames_1024_1522: 49390
      dev.igb.1.mac_stats.rx_frames_512_1023: 61271
      dev.igb.1.mac_stats.rx_frames_256_511: 60178
      dev.igb.1.mac_stats.rx_frames_128_255: 132406
      dev.igb.1.mac_stats.rx_frames_65_127: 2127900
      dev.igb.1.mac_stats.rx_frames_64: 490087
      dev.igb.1.mac_stats.mcast_pkts_recvd: 0
      dev.igb.1.mac_stats.bcast_pkts_recvd: 4
      dev.igb.1.mac_stats.good_pkts_recvd: 2921232
      dev.igb.1.mac_stats.total_pkts_recvd: 2921232
      dev.igb.1.mac_stats.mgmt_pkts_txd: 0
      dev.igb.1.mac_stats.mgmt_pkts_drop: 0
      dev.igb.1.mac_stats.mgmt_pkts_recvd: 0
      dev.igb.1.mac_stats.unsupported_fc_recvd: 0
      dev.igb.1.mac_stats.xoff_txd: 0
      dev.igb.1.mac_stats.xoff_recvd: 0
      dev.igb.1.mac_stats.xon_txd: 0
      dev.igb.1.mac_stats.xon_recvd: 0
      dev.igb.1.mac_stats.coll_ext_errs: 0
      dev.igb.1.mac_stats.tx_no_crs: 0
      dev.igb.1.mac_stats.alignment_errs: 0
      dev.igb.1.mac_stats.crc_errs: 0
      dev.igb.1.mac_stats.recv_errs: 0
      dev.igb.1.mac_stats.recv_jabber: 0
      dev.igb.1.mac_stats.recv_oversize: 0
      dev.igb.1.mac_stats.recv_fragmented: 0
      dev.igb.1.mac_stats.recv_undersize: 0
      dev.igb.1.mac_stats.recv_no_buff: 0
      dev.igb.1.mac_stats.recv_length_errors: 0
      dev.igb.1.mac_stats.missed_packets: 0
      dev.igb.1.mac_stats.defer_count: 0
      dev.igb.1.mac_stats.sequence_errors: 0
      dev.igb.1.mac_stats.symbol_errors: 0
      dev.igb.1.mac_stats.collision_count: 0
      dev.igb.1.mac_stats.late_coll: 0
      dev.igb.1.mac_stats.multiple_coll: 0
      dev.igb.1.mac_stats.single_coll: 0
      dev.igb.1.mac_stats.excess_coll: 0
      dev.igb.1.queue1.lro_flushed: 0
      dev.igb.1.queue1.lro_queued: 0
      dev.igb.1.queue1.rx_bytes: 152608531
      dev.igb.1.queue1.rx_packets: 1226723
      dev.igb.1.queue1.rxd_tail: 994
      dev.igb.1.queue1.rxd_head: 995
      dev.igb.1.queue1.tx_packets: 257
      dev.igb.1.queue1.no_desc_avail: 0
      dev.igb.1.queue1.txd_tail: 339
      dev.igb.1.queue1.txd_head: 339
      dev.igb.1.queue1.interrupt_rate: 76923
      dev.igb.1.queue0.lro_flushed: 0
      dev.igb.1.queue0.lro_queued: 0
      dev.igb.1.queue0.rx_bytes: 180516276
      dev.igb.1.queue0.rx_packets: 1694509
      dev.igb.1.queue0.rxd_tail: 812
      dev.igb.1.queue0.rxd_head: 813
      dev.igb.1.queue0.tx_packets: 4315647
      dev.igb.1.queue0.no_desc_avail: 0
      dev.igb.1.queue0.txd_tail: 442
      dev.igb.1.queue0.txd_head: 442
      dev.igb.1.queue0.interrupt_rate: 90909
      dev.igb.1.fc_low_water: 29480
      dev.igb.1.fc_high_water: 29488
      dev.igb.1.rx_buf_alloc: 34
      dev.igb.1.tx_buf_alloc: 14
      dev.igb.1.extended_int_mask: 2147484419
      dev.igb.1.interrupt_mask: 4
      dev.igb.1.rx_control: 67141658
      dev.igb.1.device_control: 1087373896
      dev.igb.1.watchdog_timeouts: 0
      dev.igb.1.rx_overruns: 0
      dev.igb.1.tx_dma_fail: 0
      dev.igb.1.mbuf_defrag_fail: 0
      dev.igb.1.link_irq: 2
      dev.igb.1.dropped: 0
      dev.igb.1.tx_processing_limit: -1
      dev.igb.1.rx_processing_limit: 100
      dev.igb.1.fc: 3
      dev.igb.1.enable_aim: 1
      dev.igb.1.nvm: -1
      dev.igb.1.%parent: pci3
      dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10a7 subvendor=0x8086 subdevice=0x10a7 class=0x020000
      dev.igb.1.%location: slot=0 function=1 dbsf=pci0:3:0:1
      dev.igb.1.%driver: igb
      dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
      dev.igb.0.host.header_redir_missed: 0
      dev.igb.0.host.serdes_violation_pkt: 0
      dev.igb.0.host.length_errors: 0
      dev.igb.0.host.tx_good_bytes: 702511124
      dev.igb.0.host.rx_good_bytes: 8859910607
      dev.igb.0.host.breaker_tx_pkt_drop: 0
      dev.igb.0.host.tx_good_pkt: 389
      dev.igb.0.host.breaker_rx_pkt_drop: 0
      dev.igb.0.host.breaker_rx_pkts: 0
      dev.igb.0.host.rx_pkt: 179
      dev.igb.0.host.host_tx_pkt_discard: 0
      dev.igb.0.host.breaker_tx_pkt: 0
      dev.igb.0.interrupts.rx_overrun: 0
      dev.igb.0.interrupts.rx_desc_min_thresh: 0
      dev.igb.0.interrupts.tx_queue_min_thresh: 0
      dev.igb.0.interrupts.tx_queue_empty: 8008878
      dev.igb.0.interrupts.tx_abs_timer: 0
      dev.igb.0.interrupts.tx_pkt_timer: 8009267
      dev.igb.0.interrupts.rx_abs_timer: 9004187
      dev.igb.0.interrupts.rx_pkt_timer: 9004008
      dev.igb.0.interrupts.asserts: 18858568
      dev.igb.0.mac_stats.tso_ctx_fail: 0
      dev.igb.0.mac_stats.tso_txd: 0
      dev.igb.0.mac_stats.tx_frames_1024_1522: 49679
      dev.igb.0.mac_stats.tx_frames_512_1023: 59868
      dev.igb.0.mac_stats.tx_frames_256_511: 65957
      dev.igb.0.mac_stats.tx_frames_128_255: 117544
      dev.igb.0.mac_stats.tx_frames_65_127: 4903787
      dev.igb.0.mac_stats.tx_frames_64: 2812432
      dev.igb.0.mac_stats.mcast_pkts_txd: 217
      dev.igb.0.mac_stats.bcast_pkts_txd: 321
      dev.igb.0.mac_stats.good_pkts_txd: 8009267
      dev.igb.0.mac_stats.total_pkts_txd: 8009267
      dev.igb.0.mac_stats.total_octets_txd: 702511679
      dev.igb.0.mac_stats.good_octets_txd: 702510340
      dev.igb.0.mac_stats.total_octets_recvd: 8859907035
      dev.igb.0.mac_stats.good_octets_recvd: 8859915684
      dev.igb.0.mac_stats.rx_frames_1024_1522: 5630206
      dev.igb.0.mac_stats.rx_frames_512_1023: 67795
      dev.igb.0.mac_stats.rx_frames_256_511: 155128
      dev.igb.0.mac_stats.rx_frames_128_255: 445051
      dev.igb.0.mac_stats.rx_frames_65_127: 765396
      dev.igb.0.mac_stats.rx_frames_64: 1940609
      dev.igb.0.mac_stats.mcast_pkts_recvd: 218995
      dev.igb.0.mac_stats.bcast_pkts_recvd: 47673
      dev.igb.0.mac_stats.good_pkts_recvd: 9004185
      dev.igb.0.mac_stats.total_pkts_recvd: 9004224
      dev.igb.0.mac_stats.mgmt_pkts_txd: 0
      dev.igb.0.mac_stats.mgmt_pkts_drop: 0
      dev.igb.0.mac_stats.mgmt_pkts_recvd: 0
      dev.igb.0.mac_stats.unsupported_fc_recvd: 0
      dev.igb.0.mac_stats.xoff_txd: 0
      dev.igb.0.mac_stats.xoff_recvd: 1
      dev.igb.0.mac_stats.xon_txd: 0
      dev.igb.0.mac_stats.xon_recvd: 1
      dev.igb.0.mac_stats.coll_ext_errs: 0
      dev.igb.0.mac_stats.tx_no_crs: 0
      dev.igb.0.mac_stats.alignment_errs: 0
      dev.igb.0.mac_stats.crc_errs: 0
      dev.igb.0.mac_stats.recv_errs: 0
      dev.igb.0.mac_stats.recv_jabber: 0
      dev.igb.0.mac_stats.recv_oversize: 0
      dev.igb.0.mac_stats.recv_fragmented: 0
      dev.igb.0.mac_stats.recv_undersize: 0
      dev.igb.0.mac_stats.recv_no_buff: 0
      dev.igb.0.mac_stats.recv_length_errors: 0
      dev.igb.0.mac_stats.missed_packets: 0
      dev.igb.0.mac_stats.defer_count: 0
      dev.igb.0.mac_stats.sequence_errors: 0
      dev.igb.0.mac_stats.symbol_errors: 0
      dev.igb.0.mac_stats.collision_count: 0
      dev.igb.0.mac_stats.late_coll: 0
      dev.igb.0.mac_stats.multiple_coll: 0
      dev.igb.0.mac_stats.single_coll: 0
      dev.igb.0.mac_stats.excess_coll: 0
      dev.igb.0.queue1.lro_flushed: 0
      dev.igb.0.queue1.lro_queued: 0
      dev.igb.0.queue1.rx_bytes: 0
      dev.igb.0.queue1.rx_packets: 2432
      dev.igb.0.queue1.rxd_tail: 35
      dev.igb.0.queue1.rxd_head: 36
      dev.igb.0.queue1.tx_packets: 1
      dev.igb.0.queue1.no_desc_avail: 0
      dev.igb.0.queue1.txd_tail: 0
      dev.igb.0.queue1.txd_head: 0
      dev.igb.0.queue1.interrupt_rate: 16129
      dev.igb.0.queue0.lro_flushed: 0
      dev.igb.0.queue0.lro_queued: 0
      dev.igb.0.queue0.rx_bytes: 0
      dev.igb.0.queue0.rx_packets: 7244
      dev.igb.0.queue0.rxd_tail: 180
      dev.igb.0.queue0.rxd_head: 181
      dev.igb.0.queue0.tx_packets: 9386
      dev.igb.0.queue0.no_desc_avail: 0
      dev.igb.0.queue0.txd_tail: 755
      dev.igb.0.queue0.txd_head: 755
      dev.igb.0.queue0.interrupt_rate: 16129
      dev.igb.0.fc_low_water: 29480
      dev.igb.0.fc_high_water: 29488
      dev.igb.0.rx_buf_alloc: 34
      dev.igb.0.tx_buf_alloc: 14
      dev.igb.0.extended_int_mask: 2147484419
      dev.igb.0.interrupt_mask: 4
      dev.igb.0.rx_control: 67141658
      dev.igb.0.device_control: 1490027080
      dev.igb.0.watchdog_timeouts: 0
      dev.igb.0.rx_overruns: 0
      dev.igb.0.tx_dma_fail: 0
      dev.igb.0.mbuf_defrag_fail: 0
      dev.igb.0.link_irq: 70
      dev.igb.0.dropped: 0
      dev.igb.0.tx_processing_limit: -1
      dev.igb.0.rx_processing_limit: 100
      dev.igb.0.fc: 3
      dev.igb.0.enable_aim: 1
      dev.igb.0.nvm: -1
      dev.igb.0.%parent: pci3
      dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10a7 subvendor=0x8086 subdevice=0x10a7 class=0x020000
      dev.igb.0.%location: slot=0 function=0 dbsf=pci0:3:0:0
      dev.igb.0.%driver: igb
      dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
      dev.igb.%parent:
      
      Shell Output - sysctl -a | grep netmap
      device	netmap
      dev.netmap.ixl_rx_miss_bufs: 0
      dev.netmap.ixl_rx_miss: 0
      dev.netmap.iflib_rx_miss_bufs: 0
      dev.netmap.iflib_rx_miss: 0
      dev.netmap.iflib_crcstrip: 1
      dev.netmap.bridge_batch: 1024
      dev.netmap.default_pipes: 0
      dev.netmap.priv_buf_num: 4098
      dev.netmap.priv_buf_size: 2048
      dev.netmap.buf_curr_num: 163840
      dev.netmap.buf_num: 163840
      dev.netmap.buf_curr_size: 4096
      dev.netmap.buf_size: 4096
      dev.netmap.priv_ring_num: 4
      dev.netmap.priv_ring_size: 20480
      dev.netmap.ring_curr_num: 200
      dev.netmap.ring_num: 200
      dev.netmap.ring_curr_size: 36864
      dev.netmap.ring_size: 36864
      dev.netmap.priv_if_num: 1
      dev.netmap.priv_if_size: 1024
      dev.netmap.if_curr_num: 100
      dev.netmap.if_num: 100
      dev.netmap.if_curr_size: 1024
      dev.netmap.if_size: 1024
      dev.netmap.generic_rings: 1
      dev.netmap.generic_ringsize: 1024
      dev.netmap.generic_mit: 100000
      dev.netmap.admode: 0
      dev.netmap.fwd: 0
      dev.netmap.flags: 0
      dev.netmap.adaptive_io: 0
      dev.netmap.txsync_retry: 2
      dev.netmap.no_pendintr: 1
      dev.netmap.mitigate: 1
      dev.netmap.no_timestamp: 0
      dev.netmap.verbose: 0
      dev.netmap.ix_rx_miss_bufs: 0
      dev.netmap.ix_rx_miss: 0
      dev.netmap.ix_crcstrip: 0
      

      Every couple of days I get one or two netmap bad packet alert even after increasing this - netmap.buf_size: 4096. I run both Suricata and Snort on Wan and Lan however, I only enable blocking on Suricata WAN...all else are disabled. I have 8GB RAM however, I can only use 6GB as a failed processor killed a row/channel in my HP Pavilion 6242n trash find I converted into a pfSense firewall.

      pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
      pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

      1 Reply Last reply Reply Quote 0
      • B
        boobletins
        last edited by boobletins

        Under System / Advanced / Networking, is "Allow IPv6" checked?

        And how many CPU cores? Is hyperthreading enabled?

        NollipfSenseN 1 Reply Last reply Reply Quote 0
        • NollipfSenseN
          NollipfSense @boobletins
          last edited by

          @boobletins said in Suricata InLine with igb NICs:

          Under System / Advanced / Networking, is "Allow IPv6" checked?

          And how many CPU cores? Is hyperthreading enabled?

          Yes...allowed IPv6 checked...CPU Type AMD Athlon(tm) 64 X2 Dual Core Processor 4800+
          2 CPUs: 1 package(s) x 2 core(s)
          AES-NI CPU Crypto: No

          No sure where to check for hyperthread...now I will disable IPv6...thought I did.

          pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
          pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

          B 1 Reply Last reply Reply Quote 0
          • B
            boobletins @NollipfSense
            last edited by boobletins

            @nollipfsense

            So here are some initial suggestions. Please keep in mind that I've been working on this for ~1 week (in other words: not long), and I'm not a FreeBSD, pfSense, or Suricata expert.

            Start by making a backup of your configuration.

            Do these first:
            My understanding is that flow control should be off on any netmap interface. You have bi-directional flow control enabled:

            dev.igb.0.fc: 3
            

            Disable flow control on all active interfaces using system tunables. Set dev.igb.0.fc=0 (and dev.igb.1.fc=0)

            Actively set energy efficient ethernet to disabled:
            dev.igb.0.eee_disabled=1

            Actively force IPv6_TXCSUM6 off by adding the following to config.xml in a shellcmd tag:

            ifconfig igb0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso
            

            (see above in this thread for a link on where/how to do that).

            Edit:
            To be clear: anywhere I have a command that says "igb0" or "igb.0" you will want to duplicate that for igb1 and any other interface you're running netmap on.

            So you will need 2 shellcmd lines in config.xml, and two new system tunables for flow control, etc


            Consider changing later:

            Set rx processing limit:
            dev.igb.0.rx_processing_limit: -1

            It looks like your txd and rxd are both set to 1024 currently, I suggest you move those to 4096:
            hw.igb.txd=4096
            hw.igb.rxd=4096

            By changing your txd and rxd we may need to revisit your netmap buf/ring (memory settings).

            We may also revisit your interrupt and queue settings.

            NollipfSenseN 1 Reply Last reply Reply Quote 0
            • B
              boobletins
              last edited by

              It would be great if you could let me know what happens after the initial changes -- if you continue to get netmap errors or not.

              If you do, don't jump right to the 2nd section of changes, we should verify that the changes we made above took properly. I learned the hard way that I was putting some settings in the wrong places.

              1 Reply Last reply Reply Quote 0
              • N
                newUser2pfSense
                last edited by newUser2pfSense

                boobletins...I apologize for not getting back sooner; other projects. I added the shellcmd line to the /cf/conf/config.xml file as you suggested. I re-enabled Suricata in Inline IPS Mode and restarted pfSense. I ran ifconfig against all four ethernet interfaces on my SuperMicro adapter and TXCSUM_IPV6 was not listed.

                One thing I find interesting is in the Services > Suricata > Alerts tab, all of the text is now black in color when before making the above change it was all red in color in Inline IPS Mode. As well, there are no entries in the Blocks tab when before making the change it was automagically populated with over 300 blocked IP addresses in Inline IPS Mode. I don't know if this is normal or not. I didn't change any of the Suricata WAN Categories.

                1 Reply Last reply Reply Quote 0
                • B
                  boobletins
                  last edited by

                  If you are running Suricata in inline mode, you will not see blocked IP addresses in the blocked tab, as any traffic that conforms to your "drop" rules is automatically intercepted and dropped (as opposed to initially logged, then ip banned as in Legacy mode).

                  The red text in the Alerts tab is letting you know that the traffic was indeed intercepted and dropped (since you don't have any information in the Block tab anymore).

                  That you are missing both blocks and red text means that either no traffic has conformed to your block rules yet, or something has gone wrong.

                  Double check which mode Suricata is running in. Then double check that you have some drop rules defined.

                  But originally the issue was netmap, yes? If so, have you see any netmap errors? Can you complete a speedtest with Suricata enabled in inline mode now?

                  1 Reply Last reply Reply Quote 0
                  • N
                    newUser2pfSense
                    last edited by newUser2pfSense

                    I guess my "drop" rules are just based on the Suricata WAN Categories I selected/checked? When you checkmark what you want, Suricata applies those rules to the incoming data and drops it?

                    I'm definitely running in Inline IPS Mode and all of the Alerts entries are in black colored text. As well, all of the text I typed into the loader.conf.local file is commented out.

                    Yes, the original issue was many, many, netmap_grab_packets console/log messages. At the time of this post, I don't have any netmap errors, yet ☺ . I can actually complete a speedtest in Inline IPS Mode now.

                    bmeeksB 1 Reply Last reply Reply Quote 1
                    • B
                      boobletins
                      last edited by

                      So Suricata is complex and can do many things.

                      I suggest you read about SID management and check out the SID management and categories tabs.

                      If I had to guess, I would say that you have a minimal set of "drop" rules in place and just haven't had blocked traffic hit your server yet. When you do, you'll see the red text.

                      If you want to convert "alerts" to "drops," you'll typically do that in SID management.

                      1 Reply Last reply Reply Quote 0
                      • N
                        newUser2pfSense
                        last edited by newUser2pfSense

                        Unfortunately, I just received several netmap_grap_packets errors:

                        netmap_grab_packets bad pkt at 465 len 2147
                        netmap_grab_packets bad pkt at 116 len 2154
                        netmap_grab_packets bad pkt at 39 len 2147
                        netmap_grab_packets bad pkt at 872 len 2147
                        netmap_grab_packets bad pkt at 860 len 2147
                        netmap_grab_packets bad pkt at 838 len 2147
                        netmap_grab_packets bad pkt at 777 len 2154

                        Apparently the shellcmd didn't work after all.

                        B 1 Reply Last reply Reply Quote 0
                        • B
                          boobletins
                          last edited by

                          Could you give me the same info I asked for from NollipfSense above?

                          The results of these commands:
                          ifconfig igb0 | grep CSUM
                          sysctl -a | grep igb
                          sysctl -a | grep netmap

                          1 Reply Last reply Reply Quote 0
                          • bmeeksB
                            bmeeks @newUser2pfSense
                            last edited by

                            @newuser2pfsense said in Suricata InLine with igb NICs:

                            I guess my "drop" rules are just based on the Suricata WAN Categories I selected/checked? When you checkmark what you want, Suricata applies those rules to the incoming data and drops it?

                            I'm definitely running in Inline IPS Mode and all of the Alerts entries are in black colored text. As well, all of the text I typed into the loader.conf.local file is commented out.

                            Yes, the original issue was many, many, netmap_grab_packets console/log messages. At the time of this post, I don't have any netmap errors, yet ☺ . I can actually complete a speedtest in Inline IPS Mode now.

                            When you run with Inline IPS Mode enabled, you have to manually change rules you want to block traffic from ALERT to DROP. You can do that in two places: (1) on the RULES tab with the selected category displayed; (2) using SID Management keywords on the SID MGMT tab. Sounds like you do not have any DROP rules enabled if you are getting no red alert entries. Either that, or as @boobletins stated, your enabled DROP rules have not been triggered.

                            1 Reply Last reply Reply Quote 0
                            • N
                              newUser2pfSense
                              last edited by newUser2pfSense

                              boobletins...Here is the output of the commands you requested.

                              ifconfig igbX | grep CSUM output -

                              for igb0:  options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
                              for igb1:  options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
                              for igb2 [not used]:  options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
                              for igb3 [WAN]:  options=1000b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,NETMAP>
                              for em0 [not used]:  options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
                              

                              sysctl -a | grep igb output -

                              device	igb
                              hw.igb.tx_process_limit: -1
                              hw.igb.rx_process_limit: 100
                              hw.igb.num_queues: 0
                              hw.igb.header_split: 0
                              hw.igb.max_interrupt_rate: 8000
                              hw.igb.enable_msix: 1
                              hw.igb.enable_aim: 1
                              hw.igb.txd: 1024
                              hw.igb.rxd: 1024
                              dev.igb.3.host.header_redir_missed: 0
                              dev.igb.3.host.serdes_violation_pkt: 0
                              dev.igb.3.host.length_errors: 3
                              dev.igb.3.host.tx_good_bytes: 213941
                              dev.igb.3.host.rx_good_bytes: 3967552
                              dev.igb.3.host.breaker_tx_pkt_drop: 0
                              dev.igb.3.host.tx_good_pkt: 0
                              dev.igb.3.host.breaker_rx_pkt_drop: 0
                              dev.igb.3.host.breaker_rx_pkts: 0
                              dev.igb.3.host.rx_pkt: 3
                              dev.igb.3.host.host_tx_pkt_discard: 0
                              dev.igb.3.host.breaker_tx_pkt: 0
                              dev.igb.3.interrupts.rx_overrun: 0
                              dev.igb.3.interrupts.rx_desc_min_thresh: 0
                              dev.igb.3.interrupts.tx_queue_min_thresh: 231
                              dev.igb.3.interrupts.tx_queue_empty: 2417
                              dev.igb.3.interrupts.tx_abs_timer: 0
                              dev.igb.3.interrupts.tx_pkt_timer: 0
                              dev.igb.3.interrupts.rx_abs_timer: 0
                              dev.igb.3.interrupts.rx_pkt_timer: 18194
                              dev.igb.3.interrupts.asserts: 22095
                              dev.igb.3.mac_stats.tso_ctx_fail: 0
                              dev.igb.3.mac_stats.tso_txd: 0
                              dev.igb.3.mac_stats.tx_frames_1024_1522: 3
                              dev.igb.3.mac_stats.tx_frames_512_1023: 16
                              dev.igb.3.mac_stats.tx_frames_256_511: 21
                              dev.igb.3.mac_stats.tx_frames_128_255: 90
                              dev.igb.3.mac_stats.tx_frames_65_127: 1502
                              dev.igb.3.mac_stats.tx_frames_64: 785
                              dev.igb.3.mac_stats.mcast_pkts_txd: 3
                              dev.igb.3.mac_stats.bcast_pkts_txd: 10
                              dev.igb.3.mac_stats.good_pkts_txd: 2417
                              dev.igb.3.mac_stats.total_pkts_txd: 2417
                              dev.igb.3.mac_stats.total_octets_txd: 213941
                              dev.igb.3.mac_stats.good_octets_txd: 213941
                              dev.igb.3.mac_stats.total_octets_recvd: 4000401
                              dev.igb.3.mac_stats.good_octets_recvd: 3967552
                              dev.igb.3.mac_stats.rx_frames_1024_1522: 1363
                              dev.igb.3.mac_stats.rx_frames_512_1023: 339
                              dev.igb.3.mac_stats.rx_frames_256_511: 1232
                              dev.igb.3.mac_stats.rx_frames_128_255: 1117
                              dev.igb.3.mac_stats.rx_frames_65_127: 6557
                              dev.igb.3.mac_stats.rx_frames_64: 7589
                              dev.igb.3.mac_stats.mcast_pkts_recvd: 11704
                              dev.igb.3.mac_stats.bcast_pkts_recvd: 3585
                              dev.igb.3.mac_stats.good_pkts_recvd: 18197
                              dev.igb.3.mac_stats.total_pkts_recvd: 18528
                              dev.igb.3.mac_stats.mgmt_pkts_txd: 0
                              dev.igb.3.mac_stats.mgmt_pkts_drop: 0
                              dev.igb.3.mac_stats.mgmt_pkts_recvd: 0
                              dev.igb.3.mac_stats.unsupported_fc_recvd: 0
                              dev.igb.3.mac_stats.xoff_txd: 0
                              dev.igb.3.mac_stats.xoff_recvd: 0
                              dev.igb.3.mac_stats.xon_txd: 0
                              dev.igb.3.mac_stats.xon_recvd: 0
                              dev.igb.3.mac_stats.coll_ext_errs: 0
                              dev.igb.3.mac_stats.tx_no_crs: 0
                              dev.igb.3.mac_stats.alignment_errs: 0
                              dev.igb.3.mac_stats.crc_errs: 0
                              dev.igb.3.mac_stats.recv_errs: 0
                              dev.igb.3.mac_stats.recv_jabber: 0
                              dev.igb.3.mac_stats.recv_oversize: 0
                              dev.igb.3.mac_stats.recv_fragmented: 0
                              dev.igb.3.mac_stats.recv_undersize: 0
                              dev.igb.3.mac_stats.recv_no_buff: 0
                              dev.igb.3.mac_stats.recv_length_errors: 0
                              dev.igb.3.mac_stats.missed_packets: 0
                              dev.igb.3.mac_stats.defer_count: 0
                              dev.igb.3.mac_stats.sequence_errors: 0
                              dev.igb.3.mac_stats.symbol_errors: 0
                              dev.igb.3.mac_stats.collision_count: 0
                              dev.igb.3.mac_stats.late_coll: 0
                              dev.igb.3.mac_stats.multiple_coll: 0
                              dev.igb.3.mac_stats.single_coll: 0
                              dev.igb.3.mac_stats.excess_coll: 0
                              dev.igb.3.queue7.lro_flushed: 0
                              dev.igb.3.queue7.lro_queued: 0
                              dev.igb.3.queue7.rx_bytes: 0
                              dev.igb.3.queue7.rx_packets: 129
                              dev.igb.3.queue7.rxd_tail: 561
                              dev.igb.3.queue7.rxd_head: 562
                              dev.igb.3.queue7.tx_packets: 0
                              dev.igb.3.queue7.no_desc_avail: 0
                              dev.igb.3.queue7.txd_tail: 0
                              dev.igb.3.queue7.txd_head: 0
                              dev.igb.3.queue7.interrupt_rate: 8000
                              dev.igb.3.queue6.lro_flushed: 0
                              dev.igb.3.queue6.lro_queued: 0
                              dev.igb.3.queue6.rx_bytes: 0
                              dev.igb.3.queue6.rx_packets: 30
                              dev.igb.3.queue6.rxd_tail: 902
                              dev.igb.3.queue6.rxd_head: 903
                              dev.igb.3.queue6.tx_packets: 0
                              dev.igb.3.queue6.no_desc_avail: 0
                              dev.igb.3.queue6.txd_tail: 0
                              dev.igb.3.queue6.txd_head: 0
                              dev.igb.3.queue6.interrupt_rate: 8000
                              dev.igb.3.queue5.lro_flushed: 0
                              dev.igb.3.queue5.lro_queued: 0
                              dev.igb.3.queue5.rx_bytes: 0
                              dev.igb.3.queue5.rx_packets: 30
                              dev.igb.3.queue5.rxd_tail: 344
                              dev.igb.3.queue5.rxd_head: 345
                              dev.igb.3.queue5.tx_packets: 0
                              dev.igb.3.queue5.no_desc_avail: 0
                              dev.igb.3.queue5.txd_tail: 0
                              dev.igb.3.queue5.txd_head: 0
                              dev.igb.3.queue5.interrupt_rate: 8000
                              dev.igb.3.queue4.lro_flushed: 0
                              dev.igb.3.queue4.lro_queued: 0
                              dev.igb.3.queue4.rx_bytes: 0
                              dev.igb.3.queue4.rx_packets: 87
                              dev.igb.3.queue4.rxd_tail: 524
                              dev.igb.3.queue4.rxd_head: 525
                              dev.igb.3.queue4.tx_packets: 0
                              dev.igb.3.queue4.no_desc_avail: 0
                              dev.igb.3.queue4.txd_tail: 0
                              dev.igb.3.queue4.txd_head: 0
                              dev.igb.3.queue4.interrupt_rate: 8000
                              dev.igb.3.queue3.lro_flushed: 0
                              dev.igb.3.queue3.lro_queued: 0
                              dev.igb.3.queue3.rx_bytes: 0
                              dev.igb.3.queue3.rx_packets: 29
                              dev.igb.3.queue3.rxd_tail: 453
                              dev.igb.3.queue3.rxd_head: 454
                              dev.igb.3.queue3.tx_packets: 0
                              dev.igb.3.queue3.no_desc_avail: 0
                              dev.igb.3.queue3.txd_tail: 0
                              dev.igb.3.queue3.txd_head: 0
                              dev.igb.3.queue3.interrupt_rate: 76923
                              dev.igb.3.queue2.lro_flushed: 0
                              dev.igb.3.queue2.lro_queued: 0
                              dev.igb.3.queue2.rx_bytes: 0
                              dev.igb.3.queue2.rx_packets: 28
                              dev.igb.3.queue2.rxd_tail: 876
                              dev.igb.3.queue2.rxd_head: 877
                              dev.igb.3.queue2.tx_packets: 0
                              dev.igb.3.queue2.no_desc_avail: 0
                              dev.igb.3.queue2.txd_tail: 0
                              dev.igb.3.queue2.txd_head: 0
                              dev.igb.3.queue2.interrupt_rate: 8000
                              dev.igb.3.queue1.lro_flushed: 0
                              dev.igb.3.queue1.lro_queued: 0
                              dev.igb.3.queue1.rx_bytes: 0
                              dev.igb.3.queue1.rx_packets: 17
                              dev.igb.3.queue1.rxd_tail: 482
                              dev.igb.3.queue1.rxd_head: 483
                              dev.igb.3.queue1.tx_packets: 0
                              dev.igb.3.queue1.no_desc_avail: 0
                              dev.igb.3.queue1.txd_tail: 0
                              dev.igb.3.queue1.txd_head: 0
                              dev.igb.3.queue1.interrupt_rate: 8000
                              dev.igb.3.queue0.lro_flushed: 0
                              dev.igb.3.queue0.lro_queued: 0
                              dev.igb.3.queue0.rx_bytes: 0
                              dev.igb.3.queue0.rx_packets: 142
                              dev.igb.3.queue0.rxd_tail: 253
                              dev.igb.3.queue0.rxd_head: 254
                              dev.igb.3.queue0.tx_packets: 353
                              dev.igb.3.queue0.no_desc_avail: 0
                              dev.igb.3.queue0.txd_tail: 18
                              dev.igb.3.queue0.txd_head: 18
                              dev.igb.3.queue0.interrupt_rate: 8000
                              dev.igb.3.fc_low_water: 33152
                              dev.igb.3.fc_high_water: 33168
                              dev.igb.3.rx_buf_alloc: 0
                              dev.igb.3.tx_buf_alloc: 0
                              dev.igb.3.extended_int_mask: 2147484159
                              dev.igb.3.interrupt_mask: 4
                              dev.igb.3.rx_control: 67141658
                              dev.igb.3.device_control: 1478230593
                              dev.igb.3.watchdog_timeouts: 0
                              dev.igb.3.rx_overruns: 0
                              dev.igb.3.tx_dma_fail: 0
                              dev.igb.3.mbuf_defrag_fail: 0
                              dev.igb.3.link_irq: 7
                              dev.igb.3.dropped: 0
                              dev.igb.3.eee_disabled: 0
                              dev.igb.3.dmac: 0
                              dev.igb.3.tx_processing_limit: -1
                              dev.igb.3.rx_processing_limit: 100
                              dev.igb.3.fc: 3
                              dev.igb.3.enable_aim: 1
                              dev.igb.3.nvm: -1
                              dev.igb.3.%parent: pci2
                              dev.igb.3.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1b12 class=0x020000
                              dev.igb.3.%location: slot=0 function=3 dbsf=pci0:2:0:3
                              dev.igb.3.%driver: igb
                              dev.igb.3.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
                              dev.igb.2.host.header_redir_missed: 0
                              dev.igb.2.host.serdes_violation_pkt: 0
                              dev.igb.2.host.length_errors: 0
                              dev.igb.2.host.tx_good_bytes: 0
                              dev.igb.2.host.rx_good_bytes: 0
                              dev.igb.2.host.breaker_tx_pkt_drop: 0
                              dev.igb.2.host.tx_good_pkt: 0
                              dev.igb.2.host.breaker_rx_pkt_drop: 0
                              dev.igb.2.host.breaker_rx_pkts: 0
                              dev.igb.2.host.rx_pkt: 0
                              dev.igb.2.host.host_tx_pkt_discard: 0
                              dev.igb.2.host.breaker_tx_pkt: 0
                              dev.igb.2.interrupts.rx_overrun: 0
                              dev.igb.2.interrupts.rx_desc_min_thresh: 0
                              dev.igb.2.interrupts.tx_queue_min_thresh: 0
                              dev.igb.2.interrupts.tx_queue_empty: 0
                              dev.igb.2.interrupts.tx_abs_timer: 0
                              dev.igb.2.interrupts.tx_pkt_timer: 0
                              dev.igb.2.interrupts.rx_abs_timer: 0
                              dev.igb.2.interrupts.rx_pkt_timer: 0
                              dev.igb.2.interrupts.asserts: 0
                              dev.igb.2.mac_stats.tso_ctx_fail: 0
                              dev.igb.2.mac_stats.tso_txd: 0
                              dev.igb.2.mac_stats.tx_frames_1024_1522: 0
                              dev.igb.2.mac_stats.tx_frames_512_1023: 0
                              dev.igb.2.mac_stats.tx_frames_256_511: 0
                              dev.igb.2.mac_stats.tx_frames_128_255: 0
                              dev.igb.2.mac_stats.tx_frames_65_127: 0
                              dev.igb.2.mac_stats.tx_frames_64: 0
                              dev.igb.2.mac_stats.mcast_pkts_txd: 0
                              dev.igb.2.mac_stats.bcast_pkts_txd: 0
                              dev.igb.2.mac_stats.good_pkts_txd: 0
                              dev.igb.2.mac_stats.total_pkts_txd: 0
                              dev.igb.2.mac_stats.total_octets_txd: 0
                              dev.igb.2.mac_stats.good_octets_txd: 0
                              dev.igb.2.mac_stats.total_octets_recvd: 0
                              dev.igb.2.mac_stats.good_octets_recvd: 0
                              dev.igb.2.mac_stats.rx_frames_1024_1522: 0
                              dev.igb.2.mac_stats.rx_frames_512_1023: 0
                              dev.igb.2.mac_stats.rx_frames_256_511: 0
                              dev.igb.2.mac_stats.rx_frames_128_255: 0
                              dev.igb.2.mac_stats.rx_frames_65_127: 0
                              dev.igb.2.mac_stats.rx_frames_64: 0
                              dev.igb.2.mac_stats.mcast_pkts_recvd: 0
                              dev.igb.2.mac_stats.bcast_pkts_recvd: 0
                              dev.igb.2.mac_stats.good_pkts_recvd: 0
                              dev.igb.2.mac_stats.total_pkts_recvd: 0
                              dev.igb.2.mac_stats.mgmt_pkts_txd: 0
                              dev.igb.2.mac_stats.mgmt_pkts_drop: 0
                              dev.igb.2.mac_stats.mgmt_pkts_recvd: 0
                              dev.igb.2.mac_stats.unsupported_fc_recvd: 0
                              dev.igb.2.mac_stats.xoff_txd: 0
                              dev.igb.2.mac_stats.xoff_recvd: 0
                              dev.igb.2.mac_stats.xon_txd: 0
                              dev.igb.2.mac_stats.xon_recvd: 0
                              dev.igb.2.mac_stats.coll_ext_errs: 0
                              dev.igb.2.mac_stats.tx_no_crs: 0
                              dev.igb.2.mac_stats.alignment_errs: 0
                              dev.igb.2.mac_stats.crc_errs: 0
                              dev.igb.2.mac_stats.recv_errs: 0
                              dev.igb.2.mac_stats.recv_jabber: 0
                              dev.igb.2.mac_stats.recv_oversize: 0
                              dev.igb.2.mac_stats.recv_fragmented: 0
                              dev.igb.2.mac_stats.recv_undersize: 0
                              dev.igb.2.mac_stats.recv_no_buff: 0
                              dev.igb.2.mac_stats.recv_length_errors: 0
                              dev.igb.2.mac_stats.missed_packets: 0
                              dev.igb.2.mac_stats.defer_count: 0
                              dev.igb.2.mac_stats.sequence_errors: 0
                              dev.igb.2.mac_stats.symbol_errors: 0
                              dev.igb.2.mac_stats.collision_count: 0
                              dev.igb.2.mac_stats.late_coll: 0
                              dev.igb.2.mac_stats.multiple_coll: 0
                              dev.igb.2.mac_stats.single_coll: 0
                              dev.igb.2.mac_stats.excess_coll: 0
                              dev.igb.2.queue7.lro_flushed: 0
                              dev.igb.2.queue7.lro_queued: 0
                              dev.igb.2.queue7.rx_bytes: 0
                              dev.igb.2.queue7.rx_packets: 0
                              dev.igb.2.queue7.rxd_tail: 0
                              dev.igb.2.queue7.rxd_head: 0
                              dev.igb.2.queue7.tx_packets: 0
                              dev.igb.2.queue7.no_desc_avail: 0
                              dev.igb.2.queue7.txd_tail: 0
                              dev.igb.2.queue7.txd_head: 0
                              dev.igb.2.queue7.interrupt_rate: 0
                              dev.igb.2.queue6.lro_flushed: 0
                              dev.igb.2.queue6.lro_queued: 0
                              dev.igb.2.queue6.rx_bytes: 0
                              dev.igb.2.queue6.rx_packets: 0
                              dev.igb.2.queue6.rxd_tail: 0
                              dev.igb.2.queue6.rxd_head: 0
                              dev.igb.2.queue6.tx_packets: 0
                              dev.igb.2.queue6.no_desc_avail: 0
                              dev.igb.2.queue6.txd_tail: 0
                              dev.igb.2.queue6.txd_head: 0
                              dev.igb.2.queue6.interrupt_rate: 0
                              dev.igb.2.queue5.lro_flushed: 0
                              dev.igb.2.queue5.lro_queued: 0
                              dev.igb.2.queue5.rx_bytes: 0
                              dev.igb.2.queue5.rx_packets: 0
                              dev.igb.2.queue5.rxd_tail: 0
                              dev.igb.2.queue5.rxd_head: 0
                              dev.igb.2.queue5.tx_packets: 0
                              dev.igb.2.queue5.no_desc_avail: 0
                              dev.igb.2.queue5.txd_tail: 0
                              dev.igb.2.queue5.txd_head: 0
                              dev.igb.2.queue5.interrupt_rate: 0
                              dev.igb.2.queue4.lro_flushed: 0
                              dev.igb.2.queue4.lro_queued: 0
                              dev.igb.2.queue4.rx_bytes: 0
                              dev.igb.2.queue4.rx_packets: 0
                              dev.igb.2.queue4.rxd_tail: 0
                              dev.igb.2.queue4.rxd_head: 0
                              dev.igb.2.queue4.tx_packets: 0
                              dev.igb.2.queue4.no_desc_avail: 0
                              dev.igb.2.queue4.txd_tail: 0
                              dev.igb.2.queue4.txd_head: 0
                              dev.igb.2.queue4.interrupt_rate: 0
                              dev.igb.2.queue3.lro_flushed: 0
                              dev.igb.2.queue3.lro_queued: 0
                              dev.igb.2.queue3.rx_bytes: 0
                              dev.igb.2.queue3.rx_packets: 0
                              dev.igb.2.queue3.rxd_tail: 0
                              dev.igb.2.queue3.rxd_head: 0
                              dev.igb.2.queue3.tx_packets: 0
                              dev.igb.2.queue3.no_desc_avail: 0
                              dev.igb.2.queue3.txd_tail: 0
                              dev.igb.2.queue3.txd_head: 0
                              dev.igb.2.queue3.interrupt_rate: 0
                              dev.igb.2.queue2.lro_flushed: 0
                              dev.igb.2.queue2.lro_queued: 0
                              dev.igb.2.queue2.rx_bytes: 0
                              dev.igb.2.queue2.rx_packets: 0
                              dev.igb.2.queue2.rxd_tail: 0
                              dev.igb.2.queue2.rxd_head: 0
                              dev.igb.2.queue2.tx_packets: 0
                              dev.igb.2.queue2.no_desc_avail: 0
                              dev.igb.2.queue2.txd_tail: 0
                              dev.igb.2.queue2.txd_head: 0
                              dev.igb.2.queue2.interrupt_rate: 0
                              dev.igb.2.queue1.lro_flushed: 0
                              dev.igb.2.queue1.lro_queued: 0
                              dev.igb.2.queue1.rx_bytes: 0
                              dev.igb.2.queue1.rx_packets: 0
                              dev.igb.2.queue1.rxd_tail: 0
                              dev.igb.2.queue1.rxd_head: 0
                              dev.igb.2.queue1.tx_packets: 0
                              dev.igb.2.queue1.no_desc_avail: 0
                              dev.igb.2.queue1.txd_tail: 0
                              dev.igb.2.queue1.txd_head: 0
                              dev.igb.2.queue1.interrupt_rate: 0
                              dev.igb.2.queue0.lro_flushed: 0
                              dev.igb.2.queue0.lro_queued: 0
                              dev.igb.2.queue0.rx_bytes: 0
                              dev.igb.2.queue0.rx_packets: 0
                              dev.igb.2.queue0.rxd_tail: 0
                              dev.igb.2.queue0.rxd_head: 0
                              dev.igb.2.queue0.tx_packets: 0
                              dev.igb.2.queue0.no_desc_avail: 0
                              dev.igb.2.queue0.txd_tail: 0
                              dev.igb.2.queue0.txd_head: 0
                              dev.igb.2.queue0.interrupt_rate: 0
                              dev.igb.2.fc_low_water: 33152
                              dev.igb.2.fc_high_water: 33168
                              dev.igb.2.rx_buf_alloc: 0
                              dev.igb.2.tx_buf_alloc: 0
                              dev.igb.2.extended_int_mask: 2147483648
                              dev.igb.2.interrupt_mask: 0
                              dev.igb.2.rx_control: 0
                              dev.igb.2.device_control: 136053313
                              dev.igb.2.watchdog_timeouts: 0
                              dev.igb.2.rx_overruns: 0
                              dev.igb.2.tx_dma_fail: 0
                              dev.igb.2.mbuf_defrag_fail: 0
                              dev.igb.2.link_irq: 0
                              dev.igb.2.dropped: 0
                              dev.igb.2.eee_disabled: 0
                              dev.igb.2.dmac: 0
                              dev.igb.2.tx_processing_limit: -1
                              dev.igb.2.rx_processing_limit: 100
                              dev.igb.2.fc: 3
                              dev.igb.2.enable_aim: 1
                              dev.igb.2.nvm: -1
                              dev.igb.2.%parent: pci2
                              dev.igb.2.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1b12 class=0x020000
                              dev.igb.2.%location: slot=0 function=2 dbsf=pci0:2:0:2
                              dev.igb.2.%driver: igb
                              dev.igb.2.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
                              dev.igb.1.host.header_redir_missed: 0
                              dev.igb.1.host.serdes_violation_pkt: 0
                              dev.igb.1.host.length_errors: 0
                              dev.igb.1.host.tx_good_bytes: 387524
                              dev.igb.1.host.rx_good_bytes: 58810
                              dev.igb.1.host.breaker_tx_pkt_drop: 0
                              dev.igb.1.host.tx_good_pkt: 1
                              dev.igb.1.host.breaker_rx_pkt_drop: 0
                              dev.igb.1.host.breaker_rx_pkts: 0
                              dev.igb.1.host.rx_pkt: 2
                              dev.igb.1.host.host_tx_pkt_discard: 0
                              dev.igb.1.host.breaker_tx_pkt: 0
                              dev.igb.1.interrupts.rx_overrun: 0
                              dev.igb.1.interrupts.rx_desc_min_thresh: 0
                              dev.igb.1.interrupts.tx_queue_min_thresh: 0
                              dev.igb.1.interrupts.tx_queue_empty: 584
                              dev.igb.1.interrupts.tx_abs_timer: 0
                              dev.igb.1.interrupts.tx_pkt_timer: 0
                              dev.igb.1.interrupts.rx_abs_timer: 0
                              dev.igb.1.interrupts.rx_pkt_timer: 381
                              dev.igb.1.interrupts.asserts: 3808
                              dev.igb.1.mac_stats.tso_ctx_fail: 0
                              dev.igb.1.mac_stats.tso_txd: 0
                              dev.igb.1.mac_stats.tx_frames_1024_1522: 213
                              dev.igb.1.mac_stats.tx_frames_512_1023: 25
                              dev.igb.1.mac_stats.tx_frames_256_511: 87
                              dev.igb.1.mac_stats.tx_frames_128_255: 9
                              dev.igb.1.mac_stats.tx_frames_65_127: 239
                              dev.igb.1.mac_stats.tx_frames_64: 12
                              dev.igb.1.mac_stats.mcast_pkts_txd: 7
                              dev.igb.1.mac_stats.bcast_pkts_txd: 5
                              dev.igb.1.mac_stats.good_pkts_txd: 585
                              dev.igb.1.mac_stats.total_pkts_txd: 585
                              dev.igb.1.mac_stats.total_octets_txd: 387524
                              dev.igb.1.mac_stats.good_octets_txd: 387524
                              dev.igb.1.mac_stats.total_octets_recvd: 70094
                              dev.igb.1.mac_stats.good_octets_recvd: 58810
                              dev.igb.1.mac_stats.rx_frames_1024_1522: 5
                              dev.igb.1.mac_stats.rx_frames_512_1023: 8
                              dev.igb.1.mac_stats.rx_frames_256_511: 17
                              dev.igb.1.mac_stats.rx_frames_128_255: 124
                              dev.igb.1.mac_stats.rx_frames_65_127: 217
                              dev.igb.1.mac_stats.rx_frames_64: 12
                              dev.igb.1.mac_stats.mcast_pkts_recvd: 20
                              dev.igb.1.mac_stats.bcast_pkts_recvd: 12
                              dev.igb.1.mac_stats.good_pkts_recvd: 383
                              dev.igb.1.mac_stats.total_pkts_recvd: 411
                              dev.igb.1.mac_stats.mgmt_pkts_txd: 0
                              dev.igb.1.mac_stats.mgmt_pkts_drop: 0
                              dev.igb.1.mac_stats.mgmt_pkts_recvd: 0
                              dev.igb.1.mac_stats.unsupported_fc_recvd: 0
                              dev.igb.1.mac_stats.xoff_txd: 0
                              dev.igb.1.mac_stats.xoff_recvd: 0
                              dev.igb.1.mac_stats.xon_txd: 0
                              dev.igb.1.mac_stats.xon_recvd: 0
                              dev.igb.1.mac_stats.coll_ext_errs: 0
                              dev.igb.1.mac_stats.tx_no_crs: 0
                              dev.igb.1.mac_stats.alignment_errs: 0
                              dev.igb.1.mac_stats.crc_errs: 0
                              dev.igb.1.mac_stats.recv_errs: 0
                              dev.igb.1.mac_stats.recv_jabber: 0
                              dev.igb.1.mac_stats.recv_oversize: 0
                              dev.igb.1.mac_stats.recv_fragmented: 0
                              dev.igb.1.mac_stats.recv_undersize: 0
                              dev.igb.1.mac_stats.recv_no_buff: 0
                              dev.igb.1.mac_stats.recv_length_errors: 0
                              dev.igb.1.mac_stats.missed_packets: 0
                              dev.igb.1.mac_stats.defer_count: 0
                              dev.igb.1.mac_stats.sequence_errors: 0
                              dev.igb.1.mac_stats.symbol_errors: 0
                              dev.igb.1.mac_stats.collision_count: 0
                              dev.igb.1.mac_stats.late_coll: 0
                              dev.igb.1.mac_stats.multiple_coll: 0
                              dev.igb.1.mac_stats.single_coll: 0
                              dev.igb.1.mac_stats.excess_coll: 0
                              dev.igb.1.queue7.lro_flushed: 0
                              dev.igb.1.queue7.lro_queued: 0
                              dev.igb.1.queue7.rx_bytes: 22234
                              dev.igb.1.queue7.rx_packets: 154
                              dev.igb.1.queue7.rxd_tail: 153
                              dev.igb.1.queue7.rxd_head: 154
                              dev.igb.1.queue7.tx_packets: 0
                              dev.igb.1.queue7.no_desc_avail: 0
                              dev.igb.1.queue7.txd_tail: 0
                              dev.igb.1.queue7.txd_head: 0
                              dev.igb.1.queue7.interrupt_rate: 62500
                              dev.igb.1.queue6.lro_flushed: 0
                              dev.igb.1.queue6.lro_queued: 0
                              dev.igb.1.queue6.rx_bytes: 4146
                              dev.igb.1.queue6.rx_packets: 22
                              dev.igb.1.queue6.rxd_tail: 21
                              dev.igb.1.queue6.rxd_head: 22
                              dev.igb.1.queue6.tx_packets: 0
                              dev.igb.1.queue6.no_desc_avail: 0
                              dev.igb.1.queue6.txd_tail: 0
                              dev.igb.1.queue6.txd_head: 0
                              dev.igb.1.queue6.interrupt_rate: 76923
                              dev.igb.1.queue5.lro_flushed: 0
                              dev.igb.1.queue5.lro_queued: 0
                              dev.igb.1.queue5.rx_bytes: 120
                              dev.igb.1.queue5.rx_packets: 2
                              dev.igb.1.queue5.rxd_tail: 1
                              dev.igb.1.queue5.rxd_head: 2
                              dev.igb.1.queue5.tx_packets: 0
                              dev.igb.1.queue5.no_desc_avail: 0
                              dev.igb.1.queue5.txd_tail: 0
                              dev.igb.1.queue5.txd_head: 0
                              dev.igb.1.queue5.interrupt_rate: 100000
                              dev.igb.1.queue4.lro_flushed: 0
                              dev.igb.1.queue4.lro_queued: 0
                              dev.igb.1.queue4.rx_bytes: 254
                              dev.igb.1.queue4.rx_packets: 4
                              dev.igb.1.queue4.rxd_tail: 3
                              dev.igb.1.queue4.rxd_head: 4
                              dev.igb.1.queue4.tx_packets: 0
                              dev.igb.1.queue4.no_desc_avail: 0
                              dev.igb.1.queue4.txd_tail: 0
                              dev.igb.1.queue4.txd_head: 0
                              dev.igb.1.queue4.interrupt_rate: 100000
                              dev.igb.1.queue3.lro_flushed: 0
                              dev.igb.1.queue3.lro_queued: 0
                              dev.igb.1.queue3.rx_bytes: 24026
                              dev.igb.1.queue3.rx_packets: 144
                              dev.igb.1.queue3.rxd_tail: 143
                              dev.igb.1.queue3.rxd_head: 144
                              dev.igb.1.queue3.tx_packets: 0
                              dev.igb.1.queue3.no_desc_avail: 0
                              dev.igb.1.queue3.txd_tail: 0
                              dev.igb.1.queue3.txd_head: 0
                              dev.igb.1.queue3.interrupt_rate: 6024
                              dev.igb.1.queue2.lro_flushed: 0
                              dev.igb.1.queue2.lro_queued: 0
                              dev.igb.1.queue2.rx_bytes: 3022
                              dev.igb.1.queue2.rx_packets: 14
                              dev.igb.1.queue2.rxd_tail: 13
                              dev.igb.1.queue2.rxd_head: 14
                              dev.igb.1.queue2.tx_packets: 0
                              dev.igb.1.queue2.no_desc_avail: 0
                              dev.igb.1.queue2.txd_tail: 0
                              dev.igb.1.queue2.txd_head: 0
                              dev.igb.1.queue2.interrupt_rate: 29411
                              dev.igb.1.queue1.lro_flushed: 0
                              dev.igb.1.queue1.lro_queued: 0
                              dev.igb.1.queue1.rx_bytes: 874
                              dev.igb.1.queue1.rx_packets: 7
                              dev.igb.1.queue1.rxd_tail: 6
                              dev.igb.1.queue1.rxd_head: 7
                              dev.igb.1.queue1.tx_packets: 0
                              dev.igb.1.queue1.no_desc_avail: 0
                              dev.igb.1.queue1.txd_tail: 0
                              dev.igb.1.queue1.txd_head: 0
                              dev.igb.1.queue1.interrupt_rate: 66666
                              dev.igb.1.queue0.lro_flushed: 0
                              dev.igb.1.queue0.lro_queued: 0
                              dev.igb.1.queue0.rx_bytes: 4193
                              dev.igb.1.queue0.rx_packets: 39
                              dev.igb.1.queue0.rxd_tail: 38
                              dev.igb.1.queue0.rxd_head: 39
                              dev.igb.1.queue0.tx_packets: 588
                              dev.igb.1.queue0.no_desc_avail: 0
                              dev.igb.1.queue0.txd_tail: 911
                              dev.igb.1.queue0.txd_head: 911
                              dev.igb.1.queue0.interrupt_rate: 90909
                              dev.igb.1.fc_low_water: 33152
                              dev.igb.1.fc_high_water: 33168
                              dev.igb.1.rx_buf_alloc: 0
                              dev.igb.1.tx_buf_alloc: 0
                              dev.igb.1.extended_int_mask: 2147484159
                              dev.igb.1.interrupt_mask: 4
                              dev.igb.1.rx_control: 67141634
                              dev.igb.1.device_control: 1478230593
                              dev.igb.1.watchdog_timeouts: 0
                              dev.igb.1.rx_overruns: 0
                              dev.igb.1.tx_dma_fail: 0
                              dev.igb.1.mbuf_defrag_fail: 0
                              dev.igb.1.link_irq: 4
                              dev.igb.1.dropped: 0
                              dev.igb.1.eee_disabled: 0
                              dev.igb.1.dmac: 0
                              dev.igb.1.tx_processing_limit: -1
                              dev.igb.1.rx_processing_limit: 100
                              dev.igb.1.fc: 3
                              dev.igb.1.enable_aim: 1
                              dev.igb.1.nvm: -1
                              dev.igb.1.%parent: pci2
                              dev.igb.1.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1b12 class=0x020000
                              dev.igb.1.%location: slot=0 function=1 dbsf=pci0:2:0:1
                              dev.igb.1.%driver: igb
                              dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
                              dev.igb.0.wake: 0
                              dev.igb.0.host.header_redir_missed: 0
                              dev.igb.0.host.serdes_violation_pkt: 0
                              dev.igb.0.host.length_errors: 0
                              dev.igb.0.host.tx_good_bytes: 1710353
                              dev.igb.0.host.rx_good_bytes: 377365
                              dev.igb.0.host.breaker_tx_pkt_drop: 0
                              dev.igb.0.host.tx_good_pkt: 0
                              dev.igb.0.host.breaker_rx_pkt_drop: 0
                              dev.igb.0.host.breaker_rx_pkts: 0
                              dev.igb.0.host.rx_pkt: 0
                              dev.igb.0.host.host_tx_pkt_discard: 0
                              dev.igb.0.host.breaker_tx_pkt: 0
                              dev.igb.0.interrupts.rx_overrun: 0
                              dev.igb.0.interrupts.rx_desc_min_thresh: 0
                              dev.igb.0.interrupts.tx_queue_min_thresh: 23214
                              dev.igb.0.interrupts.tx_queue_empty: 1413
                              dev.igb.0.interrupts.tx_abs_timer: 0
                              dev.igb.0.interrupts.tx_pkt_timer: 0
                              dev.igb.0.interrupts.rx_abs_timer: 0
                              dev.igb.0.interrupts.rx_pkt_timer: 4886
                              dev.igb.0.interrupts.asserts: 8193
                              dev.igb.0.mac_stats.tso_ctx_fail: 0
                              dev.igb.0.mac_stats.tso_txd: 0
                              dev.igb.0.mac_stats.tx_frames_1024_1522: 1093
                              dev.igb.0.mac_stats.tx_frames_512_1023: 30
                              dev.igb.0.mac_stats.tx_frames_256_511: 30
                              dev.igb.0.mac_stats.tx_frames_128_255: 10
                              dev.igb.0.mac_stats.tx_frames_65_127: 42
                              dev.igb.0.mac_stats.tx_frames_64: 208
                              dev.igb.0.mac_stats.mcast_pkts_txd: 1
                              dev.igb.0.mac_stats.bcast_pkts_txd: 10
                              dev.igb.0.mac_stats.good_pkts_txd: 1413
                              dev.igb.0.mac_stats.total_pkts_txd: 1413
                              dev.igb.0.mac_stats.total_octets_txd: 1710353
                              dev.igb.0.mac_stats.good_octets_txd: 1710353
                              dev.igb.0.mac_stats.total_octets_recvd: 384601
                              dev.igb.0.mac_stats.good_octets_recvd: 377365
                              dev.igb.0.mac_stats.rx_frames_1024_1522: 3
                              dev.igb.0.mac_stats.rx_frames_512_1023: 1
                              dev.igb.0.mac_stats.rx_frames_256_511: 22
                              dev.igb.0.mac_stats.rx_frames_128_255: 385
                              dev.igb.0.mac_stats.rx_frames_65_127: 1331
                              dev.igb.0.mac_stats.rx_frames_64: 3144
                              dev.igb.0.mac_stats.mcast_pkts_recvd: 412
                              dev.igb.0.mac_stats.bcast_pkts_recvd: 2902
                              dev.igb.0.mac_stats.good_pkts_recvd: 4886
                              dev.igb.0.mac_stats.total_pkts_recvd: 4902
                              dev.igb.0.mac_stats.mgmt_pkts_txd: 0
                              dev.igb.0.mac_stats.mgmt_pkts_drop: 0
                              dev.igb.0.mac_stats.mgmt_pkts_recvd: 0
                              dev.igb.0.mac_stats.unsupported_fc_recvd: 0
                              dev.igb.0.mac_stats.xoff_txd: 0
                              dev.igb.0.mac_stats.xoff_recvd: 0
                              dev.igb.0.mac_stats.xon_txd: 0
                              dev.igb.0.mac_stats.xon_recvd: 0
                              dev.igb.0.mac_stats.coll_ext_errs: 0
                              dev.igb.0.mac_stats.tx_no_crs: 0
                              dev.igb.0.mac_stats.alignment_errs: 0
                              dev.igb.0.mac_stats.crc_errs: 0
                              dev.igb.0.mac_stats.recv_errs: 0
                              dev.igb.0.mac_stats.recv_jabber: 0
                              dev.igb.0.mac_stats.recv_oversize: 0
                              dev.igb.0.mac_stats.recv_fragmented: 0
                              dev.igb.0.mac_stats.recv_undersize: 0
                              dev.igb.0.mac_stats.recv_no_buff: 0
                              dev.igb.0.mac_stats.recv_length_errors: 0
                              dev.igb.0.mac_stats.missed_packets: 0
                              dev.igb.0.mac_stats.defer_count: 0
                              dev.igb.0.mac_stats.sequence_errors: 0
                              dev.igb.0.mac_stats.symbol_errors: 0
                              dev.igb.0.mac_stats.collision_count: 0
                              dev.igb.0.mac_stats.late_coll: 0
                              dev.igb.0.mac_stats.multiple_coll: 0
                              dev.igb.0.mac_stats.single_coll: 0
                              dev.igb.0.mac_stats.excess_coll: 0
                              dev.igb.0.queue7.lro_flushed: 0
                              dev.igb.0.queue7.lro_queued: 0
                              dev.igb.0.queue7.rx_bytes: 61172
                              dev.igb.0.queue7.rx_packets: 437
                              dev.igb.0.queue7.rxd_tail: 436
                              dev.igb.0.queue7.rxd_head: 437
                              dev.igb.0.queue7.tx_packets: 0
                              dev.igb.0.queue7.no_desc_avail: 0
                              dev.igb.0.queue7.txd_tail: 0
                              dev.igb.0.queue7.txd_head: 0
                              dev.igb.0.queue7.interrupt_rate: 50000
                              dev.igb.0.queue6.lro_flushed: 0
                              dev.igb.0.queue6.lro_queued: 0
                              dev.igb.0.queue6.rx_bytes: 6689
                              dev.igb.0.queue6.rx_packets: 78
                              dev.igb.0.queue6.rxd_tail: 77
                              dev.igb.0.queue6.rxd_head: 78
                              dev.igb.0.queue6.tx_packets: 0
                              dev.igb.0.queue6.no_desc_avail: 0
                              dev.igb.0.queue6.txd_tail: 0
                              dev.igb.0.queue6.txd_head: 0
                              dev.igb.0.queue6.interrupt_rate: 90909
                              dev.igb.0.queue5.lro_flushed: 0
                              dev.igb.0.queue5.lro_queued: 0
                              dev.igb.0.queue5.rx_bytes: 79329
                              dev.igb.0.queue5.rx_packets: 955
                              dev.igb.0.queue5.rxd_tail: 954
                              dev.igb.0.queue5.rxd_head: 955
                              dev.igb.0.queue5.tx_packets: 0
                              dev.igb.0.queue5.no_desc_avail: 0
                              dev.igb.0.queue5.txd_tail: 0
                              dev.igb.0.queue5.txd_head: 0
                              dev.igb.0.queue5.interrupt_rate: 90909
                              dev.igb.0.queue4.lro_flushed: 0
                              dev.igb.0.queue4.lro_queued: 0
                              dev.igb.0.queue4.rx_bytes: 2661
                              dev.igb.0.queue4.rx_packets: 39
                              dev.igb.0.queue4.rxd_tail: 38
                              dev.igb.0.queue4.rxd_head: 39
                              dev.igb.0.queue4.tx_packets: 0
                              dev.igb.0.queue4.no_desc_avail: 0
                              dev.igb.0.queue4.txd_tail: 0
                              dev.igb.0.queue4.txd_head: 0
                              dev.igb.0.queue4.interrupt_rate: 76923
                              dev.igb.0.queue3.lro_flushed: 0
                              dev.igb.0.queue3.lro_queued: 0
                              dev.igb.0.queue3.rx_bytes: 4055
                              dev.igb.0.queue3.rx_packets: 60
                              dev.igb.0.queue3.rxd_tail: 59
                              dev.igb.0.queue3.rxd_head: 60
                              dev.igb.0.queue3.tx_packets: 0
                              dev.igb.0.queue3.no_desc_avail: 0
                              dev.igb.0.queue3.txd_tail: 0
                              dev.igb.0.queue3.txd_head: 0
                              dev.igb.0.queue3.interrupt_rate: 90909
                              dev.igb.0.queue2.lro_flushed: 0
                              dev.igb.0.queue2.lro_queued: 0
                              dev.igb.0.queue2.rx_bytes: 7874
                              dev.igb.0.queue2.rx_packets: 107
                              dev.igb.0.queue2.rxd_tail: 106
                              dev.igb.0.queue2.rxd_head: 107
                              dev.igb.0.queue2.tx_packets: 0
                              dev.igb.0.queue2.no_desc_avail: 0
                              dev.igb.0.queue2.txd_tail: 0
                              dev.igb.0.queue2.txd_head: 0
                              dev.igb.0.queue2.interrupt_rate: 90909
                              dev.igb.0.queue1.lro_flushed: 0
                              dev.igb.0.queue1.lro_queued: 0
                              dev.igb.0.queue1.rx_bytes: 7373
                              dev.igb.0.queue1.rx_packets: 69
                              dev.igb.0.queue1.rxd_tail: 68
                              dev.igb.0.queue1.rxd_head: 69
                              dev.igb.0.queue1.tx_packets: 0
                              dev.igb.0.queue1.no_desc_avail: 0
                              dev.igb.0.queue1.txd_tail: 0
                              dev.igb.0.queue1.txd_head: 0
                              dev.igb.0.queue1.interrupt_rate: 90909
                              dev.igb.0.queue0.lro_flushed: 0
                              dev.igb.0.queue0.lro_queued: 0
                              dev.igb.0.queue0.rx_bytes: 189284
                              dev.igb.0.queue0.rx_packets: 3150
                              dev.igb.0.queue0.rxd_tail: 77
                              dev.igb.0.queue0.rxd_head: 78
                              dev.igb.0.queue0.tx_packets: 1413
                              dev.igb.0.queue0.no_desc_avail: 0
                              dev.igb.0.queue0.txd_tail: 393
                              dev.igb.0.queue0.txd_head: 393
                              dev.igb.0.queue0.interrupt_rate: 100000
                              dev.igb.0.fc_low_water: 33152
                              dev.igb.0.fc_high_water: 33168
                              dev.igb.0.rx_buf_alloc: 0
                              dev.igb.0.tx_buf_alloc: 0
                              dev.igb.0.extended_int_mask: 2147484159
                              dev.igb.0.interrupt_mask: 4
                              dev.igb.0.rx_control: 67141634
                              dev.igb.0.device_control: 1478230593
                              dev.igb.0.watchdog_timeouts: 0
                              dev.igb.0.rx_overruns: 0
                              dev.igb.0.tx_dma_fail: 0
                              dev.igb.0.mbuf_defrag_fail: 0
                              dev.igb.0.link_irq: 4
                              dev.igb.0.dropped: 0
                              dev.igb.0.eee_disabled: 0
                              dev.igb.0.dmac: 0
                              dev.igb.0.tx_processing_limit: -1
                              dev.igb.0.rx_processing_limit: 100
                              dev.igb.0.fc: 3
                              dev.igb.0.enable_aim: 1
                              dev.igb.0.nvm: -1
                              dev.igb.0.%parent: pci2
                              dev.igb.0.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1b12 class=0x020000
                              dev.igb.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 handle=\_SB_.PCI0.RP05.PXSX
                              dev.igb.0.%driver: igb
                              dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
                              dev.igb.%parent:
                              

                              sysctl -a | grep netmap output -

                              device	netmap
                              dev.netmap.ixl_rx_miss_bufs: 0
                              dev.netmap.ixl_rx_miss: 0
                              dev.netmap.iflib_rx_miss_bufs: 0
                              dev.netmap.iflib_rx_miss: 0
                              dev.netmap.iflib_crcstrip: 1
                              dev.netmap.bridge_batch: 1024
                              dev.netmap.default_pipes: 0
                              dev.netmap.priv_buf_num: 4098
                              dev.netmap.priv_buf_size: 2048
                              dev.netmap.buf_curr_num: 163840
                              dev.netmap.buf_num: 163840
                              dev.netmap.buf_curr_size: 2048
                              dev.netmap.buf_size: 2048
                              dev.netmap.priv_ring_num: 4
                              dev.netmap.priv_ring_size: 20480
                              dev.netmap.ring_curr_num: 200
                              dev.netmap.ring_num: 200
                              dev.netmap.ring_curr_size: 36864
                              dev.netmap.ring_size: 36864
                              dev.netmap.priv_if_num: 1
                              dev.netmap.priv_if_size: 1024
                              dev.netmap.if_curr_num: 100
                              dev.netmap.if_num: 100
                              dev.netmap.if_curr_size: 1024
                              dev.netmap.if_size: 1024
                              dev.netmap.generic_rings: 1
                              dev.netmap.generic_ringsize: 1024
                              dev.netmap.generic_mit: 100000
                              dev.netmap.admode: 0
                              dev.netmap.fwd: 0
                              dev.netmap.flags: 0
                              dev.netmap.adaptive_io: 0
                              dev.netmap.txsync_retry: 2
                              dev.netmap.no_pendintr: 1
                              dev.netmap.mitigate: 1
                              dev.netmap.no_timestamp: 0
                              dev.netmap.verbose: 0
                              dev.netmap.ix_rx_miss_bufs: 0
                              dev.netmap.ix_rx_miss: 0
                              dev.netmap.ix_crcstrip: 0
                              
                              1 Reply Last reply Reply Quote 0
                              • N
                                newUser2pfSense
                                last edited by

                                bmeeks...Thank you for the Suricata rules explanation. I did a little further digging, and even though I haven't been able to read all of it yet, I found a post of yours:
                                https://forum.netgate.com/topic/128480/how-automatic-sid-management-and-user-rule-overrides-work-in-snort-and-suricata

                                1 Reply Last reply Reply Quote 0
                                • NollipfSenseN
                                  NollipfSense @boobletins
                                  last edited by

                                  @boobletins said in Suricata InLine with igb NICs:

                                  @nollipfsense

                                  So here are some initial suggestions. Please keep in mind that I've been working on this for ~1 week (in other words: not long), and I'm not a FreeBSD, pfSense, or Suricata expert.

                                  Start by making a backup of your configuration.

                                  Do these first:
                                  My understanding is that flow control should be off on any netmap interface. You have bi-directional flow control enabled:

                                  dev.igb.0.fc: 3
                                  

                                  Disable flow control on all active interfaces using system tunables. Set dev.igb.0.fc=0 (and dev.igb.1.fc=0)

                                  Actively set energy efficient ethernet to disabled:
                                  dev.igb.0.eee_disabled=1

                                  Actively force IPv6_TXCSUM6 off by adding the following to config.xml in a shellcmd tag:

                                  ifconfig igb0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso
                                  

                                  (see above in this thread for a link on where/how to do that).

                                  Edit:
                                  To be clear: anywhere I have a command that says "igb0" or "igb.0" you will want to duplicate that for igb1 and any other interface you're running netmap on.

                                  So you will need 2 shellcmd lines in config.xml, and two new system tunables for flow control, etc


                                  Consider changing later:

                                  Set rx processing limit:
                                  dev.igb.0.rx_processing_limit: -1

                                  It looks like your txd and rxd are both set to 1024 currently, I suggest you move those to 4096:
                                  hw.igb.txd=4096
                                  hw.igb.rxd=4096

                                  By changing your txd and rxd we may need to revisit your netmap buf/ring (memory settings).

                                  We may also revisit your interrupt and queue settings.

                                  Boobletins, I will need to revisit later...currently, I am happy with just making adjustments to the buf_size:4096 and disable IPv6...haven't got any alert since and my Internet will be down for a while because of moving.

                                  pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
                                  pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

                                  1 Reply Last reply Reply Quote 0
                                  • B
                                    boobletins @newUser2pfSense
                                    last edited by boobletins

                                    So you're running netmap/IPS mode on igb0 (LAN), igb1 (OPT?), and igb3 (WAN)?

                                    What type of CPU is in the machine (# of cores?, is hyper-threading enabled)? How much RAM?

                                    Are you saturating all 3 active interfaces? Or just 2?

                                    Start by making a backup of your configuration.

                                    First disable flow control (as discussed above):
                                    You have the following on all igb interfaces which means bi-directional flow control is enabled.:

                                    dev.igb.0.fc: 3
                                    

                                    Change to fc=0 on all netmap interfaces in system tunables. This will take ethernet flow control out of the picture in favor of higher level flow control (TCP) which is less likely to mess with buffering and clog things up.

                                    Let's look at what generates this particular netmap error:
                                    From http://web.mit.edu/freebsd/head/sys/dev/netmap/netmap.c

                                    /*
                                     * put a copy of the buffers marked NS_FORWARD into an mbuf chain.
                                     * Take packets from hwcur to ring->head marked NS_FORWARD (or forced)
                                     * and pass them up. Drop remaining packets in the unlikely event
                                     * of an mbuf shortage.
                                     */
                                    static void
                                    netmap_grab_packets(struct netmap_kring *kring, struct mbq *q, int force)
                                    {
                                    	u_int const lim = kring->nkr_num_slots - 1;
                                    	u_int const head = kring->ring->head;
                                    	u_int n;
                                    	struct netmap_adapter *na = kring->na;
                                    
                                    	for (n = kring->nr_hwcur; n != head; n = nm_next(n, lim)) {
                                    		struct mbuf *m;
                                    		struct netmap_slot *slot = &kring->ring->slot[n];
                                    
                                    		if ((slot->flags & NS_FORWARD) == 0 && !force)
                                    			continue;
                                    		if (slot->len < 14 || slot->len > NETMAP_BUF_SIZE(na)) {
                                    			RD(5, "bad pkt at %d len %d", n, slot->len);
                                    			continue;
                                    		}
                                    		slot->flags &= ~NS_FORWARD; // XXX needed ?
                                    		/* XXX TODO: adapt to the case of a multisegment packet */
                                    		m = m_devget(NMB(na, slot), slot->len, 0, na->ifp, NULL);
                                    
                                    		if (m == NULL)
                                    			break;
                                    		mbq_enqueue(q, m);
                                    	}
                                    }
                                    

                                    I'm no C expert, but as I read this code there are 2 ways to generate your error in netmap:

                                    1. a slot is of size less than 14
                                    2. a slot is of size greater than the netmap buffer can handle

                                    I don't know what the magic number 14 represents, but let's assume it's some kind of minimum packet size we can't control. If that's the case, then the bad_pkt error is generated from packets that are actually bad.

                                    That's not what you have. The error is telling us the current hwcur value (the first number - the slot number in the ring) and the length or size of the slot (eg #777 with len 2154).

                                    So this is a memory issue. The error would be better off saying something like "dropped a packet because it was too short or too large!" -- but that would be useful to others and is thus verboten ;)

                                    edited: Removed incorrect speculation. Skip to my latest post.

                                    1 Reply Last reply Reply Quote 0
                                    • B
                                      boobletins
                                      last edited by boobletins

                                      This post is deleted!
                                      B 1 Reply Last reply Reply Quote 0
                                      • B
                                        boobletins @boobletins
                                        last edited by boobletins

                                        @boobletins said in Suricata InLine with igb NICs:

                                        I guess it depends on what NETMAP_BUF_SIZE(na) is returning. It should be either the available memory for netmap buffers, or the available kernel buffers (for the host adapter).

                                        From: https://github.com/luigirizzo/netmap/blob/master/sys/dev/netmap/netmap_kern.h

                                        #define NETMAP_BUF_SIZE(_na)	((_na)->na_lut.objsize)
                                        
                                        ...
                                        
                                        struct netmap_adapter {
                                        	...
                                        
                                        	struct netmap_lut {
                                        		struct lut_entry *lut;
                                        		struct plut_entry *plut;
                                        		uint32_t objtotal;	/* max buffer index */
                                        		uint32_t objsize;	/* buffer size */
                                        	};
                                        
                                        
                                        	/* memory allocator (opaque)
                                        	 * We also cache a pointer to the lut_entry for translating
                                        	 * buffer addresses, the total number of buffers and the buffer size.
                                        	 */
                                         	struct netmap_mem_d *nm_mem;
                                        	struct netmap_mem_d *nm_mem_prev;
                                        	struct netmap_lut na_lut;
                                        

                                        It's returning netmap adapter buffer size.

                                        Let's see.

                                        Your dev.netmap.buf_size=2048 and the length of the slot it was trying to process were all > 2048 when the error was generated.

                                        That makes a certain kind of sense. Why were the slots larger..

                                        Wait. What's your MTU set to on these interfaces? It has to be > 2048? Check this with 'ifconfig igb0' for each interface.

                                        Some sanity checks when enabling netmap would save people a lot of headaches. If your MTU is 10000 and your dev.netmap.buf_size=2048, then netmap will always choke.

                                        Know that if you set dev.netmap.buf_size to some obscenely high number to cover an equally high MTU, netmap will preallocate all of that memory and sit on it.

                                        1 Reply Last reply Reply Quote 0
                                        • N
                                          newUser2pfSense
                                          last edited by newUser2pfSense

                                          boobletins...Presently I'm using Inline IPS Mode and I only have Suricata running on my WAN and that's igb3. I'm using igb0 and igb1 as well for my WLAN and LAN.

                                          CPU:
                                          Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
                                          Current: 4000 MHz, Max: 4001 MHz
                                          8 CPUs: 1 package(s) x 4 core(s) x 2 hardware threads
                                          AES-NI CPU Crypto: Yes (active)

                                          Memory:
                                          64 Gig

                                          System Tunables addition:
                                          Tunable Name Description Value
                                          dev.igb.0.fc disable flow control 0
                                          dev.igb.1.fc disable flow control 0
                                          dev.igb.2.fc disable flow control 0
                                          dev.igb.3.fc disable flow control 0
                                          dev.igb.0.eee_disabled disable energy efficient ethernet 1
                                          dev.igb.1.eee_disabled disable energy efficient ethernet 1
                                          dev.igb.2.eee_disabled disable energy efficient ethernet 1
                                          dev.igb.3.eee_disabled disable energy efficient ethernet 1

                                          config.xml addition (I had to take the beginning < and ending > out to get it to display):
                                          shellcmd>ifconfig igb0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso</shellcmd
                                          shellcmd>ifconfig igb1 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso</shellcmd
                                          shellcmd>ifconfig igb2 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso</shellcmd
                                          shellcmd>ifconfig igb3 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso</shellcmd
                                          shellcmd>ifconfig em0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso</shellcmd

                                          igb0,1,2,3 all have an MTU of 1500 which I believe is default. I haven't set any values for this myself.

                                          B 1 Reply Last reply Reply Quote 2
                                          • B
                                            boobletins @newUser2pfSense
                                            last edited by

                                            @newuser2pfsense said in Suricata InLine with igb NICs:

                                            boobletins...Presently I'm using Inline IPS Mode and I only have Suricata running on my WAN and that's igb3. I'm using igb0 and igb1 as well for my WLAN and LAN.
                                            dev.igb.3.fc disable flow control 0

                                            Previously you had dev.igb.3.fc=3 Does the "bad pkt" error persist with dev.igb.3.fc=0?

                                            Just to confirm, could you double check and paste me the full output from

                                            ifconfig igb3
                                            

                                            Please paste any additional system turntables you've set via the ui and your full loader.conf.local (minus any sensitive data).

                                            Please then manually double check and paste the output from these commands:

                                            sysctl -a | grep nmbclusters
                                            sysctl -a | grep msi
                                            sysctl -a | grep num_queues
                                            dmesg | grep igb3
                                            

                                            Above is not busy work, I'm having you manually confirm because with a few commands I found that when I set them in loader.conf.local they didn't take effect. I needed to put some in the ui system tuneables.

                                            We have more settings to tinker with, I made a bunch of changes before the errors went away, but I'm trying to narrow down the issue before just throwing a bunch of new settings at you. I'm pretty confident we can get this working on your igb since its working on mine with 0 errors for over a week now.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.