Suricata InLine with igb NICs

NollipfSense

@boobletins said in Suricata InLine with igb NICs:

Under System / Advanced / Networking, is "Allow IPv6" checked?

And how many CPU cores? Is hyperthreading enabled?

Yes...allowed IPv6 checked...CPU Type AMD Athlon(tm) 64 X2 Dual Core Processor 4800+
2 CPUs: 1 package(s) x 2 core(s)
AES-NI CPU Crypto: No

No sure where to check for hyperthread...now I will disable IPv6...thought I did.

boobletins

@nollipfsense

So here are some initial suggestions. Please keep in mind that I've been working on this for ~1 week (in other words: not long), and I'm not a FreeBSD, pfSense, or Suricata expert.

Start by making a backup of your configuration.

Do these first:
My understanding is that flow control should be off on any netmap interface. You have bi-directional flow control enabled:

dev.igb.0.fc: 3

Disable flow control on all active interfaces using system tunables. Set dev.igb.0.fc=0 (and dev.igb.1.fc=0)

Actively set energy efficient ethernet to disabled:
dev.igb.0.eee_disabled=1

Actively force IPv6_TXCSUM6 off by adding the following to config.xml in a shellcmd tag:

ifconfig igb0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso

(see above in this thread for a link on where/how to do that).

Edit:
To be clear: anywhere I have a command that says "igb0" or "igb.0" you will want to duplicate that for igb1 and any other interface you're running netmap on.

So you will need 2 shellcmd lines in config.xml, and two new system tunables for flow control, etc

Consider changing later:

Set rx processing limit:
dev.igb.0.rx_processing_limit: -1

It looks like your txd and rxd are both set to 1024 currently, I suggest you move those to 4096:
hw.igb.txd=4096
hw.igb.rxd=4096

By changing your txd and rxd we may need to revisit your netmap buf/ring (memory settings).

We may also revisit your interrupt and queue settings.

boobletins

It would be great if you could let me know what happens after the initial changes -- if you continue to get netmap errors or not.

If you do, don't jump right to the 2nd section of changes, we should verify that the changes we made above took properly. I learned the hard way that I was putting some settings in the wrong places.

newUser2pfSense

boobletins...I apologize for not getting back sooner; other projects. I added the shellcmd line to the /cf/conf/config.xml file as you suggested. I re-enabled Suricata in Inline IPS Mode and restarted pfSense. I ran ifconfig against all four ethernet interfaces on my SuperMicro adapter and TXCSUM_IPV6 was not listed.

One thing I find interesting is in the Services > Suricata > Alerts tab, all of the text is now black in color when before making the above change it was all red in color in Inline IPS Mode. As well, there are no entries in the Blocks tab when before making the change it was automagically populated with over 300 blocked IP addresses in Inline IPS Mode. I don't know if this is normal or not. I didn't change any of the Suricata WAN Categories.

boobletins

If you are running Suricata in inline mode, you will not see blocked IP addresses in the blocked tab, as any traffic that conforms to your "drop" rules is automatically intercepted and dropped (as opposed to initially logged, then ip banned as in Legacy mode).

The red text in the Alerts tab is letting you know that the traffic was indeed intercepted and dropped (since you don't have any information in the Block tab anymore).

That you are missing both blocks and red text means that either no traffic has conformed to your block rules yet, or something has gone wrong.

Double check which mode Suricata is running in. Then double check that you have some drop rules defined.

But originally the issue was netmap, yes? If so, have you see any netmap errors? Can you complete a speedtest with Suricata enabled in inline mode now?

newUser2pfSense

I guess my "drop" rules are just based on the Suricata WAN Categories I selected/checked? When you checkmark what you want, Suricata applies those rules to the incoming data and drops it?

I'm definitely running in Inline IPS Mode and all of the Alerts entries are in black colored text. As well, all of the text I typed into the loader.conf.local file is commented out.

Yes, the original issue was many, many, netmap_grab_packets console/log messages. At the time of this post, I don't have any netmap errors, yet . I can actually complete a speedtest in Inline IPS Mode now.

boobletins

So Suricata is complex and can do many things.

I suggest you read about SID management and check out the SID management and categories tabs.

If I had to guess, I would say that you have a minimal set of "drop" rules in place and just haven't had blocked traffic hit your server yet. When you do, you'll see the red text.

If you want to convert "alerts" to "drops," you'll typically do that in SID management.

newUser2pfSense

Unfortunately, I just received several netmap_grap_packets errors:

netmap_grab_packets bad pkt at 465 len 2147
netmap_grab_packets bad pkt at 116 len 2154
netmap_grab_packets bad pkt at 39 len 2147
netmap_grab_packets bad pkt at 872 len 2147
netmap_grab_packets bad pkt at 860 len 2147
netmap_grab_packets bad pkt at 838 len 2147
netmap_grab_packets bad pkt at 777 len 2154

Apparently the shellcmd didn't work after all.

boobletins

Could you give me the same info I asked for from NollipfSense above?

The results of these commands:
ifconfig igb0 | grep CSUM
sysctl -a | grep igb
sysctl -a | grep netmap

bmeeks

@newuser2pfsense said in Suricata InLine with igb NICs:

I guess my "drop" rules are just based on the Suricata WAN Categories I selected/checked? When you checkmark what you want, Suricata applies those rules to the incoming data and drops it?

I'm definitely running in Inline IPS Mode and all of the Alerts entries are in black colored text. As well, all of the text I typed into the loader.conf.local file is commented out.

Yes, the original issue was many, many, netmap_grab_packets console/log messages. At the time of this post, I don't have any netmap errors, yet . I can actually complete a speedtest in Inline IPS Mode now.

When you run with Inline IPS Mode enabled, you have to manually change rules you want to block traffic from ALERT to DROP. You can do that in two places: (1) on the RULES tab with the selected category displayed; (2) using SID Management keywords on the SID MGMT tab. Sounds like you do not have any DROP rules enabled if you are getting no red alert entries. Either that, or as @boobletins stated, your enabled DROP rules have not been triggered.

newUser2pfSense

boobletins...Here is the output of the commands you requested.

ifconfig igbX | grep CSUM output -

for igb0:  options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
for igb1:  options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
for igb2 [not used]:  options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
for igb3 [WAN]:  options=1000b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,NETMAP>
for em0 [not used]:  options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>

sysctl -a | grep igb output -

device	igb
hw.igb.tx_process_limit: -1
hw.igb.rx_process_limit: 100
hw.igb.num_queues: 0
hw.igb.header_split: 0
hw.igb.max_interrupt_rate: 8000
hw.igb.enable_msix: 1
hw.igb.enable_aim: 1
hw.igb.txd: 1024
hw.igb.rxd: 1024
dev.igb.3.host.header_redir_missed: 0
dev.igb.3.host.serdes_violation_pkt: 0
dev.igb.3.host.length_errors: 3
dev.igb.3.host.tx_good_bytes: 213941
dev.igb.3.host.rx_good_bytes: 3967552
dev.igb.3.host.breaker_tx_pkt_drop: 0
dev.igb.3.host.tx_good_pkt: 0
dev.igb.3.host.breaker_rx_pkt_drop: 0
dev.igb.3.host.breaker_rx_pkts: 0
dev.igb.3.host.rx_pkt: 3
dev.igb.3.host.host_tx_pkt_discard: 0
dev.igb.3.host.breaker_tx_pkt: 0
dev.igb.3.interrupts.rx_overrun: 0
dev.igb.3.interrupts.rx_desc_min_thresh: 0
dev.igb.3.interrupts.tx_queue_min_thresh: 231
dev.igb.3.interrupts.tx_queue_empty: 2417
dev.igb.3.interrupts.tx_abs_timer: 0
dev.igb.3.interrupts.tx_pkt_timer: 0
dev.igb.3.interrupts.rx_abs_timer: 0
dev.igb.3.interrupts.rx_pkt_timer: 18194
dev.igb.3.interrupts.asserts: 22095
dev.igb.3.mac_stats.tso_ctx_fail: 0
dev.igb.3.mac_stats.tso_txd: 0
dev.igb.3.mac_stats.tx_frames_1024_1522: 3
dev.igb.3.mac_stats.tx_frames_512_1023: 16
dev.igb.3.mac_stats.tx_frames_256_511: 21
dev.igb.3.mac_stats.tx_frames_128_255: 90
dev.igb.3.mac_stats.tx_frames_65_127: 1502
dev.igb.3.mac_stats.tx_frames_64: 785
dev.igb.3.mac_stats.mcast_pkts_txd: 3
dev.igb.3.mac_stats.bcast_pkts_txd: 10
dev.igb.3.mac_stats.good_pkts_txd: 2417
dev.igb.3.mac_stats.total_pkts_txd: 2417
dev.igb.3.mac_stats.total_octets_txd: 213941
dev.igb.3.mac_stats.good_octets_txd: 213941
dev.igb.3.mac_stats.total_octets_recvd: 4000401
dev.igb.3.mac_stats.good_octets_recvd: 3967552
dev.igb.3.mac_stats.rx_frames_1024_1522: 1363
dev.igb.3.mac_stats.rx_frames_512_1023: 339
dev.igb.3.mac_stats.rx_frames_256_511: 1232
dev.igb.3.mac_stats.rx_frames_128_255: 1117
dev.igb.3.mac_stats.rx_frames_65_127: 6557
dev.igb.3.mac_stats.rx_frames_64: 7589
dev.igb.3.mac_stats.mcast_pkts_recvd: 11704
dev.igb.3.mac_stats.bcast_pkts_recvd: 3585
dev.igb.3.mac_stats.good_pkts_recvd: 18197
dev.igb.3.mac_stats.total_pkts_recvd: 18528
dev.igb.3.mac_stats.mgmt_pkts_txd: 0
dev.igb.3.mac_stats.mgmt_pkts_drop: 0
dev.igb.3.mac_stats.mgmt_pkts_recvd: 0
dev.igb.3.mac_stats.unsupported_fc_recvd: 0
dev.igb.3.mac_stats.xoff_txd: 0
dev.igb.3.mac_stats.xoff_recvd: 0
dev.igb.3.mac_stats.xon_txd: 0
dev.igb.3.mac_stats.xon_recvd: 0
dev.igb.3.mac_stats.coll_ext_errs: 0
dev.igb.3.mac_stats.tx_no_crs: 0
dev.igb.3.mac_stats.alignment_errs: 0
dev.igb.3.mac_stats.crc_errs: 0
dev.igb.3.mac_stats.recv_errs: 0
dev.igb.3.mac_stats.recv_jabber: 0
dev.igb.3.mac_stats.recv_oversize: 0
dev.igb.3.mac_stats.recv_fragmented: 0
dev.igb.3.mac_stats.recv_undersize: 0
dev.igb.3.mac_stats.recv_no_buff: 0
dev.igb.3.mac_stats.recv_length_errors: 0
dev.igb.3.mac_stats.missed_packets: 0
dev.igb.3.mac_stats.defer_count: 0
dev.igb.3.mac_stats.sequence_errors: 0
dev.igb.3.mac_stats.symbol_errors: 0
dev.igb.3.mac_stats.collision_count: 0
dev.igb.3.mac_stats.late_coll: 0
dev.igb.3.mac_stats.multiple_coll: 0
dev.igb.3.mac_stats.single_coll: 0
dev.igb.3.mac_stats.excess_coll: 0
dev.igb.3.queue7.lro_flushed: 0
dev.igb.3.queue7.lro_queued: 0
dev.igb.3.queue7.rx_bytes: 0
dev.igb.3.queue7.rx_packets: 129
dev.igb.3.queue7.rxd_tail: 561
dev.igb.3.queue7.rxd_head: 562
dev.igb.3.queue7.tx_packets: 0
dev.igb.3.queue7.no_desc_avail: 0
dev.igb.3.queue7.txd_tail: 0
dev.igb.3.queue7.txd_head: 0
dev.igb.3.queue7.interrupt_rate: 8000
dev.igb.3.queue6.lro_flushed: 0
dev.igb.3.queue6.lro_queued: 0
dev.igb.3.queue6.rx_bytes: 0
dev.igb.3.queue6.rx_packets: 30
dev.igb.3.queue6.rxd_tail: 902
dev.igb.3.queue6.rxd_head: 903
dev.igb.3.queue6.tx_packets: 0
dev.igb.3.queue6.no_desc_avail: 0
dev.igb.3.queue6.txd_tail: 0
dev.igb.3.queue6.txd_head: 0
dev.igb.3.queue6.interrupt_rate: 8000
dev.igb.3.queue5.lro_flushed: 0
dev.igb.3.queue5.lro_queued: 0
dev.igb.3.queue5.rx_bytes: 0
dev.igb.3.queue5.rx_packets: 30
dev.igb.3.queue5.rxd_tail: 344
dev.igb.3.queue5.rxd_head: 345
dev.igb.3.queue5.tx_packets: 0
dev.igb.3.queue5.no_desc_avail: 0
dev.igb.3.queue5.txd_tail: 0
dev.igb.3.queue5.txd_head: 0
dev.igb.3.queue5.interrupt_rate: 8000
dev.igb.3.queue4.lro_flushed: 0
dev.igb.3.queue4.lro_queued: 0
dev.igb.3.queue4.rx_bytes: 0
dev.igb.3.queue4.rx_packets: 87
dev.igb.3.queue4.rxd_tail: 524
dev.igb.3.queue4.rxd_head: 525
dev.igb.3.queue4.tx_packets: 0
dev.igb.3.queue4.no_desc_avail: 0
dev.igb.3.queue4.txd_tail: 0
dev.igb.3.queue4.txd_head: 0
dev.igb.3.queue4.interrupt_rate: 8000
dev.igb.3.queue3.lro_flushed: 0
dev.igb.3.queue3.lro_queued: 0
dev.igb.3.queue3.rx_bytes: 0
dev.igb.3.queue3.rx_packets: 29
dev.igb.3.queue3.rxd_tail: 453
dev.igb.3.queue3.rxd_head: 454
dev.igb.3.queue3.tx_packets: 0
dev.igb.3.queue3.no_desc_avail: 0
dev.igb.3.queue3.txd_tail: 0
dev.igb.3.queue3.txd_head: 0
dev.igb.3.queue3.interrupt_rate: 76923
dev.igb.3.queue2.lro_flushed: 0
dev.igb.3.queue2.lro_queued: 0
dev.igb.3.queue2.rx_bytes: 0
dev.igb.3.queue2.rx_packets: 28
dev.igb.3.queue2.rxd_tail: 876
dev.igb.3.queue2.rxd_head: 877
dev.igb.3.queue2.tx_packets: 0
dev.igb.3.queue2.no_desc_avail: 0
dev.igb.3.queue2.txd_tail: 0
dev.igb.3.queue2.txd_head: 0
dev.igb.3.queue2.interrupt_rate: 8000
dev.igb.3.queue1.lro_flushed: 0
dev.igb.3.queue1.lro_queued: 0
dev.igb.3.queue1.rx_bytes: 0
dev.igb.3.queue1.rx_packets: 17
dev.igb.3.queue1.rxd_tail: 482
dev.igb.3.queue1.rxd_head: 483
dev.igb.3.queue1.tx_packets: 0
dev.igb.3.queue1.no_desc_avail: 0
dev.igb.3.queue1.txd_tail: 0
dev.igb.3.queue1.txd_head: 0
dev.igb.3.queue1.interrupt_rate: 8000
dev.igb.3.queue0.lro_flushed: 0
dev.igb.3.queue0.lro_queued: 0
dev.igb.3.queue0.rx_bytes: 0
dev.igb.3.queue0.rx_packets: 142
dev.igb.3.queue0.rxd_tail: 253
dev.igb.3.queue0.rxd_head: 254
dev.igb.3.queue0.tx_packets: 353
dev.igb.3.queue0.no_desc_avail: 0
dev.igb.3.queue0.txd_tail: 18
dev.igb.3.queue0.txd_head: 18
dev.igb.3.queue0.interrupt_rate: 8000
dev.igb.3.fc_low_water: 33152
dev.igb.3.fc_high_water: 33168
dev.igb.3.rx_buf_alloc: 0
dev.igb.3.tx_buf_alloc: 0
dev.igb.3.extended_int_mask: 2147484159
dev.igb.3.interrupt_mask: 4
dev.igb.3.rx_control: 67141658
dev.igb.3.device_control: 1478230593
dev.igb.3.watchdog_timeouts: 0
dev.igb.3.rx_overruns: 0
dev.igb.3.tx_dma_fail: 0
dev.igb.3.mbuf_defrag_fail: 0
dev.igb.3.link_irq: 7
dev.igb.3.dropped: 0
dev.igb.3.eee_disabled: 0
dev.igb.3.dmac: 0
dev.igb.3.tx_processing_limit: -1
dev.igb.3.rx_processing_limit: 100
dev.igb.3.fc: 3
dev.igb.3.enable_aim: 1
dev.igb.3.nvm: -1
dev.igb.3.%parent: pci2
dev.igb.3.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1b12 class=0x020000
dev.igb.3.%location: slot=0 function=3 dbsf=pci0:2:0:3
dev.igb.3.%driver: igb
dev.igb.3.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.2.host.header_redir_missed: 0
dev.igb.2.host.serdes_violation_pkt: 0
dev.igb.2.host.length_errors: 0
dev.igb.2.host.tx_good_bytes: 0
dev.igb.2.host.rx_good_bytes: 0
dev.igb.2.host.breaker_tx_pkt_drop: 0
dev.igb.2.host.tx_good_pkt: 0
dev.igb.2.host.breaker_rx_pkt_drop: 0
dev.igb.2.host.breaker_rx_pkts: 0
dev.igb.2.host.rx_pkt: 0
dev.igb.2.host.host_tx_pkt_discard: 0
dev.igb.2.host.breaker_tx_pkt: 0
dev.igb.2.interrupts.rx_overrun: 0
dev.igb.2.interrupts.rx_desc_min_thresh: 0
dev.igb.2.interrupts.tx_queue_min_thresh: 0
dev.igb.2.interrupts.tx_queue_empty: 0
dev.igb.2.interrupts.tx_abs_timer: 0
dev.igb.2.interrupts.tx_pkt_timer: 0
dev.igb.2.interrupts.rx_abs_timer: 0
dev.igb.2.interrupts.rx_pkt_timer: 0
dev.igb.2.interrupts.asserts: 0
dev.igb.2.mac_stats.tso_ctx_fail: 0
dev.igb.2.mac_stats.tso_txd: 0
dev.igb.2.mac_stats.tx_frames_1024_1522: 0
dev.igb.2.mac_stats.tx_frames_512_1023: 0
dev.igb.2.mac_stats.tx_frames_256_511: 0
dev.igb.2.mac_stats.tx_frames_128_255: 0
dev.igb.2.mac_stats.tx_frames_65_127: 0
dev.igb.2.mac_stats.tx_frames_64: 0
dev.igb.2.mac_stats.mcast_pkts_txd: 0
dev.igb.2.mac_stats.bcast_pkts_txd: 0
dev.igb.2.mac_stats.good_pkts_txd: 0
dev.igb.2.mac_stats.total_pkts_txd: 0
dev.igb.2.mac_stats.total_octets_txd: 0
dev.igb.2.mac_stats.good_octets_txd: 0
dev.igb.2.mac_stats.total_octets_recvd: 0
dev.igb.2.mac_stats.good_octets_recvd: 0
dev.igb.2.mac_stats.rx_frames_1024_1522: 0
dev.igb.2.mac_stats.rx_frames_512_1023: 0
dev.igb.2.mac_stats.rx_frames_256_511: 0
dev.igb.2.mac_stats.rx_frames_128_255: 0
dev.igb.2.mac_stats.rx_frames_65_127: 0
dev.igb.2.mac_stats.rx_frames_64: 0
dev.igb.2.mac_stats.mcast_pkts_recvd: 0
dev.igb.2.mac_stats.bcast_pkts_recvd: 0
dev.igb.2.mac_stats.good_pkts_recvd: 0
dev.igb.2.mac_stats.total_pkts_recvd: 0
dev.igb.2.mac_stats.mgmt_pkts_txd: 0
dev.igb.2.mac_stats.mgmt_pkts_drop: 0
dev.igb.2.mac_stats.mgmt_pkts_recvd: 0
dev.igb.2.mac_stats.unsupported_fc_recvd: 0
dev.igb.2.mac_stats.xoff_txd: 0
dev.igb.2.mac_stats.xoff_recvd: 0
dev.igb.2.mac_stats.xon_txd: 0
dev.igb.2.mac_stats.xon_recvd: 0
dev.igb.2.mac_stats.coll_ext_errs: 0
dev.igb.2.mac_stats.tx_no_crs: 0
dev.igb.2.mac_stats.alignment_errs: 0
dev.igb.2.mac_stats.crc_errs: 0
dev.igb.2.mac_stats.recv_errs: 0
dev.igb.2.mac_stats.recv_jabber: 0
dev.igb.2.mac_stats.recv_oversize: 0
dev.igb.2.mac_stats.recv_fragmented: 0
dev.igb.2.mac_stats.recv_undersize: 0
dev.igb.2.mac_stats.recv_no_buff: 0
dev.igb.2.mac_stats.recv_length_errors: 0
dev.igb.2.mac_stats.missed_packets: 0
dev.igb.2.mac_stats.defer_count: 0
dev.igb.2.mac_stats.sequence_errors: 0
dev.igb.2.mac_stats.symbol_errors: 0
dev.igb.2.mac_stats.collision_count: 0
dev.igb.2.mac_stats.late_coll: 0
dev.igb.2.mac_stats.multiple_coll: 0
dev.igb.2.mac_stats.single_coll: 0
dev.igb.2.mac_stats.excess_coll: 0
dev.igb.2.queue7.lro_flushed: 0
dev.igb.2.queue7.lro_queued: 0
dev.igb.2.queue7.rx_bytes: 0
dev.igb.2.queue7.rx_packets: 0
dev.igb.2.queue7.rxd_tail: 0
dev.igb.2.queue7.rxd_head: 0
dev.igb.2.queue7.tx_packets: 0
dev.igb.2.queue7.no_desc_avail: 0
dev.igb.2.queue7.txd_tail: 0
dev.igb.2.queue7.txd_head: 0
dev.igb.2.queue7.interrupt_rate: 0
dev.igb.2.queue6.lro_flushed: 0
dev.igb.2.queue6.lro_queued: 0
dev.igb.2.queue6.rx_bytes: 0
dev.igb.2.queue6.rx_packets: 0
dev.igb.2.queue6.rxd_tail: 0
dev.igb.2.queue6.rxd_head: 0
dev.igb.2.queue6.tx_packets: 0
dev.igb.2.queue6.no_desc_avail: 0
dev.igb.2.queue6.txd_tail: 0
dev.igb.2.queue6.txd_head: 0
dev.igb.2.queue6.interrupt_rate: 0
dev.igb.2.queue5.lro_flushed: 0
dev.igb.2.queue5.lro_queued: 0
dev.igb.2.queue5.rx_bytes: 0
dev.igb.2.queue5.rx_packets: 0
dev.igb.2.queue5.rxd_tail: 0
dev.igb.2.queue5.rxd_head: 0
dev.igb.2.queue5.tx_packets: 0
dev.igb.2.queue5.no_desc_avail: 0
dev.igb.2.queue5.txd_tail: 0
dev.igb.2.queue5.txd_head: 0
dev.igb.2.queue5.interrupt_rate: 0
dev.igb.2.queue4.lro_flushed: 0
dev.igb.2.queue4.lro_queued: 0
dev.igb.2.queue4.rx_bytes: 0
dev.igb.2.queue4.rx_packets: 0
dev.igb.2.queue4.rxd_tail: 0
dev.igb.2.queue4.rxd_head: 0
dev.igb.2.queue4.tx_packets: 0
dev.igb.2.queue4.no_desc_avail: 0
dev.igb.2.queue4.txd_tail: 0
dev.igb.2.queue4.txd_head: 0
dev.igb.2.queue4.interrupt_rate: 0
dev.igb.2.queue3.lro_flushed: 0
dev.igb.2.queue3.lro_queued: 0
dev.igb.2.queue3.rx_bytes: 0
dev.igb.2.queue3.rx_packets: 0
dev.igb.2.queue3.rxd_tail: 0
dev.igb.2.queue3.rxd_head: 0
dev.igb.2.queue3.tx_packets: 0
dev.igb.2.queue3.no_desc_avail: 0
dev.igb.2.queue3.txd_tail: 0
dev.igb.2.queue3.txd_head: 0
dev.igb.2.queue3.interrupt_rate: 0
dev.igb.2.queue2.lro_flushed: 0
dev.igb.2.queue2.lro_queued: 0
dev.igb.2.queue2.rx_bytes: 0
dev.igb.2.queue2.rx_packets: 0
dev.igb.2.queue2.rxd_tail: 0
dev.igb.2.queue2.rxd_head: 0
dev.igb.2.queue2.tx_packets: 0
dev.igb.2.queue2.no_desc_avail: 0
dev.igb.2.queue2.txd_tail: 0
dev.igb.2.queue2.txd_head: 0
dev.igb.2.queue2.interrupt_rate: 0
dev.igb.2.queue1.lro_flushed: 0
dev.igb.2.queue1.lro_queued: 0
dev.igb.2.queue1.rx_bytes: 0
dev.igb.2.queue1.rx_packets: 0
dev.igb.2.queue1.rxd_tail: 0
dev.igb.2.queue1.rxd_head: 0
dev.igb.2.queue1.tx_packets: 0
dev.igb.2.queue1.no_desc_avail: 0
dev.igb.2.queue1.txd_tail: 0
dev.igb.2.queue1.txd_head: 0
dev.igb.2.queue1.interrupt_rate: 0
dev.igb.2.queue0.lro_flushed: 0
dev.igb.2.queue0.lro_queued: 0
dev.igb.2.queue0.rx_bytes: 0
dev.igb.2.queue0.rx_packets: 0
dev.igb.2.queue0.rxd_tail: 0
dev.igb.2.queue0.rxd_head: 0
dev.igb.2.queue0.tx_packets: 0
dev.igb.2.queue0.no_desc_avail: 0
dev.igb.2.queue0.txd_tail: 0
dev.igb.2.queue0.txd_head: 0
dev.igb.2.queue0.interrupt_rate: 0
dev.igb.2.fc_low_water: 33152
dev.igb.2.fc_high_water: 33168
dev.igb.2.rx_buf_alloc: 0
dev.igb.2.tx_buf_alloc: 0
dev.igb.2.extended_int_mask: 2147483648
dev.igb.2.interrupt_mask: 0
dev.igb.2.rx_control: 0
dev.igb.2.device_control: 136053313
dev.igb.2.watchdog_timeouts: 0
dev.igb.2.rx_overruns: 0
dev.igb.2.tx_dma_fail: 0
dev.igb.2.mbuf_defrag_fail: 0
dev.igb.2.link_irq: 0
dev.igb.2.dropped: 0
dev.igb.2.eee_disabled: 0
dev.igb.2.dmac: 0
dev.igb.2.tx_processing_limit: -1
dev.igb.2.rx_processing_limit: 100
dev.igb.2.fc: 3
dev.igb.2.enable_aim: 1
dev.igb.2.nvm: -1
dev.igb.2.%parent: pci2
dev.igb.2.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1b12 class=0x020000
dev.igb.2.%location: slot=0 function=2 dbsf=pci0:2:0:2
dev.igb.2.%driver: igb
dev.igb.2.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.1.host.header_redir_missed: 0
dev.igb.1.host.serdes_violation_pkt: 0
dev.igb.1.host.length_errors: 0
dev.igb.1.host.tx_good_bytes: 387524
dev.igb.1.host.rx_good_bytes: 58810
dev.igb.1.host.breaker_tx_pkt_drop: 0
dev.igb.1.host.tx_good_pkt: 1
dev.igb.1.host.breaker_rx_pkt_drop: 0
dev.igb.1.host.breaker_rx_pkts: 0
dev.igb.1.host.rx_pkt: 2
dev.igb.1.host.host_tx_pkt_discard: 0
dev.igb.1.host.breaker_tx_pkt: 0
dev.igb.1.interrupts.rx_overrun: 0
dev.igb.1.interrupts.rx_desc_min_thresh: 0
dev.igb.1.interrupts.tx_queue_min_thresh: 0
dev.igb.1.interrupts.tx_queue_empty: 584
dev.igb.1.interrupts.tx_abs_timer: 0
dev.igb.1.interrupts.tx_pkt_timer: 0
dev.igb.1.interrupts.rx_abs_timer: 0
dev.igb.1.interrupts.rx_pkt_timer: 381
dev.igb.1.interrupts.asserts: 3808
dev.igb.1.mac_stats.tso_ctx_fail: 0
dev.igb.1.mac_stats.tso_txd: 0
dev.igb.1.mac_stats.tx_frames_1024_1522: 213
dev.igb.1.mac_stats.tx_frames_512_1023: 25
dev.igb.1.mac_stats.tx_frames_256_511: 87
dev.igb.1.mac_stats.tx_frames_128_255: 9
dev.igb.1.mac_stats.tx_frames_65_127: 239
dev.igb.1.mac_stats.tx_frames_64: 12
dev.igb.1.mac_stats.mcast_pkts_txd: 7
dev.igb.1.mac_stats.bcast_pkts_txd: 5
dev.igb.1.mac_stats.good_pkts_txd: 585
dev.igb.1.mac_stats.total_pkts_txd: 585
dev.igb.1.mac_stats.total_octets_txd: 387524
dev.igb.1.mac_stats.good_octets_txd: 387524
dev.igb.1.mac_stats.total_octets_recvd: 70094
dev.igb.1.mac_stats.good_octets_recvd: 58810
dev.igb.1.mac_stats.rx_frames_1024_1522: 5
dev.igb.1.mac_stats.rx_frames_512_1023: 8
dev.igb.1.mac_stats.rx_frames_256_511: 17
dev.igb.1.mac_stats.rx_frames_128_255: 124
dev.igb.1.mac_stats.rx_frames_65_127: 217
dev.igb.1.mac_stats.rx_frames_64: 12
dev.igb.1.mac_stats.mcast_pkts_recvd: 20
dev.igb.1.mac_stats.bcast_pkts_recvd: 12
dev.igb.1.mac_stats.good_pkts_recvd: 383
dev.igb.1.mac_stats.total_pkts_recvd: 411
dev.igb.1.mac_stats.mgmt_pkts_txd: 0
dev.igb.1.mac_stats.mgmt_pkts_drop: 0
dev.igb.1.mac_stats.mgmt_pkts_recvd: 0
dev.igb.1.mac_stats.unsupported_fc_recvd: 0
dev.igb.1.mac_stats.xoff_txd: 0
dev.igb.1.mac_stats.xoff_recvd: 0
dev.igb.1.mac_stats.xon_txd: 0
dev.igb.1.mac_stats.xon_recvd: 0
dev.igb.1.mac_stats.coll_ext_errs: 0
dev.igb.1.mac_stats.tx_no_crs: 0
dev.igb.1.mac_stats.alignment_errs: 0
dev.igb.1.mac_stats.crc_errs: 0
dev.igb.1.mac_stats.recv_errs: 0
dev.igb.1.mac_stats.recv_jabber: 0
dev.igb.1.mac_stats.recv_oversize: 0
dev.igb.1.mac_stats.recv_fragmented: 0
dev.igb.1.mac_stats.recv_undersize: 0
dev.igb.1.mac_stats.recv_no_buff: 0
dev.igb.1.mac_stats.recv_length_errors: 0
dev.igb.1.mac_stats.missed_packets: 0
dev.igb.1.mac_stats.defer_count: 0
dev.igb.1.mac_stats.sequence_errors: 0
dev.igb.1.mac_stats.symbol_errors: 0
dev.igb.1.mac_stats.collision_count: 0
dev.igb.1.mac_stats.late_coll: 0
dev.igb.1.mac_stats.multiple_coll: 0
dev.igb.1.mac_stats.single_coll: 0
dev.igb.1.mac_stats.excess_coll: 0
dev.igb.1.queue7.lro_flushed: 0
dev.igb.1.queue7.lro_queued: 0
dev.igb.1.queue7.rx_bytes: 22234
dev.igb.1.queue7.rx_packets: 154
dev.igb.1.queue7.rxd_tail: 153
dev.igb.1.queue7.rxd_head: 154
dev.igb.1.queue7.tx_packets: 0
dev.igb.1.queue7.no_desc_avail: 0
dev.igb.1.queue7.txd_tail: 0
dev.igb.1.queue7.txd_head: 0
dev.igb.1.queue7.interrupt_rate: 62500
dev.igb.1.queue6.lro_flushed: 0
dev.igb.1.queue6.lro_queued: 0
dev.igb.1.queue6.rx_bytes: 4146
dev.igb.1.queue6.rx_packets: 22
dev.igb.1.queue6.rxd_tail: 21
dev.igb.1.queue6.rxd_head: 22
dev.igb.1.queue6.tx_packets: 0
dev.igb.1.queue6.no_desc_avail: 0
dev.igb.1.queue6.txd_tail: 0
dev.igb.1.queue6.txd_head: 0
dev.igb.1.queue6.interrupt_rate: 76923
dev.igb.1.queue5.lro_flushed: 0
dev.igb.1.queue5.lro_queued: 0
dev.igb.1.queue5.rx_bytes: 120
dev.igb.1.queue5.rx_packets: 2
dev.igb.1.queue5.rxd_tail: 1
dev.igb.1.queue5.rxd_head: 2
dev.igb.1.queue5.tx_packets: 0
dev.igb.1.queue5.no_desc_avail: 0
dev.igb.1.queue5.txd_tail: 0
dev.igb.1.queue5.txd_head: 0
dev.igb.1.queue5.interrupt_rate: 100000
dev.igb.1.queue4.lro_flushed: 0
dev.igb.1.queue4.lro_queued: 0
dev.igb.1.queue4.rx_bytes: 254
dev.igb.1.queue4.rx_packets: 4
dev.igb.1.queue4.rxd_tail: 3
dev.igb.1.queue4.rxd_head: 4
dev.igb.1.queue4.tx_packets: 0
dev.igb.1.queue4.no_desc_avail: 0
dev.igb.1.queue4.txd_tail: 0
dev.igb.1.queue4.txd_head: 0
dev.igb.1.queue4.interrupt_rate: 100000
dev.igb.1.queue3.lro_flushed: 0
dev.igb.1.queue3.lro_queued: 0
dev.igb.1.queue3.rx_bytes: 24026
dev.igb.1.queue3.rx_packets: 144
dev.igb.1.queue3.rxd_tail: 143
dev.igb.1.queue3.rxd_head: 144
dev.igb.1.queue3.tx_packets: 0
dev.igb.1.queue3.no_desc_avail: 0
dev.igb.1.queue3.txd_tail: 0
dev.igb.1.queue3.txd_head: 0
dev.igb.1.queue3.interrupt_rate: 6024
dev.igb.1.queue2.lro_flushed: 0
dev.igb.1.queue2.lro_queued: 0
dev.igb.1.queue2.rx_bytes: 3022
dev.igb.1.queue2.rx_packets: 14
dev.igb.1.queue2.rxd_tail: 13
dev.igb.1.queue2.rxd_head: 14
dev.igb.1.queue2.tx_packets: 0
dev.igb.1.queue2.no_desc_avail: 0
dev.igb.1.queue2.txd_tail: 0
dev.igb.1.queue2.txd_head: 0
dev.igb.1.queue2.interrupt_rate: 29411
dev.igb.1.queue1.lro_flushed: 0
dev.igb.1.queue1.lro_queued: 0
dev.igb.1.queue1.rx_bytes: 874
dev.igb.1.queue1.rx_packets: 7
dev.igb.1.queue1.rxd_tail: 6
dev.igb.1.queue1.rxd_head: 7
dev.igb.1.queue1.tx_packets: 0
dev.igb.1.queue1.no_desc_avail: 0
dev.igb.1.queue1.txd_tail: 0
dev.igb.1.queue1.txd_head: 0
dev.igb.1.queue1.interrupt_rate: 66666
dev.igb.1.queue0.lro_flushed: 0
dev.igb.1.queue0.lro_queued: 0
dev.igb.1.queue0.rx_bytes: 4193
dev.igb.1.queue0.rx_packets: 39
dev.igb.1.queue0.rxd_tail: 38
dev.igb.1.queue0.rxd_head: 39
dev.igb.1.queue0.tx_packets: 588
dev.igb.1.queue0.no_desc_avail: 0
dev.igb.1.queue0.txd_tail: 911
dev.igb.1.queue0.txd_head: 911
dev.igb.1.queue0.interrupt_rate: 90909
dev.igb.1.fc_low_water: 33152
dev.igb.1.fc_high_water: 33168
dev.igb.1.rx_buf_alloc: 0
dev.igb.1.tx_buf_alloc: 0
dev.igb.1.extended_int_mask: 2147484159
dev.igb.1.interrupt_mask: 4
dev.igb.1.rx_control: 67141634
dev.igb.1.device_control: 1478230593
dev.igb.1.watchdog_timeouts: 0
dev.igb.1.rx_overruns: 0
dev.igb.1.tx_dma_fail: 0
dev.igb.1.mbuf_defrag_fail: 0
dev.igb.1.link_irq: 4
dev.igb.1.dropped: 0
dev.igb.1.eee_disabled: 0
dev.igb.1.dmac: 0
dev.igb.1.tx_processing_limit: -1
dev.igb.1.rx_processing_limit: 100
dev.igb.1.fc: 3
dev.igb.1.enable_aim: 1
dev.igb.1.nvm: -1
dev.igb.1.%parent: pci2
dev.igb.1.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1b12 class=0x020000
dev.igb.1.%location: slot=0 function=1 dbsf=pci0:2:0:1
dev.igb.1.%driver: igb
dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.0.wake: 0
dev.igb.0.host.header_redir_missed: 0
dev.igb.0.host.serdes_violation_pkt: 0
dev.igb.0.host.length_errors: 0
dev.igb.0.host.tx_good_bytes: 1710353
dev.igb.0.host.rx_good_bytes: 377365
dev.igb.0.host.breaker_tx_pkt_drop: 0
dev.igb.0.host.tx_good_pkt: 0
dev.igb.0.host.breaker_rx_pkt_drop: 0
dev.igb.0.host.breaker_rx_pkts: 0
dev.igb.0.host.rx_pkt: 0
dev.igb.0.host.host_tx_pkt_discard: 0
dev.igb.0.host.breaker_tx_pkt: 0
dev.igb.0.interrupts.rx_overrun: 0
dev.igb.0.interrupts.rx_desc_min_thresh: 0
dev.igb.0.interrupts.tx_queue_min_thresh: 23214
dev.igb.0.interrupts.tx_queue_empty: 1413
dev.igb.0.interrupts.tx_abs_timer: 0
dev.igb.0.interrupts.tx_pkt_timer: 0
dev.igb.0.interrupts.rx_abs_timer: 0
dev.igb.0.interrupts.rx_pkt_timer: 4886
dev.igb.0.interrupts.asserts: 8193
dev.igb.0.mac_stats.tso_ctx_fail: 0
dev.igb.0.mac_stats.tso_txd: 0
dev.igb.0.mac_stats.tx_frames_1024_1522: 1093
dev.igb.0.mac_stats.tx_frames_512_1023: 30
dev.igb.0.mac_stats.tx_frames_256_511: 30
dev.igb.0.mac_stats.tx_frames_128_255: 10
dev.igb.0.mac_stats.tx_frames_65_127: 42
dev.igb.0.mac_stats.tx_frames_64: 208
dev.igb.0.mac_stats.mcast_pkts_txd: 1
dev.igb.0.mac_stats.bcast_pkts_txd: 10
dev.igb.0.mac_stats.good_pkts_txd: 1413
dev.igb.0.mac_stats.total_pkts_txd: 1413
dev.igb.0.mac_stats.total_octets_txd: 1710353
dev.igb.0.mac_stats.good_octets_txd: 1710353
dev.igb.0.mac_stats.total_octets_recvd: 384601
dev.igb.0.mac_stats.good_octets_recvd: 377365
dev.igb.0.mac_stats.rx_frames_1024_1522: 3
dev.igb.0.mac_stats.rx_frames_512_1023: 1
dev.igb.0.mac_stats.rx_frames_256_511: 22
dev.igb.0.mac_stats.rx_frames_128_255: 385
dev.igb.0.mac_stats.rx_frames_65_127: 1331
dev.igb.0.mac_stats.rx_frames_64: 3144
dev.igb.0.mac_stats.mcast_pkts_recvd: 412
dev.igb.0.mac_stats.bcast_pkts_recvd: 2902
dev.igb.0.mac_stats.good_pkts_recvd: 4886
dev.igb.0.mac_stats.total_pkts_recvd: 4902
dev.igb.0.mac_stats.mgmt_pkts_txd: 0
dev.igb.0.mac_stats.mgmt_pkts_drop: 0
dev.igb.0.mac_stats.mgmt_pkts_recvd: 0
dev.igb.0.mac_stats.unsupported_fc_recvd: 0
dev.igb.0.mac_stats.xoff_txd: 0
dev.igb.0.mac_stats.xoff_recvd: 0
dev.igb.0.mac_stats.xon_txd: 0
dev.igb.0.mac_stats.xon_recvd: 0
dev.igb.0.mac_stats.coll_ext_errs: 0
dev.igb.0.mac_stats.tx_no_crs: 0
dev.igb.0.mac_stats.alignment_errs: 0
dev.igb.0.mac_stats.crc_errs: 0
dev.igb.0.mac_stats.recv_errs: 0
dev.igb.0.mac_stats.recv_jabber: 0
dev.igb.0.mac_stats.recv_oversize: 0
dev.igb.0.mac_stats.recv_fragmented: 0
dev.igb.0.mac_stats.recv_undersize: 0
dev.igb.0.mac_stats.recv_no_buff: 0
dev.igb.0.mac_stats.recv_length_errors: 0
dev.igb.0.mac_stats.missed_packets: 0
dev.igb.0.mac_stats.defer_count: 0
dev.igb.0.mac_stats.sequence_errors: 0
dev.igb.0.mac_stats.symbol_errors: 0
dev.igb.0.mac_stats.collision_count: 0
dev.igb.0.mac_stats.late_coll: 0
dev.igb.0.mac_stats.multiple_coll: 0
dev.igb.0.mac_stats.single_coll: 0
dev.igb.0.mac_stats.excess_coll: 0
dev.igb.0.queue7.lro_flushed: 0
dev.igb.0.queue7.lro_queued: 0
dev.igb.0.queue7.rx_bytes: 61172
dev.igb.0.queue7.rx_packets: 437
dev.igb.0.queue7.rxd_tail: 436
dev.igb.0.queue7.rxd_head: 437
dev.igb.0.queue7.tx_packets: 0
dev.igb.0.queue7.no_desc_avail: 0
dev.igb.0.queue7.txd_tail: 0
dev.igb.0.queue7.txd_head: 0
dev.igb.0.queue7.interrupt_rate: 50000
dev.igb.0.queue6.lro_flushed: 0
dev.igb.0.queue6.lro_queued: 0
dev.igb.0.queue6.rx_bytes: 6689
dev.igb.0.queue6.rx_packets: 78
dev.igb.0.queue6.rxd_tail: 77
dev.igb.0.queue6.rxd_head: 78
dev.igb.0.queue6.tx_packets: 0
dev.igb.0.queue6.no_desc_avail: 0
dev.igb.0.queue6.txd_tail: 0
dev.igb.0.queue6.txd_head: 0
dev.igb.0.queue6.interrupt_rate: 90909
dev.igb.0.queue5.lro_flushed: 0
dev.igb.0.queue5.lro_queued: 0
dev.igb.0.queue5.rx_bytes: 79329
dev.igb.0.queue5.rx_packets: 955
dev.igb.0.queue5.rxd_tail: 954
dev.igb.0.queue5.rxd_head: 955
dev.igb.0.queue5.tx_packets: 0
dev.igb.0.queue5.no_desc_avail: 0
dev.igb.0.queue5.txd_tail: 0
dev.igb.0.queue5.txd_head: 0
dev.igb.0.queue5.interrupt_rate: 90909
dev.igb.0.queue4.lro_flushed: 0
dev.igb.0.queue4.lro_queued: 0
dev.igb.0.queue4.rx_bytes: 2661
dev.igb.0.queue4.rx_packets: 39
dev.igb.0.queue4.rxd_tail: 38
dev.igb.0.queue4.rxd_head: 39
dev.igb.0.queue4.tx_packets: 0
dev.igb.0.queue4.no_desc_avail: 0
dev.igb.0.queue4.txd_tail: 0
dev.igb.0.queue4.txd_head: 0
dev.igb.0.queue4.interrupt_rate: 76923
dev.igb.0.queue3.lro_flushed: 0
dev.igb.0.queue3.lro_queued: 0
dev.igb.0.queue3.rx_bytes: 4055
dev.igb.0.queue3.rx_packets: 60
dev.igb.0.queue3.rxd_tail: 59
dev.igb.0.queue3.rxd_head: 60
dev.igb.0.queue3.tx_packets: 0
dev.igb.0.queue3.no_desc_avail: 0
dev.igb.0.queue3.txd_tail: 0
dev.igb.0.queue3.txd_head: 0
dev.igb.0.queue3.interrupt_rate: 90909
dev.igb.0.queue2.lro_flushed: 0
dev.igb.0.queue2.lro_queued: 0
dev.igb.0.queue2.rx_bytes: 7874
dev.igb.0.queue2.rx_packets: 107
dev.igb.0.queue2.rxd_tail: 106
dev.igb.0.queue2.rxd_head: 107
dev.igb.0.queue2.tx_packets: 0
dev.igb.0.queue2.no_desc_avail: 0
dev.igb.0.queue2.txd_tail: 0
dev.igb.0.queue2.txd_head: 0
dev.igb.0.queue2.interrupt_rate: 90909
dev.igb.0.queue1.lro_flushed: 0
dev.igb.0.queue1.lro_queued: 0
dev.igb.0.queue1.rx_bytes: 7373
dev.igb.0.queue1.rx_packets: 69
dev.igb.0.queue1.rxd_tail: 68
dev.igb.0.queue1.rxd_head: 69
dev.igb.0.queue1.tx_packets: 0
dev.igb.0.queue1.no_desc_avail: 0
dev.igb.0.queue1.txd_tail: 0
dev.igb.0.queue1.txd_head: 0
dev.igb.0.queue1.interrupt_rate: 90909
dev.igb.0.queue0.lro_flushed: 0
dev.igb.0.queue0.lro_queued: 0
dev.igb.0.queue0.rx_bytes: 189284
dev.igb.0.queue0.rx_packets: 3150
dev.igb.0.queue0.rxd_tail: 77
dev.igb.0.queue0.rxd_head: 78
dev.igb.0.queue0.tx_packets: 1413
dev.igb.0.queue0.no_desc_avail: 0
dev.igb.0.queue0.txd_tail: 393
dev.igb.0.queue0.txd_head: 393
dev.igb.0.queue0.interrupt_rate: 100000
dev.igb.0.fc_low_water: 33152
dev.igb.0.fc_high_water: 33168
dev.igb.0.rx_buf_alloc: 0
dev.igb.0.tx_buf_alloc: 0
dev.igb.0.extended_int_mask: 2147484159
dev.igb.0.interrupt_mask: 4
dev.igb.0.rx_control: 67141634
dev.igb.0.device_control: 1478230593
dev.igb.0.watchdog_timeouts: 0
dev.igb.0.rx_overruns: 0
dev.igb.0.tx_dma_fail: 0
dev.igb.0.mbuf_defrag_fail: 0
dev.igb.0.link_irq: 4
dev.igb.0.dropped: 0
dev.igb.0.eee_disabled: 0
dev.igb.0.dmac: 0
dev.igb.0.tx_processing_limit: -1
dev.igb.0.rx_processing_limit: 100
dev.igb.0.fc: 3
dev.igb.0.enable_aim: 1
dev.igb.0.nvm: -1
dev.igb.0.%parent: pci2
dev.igb.0.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1b12 class=0x020000
dev.igb.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 handle=\_SB_.PCI0.RP05.PXSX
dev.igb.0.%driver: igb
dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.%parent:

sysctl -a | grep netmap output -

device	netmap
dev.netmap.ixl_rx_miss_bufs: 0
dev.netmap.ixl_rx_miss: 0
dev.netmap.iflib_rx_miss_bufs: 0
dev.netmap.iflib_rx_miss: 0
dev.netmap.iflib_crcstrip: 1
dev.netmap.bridge_batch: 1024
dev.netmap.default_pipes: 0
dev.netmap.priv_buf_num: 4098
dev.netmap.priv_buf_size: 2048
dev.netmap.buf_curr_num: 163840
dev.netmap.buf_num: 163840
dev.netmap.buf_curr_size: 2048
dev.netmap.buf_size: 2048
dev.netmap.priv_ring_num: 4
dev.netmap.priv_ring_size: 20480
dev.netmap.ring_curr_num: 200
dev.netmap.ring_num: 200
dev.netmap.ring_curr_size: 36864
dev.netmap.ring_size: 36864
dev.netmap.priv_if_num: 1
dev.netmap.priv_if_size: 1024
dev.netmap.if_curr_num: 100
dev.netmap.if_num: 100
dev.netmap.if_curr_size: 1024
dev.netmap.if_size: 1024
dev.netmap.generic_rings: 1
dev.netmap.generic_ringsize: 1024
dev.netmap.generic_mit: 100000
dev.netmap.admode: 0
dev.netmap.fwd: 0
dev.netmap.flags: 0
dev.netmap.adaptive_io: 0
dev.netmap.txsync_retry: 2
dev.netmap.no_pendintr: 1
dev.netmap.mitigate: 1
dev.netmap.no_timestamp: 0
dev.netmap.verbose: 0
dev.netmap.ix_rx_miss_bufs: 0
dev.netmap.ix_rx_miss: 0
dev.netmap.ix_crcstrip: 0

newUser2pfSense

bmeeks...Thank you for the Suricata rules explanation. I did a little further digging, and even though I haven't been able to read all of it yet, I found a post of yours:
https://forum.netgate.com/topic/128480/how-automatic-sid-management-and-user-rule-overrides-work-in-snort-and-suricata

NollipfSense

@boobletins said in Suricata InLine with igb NICs:

@nollipfsense

So here are some initial suggestions. Please keep in mind that I've been working on this for ~1 week (in other words: not long), and I'm not a FreeBSD, pfSense, or Suricata expert.

Start by making a backup of your configuration.

Do these first:
My understanding is that flow control should be off on any netmap interface. You have bi-directional flow control enabled:
dev.igb.0.fc: 3
Disable flow control on all active interfaces using system tunables. Set dev.igb.0.fc=0 (and dev.igb.1.fc=0)

Actively set energy efficient ethernet to disabled:
dev.igb.0.eee_disabled=1

Actively force IPv6_TXCSUM6 off by adding the following to config.xml in a shellcmd tag:
ifconfig igb0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso
(see above in this thread for a link on where/how to do that).

Edit:
To be clear: anywhere I have a command that says "igb0" or "igb.0" you will want to duplicate that for igb1 and any other interface you're running netmap on.

So you will need 2 shellcmd lines in config.xml, and two new system tunables for flow control, etc

Consider changing later:

Set rx processing limit:
dev.igb.0.rx_processing_limit: -1

It looks like your txd and rxd are both set to 1024 currently, I suggest you move those to 4096:
hw.igb.txd=4096
hw.igb.rxd=4096

By changing your txd and rxd we may need to revisit your netmap buf/ring (memory settings).

We may also revisit your interrupt and queue settings.

Boobletins, I will need to revisit later...currently, I am happy with just making adjustments to the buf_size:4096 and disable IPv6...haven't got any alert since and my Internet will be down for a while because of moving.

boobletins

So you're running netmap/IPS mode on igb0 (LAN), igb1 (OPT?), and igb3 (WAN)?

What type of CPU is in the machine (# of cores?, is hyper-threading enabled)? How much RAM?

Are you saturating all 3 active interfaces? Or just 2?

Start by making a backup of your configuration.

First disable flow control (as discussed above):
You have the following on all igb interfaces which means bi-directional flow control is enabled.:

dev.igb.0.fc: 3

Change to fc=0 on all netmap interfaces in system tunables. This will take ethernet flow control out of the picture in favor of higher level flow control (TCP) which is less likely to mess with buffering and clog things up.

Let's look at what generates this particular netmap error:
From http://web.mit.edu/freebsd/head/sys/dev/netmap/netmap.c

/*
 * put a copy of the buffers marked NS_FORWARD into an mbuf chain.
 * Take packets from hwcur to ring->head marked NS_FORWARD (or forced)
 * and pass them up. Drop remaining packets in the unlikely event
 * of an mbuf shortage.
 */
static void
netmap_grab_packets(struct netmap_kring *kring, struct mbq *q, int force)
{
	u_int const lim = kring->nkr_num_slots - 1;
	u_int const head = kring->ring->head;
	u_int n;
	struct netmap_adapter *na = kring->na;

	for (n = kring->nr_hwcur; n != head; n = nm_next(n, lim)) {
		struct mbuf *m;
		struct netmap_slot *slot = &kring->ring->slot[n];

		if ((slot->flags & NS_FORWARD) == 0 && !force)
			continue;
		if (slot->len < 14 || slot->len > NETMAP_BUF_SIZE(na)) {
			RD(5, "bad pkt at %d len %d", n, slot->len);
			continue;
		}
		slot->flags &= ~NS_FORWARD; // XXX needed ?
		/* XXX TODO: adapt to the case of a multisegment packet */
		m = m_devget(NMB(na, slot), slot->len, 0, na->ifp, NULL);

		if (m == NULL)
			break;
		mbq_enqueue(q, m);
	}
}

I'm no C expert, but as I read this code there are 2 ways to generate your error in netmap:

a slot is of size less than 14
a slot is of size greater than the netmap buffer can handle

I don't know what the magic number 14 represents, but let's assume it's some kind of minimum packet size we can't control. If that's the case, then the bad_pkt error is generated from packets that are actually bad.

That's not what you have. The error is telling us the current hwcur value (the first number - the slot number in the ring) and the length or size of the slot (eg #777 with len 2154).

So this is a memory issue. The error would be better off saying something like "dropped a packet because it was too short or too large!" -- but that would be useful to others and is thus verboten ;)

edited: Removed incorrect speculation. Skip to my latest post.

boobletins

This post is deleted!

boobletins

@boobletins said in Suricata InLine with igb NICs:

I guess it depends on what NETMAP_BUF_SIZE(na) is returning. It should be either the available memory for netmap buffers, or the available kernel buffers (for the host adapter).

From: https://github.com/luigirizzo/netmap/blob/master/sys/dev/netmap/netmap_kern.h

#define NETMAP_BUF_SIZE(_na)	((_na)->na_lut.objsize)

...

struct netmap_adapter {
	...

	struct netmap_lut {
		struct lut_entry *lut;
		struct plut_entry *plut;
		uint32_t objtotal;	/* max buffer index */
		uint32_t objsize;	/* buffer size */
	};


	/* memory allocator (opaque)
	 * We also cache a pointer to the lut_entry for translating
	 * buffer addresses, the total number of buffers and the buffer size.
	 */
 	struct netmap_mem_d *nm_mem;
	struct netmap_mem_d *nm_mem_prev;
	struct netmap_lut na_lut;

It's returning netmap adapter buffer size.

Let's see.

Your dev.netmap.buf_size=2048 and the length of the slot it was trying to process were all > 2048 when the error was generated.

That makes a certain kind of sense. Why were the slots larger..

Wait. What's your MTU set to on these interfaces? It has to be > 2048? Check this with 'ifconfig igb0' for each interface.

Some sanity checks when enabling netmap would save people a lot of headaches. If your MTU is 10000 and your dev.netmap.buf_size=2048, then netmap will always choke.

Know that if you set dev.netmap.buf_size to some obscenely high number to cover an equally high MTU, netmap will preallocate all of that memory and sit on it.

newUser2pfSense

boobletins...Presently I'm using Inline IPS Mode and I only have Suricata running on my WAN and that's igb3. I'm using igb0 and igb1 as well for my WLAN and LAN.

CPU:
Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
Current: 4000 MHz, Max: 4001 MHz
8 CPUs: 1 package(s) x 4 core(s) x 2 hardware threads
AES-NI CPU Crypto: Yes (active)

Memory:
64 Gig

System Tunables addition:
Tunable Name Description Value
dev.igb.0.fc disable flow control 0
dev.igb.1.fc disable flow control 0
dev.igb.2.fc disable flow control 0
dev.igb.3.fc disable flow control 0
dev.igb.0.eee_disabled disable energy efficient ethernet 1
dev.igb.1.eee_disabled disable energy efficient ethernet 1
dev.igb.2.eee_disabled disable energy efficient ethernet 1
dev.igb.3.eee_disabled disable energy efficient ethernet 1

config.xml addition (I had to take the beginning < and ending > out to get it to display):
shellcmd>ifconfig igb0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso</shellcmd
shellcmd>ifconfig igb1 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso</shellcmd
shellcmd>ifconfig igb2 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso</shellcmd
shellcmd>ifconfig igb3 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso</shellcmd
shellcmd>ifconfig em0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso</shellcmd

igb0,1,2,3 all have an MTU of 1500 which I believe is default. I haven't set any values for this myself.

boobletins

@newuser2pfsense said in Suricata InLine with igb NICs:

boobletins...Presently I'm using Inline IPS Mode and I only have Suricata running on my WAN and that's igb3. I'm using igb0 and igb1 as well for my WLAN and LAN.
dev.igb.3.fc disable flow control 0

Previously you had dev.igb.3.fc=3 Does the "bad pkt" error persist with dev.igb.3.fc=0?

Just to confirm, could you double check and paste me the full output from

ifconfig igb3

Please paste any additional system turntables you've set via the ui and your full loader.conf.local (minus any sensitive data).

Please then manually double check and paste the output from these commands:

sysctl -a | grep nmbclusters
sysctl -a | grep msi
sysctl -a | grep num_queues
dmesg | grep igb3

Above is not busy work, I'm having you manually confirm because with a few commands I found that when I set them in loader.conf.local they didn't take effect. I needed to put some in the ui system tuneables.

We have more settings to tinker with, I made a bunch of changes before the errors went away, but I'm trying to narrow down the issue before just throwing a bunch of new settings at you. I'm pretty confident we can get this working on your igb since its working on mine with 0 errors for over a week now.

newUser2pfSense

boobletins...I keep getting the following error message from the page when posting the information you requested; frustrating to say the least:

Error
Post content was flagged as spam by Akismet.com

I'll do what I can to get the information in.

The errors do persist:
408.786592 [1071] netmap_grab_packets bad pkt at 186 len 2154
950.583865 [1071] netmap_grab_packets bad pkt at 433 len 2154
530.551894 [1071] netmap_grab_packets bad pkt at 810 len 2147
530.547133 [1071] netmap_grab_packets bad pkt at 807 len 2147
360.440859 [1071] netmap_grab_packets bad pkt at 728 len 2154
764.263927 [1071] netmap_grab_packets bad pkt at 311 len 2154

boobletins

Ok -- I tried to thumbs-up some of your posts hoping that will help with Akismet.

I am interested in those results -- mostly because I think something is putting packets into your hardware buffers that are greater than 2048. They also seem to be consistently in the 2100 range. I can't explain what is doing that or why if your MTU is actually 1500. Maybe there's some kind of overhead with vlan tagging, qos, etc that I'm not aware of.

The why doesn't really matter if all you want is a fix. If you raise the buffer_size of netmap (and the packet sizes stay below those new maximums) then the errors should disappear.

Currently your dev.netmap.buf_size is set to 2048. If you, for example, double that to 4096, then all of the current errors would be covered by the new larger buffer_size in netmap (do this in the ui under system tuneables).

Since I don't understand how you're getting packets that are > 2048 with an MTU of 1500, I can't promise it won't come back with even larger numbers, but that change would cover all of the errors you've pasted so far.

As I say above, you may get additional errors by changing dev.netmap.buf_size -- let me know if that's the case.

For the record: I have an MTU of 1500 and a dev.netmap.buf_size of 1920 is enough to prevent errors.