-
boobletins...Here is the output of the commands you requested.
ifconfig igbX | grep CSUM output -
for igb0: options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM> for igb1: options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM> for igb2 [not used]: options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM> for igb3 [WAN]: options=1000b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,NETMAP> for em0 [not used]: options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>sysctl -a | grep igb output -
device igb hw.igb.tx_process_limit: -1 hw.igb.rx_process_limit: 100 hw.igb.num_queues: 0 hw.igb.header_split: 0 hw.igb.max_interrupt_rate: 8000 hw.igb.enable_msix: 1 hw.igb.enable_aim: 1 hw.igb.txd: 1024 hw.igb.rxd: 1024 dev.igb.3.host.header_redir_missed: 0 dev.igb.3.host.serdes_violation_pkt: 0 dev.igb.3.host.length_errors: 3 dev.igb.3.host.tx_good_bytes: 213941 dev.igb.3.host.rx_good_bytes: 3967552 dev.igb.3.host.breaker_tx_pkt_drop: 0 dev.igb.3.host.tx_good_pkt: 0 dev.igb.3.host.breaker_rx_pkt_drop: 0 dev.igb.3.host.breaker_rx_pkts: 0 dev.igb.3.host.rx_pkt: 3 dev.igb.3.host.host_tx_pkt_discard: 0 dev.igb.3.host.breaker_tx_pkt: 0 dev.igb.3.interrupts.rx_overrun: 0 dev.igb.3.interrupts.rx_desc_min_thresh: 0 dev.igb.3.interrupts.tx_queue_min_thresh: 231 dev.igb.3.interrupts.tx_queue_empty: 2417 dev.igb.3.interrupts.tx_abs_timer: 0 dev.igb.3.interrupts.tx_pkt_timer: 0 dev.igb.3.interrupts.rx_abs_timer: 0 dev.igb.3.interrupts.rx_pkt_timer: 18194 dev.igb.3.interrupts.asserts: 22095 dev.igb.3.mac_stats.tso_ctx_fail: 0 dev.igb.3.mac_stats.tso_txd: 0 dev.igb.3.mac_stats.tx_frames_1024_1522: 3 dev.igb.3.mac_stats.tx_frames_512_1023: 16 dev.igb.3.mac_stats.tx_frames_256_511: 21 dev.igb.3.mac_stats.tx_frames_128_255: 90 dev.igb.3.mac_stats.tx_frames_65_127: 1502 dev.igb.3.mac_stats.tx_frames_64: 785 dev.igb.3.mac_stats.mcast_pkts_txd: 3 dev.igb.3.mac_stats.bcast_pkts_txd: 10 dev.igb.3.mac_stats.good_pkts_txd: 2417 dev.igb.3.mac_stats.total_pkts_txd: 2417 dev.igb.3.mac_stats.total_octets_txd: 213941 dev.igb.3.mac_stats.good_octets_txd: 213941 dev.igb.3.mac_stats.total_octets_recvd: 4000401 dev.igb.3.mac_stats.good_octets_recvd: 3967552 dev.igb.3.mac_stats.rx_frames_1024_1522: 1363 dev.igb.3.mac_stats.rx_frames_512_1023: 339 dev.igb.3.mac_stats.rx_frames_256_511: 1232 dev.igb.3.mac_stats.rx_frames_128_255: 1117 dev.igb.3.mac_stats.rx_frames_65_127: 6557 dev.igb.3.mac_stats.rx_frames_64: 7589 dev.igb.3.mac_stats.mcast_pkts_recvd: 11704 dev.igb.3.mac_stats.bcast_pkts_recvd: 3585 dev.igb.3.mac_stats.good_pkts_recvd: 18197 dev.igb.3.mac_stats.total_pkts_recvd: 18528 dev.igb.3.mac_stats.mgmt_pkts_txd: 0 dev.igb.3.mac_stats.mgmt_pkts_drop: 0 dev.igb.3.mac_stats.mgmt_pkts_recvd: 0 dev.igb.3.mac_stats.unsupported_fc_recvd: 0 dev.igb.3.mac_stats.xoff_txd: 0 dev.igb.3.mac_stats.xoff_recvd: 0 dev.igb.3.mac_stats.xon_txd: 0 dev.igb.3.mac_stats.xon_recvd: 0 dev.igb.3.mac_stats.coll_ext_errs: 0 dev.igb.3.mac_stats.tx_no_crs: 0 dev.igb.3.mac_stats.alignment_errs: 0 dev.igb.3.mac_stats.crc_errs: 0 dev.igb.3.mac_stats.recv_errs: 0 dev.igb.3.mac_stats.recv_jabber: 0 dev.igb.3.mac_stats.recv_oversize: 0 dev.igb.3.mac_stats.recv_fragmented: 0 dev.igb.3.mac_stats.recv_undersize: 0 dev.igb.3.mac_stats.recv_no_buff: 0 dev.igb.3.mac_stats.recv_length_errors: 0 dev.igb.3.mac_stats.missed_packets: 0 dev.igb.3.mac_stats.defer_count: 0 dev.igb.3.mac_stats.sequence_errors: 0 dev.igb.3.mac_stats.symbol_errors: 0 dev.igb.3.mac_stats.collision_count: 0 dev.igb.3.mac_stats.late_coll: 0 dev.igb.3.mac_stats.multiple_coll: 0 dev.igb.3.mac_stats.single_coll: 0 dev.igb.3.mac_stats.excess_coll: 0 dev.igb.3.queue7.lro_flushed: 0 dev.igb.3.queue7.lro_queued: 0 dev.igb.3.queue7.rx_bytes: 0 dev.igb.3.queue7.rx_packets: 129 dev.igb.3.queue7.rxd_tail: 561 dev.igb.3.queue7.rxd_head: 562 dev.igb.3.queue7.tx_packets: 0 dev.igb.3.queue7.no_desc_avail: 0 dev.igb.3.queue7.txd_tail: 0 dev.igb.3.queue7.txd_head: 0 dev.igb.3.queue7.interrupt_rate: 8000 dev.igb.3.queue6.lro_flushed: 0 dev.igb.3.queue6.lro_queued: 0 dev.igb.3.queue6.rx_bytes: 0 dev.igb.3.queue6.rx_packets: 30 dev.igb.3.queue6.rxd_tail: 902 dev.igb.3.queue6.rxd_head: 903 dev.igb.3.queue6.tx_packets: 0 dev.igb.3.queue6.no_desc_avail: 0 dev.igb.3.queue6.txd_tail: 0 dev.igb.3.queue6.txd_head: 0 dev.igb.3.queue6.interrupt_rate: 8000 dev.igb.3.queue5.lro_flushed: 0 dev.igb.3.queue5.lro_queued: 0 dev.igb.3.queue5.rx_bytes: 0 dev.igb.3.queue5.rx_packets: 30 dev.igb.3.queue5.rxd_tail: 344 dev.igb.3.queue5.rxd_head: 345 dev.igb.3.queue5.tx_packets: 0 dev.igb.3.queue5.no_desc_avail: 0 dev.igb.3.queue5.txd_tail: 0 dev.igb.3.queue5.txd_head: 0 dev.igb.3.queue5.interrupt_rate: 8000 dev.igb.3.queue4.lro_flushed: 0 dev.igb.3.queue4.lro_queued: 0 dev.igb.3.queue4.rx_bytes: 0 dev.igb.3.queue4.rx_packets: 87 dev.igb.3.queue4.rxd_tail: 524 dev.igb.3.queue4.rxd_head: 525 dev.igb.3.queue4.tx_packets: 0 dev.igb.3.queue4.no_desc_avail: 0 dev.igb.3.queue4.txd_tail: 0 dev.igb.3.queue4.txd_head: 0 dev.igb.3.queue4.interrupt_rate: 8000 dev.igb.3.queue3.lro_flushed: 0 dev.igb.3.queue3.lro_queued: 0 dev.igb.3.queue3.rx_bytes: 0 dev.igb.3.queue3.rx_packets: 29 dev.igb.3.queue3.rxd_tail: 453 dev.igb.3.queue3.rxd_head: 454 dev.igb.3.queue3.tx_packets: 0 dev.igb.3.queue3.no_desc_avail: 0 dev.igb.3.queue3.txd_tail: 0 dev.igb.3.queue3.txd_head: 0 dev.igb.3.queue3.interrupt_rate: 76923 dev.igb.3.queue2.lro_flushed: 0 dev.igb.3.queue2.lro_queued: 0 dev.igb.3.queue2.rx_bytes: 0 dev.igb.3.queue2.rx_packets: 28 dev.igb.3.queue2.rxd_tail: 876 dev.igb.3.queue2.rxd_head: 877 dev.igb.3.queue2.tx_packets: 0 dev.igb.3.queue2.no_desc_avail: 0 dev.igb.3.queue2.txd_tail: 0 dev.igb.3.queue2.txd_head: 0 dev.igb.3.queue2.interrupt_rate: 8000 dev.igb.3.queue1.lro_flushed: 0 dev.igb.3.queue1.lro_queued: 0 dev.igb.3.queue1.rx_bytes: 0 dev.igb.3.queue1.rx_packets: 17 dev.igb.3.queue1.rxd_tail: 482 dev.igb.3.queue1.rxd_head: 483 dev.igb.3.queue1.tx_packets: 0 dev.igb.3.queue1.no_desc_avail: 0 dev.igb.3.queue1.txd_tail: 0 dev.igb.3.queue1.txd_head: 0 dev.igb.3.queue1.interrupt_rate: 8000 dev.igb.3.queue0.lro_flushed: 0 dev.igb.3.queue0.lro_queued: 0 dev.igb.3.queue0.rx_bytes: 0 dev.igb.3.queue0.rx_packets: 142 dev.igb.3.queue0.rxd_tail: 253 dev.igb.3.queue0.rxd_head: 254 dev.igb.3.queue0.tx_packets: 353 dev.igb.3.queue0.no_desc_avail: 0 dev.igb.3.queue0.txd_tail: 18 dev.igb.3.queue0.txd_head: 18 dev.igb.3.queue0.interrupt_rate: 8000 dev.igb.3.fc_low_water: 33152 dev.igb.3.fc_high_water: 33168 dev.igb.3.rx_buf_alloc: 0 dev.igb.3.tx_buf_alloc: 0 dev.igb.3.extended_int_mask: 2147484159 dev.igb.3.interrupt_mask: 4 dev.igb.3.rx_control: 67141658 dev.igb.3.device_control: 1478230593 dev.igb.3.watchdog_timeouts: 0 dev.igb.3.rx_overruns: 0 dev.igb.3.tx_dma_fail: 0 dev.igb.3.mbuf_defrag_fail: 0 dev.igb.3.link_irq: 7 dev.igb.3.dropped: 0 dev.igb.3.eee_disabled: 0 dev.igb.3.dmac: 0 dev.igb.3.tx_processing_limit: -1 dev.igb.3.rx_processing_limit: 100 dev.igb.3.fc: 3 dev.igb.3.enable_aim: 1 dev.igb.3.nvm: -1 dev.igb.3.%parent: pci2 dev.igb.3.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1b12 class=0x020000 dev.igb.3.%location: slot=0 function=3 dbsf=pci0:2:0:3 dev.igb.3.%driver: igb dev.igb.3.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k dev.igb.2.host.header_redir_missed: 0 dev.igb.2.host.serdes_violation_pkt: 0 dev.igb.2.host.length_errors: 0 dev.igb.2.host.tx_good_bytes: 0 dev.igb.2.host.rx_good_bytes: 0 dev.igb.2.host.breaker_tx_pkt_drop: 0 dev.igb.2.host.tx_good_pkt: 0 dev.igb.2.host.breaker_rx_pkt_drop: 0 dev.igb.2.host.breaker_rx_pkts: 0 dev.igb.2.host.rx_pkt: 0 dev.igb.2.host.host_tx_pkt_discard: 0 dev.igb.2.host.breaker_tx_pkt: 0 dev.igb.2.interrupts.rx_overrun: 0 dev.igb.2.interrupts.rx_desc_min_thresh: 0 dev.igb.2.interrupts.tx_queue_min_thresh: 0 dev.igb.2.interrupts.tx_queue_empty: 0 dev.igb.2.interrupts.tx_abs_timer: 0 dev.igb.2.interrupts.tx_pkt_timer: 0 dev.igb.2.interrupts.rx_abs_timer: 0 dev.igb.2.interrupts.rx_pkt_timer: 0 dev.igb.2.interrupts.asserts: 0 dev.igb.2.mac_stats.tso_ctx_fail: 0 dev.igb.2.mac_stats.tso_txd: 0 dev.igb.2.mac_stats.tx_frames_1024_1522: 0 dev.igb.2.mac_stats.tx_frames_512_1023: 0 dev.igb.2.mac_stats.tx_frames_256_511: 0 dev.igb.2.mac_stats.tx_frames_128_255: 0 dev.igb.2.mac_stats.tx_frames_65_127: 0 dev.igb.2.mac_stats.tx_frames_64: 0 dev.igb.2.mac_stats.mcast_pkts_txd: 0 dev.igb.2.mac_stats.bcast_pkts_txd: 0 dev.igb.2.mac_stats.good_pkts_txd: 0 dev.igb.2.mac_stats.total_pkts_txd: 0 dev.igb.2.mac_stats.total_octets_txd: 0 dev.igb.2.mac_stats.good_octets_txd: 0 dev.igb.2.mac_stats.total_octets_recvd: 0 dev.igb.2.mac_stats.good_octets_recvd: 0 dev.igb.2.mac_stats.rx_frames_1024_1522: 0 dev.igb.2.mac_stats.rx_frames_512_1023: 0 dev.igb.2.mac_stats.rx_frames_256_511: 0 dev.igb.2.mac_stats.rx_frames_128_255: 0 dev.igb.2.mac_stats.rx_frames_65_127: 0 dev.igb.2.mac_stats.rx_frames_64: 0 dev.igb.2.mac_stats.mcast_pkts_recvd: 0 dev.igb.2.mac_stats.bcast_pkts_recvd: 0 dev.igb.2.mac_stats.good_pkts_recvd: 0 dev.igb.2.mac_stats.total_pkts_recvd: 0 dev.igb.2.mac_stats.mgmt_pkts_txd: 0 dev.igb.2.mac_stats.mgmt_pkts_drop: 0 dev.igb.2.mac_stats.mgmt_pkts_recvd: 0 dev.igb.2.mac_stats.unsupported_fc_recvd: 0 dev.igb.2.mac_stats.xoff_txd: 0 dev.igb.2.mac_stats.xoff_recvd: 0 dev.igb.2.mac_stats.xon_txd: 0 dev.igb.2.mac_stats.xon_recvd: 0 dev.igb.2.mac_stats.coll_ext_errs: 0 dev.igb.2.mac_stats.tx_no_crs: 0 dev.igb.2.mac_stats.alignment_errs: 0 dev.igb.2.mac_stats.crc_errs: 0 dev.igb.2.mac_stats.recv_errs: 0 dev.igb.2.mac_stats.recv_jabber: 0 dev.igb.2.mac_stats.recv_oversize: 0 dev.igb.2.mac_stats.recv_fragmented: 0 dev.igb.2.mac_stats.recv_undersize: 0 dev.igb.2.mac_stats.recv_no_buff: 0 dev.igb.2.mac_stats.recv_length_errors: 0 dev.igb.2.mac_stats.missed_packets: 0 dev.igb.2.mac_stats.defer_count: 0 dev.igb.2.mac_stats.sequence_errors: 0 dev.igb.2.mac_stats.symbol_errors: 0 dev.igb.2.mac_stats.collision_count: 0 dev.igb.2.mac_stats.late_coll: 0 dev.igb.2.mac_stats.multiple_coll: 0 dev.igb.2.mac_stats.single_coll: 0 dev.igb.2.mac_stats.excess_coll: 0 dev.igb.2.queue7.lro_flushed: 0 dev.igb.2.queue7.lro_queued: 0 dev.igb.2.queue7.rx_bytes: 0 dev.igb.2.queue7.rx_packets: 0 dev.igb.2.queue7.rxd_tail: 0 dev.igb.2.queue7.rxd_head: 0 dev.igb.2.queue7.tx_packets: 0 dev.igb.2.queue7.no_desc_avail: 0 dev.igb.2.queue7.txd_tail: 0 dev.igb.2.queue7.txd_head: 0 dev.igb.2.queue7.interrupt_rate: 0 dev.igb.2.queue6.lro_flushed: 0 dev.igb.2.queue6.lro_queued: 0 dev.igb.2.queue6.rx_bytes: 0 dev.igb.2.queue6.rx_packets: 0 dev.igb.2.queue6.rxd_tail: 0 dev.igb.2.queue6.rxd_head: 0 dev.igb.2.queue6.tx_packets: 0 dev.igb.2.queue6.no_desc_avail: 0 dev.igb.2.queue6.txd_tail: 0 dev.igb.2.queue6.txd_head: 0 dev.igb.2.queue6.interrupt_rate: 0 dev.igb.2.queue5.lro_flushed: 0 dev.igb.2.queue5.lro_queued: 0 dev.igb.2.queue5.rx_bytes: 0 dev.igb.2.queue5.rx_packets: 0 dev.igb.2.queue5.rxd_tail: 0 dev.igb.2.queue5.rxd_head: 0 dev.igb.2.queue5.tx_packets: 0 dev.igb.2.queue5.no_desc_avail: 0 dev.igb.2.queue5.txd_tail: 0 dev.igb.2.queue5.txd_head: 0 dev.igb.2.queue5.interrupt_rate: 0 dev.igb.2.queue4.lro_flushed: 0 dev.igb.2.queue4.lro_queued: 0 dev.igb.2.queue4.rx_bytes: 0 dev.igb.2.queue4.rx_packets: 0 dev.igb.2.queue4.rxd_tail: 0 dev.igb.2.queue4.rxd_head: 0 dev.igb.2.queue4.tx_packets: 0 dev.igb.2.queue4.no_desc_avail: 0 dev.igb.2.queue4.txd_tail: 0 dev.igb.2.queue4.txd_head: 0 dev.igb.2.queue4.interrupt_rate: 0 dev.igb.2.queue3.lro_flushed: 0 dev.igb.2.queue3.lro_queued: 0 dev.igb.2.queue3.rx_bytes: 0 dev.igb.2.queue3.rx_packets: 0 dev.igb.2.queue3.rxd_tail: 0 dev.igb.2.queue3.rxd_head: 0 dev.igb.2.queue3.tx_packets: 0 dev.igb.2.queue3.no_desc_avail: 0 dev.igb.2.queue3.txd_tail: 0 dev.igb.2.queue3.txd_head: 0 dev.igb.2.queue3.interrupt_rate: 0 dev.igb.2.queue2.lro_flushed: 0 dev.igb.2.queue2.lro_queued: 0 dev.igb.2.queue2.rx_bytes: 0 dev.igb.2.queue2.rx_packets: 0 dev.igb.2.queue2.rxd_tail: 0 dev.igb.2.queue2.rxd_head: 0 dev.igb.2.queue2.tx_packets: 0 dev.igb.2.queue2.no_desc_avail: 0 dev.igb.2.queue2.txd_tail: 0 dev.igb.2.queue2.txd_head: 0 dev.igb.2.queue2.interrupt_rate: 0 dev.igb.2.queue1.lro_flushed: 0 dev.igb.2.queue1.lro_queued: 0 dev.igb.2.queue1.rx_bytes: 0 dev.igb.2.queue1.rx_packets: 0 dev.igb.2.queue1.rxd_tail: 0 dev.igb.2.queue1.rxd_head: 0 dev.igb.2.queue1.tx_packets: 0 dev.igb.2.queue1.no_desc_avail: 0 dev.igb.2.queue1.txd_tail: 0 dev.igb.2.queue1.txd_head: 0 dev.igb.2.queue1.interrupt_rate: 0 dev.igb.2.queue0.lro_flushed: 0 dev.igb.2.queue0.lro_queued: 0 dev.igb.2.queue0.rx_bytes: 0 dev.igb.2.queue0.rx_packets: 0 dev.igb.2.queue0.rxd_tail: 0 dev.igb.2.queue0.rxd_head: 0 dev.igb.2.queue0.tx_packets: 0 dev.igb.2.queue0.no_desc_avail: 0 dev.igb.2.queue0.txd_tail: 0 dev.igb.2.queue0.txd_head: 0 dev.igb.2.queue0.interrupt_rate: 0 dev.igb.2.fc_low_water: 33152 dev.igb.2.fc_high_water: 33168 dev.igb.2.rx_buf_alloc: 0 dev.igb.2.tx_buf_alloc: 0 dev.igb.2.extended_int_mask: 2147483648 dev.igb.2.interrupt_mask: 0 dev.igb.2.rx_control: 0 dev.igb.2.device_control: 136053313 dev.igb.2.watchdog_timeouts: 0 dev.igb.2.rx_overruns: 0 dev.igb.2.tx_dma_fail: 0 dev.igb.2.mbuf_defrag_fail: 0 dev.igb.2.link_irq: 0 dev.igb.2.dropped: 0 dev.igb.2.eee_disabled: 0 dev.igb.2.dmac: 0 dev.igb.2.tx_processing_limit: -1 dev.igb.2.rx_processing_limit: 100 dev.igb.2.fc: 3 dev.igb.2.enable_aim: 1 dev.igb.2.nvm: -1 dev.igb.2.%parent: pci2 dev.igb.2.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1b12 class=0x020000 dev.igb.2.%location: slot=0 function=2 dbsf=pci0:2:0:2 dev.igb.2.%driver: igb dev.igb.2.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k dev.igb.1.host.header_redir_missed: 0 dev.igb.1.host.serdes_violation_pkt: 0 dev.igb.1.host.length_errors: 0 dev.igb.1.host.tx_good_bytes: 387524 dev.igb.1.host.rx_good_bytes: 58810 dev.igb.1.host.breaker_tx_pkt_drop: 0 dev.igb.1.host.tx_good_pkt: 1 dev.igb.1.host.breaker_rx_pkt_drop: 0 dev.igb.1.host.breaker_rx_pkts: 0 dev.igb.1.host.rx_pkt: 2 dev.igb.1.host.host_tx_pkt_discard: 0 dev.igb.1.host.breaker_tx_pkt: 0 dev.igb.1.interrupts.rx_overrun: 0 dev.igb.1.interrupts.rx_desc_min_thresh: 0 dev.igb.1.interrupts.tx_queue_min_thresh: 0 dev.igb.1.interrupts.tx_queue_empty: 584 dev.igb.1.interrupts.tx_abs_timer: 0 dev.igb.1.interrupts.tx_pkt_timer: 0 dev.igb.1.interrupts.rx_abs_timer: 0 dev.igb.1.interrupts.rx_pkt_timer: 381 dev.igb.1.interrupts.asserts: 3808 dev.igb.1.mac_stats.tso_ctx_fail: 0 dev.igb.1.mac_stats.tso_txd: 0 dev.igb.1.mac_stats.tx_frames_1024_1522: 213 dev.igb.1.mac_stats.tx_frames_512_1023: 25 dev.igb.1.mac_stats.tx_frames_256_511: 87 dev.igb.1.mac_stats.tx_frames_128_255: 9 dev.igb.1.mac_stats.tx_frames_65_127: 239 dev.igb.1.mac_stats.tx_frames_64: 12 dev.igb.1.mac_stats.mcast_pkts_txd: 7 dev.igb.1.mac_stats.bcast_pkts_txd: 5 dev.igb.1.mac_stats.good_pkts_txd: 585 dev.igb.1.mac_stats.total_pkts_txd: 585 dev.igb.1.mac_stats.total_octets_txd: 387524 dev.igb.1.mac_stats.good_octets_txd: 387524 dev.igb.1.mac_stats.total_octets_recvd: 70094 dev.igb.1.mac_stats.good_octets_recvd: 58810 dev.igb.1.mac_stats.rx_frames_1024_1522: 5 dev.igb.1.mac_stats.rx_frames_512_1023: 8 dev.igb.1.mac_stats.rx_frames_256_511: 17 dev.igb.1.mac_stats.rx_frames_128_255: 124 dev.igb.1.mac_stats.rx_frames_65_127: 217 dev.igb.1.mac_stats.rx_frames_64: 12 dev.igb.1.mac_stats.mcast_pkts_recvd: 20 dev.igb.1.mac_stats.bcast_pkts_recvd: 12 dev.igb.1.mac_stats.good_pkts_recvd: 383 dev.igb.1.mac_stats.total_pkts_recvd: 411 dev.igb.1.mac_stats.mgmt_pkts_txd: 0 dev.igb.1.mac_stats.mgmt_pkts_drop: 0 dev.igb.1.mac_stats.mgmt_pkts_recvd: 0 dev.igb.1.mac_stats.unsupported_fc_recvd: 0 dev.igb.1.mac_stats.xoff_txd: 0 dev.igb.1.mac_stats.xoff_recvd: 0 dev.igb.1.mac_stats.xon_txd: 0 dev.igb.1.mac_stats.xon_recvd: 0 dev.igb.1.mac_stats.coll_ext_errs: 0 dev.igb.1.mac_stats.tx_no_crs: 0 dev.igb.1.mac_stats.alignment_errs: 0 dev.igb.1.mac_stats.crc_errs: 0 dev.igb.1.mac_stats.recv_errs: 0 dev.igb.1.mac_stats.recv_jabber: 0 dev.igb.1.mac_stats.recv_oversize: 0 dev.igb.1.mac_stats.recv_fragmented: 0 dev.igb.1.mac_stats.recv_undersize: 0 dev.igb.1.mac_stats.recv_no_buff: 0 dev.igb.1.mac_stats.recv_length_errors: 0 dev.igb.1.mac_stats.missed_packets: 0 dev.igb.1.mac_stats.defer_count: 0 dev.igb.1.mac_stats.sequence_errors: 0 dev.igb.1.mac_stats.symbol_errors: 0 dev.igb.1.mac_stats.collision_count: 0 dev.igb.1.mac_stats.late_coll: 0 dev.igb.1.mac_stats.multiple_coll: 0 dev.igb.1.mac_stats.single_coll: 0 dev.igb.1.mac_stats.excess_coll: 0 dev.igb.1.queue7.lro_flushed: 0 dev.igb.1.queue7.lro_queued: 0 dev.igb.1.queue7.rx_bytes: 22234 dev.igb.1.queue7.rx_packets: 154 dev.igb.1.queue7.rxd_tail: 153 dev.igb.1.queue7.rxd_head: 154 dev.igb.1.queue7.tx_packets: 0 dev.igb.1.queue7.no_desc_avail: 0 dev.igb.1.queue7.txd_tail: 0 dev.igb.1.queue7.txd_head: 0 dev.igb.1.queue7.interrupt_rate: 62500 dev.igb.1.queue6.lro_flushed: 0 dev.igb.1.queue6.lro_queued: 0 dev.igb.1.queue6.rx_bytes: 4146 dev.igb.1.queue6.rx_packets: 22 dev.igb.1.queue6.rxd_tail: 21 dev.igb.1.queue6.rxd_head: 22 dev.igb.1.queue6.tx_packets: 0 dev.igb.1.queue6.no_desc_avail: 0 dev.igb.1.queue6.txd_tail: 0 dev.igb.1.queue6.txd_head: 0 dev.igb.1.queue6.interrupt_rate: 76923 dev.igb.1.queue5.lro_flushed: 0 dev.igb.1.queue5.lro_queued: 0 dev.igb.1.queue5.rx_bytes: 120 dev.igb.1.queue5.rx_packets: 2 dev.igb.1.queue5.rxd_tail: 1 dev.igb.1.queue5.rxd_head: 2 dev.igb.1.queue5.tx_packets: 0 dev.igb.1.queue5.no_desc_avail: 0 dev.igb.1.queue5.txd_tail: 0 dev.igb.1.queue5.txd_head: 0 dev.igb.1.queue5.interrupt_rate: 100000 dev.igb.1.queue4.lro_flushed: 0 dev.igb.1.queue4.lro_queued: 0 dev.igb.1.queue4.rx_bytes: 254 dev.igb.1.queue4.rx_packets: 4 dev.igb.1.queue4.rxd_tail: 3 dev.igb.1.queue4.rxd_head: 4 dev.igb.1.queue4.tx_packets: 0 dev.igb.1.queue4.no_desc_avail: 0 dev.igb.1.queue4.txd_tail: 0 dev.igb.1.queue4.txd_head: 0 dev.igb.1.queue4.interrupt_rate: 100000 dev.igb.1.queue3.lro_flushed: 0 dev.igb.1.queue3.lro_queued: 0 dev.igb.1.queue3.rx_bytes: 24026 dev.igb.1.queue3.rx_packets: 144 dev.igb.1.queue3.rxd_tail: 143 dev.igb.1.queue3.rxd_head: 144 dev.igb.1.queue3.tx_packets: 0 dev.igb.1.queue3.no_desc_avail: 0 dev.igb.1.queue3.txd_tail: 0 dev.igb.1.queue3.txd_head: 0 dev.igb.1.queue3.interrupt_rate: 6024 dev.igb.1.queue2.lro_flushed: 0 dev.igb.1.queue2.lro_queued: 0 dev.igb.1.queue2.rx_bytes: 3022 dev.igb.1.queue2.rx_packets: 14 dev.igb.1.queue2.rxd_tail: 13 dev.igb.1.queue2.rxd_head: 14 dev.igb.1.queue2.tx_packets: 0 dev.igb.1.queue2.no_desc_avail: 0 dev.igb.1.queue2.txd_tail: 0 dev.igb.1.queue2.txd_head: 0 dev.igb.1.queue2.interrupt_rate: 29411 dev.igb.1.queue1.lro_flushed: 0 dev.igb.1.queue1.lro_queued: 0 dev.igb.1.queue1.rx_bytes: 874 dev.igb.1.queue1.rx_packets: 7 dev.igb.1.queue1.rxd_tail: 6 dev.igb.1.queue1.rxd_head: 7 dev.igb.1.queue1.tx_packets: 0 dev.igb.1.queue1.no_desc_avail: 0 dev.igb.1.queue1.txd_tail: 0 dev.igb.1.queue1.txd_head: 0 dev.igb.1.queue1.interrupt_rate: 66666 dev.igb.1.queue0.lro_flushed: 0 dev.igb.1.queue0.lro_queued: 0 dev.igb.1.queue0.rx_bytes: 4193 dev.igb.1.queue0.rx_packets: 39 dev.igb.1.queue0.rxd_tail: 38 dev.igb.1.queue0.rxd_head: 39 dev.igb.1.queue0.tx_packets: 588 dev.igb.1.queue0.no_desc_avail: 0 dev.igb.1.queue0.txd_tail: 911 dev.igb.1.queue0.txd_head: 911 dev.igb.1.queue0.interrupt_rate: 90909 dev.igb.1.fc_low_water: 33152 dev.igb.1.fc_high_water: 33168 dev.igb.1.rx_buf_alloc: 0 dev.igb.1.tx_buf_alloc: 0 dev.igb.1.extended_int_mask: 2147484159 dev.igb.1.interrupt_mask: 4 dev.igb.1.rx_control: 67141634 dev.igb.1.device_control: 1478230593 dev.igb.1.watchdog_timeouts: 0 dev.igb.1.rx_overruns: 0 dev.igb.1.tx_dma_fail: 0 dev.igb.1.mbuf_defrag_fail: 0 dev.igb.1.link_irq: 4 dev.igb.1.dropped: 0 dev.igb.1.eee_disabled: 0 dev.igb.1.dmac: 0 dev.igb.1.tx_processing_limit: -1 dev.igb.1.rx_processing_limit: 100 dev.igb.1.fc: 3 dev.igb.1.enable_aim: 1 dev.igb.1.nvm: -1 dev.igb.1.%parent: pci2 dev.igb.1.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1b12 class=0x020000 dev.igb.1.%location: slot=0 function=1 dbsf=pci0:2:0:1 dev.igb.1.%driver: igb dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k dev.igb.0.wake: 0 dev.igb.0.host.header_redir_missed: 0 dev.igb.0.host.serdes_violation_pkt: 0 dev.igb.0.host.length_errors: 0 dev.igb.0.host.tx_good_bytes: 1710353 dev.igb.0.host.rx_good_bytes: 377365 dev.igb.0.host.breaker_tx_pkt_drop: 0 dev.igb.0.host.tx_good_pkt: 0 dev.igb.0.host.breaker_rx_pkt_drop: 0 dev.igb.0.host.breaker_rx_pkts: 0 dev.igb.0.host.rx_pkt: 0 dev.igb.0.host.host_tx_pkt_discard: 0 dev.igb.0.host.breaker_tx_pkt: 0 dev.igb.0.interrupts.rx_overrun: 0 dev.igb.0.interrupts.rx_desc_min_thresh: 0 dev.igb.0.interrupts.tx_queue_min_thresh: 23214 dev.igb.0.interrupts.tx_queue_empty: 1413 dev.igb.0.interrupts.tx_abs_timer: 0 dev.igb.0.interrupts.tx_pkt_timer: 0 dev.igb.0.interrupts.rx_abs_timer: 0 dev.igb.0.interrupts.rx_pkt_timer: 4886 dev.igb.0.interrupts.asserts: 8193 dev.igb.0.mac_stats.tso_ctx_fail: 0 dev.igb.0.mac_stats.tso_txd: 0 dev.igb.0.mac_stats.tx_frames_1024_1522: 1093 dev.igb.0.mac_stats.tx_frames_512_1023: 30 dev.igb.0.mac_stats.tx_frames_256_511: 30 dev.igb.0.mac_stats.tx_frames_128_255: 10 dev.igb.0.mac_stats.tx_frames_65_127: 42 dev.igb.0.mac_stats.tx_frames_64: 208 dev.igb.0.mac_stats.mcast_pkts_txd: 1 dev.igb.0.mac_stats.bcast_pkts_txd: 10 dev.igb.0.mac_stats.good_pkts_txd: 1413 dev.igb.0.mac_stats.total_pkts_txd: 1413 dev.igb.0.mac_stats.total_octets_txd: 1710353 dev.igb.0.mac_stats.good_octets_txd: 1710353 dev.igb.0.mac_stats.total_octets_recvd: 384601 dev.igb.0.mac_stats.good_octets_recvd: 377365 dev.igb.0.mac_stats.rx_frames_1024_1522: 3 dev.igb.0.mac_stats.rx_frames_512_1023: 1 dev.igb.0.mac_stats.rx_frames_256_511: 22 dev.igb.0.mac_stats.rx_frames_128_255: 385 dev.igb.0.mac_stats.rx_frames_65_127: 1331 dev.igb.0.mac_stats.rx_frames_64: 3144 dev.igb.0.mac_stats.mcast_pkts_recvd: 412 dev.igb.0.mac_stats.bcast_pkts_recvd: 2902 dev.igb.0.mac_stats.good_pkts_recvd: 4886 dev.igb.0.mac_stats.total_pkts_recvd: 4902 dev.igb.0.mac_stats.mgmt_pkts_txd: 0 dev.igb.0.mac_stats.mgmt_pkts_drop: 0 dev.igb.0.mac_stats.mgmt_pkts_recvd: 0 dev.igb.0.mac_stats.unsupported_fc_recvd: 0 dev.igb.0.mac_stats.xoff_txd: 0 dev.igb.0.mac_stats.xoff_recvd: 0 dev.igb.0.mac_stats.xon_txd: 0 dev.igb.0.mac_stats.xon_recvd: 0 dev.igb.0.mac_stats.coll_ext_errs: 0 dev.igb.0.mac_stats.tx_no_crs: 0 dev.igb.0.mac_stats.alignment_errs: 0 dev.igb.0.mac_stats.crc_errs: 0 dev.igb.0.mac_stats.recv_errs: 0 dev.igb.0.mac_stats.recv_jabber: 0 dev.igb.0.mac_stats.recv_oversize: 0 dev.igb.0.mac_stats.recv_fragmented: 0 dev.igb.0.mac_stats.recv_undersize: 0 dev.igb.0.mac_stats.recv_no_buff: 0 dev.igb.0.mac_stats.recv_length_errors: 0 dev.igb.0.mac_stats.missed_packets: 0 dev.igb.0.mac_stats.defer_count: 0 dev.igb.0.mac_stats.sequence_errors: 0 dev.igb.0.mac_stats.symbol_errors: 0 dev.igb.0.mac_stats.collision_count: 0 dev.igb.0.mac_stats.late_coll: 0 dev.igb.0.mac_stats.multiple_coll: 0 dev.igb.0.mac_stats.single_coll: 0 dev.igb.0.mac_stats.excess_coll: 0 dev.igb.0.queue7.lro_flushed: 0 dev.igb.0.queue7.lro_queued: 0 dev.igb.0.queue7.rx_bytes: 61172 dev.igb.0.queue7.rx_packets: 437 dev.igb.0.queue7.rxd_tail: 436 dev.igb.0.queue7.rxd_head: 437 dev.igb.0.queue7.tx_packets: 0 dev.igb.0.queue7.no_desc_avail: 0 dev.igb.0.queue7.txd_tail: 0 dev.igb.0.queue7.txd_head: 0 dev.igb.0.queue7.interrupt_rate: 50000 dev.igb.0.queue6.lro_flushed: 0 dev.igb.0.queue6.lro_queued: 0 dev.igb.0.queue6.rx_bytes: 6689 dev.igb.0.queue6.rx_packets: 78 dev.igb.0.queue6.rxd_tail: 77 dev.igb.0.queue6.rxd_head: 78 dev.igb.0.queue6.tx_packets: 0 dev.igb.0.queue6.no_desc_avail: 0 dev.igb.0.queue6.txd_tail: 0 dev.igb.0.queue6.txd_head: 0 dev.igb.0.queue6.interrupt_rate: 90909 dev.igb.0.queue5.lro_flushed: 0 dev.igb.0.queue5.lro_queued: 0 dev.igb.0.queue5.rx_bytes: 79329 dev.igb.0.queue5.rx_packets: 955 dev.igb.0.queue5.rxd_tail: 954 dev.igb.0.queue5.rxd_head: 955 dev.igb.0.queue5.tx_packets: 0 dev.igb.0.queue5.no_desc_avail: 0 dev.igb.0.queue5.txd_tail: 0 dev.igb.0.queue5.txd_head: 0 dev.igb.0.queue5.interrupt_rate: 90909 dev.igb.0.queue4.lro_flushed: 0 dev.igb.0.queue4.lro_queued: 0 dev.igb.0.queue4.rx_bytes: 2661 dev.igb.0.queue4.rx_packets: 39 dev.igb.0.queue4.rxd_tail: 38 dev.igb.0.queue4.rxd_head: 39 dev.igb.0.queue4.tx_packets: 0 dev.igb.0.queue4.no_desc_avail: 0 dev.igb.0.queue4.txd_tail: 0 dev.igb.0.queue4.txd_head: 0 dev.igb.0.queue4.interrupt_rate: 76923 dev.igb.0.queue3.lro_flushed: 0 dev.igb.0.queue3.lro_queued: 0 dev.igb.0.queue3.rx_bytes: 4055 dev.igb.0.queue3.rx_packets: 60 dev.igb.0.queue3.rxd_tail: 59 dev.igb.0.queue3.rxd_head: 60 dev.igb.0.queue3.tx_packets: 0 dev.igb.0.queue3.no_desc_avail: 0 dev.igb.0.queue3.txd_tail: 0 dev.igb.0.queue3.txd_head: 0 dev.igb.0.queue3.interrupt_rate: 90909 dev.igb.0.queue2.lro_flushed: 0 dev.igb.0.queue2.lro_queued: 0 dev.igb.0.queue2.rx_bytes: 7874 dev.igb.0.queue2.rx_packets: 107 dev.igb.0.queue2.rxd_tail: 106 dev.igb.0.queue2.rxd_head: 107 dev.igb.0.queue2.tx_packets: 0 dev.igb.0.queue2.no_desc_avail: 0 dev.igb.0.queue2.txd_tail: 0 dev.igb.0.queue2.txd_head: 0 dev.igb.0.queue2.interrupt_rate: 90909 dev.igb.0.queue1.lro_flushed: 0 dev.igb.0.queue1.lro_queued: 0 dev.igb.0.queue1.rx_bytes: 7373 dev.igb.0.queue1.rx_packets: 69 dev.igb.0.queue1.rxd_tail: 68 dev.igb.0.queue1.rxd_head: 69 dev.igb.0.queue1.tx_packets: 0 dev.igb.0.queue1.no_desc_avail: 0 dev.igb.0.queue1.txd_tail: 0 dev.igb.0.queue1.txd_head: 0 dev.igb.0.queue1.interrupt_rate: 90909 dev.igb.0.queue0.lro_flushed: 0 dev.igb.0.queue0.lro_queued: 0 dev.igb.0.queue0.rx_bytes: 189284 dev.igb.0.queue0.rx_packets: 3150 dev.igb.0.queue0.rxd_tail: 77 dev.igb.0.queue0.rxd_head: 78 dev.igb.0.queue0.tx_packets: 1413 dev.igb.0.queue0.no_desc_avail: 0 dev.igb.0.queue0.txd_tail: 393 dev.igb.0.queue0.txd_head: 393 dev.igb.0.queue0.interrupt_rate: 100000 dev.igb.0.fc_low_water: 33152 dev.igb.0.fc_high_water: 33168 dev.igb.0.rx_buf_alloc: 0 dev.igb.0.tx_buf_alloc: 0 dev.igb.0.extended_int_mask: 2147484159 dev.igb.0.interrupt_mask: 4 dev.igb.0.rx_control: 67141634 dev.igb.0.device_control: 1478230593 dev.igb.0.watchdog_timeouts: 0 dev.igb.0.rx_overruns: 0 dev.igb.0.tx_dma_fail: 0 dev.igb.0.mbuf_defrag_fail: 0 dev.igb.0.link_irq: 4 dev.igb.0.dropped: 0 dev.igb.0.eee_disabled: 0 dev.igb.0.dmac: 0 dev.igb.0.tx_processing_limit: -1 dev.igb.0.rx_processing_limit: 100 dev.igb.0.fc: 3 dev.igb.0.enable_aim: 1 dev.igb.0.nvm: -1 dev.igb.0.%parent: pci2 dev.igb.0.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1b12 class=0x020000 dev.igb.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 handle=\_SB_.PCI0.RP05.PXSX dev.igb.0.%driver: igb dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k dev.igb.%parent:sysctl -a | grep netmap output -
device netmap dev.netmap.ixl_rx_miss_bufs: 0 dev.netmap.ixl_rx_miss: 0 dev.netmap.iflib_rx_miss_bufs: 0 dev.netmap.iflib_rx_miss: 0 dev.netmap.iflib_crcstrip: 1 dev.netmap.bridge_batch: 1024 dev.netmap.default_pipes: 0 dev.netmap.priv_buf_num: 4098 dev.netmap.priv_buf_size: 2048 dev.netmap.buf_curr_num: 163840 dev.netmap.buf_num: 163840 dev.netmap.buf_curr_size: 2048 dev.netmap.buf_size: 2048 dev.netmap.priv_ring_num: 4 dev.netmap.priv_ring_size: 20480 dev.netmap.ring_curr_num: 200 dev.netmap.ring_num: 200 dev.netmap.ring_curr_size: 36864 dev.netmap.ring_size: 36864 dev.netmap.priv_if_num: 1 dev.netmap.priv_if_size: 1024 dev.netmap.if_curr_num: 100 dev.netmap.if_num: 100 dev.netmap.if_curr_size: 1024 dev.netmap.if_size: 1024 dev.netmap.generic_rings: 1 dev.netmap.generic_ringsize: 1024 dev.netmap.generic_mit: 100000 dev.netmap.admode: 0 dev.netmap.fwd: 0 dev.netmap.flags: 0 dev.netmap.adaptive_io: 0 dev.netmap.txsync_retry: 2 dev.netmap.no_pendintr: 1 dev.netmap.mitigate: 1 dev.netmap.no_timestamp: 0 dev.netmap.verbose: 0 dev.netmap.ix_rx_miss_bufs: 0 dev.netmap.ix_rx_miss: 0 dev.netmap.ix_crcstrip: 0 -
bmeeks...Thank you for the Suricata rules explanation. I did a little further digging, and even though I haven't been able to read all of it yet, I found a post of yours:
https://forum.netgate.com/topic/128480/how-automatic-sid-management-and-user-rule-overrides-work-in-snort-and-suricata -
@boobletins said in Suricata InLine with igb NICs:
So here are some initial suggestions. Please keep in mind that I've been working on this for ~1 week (in other words: not long), and I'm not a FreeBSD, pfSense, or Suricata expert.
Start by making a backup of your configuration.
Do these first:
My understanding is that flow control should be off on any netmap interface. You have bi-directional flow control enabled:dev.igb.0.fc: 3Disable flow control on all active interfaces using system tunables. Set dev.igb.0.fc=0 (and dev.igb.1.fc=0)
Actively set energy efficient ethernet to disabled:
dev.igb.0.eee_disabled=1Actively force IPv6_TXCSUM6 off by adding the following to config.xml in a shellcmd tag:
ifconfig igb0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso(see above in this thread for a link on where/how to do that).
Edit:
To be clear: anywhere I have a command that says "igb0" or "igb.0" you will want to duplicate that for igb1 and any other interface you're running netmap on.So you will need 2 shellcmd lines in config.xml, and two new system tunables for flow control, etc
Consider changing later:
Set rx processing limit:
dev.igb.0.rx_processing_limit: -1It looks like your txd and rxd are both set to 1024 currently, I suggest you move those to 4096:
hw.igb.txd=4096
hw.igb.rxd=4096By changing your txd and rxd we may need to revisit your netmap buf/ring (memory settings).
We may also revisit your interrupt and queue settings.
Boobletins, I will need to revisit later...currently, I am happy with just making adjustments to the buf_size:4096 and disable IPv6...haven't got any alert since and my Internet will be down for a while because of moving.
-
So you're running netmap/IPS mode on igb0 (LAN), igb1 (OPT?), and igb3 (WAN)?
What type of CPU is in the machine (# of cores?, is hyper-threading enabled)? How much RAM?
Are you saturating all 3 active interfaces? Or just 2?
Start by making a backup of your configuration.
First disable flow control (as discussed above):
You have the following on all igb interfaces which means bi-directional flow control is enabled.:dev.igb.0.fc: 3Change to fc=0 on all netmap interfaces in system tunables. This will take ethernet flow control out of the picture in favor of higher level flow control (TCP) which is less likely to mess with buffering and clog things up.
Let's look at what generates this particular netmap error:
From http://web.mit.edu/freebsd/head/sys/dev/netmap/netmap.c/* * put a copy of the buffers marked NS_FORWARD into an mbuf chain. * Take packets from hwcur to ring->head marked NS_FORWARD (or forced) * and pass them up. Drop remaining packets in the unlikely event * of an mbuf shortage. */ static void netmap_grab_packets(struct netmap_kring *kring, struct mbq *q, int force) { u_int const lim = kring->nkr_num_slots - 1; u_int const head = kring->ring->head; u_int n; struct netmap_adapter *na = kring->na; for (n = kring->nr_hwcur; n != head; n = nm_next(n, lim)) { struct mbuf *m; struct netmap_slot *slot = &kring->ring->slot[n]; if ((slot->flags & NS_FORWARD) == 0 && !force) continue; if (slot->len < 14 || slot->len > NETMAP_BUF_SIZE(na)) { RD(5, "bad pkt at %d len %d", n, slot->len); continue; } slot->flags &= ~NS_FORWARD; // XXX needed ? /* XXX TODO: adapt to the case of a multisegment packet */ m = m_devget(NMB(na, slot), slot->len, 0, na->ifp, NULL); if (m == NULL) break; mbq_enqueue(q, m); } }I'm no C expert, but as I read this code there are 2 ways to generate your error in netmap:
- a slot is of size less than 14
- a slot is of size greater than the netmap buffer can handle
I don't know what the magic number 14 represents, but let's assume it's some kind of minimum packet size we can't control. If that's the case, then the bad_pkt error is generated from packets that are actually bad.
That's not what you have. The error is telling us the current hwcur value (the first number - the slot number in the ring) and the length or size of the slot (eg #777 with len 2154).
So this is a memory issue. The error would be better off saying something like "dropped a packet because it was too short or too large!" -- but that would be useful to others and is thus verboten ;)
edited: Removed incorrect speculation. Skip to my latest post.
-
This post is deleted! -
@boobletins said in Suricata InLine with igb NICs:
I guess it depends on what NETMAP_BUF_SIZE(na) is returning. It should be either the available memory for netmap buffers, or the available kernel buffers (for the host adapter).
From: https://github.com/luigirizzo/netmap/blob/master/sys/dev/netmap/netmap_kern.h
#define NETMAP_BUF_SIZE(_na) ((_na)->na_lut.objsize) ... struct netmap_adapter { ... struct netmap_lut { struct lut_entry *lut; struct plut_entry *plut; uint32_t objtotal; /* max buffer index */ uint32_t objsize; /* buffer size */ }; /* memory allocator (opaque) * We also cache a pointer to the lut_entry for translating * buffer addresses, the total number of buffers and the buffer size. */ struct netmap_mem_d *nm_mem; struct netmap_mem_d *nm_mem_prev; struct netmap_lut na_lut;It's returning netmap adapter buffer size.
Let's see.
Your dev.netmap.buf_size=2048 and the length of the slot it was trying to process were all > 2048 when the error was generated.
That makes a certain kind of sense. Why were the slots larger..
Wait. What's your MTU set to on these interfaces? It has to be > 2048? Check this with 'ifconfig igb0' for each interface.
Some sanity checks when enabling netmap would save people a lot of headaches. If your MTU is 10000 and your dev.netmap.buf_size=2048, then netmap will always choke.
Know that if you set dev.netmap.buf_size to some obscenely high number to cover an equally high MTU, netmap will preallocate all of that memory and sit on it.
-
boobletins...Presently I'm using Inline IPS Mode and I only have Suricata running on my WAN and that's igb3. I'm using igb0 and igb1 as well for my WLAN and LAN.
CPU:
Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
Current: 4000 MHz, Max: 4001 MHz
8 CPUs: 1 package(s) x 4 core(s) x 2 hardware threads
AES-NI CPU Crypto: Yes (active)Memory:
64 GigSystem Tunables addition:
Tunable Name Description Value
dev.igb.0.fc disable flow control 0
dev.igb.1.fc disable flow control 0
dev.igb.2.fc disable flow control 0
dev.igb.3.fc disable flow control 0
dev.igb.0.eee_disabled disable energy efficient ethernet 1
dev.igb.1.eee_disabled disable energy efficient ethernet 1
dev.igb.2.eee_disabled disable energy efficient ethernet 1
dev.igb.3.eee_disabled disable energy efficient ethernet 1config.xml addition (I had to take the beginning < and ending > out to get it to display):
shellcmd>ifconfig igb0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso</shellcmd
shellcmd>ifconfig igb1 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso</shellcmd
shellcmd>ifconfig igb2 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso</shellcmd
shellcmd>ifconfig igb3 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso</shellcmd
shellcmd>ifconfig em0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso</shellcmdigb0,1,2,3 all have an MTU of 1500 which I believe is default. I haven't set any values for this myself.
-
@newuser2pfsense said in Suricata InLine with igb NICs:
boobletins...Presently I'm using Inline IPS Mode and I only have Suricata running on my WAN and that's igb3. I'm using igb0 and igb1 as well for my WLAN and LAN.
dev.igb.3.fc disable flow control 0Previously you had dev.igb.3.fc=3 Does the "bad pkt" error persist with dev.igb.3.fc=0?
Just to confirm, could you double check and paste me the full output from
ifconfig igb3Please paste any additional system turntables you've set via the ui and your full loader.conf.local (minus any sensitive data).
Please then manually double check and paste the output from these commands:
sysctl -a | grep nmbclusters sysctl -a | grep msi sysctl -a | grep num_queues dmesg | grep igb3Above is not busy work, I'm having you manually confirm because with a few commands I found that when I set them in loader.conf.local they didn't take effect. I needed to put some in the ui system tuneables.
We have more settings to tinker with, I made a bunch of changes before the errors went away, but I'm trying to narrow down the issue before just throwing a bunch of new settings at you. I'm pretty confident we can get this working on your igb since its working on mine with 0 errors for over a week now.
-
boobletins...I keep getting the following error message from the page when posting the information you requested; frustrating to say the least:
Error
Post content was flagged as spam by Akismet.comI'll do what I can to get the information in.
The errors do persist:
408.786592 [1071] netmap_grab_packets bad pkt at 186 len 2154
950.583865 [1071] netmap_grab_packets bad pkt at 433 len 2154
530.551894 [1071] netmap_grab_packets bad pkt at 810 len 2147
530.547133 [1071] netmap_grab_packets bad pkt at 807 len 2147
360.440859 [1071] netmap_grab_packets bad pkt at 728 len 2154
764.263927 [1071] netmap_grab_packets bad pkt at 311 len 2154 -
Ok -- I tried to thumbs-up some of your posts hoping that will help with Akismet.
I am interested in those results -- mostly because I think something is putting packets into your hardware buffers that are greater than 2048. They also seem to be consistently in the 2100 range. I can't explain what is doing that or why if your MTU is actually 1500. Maybe there's some kind of overhead with vlan tagging, qos, etc that I'm not aware of.
The why doesn't really matter if all you want is a fix. If you raise the buffer_size of netmap (and the packet sizes stay below those new maximums) then the errors should disappear.
Currently your dev.netmap.buf_size is set to 2048. If you, for example, double that to 4096, then all of the current errors would be covered by the new larger buffer_size in netmap (do this in the ui under system tuneables).
Since I don't understand how you're getting packets that are > 2048 with an MTU of 1500, I can't promise it won't come back with even larger numbers, but that change would cover all of the errors you've pasted so far.
As I say above, you may get additional errors by changing dev.netmap.buf_size -- let me know if that's the case.
For the record: I have an MTU of 1500 and a dev.netmap.buf_size of 1920 is enough to prevent errors.
-
Maybe don't change this unless you run into other issues, but the remarks at the link below suggest that hyperthreading (which you have enabled) may limit your throughput.
https://calomel.org/freebsd_network_tuning.html
# Disable Hyper Threading (HT), also known as Intel's proprietary simultaneous # multithreading (SMT) because implementations typically share TLBs and L1 # caches between threads which is a security concern. SMT is likely to slow # down workloads not specifically optimized for SMT if you have a CPU with more # than two(2) real CPU cores. Secondly, multi-queue network cards are as much # as 20% slower when network queues are bound to both real CPU cores and SMT # virtual cores due to interrupt processing collisions. # machdep.hyperthreading_allowed="0" # (default 1, allow Hyper Threading (HT))That last sentence seems to apply in your situation. They note they've used the config with an i350. I don't see a lot of netmap-specific configuration in there, so ymmv.
This is unrelated to the "bad pkt" error.
-
@boobletins said in Suricata InLine with igb NICs:
Ok -- I tried to thumbs-up some of your posts hoping that will help with Akismet.
It should. Users with a reputation of 5 or more should never see Akismet.
I voted a few posts too so that is now that case.Steve
-
boobletins/stephenw10...I tried posting again but unfortunately I keep getting:
Post content was flagged as spam by Akismet.com
I apologize, I keep trying to post.
I'll try posting a little at a time again if it will let me.
-
ifconfig igb3 [I redacted out IP/MAC addresses]:
igb3: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
options=1000b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,NETMAP>
ether
hwaddr
inet6 %igb3 prefixlen 64 scopeid 0x4
inet netmask broadcast
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active -
The only other System Tunable I changed was from here:
https://www.netgate.com/docs/pfsense/hardware/tuning-and-troubleshooting-network-cards.html?highlight=tuning
net.inet.ip.intr_queue_maxlen , Maximum size of the IP input queue, 3000
I believe it was originally set to 1000. I just never changed it back. I can change it back if need be.Although I've kept some of the tunables in my loader.conf.local file for testing, I've commented them out, #, so nothing there should be loading:
#hw.igb.rxd="1024"
#hw.igb.txd="1024"
#hw.igb.enable_aim=1
#hw.igb.num_queues=0
#kern.ipc.nmbclusters="1000000"
#hw.pci.enable_msi=0
#hw.igb.max_interrupt_rate="32000"
#hw.igb.fc_setting=0
#hw.igb.txd=4096
#hw.igb.rxd=4096 -
sysctl -a | grep nmbclusters -
kern.ipc.nmbclusters: 4076726sysctl -a | grep msi -
hw.ixl.enable_msix: 1
hw.sdhci.enable_msi: 1
hw.puc.msi_disable: 0
hw.pci.honor_msi_blacklist: 1
hw.pci.msix_rewrite_table: 0
hw.pci.enable_msix: 1
hw.pci.enable_msi: 1
hw.mfi.msi: 1
hw.malo.pci.msi_disable: 0
hw.ix.enable_msix: 1
hw.igb.enable_msix: 1
hw.em.enable_msix: 1
hw.cxgb.msi_allowed: 2
hw.bce.msi_enable: 1
hw.aac.enable_msi: 1
machdep.disable_msix_migration: 0sysctl -a | grep num_queues -
hw.ix.num_queues: 0
hw.igb.num_queues: 0 -
dmesg | grep igb3 [I redacted out IP/MAC addresses] -
igb3: link state changed to UP igb3: permanently promiscuous mode enabled igb3: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe000-0xe01f mem 0xdf000000-0xdf0fffff,0xdf600000-0xdf603fff irq 19 at device 0.3 on pci2 igb3: Using MSIX interrupts with 9 vectors igb3: Ethernet address: igb3: Bound queue 0 to cpu 0 igb3: Bound queue 1 to cpu 1 igb3: Bound queue 2 to cpu 2 igb3: Bound queue 3 to cpu 3 igb3: Bound queue 4 to cpu 4 igb3: Bound queue 5 to cpu 5 igb3: Bound queue 6 to cpu 6 igb3: Bound queue 7 to cpu 7 igb3: netmap queues/slots: TX 8/4096, RX 8/4096 igb3: link state changed to UP igb3: link state changed to DOWN igb3: link state changed to UP igb3: permanently promiscuous mode enabled igb3: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe000-0xe01f mem 0xdf000000-0xdf0fffff,0xdf600000-0xdf603fff irq 19 at device 0.3 on pci2 igb3: Using MSIX interrupts with 9 vectors igb3: Ethernet address: igb3: Bound queue 0 to cpu 0 igb3: Bound queue 1 to cpu 1 igb3: Bound queue 2 to cpu 2 igb3: Bound queue 3 to cpu 3 igb3: Bound queue 4 to cpu 4 igb3: Bound queue 5 to cpu 5 igb3: Bound queue 6 to cpu 6 igb3: Bound queue 7 to cpu 7 igb3: netmap queues/slots: TX 8/1024, RX 8/1024 igb3: link state changed to UP igb3: link state changed to DOWN igb3: link state changed to UP igb3: permanently promiscuous mode enabled igb3: link state changed to DOWN igb3: link state changed to UP igb3: link state changed to DOWN igb3: link state changed to UP igb3: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe000-0xe01f mem 0xdf000000-0xdf0fffff,0xdf600000-0xdf603fff irq 19 at device 0.3 on pci2 igb3: Using MSIX interrupts with 9 vectors igb3: Ethernet address: igb3: Bound queue 0 to cpu 0 igb3: Bound queue 1 to cpu 1 igb3: Bound queue 2 to cpu 2 igb3: Bound queue 3 to cpu 3 igb3: Bound queue 4 to cpu 4 igb3: Bound queue 5 to cpu 5 igb3: Bound queue 6 to cpu 6 igb3: Bound queue 7 to cpu 7 igb3: netmap queues/slots: TX 8/1024, RX 8/1024 igb3: link state changed to UP igb3: link state changed to DOWN igb3: link state changed to UP igb3: permanently promiscuous mode enabled igb3: link state changed to DOWN igb3: link state changed to UP igb3: link state changed to DOWN igb3: link state changed to UP igb3: link state changed to DOWN igb3: link state changed to UP igb3: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe000-0xe01f mem 0xdf000000-0xdf0fffff,0xdf600000-0xdf603fff irq 19 at device 0.3 on pci2 igb3: Using MSIX interrupts with 9 vectors igb3: Ethernet address: igb3: Bound queue 0 to cpu 0 igb3: Bound queue 1 to cpu 1 igb3: Bound queue 2 to cpu 2 igb3: Bound queue 3 to cpu 3 igb3: Bound queue 4 to cpu 4 igb3: Bound queue 5 to cpu 5 igb3: Bound queue 6 to cpu 6 igb3: Bound queue 7 to cpu 7 igb3: netmap queues/slots: TX 8/1024, RX 8/1024 igb3: link state changed to UP igb3: link state changed to DOWN igb3: link state changed to UP igb3: link state changed to DOWN igb3: link state changed to UP igb3: permanently promiscuous mode enabled igb3: link state changed to DOWN arpresolve: can't allocate llinfo for on igb3 igb3: link state changed to UP igb3: link state changed to DOWN igb3: link state changed to UP igb3: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe000-0xe01f mem 0xdf000000-0xdf0fffff,0xdf600000-0xdf603fff irq 19 at device 0.3 on pci2 igb3: Using MSIX interrupts with 9 vectors igb3: Ethernet address: igb3: Bound queue 0 to cpu 0 igb3: Bound queue 1 to cpu 1 igb3: Bound queue 2 to cpu 2 igb3: Bound queue 3 to cpu 3 igb3: Bound queue 4 to cpu 4 igb3: Bound queue 5 to cpu 5 igb3: Bound queue 6 to cpu 6 igb3: Bound queue 7 to cpu 7 igb3: netmap queues/slots: TX 8/1024, RX 8/1024 igb3: link state changed to UP igb3: link state changed to DOWN igb3: link state changed to UP igb3: link state changed to DOWN igb3: link state changed to UP igb3: permanently promiscuous mode enabled igb3: link state changed to DOWN igb3: link state changed to UP arpresolve: can't allocate llinfo for on igb3I added a dev.netmap.buf_size to System Tunables and set the value to 4096. I restarted pfSense and then really throttled as much traffic going through it as I could. I didn't get any netmap_grab_packets errors. I'm now wondering if there is a maximum netmap buffer size.
I look forward to doing whatever change/testing we can to find a solution. Thanks for the continued help!
-
@newuser2pfsense said in Suricata InLine with igb NICs:
I'm now wondering if there is a maximum netmap buffer size.
With 64 GB of RAM, you should be able to take that tuneable very high, but there isn't a need unless the bad pkt error returns with higher 2nd numbers. I'd wait and see, otherwise you are locking up memory for no reason.
I can't explain how you're getting packets of size ~2100 with an MTU of 1500. Maybe JUMBO_MTU allows for that (I don't know). It could also be that something else on your network has a larger MTU setting. I'm not sure how FreeBSD handles those situations. If you're interested you can check any switches and clients to see and adjust accordingly.
If the system is handling as much throughput as you can throw at it, then I'd leave everything alone for now.
If you run into throughput or interrupt issues, then consider disabling hyperthreading. The dmesg output indicates that you're binding queues to virtual and hardware cores which may become an issue depending on how hard you're saturating the interfaces.
-
Once the applicable system tuneables are nailed down and some "good" typical values are established, this thread should be made a "sticky post" or else a new single "sticky post" created containing the relevant settings. The netmap bad packets error has plagued a lot of Suricata Inline IPS Mode users.
-
boobletins...I'll let it run for a while with all of the tweaks we've made and check it periodically for any netmap_grab_packets errors.
bmeeks...I agree.
Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.