-
Grimson...You're right. I didn't see it. Shellcmd - The shellcmd utility is used to manage commands on system startup. Now we just need the syntax to use.
-
Shellcmd just runs commands like they would be at the command line. The only difference is you often need the complete path to the command as it runs as a different user. But you could use:
ifconfig igb0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtsoThat should run OK.
Steve
-
I noticed that with the release of pfSenxe 2.4.4, there was a dramatic increase of Netmap errors (https://forum.netgate.com/topic/136191/netmap-alerts-gotten-worst-with-2-4-4). I also have an igb network card.
The major headache is this issue turns into finger pointing...pfSense says its FreeBSD 11.2, Netmap says its Suricata, FreeBSD says its the network card, and Suricata says its Netmap.
My feeling is since we're ultimately using pfSense, it's pfSense responsibility to ensure that we achieve synergy of a robust firewall with all parts working seamlessly. Inline mode is an important part of firewall intrusion detection prevention system in that it provides a more efficient screening.
-
It looks like in that thread you seemed to have solved the issue, is that the case? (I ask because I'm interested in knowing if I've solved this issue for myself as well).
If you're game, could you give me the output from the following shell commands (case sensitive):
ifconfig igb0 | grep CSUM sysctl -a | grep igb sysctl -a | grep netmap- How many CPU cores do you have?
- Is hyperthreading enabled?
- How much RAM do you have?
- Are you running Suricata on more than 1 interface? (If so, what's the second interface? Also: run the shell commands above on that interface)
I've gone a few days now without netmap errors on either my em0 or igb0 interface with Suricata in inline IPS mode and 2 speedtests / hour. I'm becoming more confident that I have a working configuration, but if we can eliminate them from yours as well that'd be some welcome evidence...
-
@boobletins said in Suricata InLine with igb NICs:
ifconfig igb0 | grep CSUM
Shell Output - ifconfig igb0 | grep CSUM
options=5400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,NETMAP,TXCSUM_IPV6>Shell Output - sysctl -a | grep igb
device igb hw.igb.tx_process_limit: -1 hw.igb.rx_process_limit: 100 hw.igb.num_queues: 0 hw.igb.header_split: 0 hw.igb.max_interrupt_rate: 8000 hw.igb.enable_msix: 1 hw.igb.enable_aim: 1 hw.igb.txd: 1024 hw.igb.rxd: 1024 dev.igb.1.host.header_redir_missed: 0 dev.igb.1.host.serdes_violation_pkt: 0 dev.igb.1.host.length_errors: 0 dev.igb.1.host.tx_good_bytes: 5014864175 dev.igb.1.host.rx_good_bytes: 344809214 dev.igb.1.host.breaker_tx_pkt_drop: 0 dev.igb.1.host.tx_good_pkt: 63 dev.igb.1.host.breaker_rx_pkt_drop: 0 dev.igb.1.host.breaker_rx_pkts: 0 dev.igb.1.host.rx_pkt: 77 dev.igb.1.host.host_tx_pkt_discard: 0 dev.igb.1.host.breaker_tx_pkt: 0 dev.igb.1.interrupts.rx_overrun: 0 dev.igb.1.interrupts.rx_desc_min_thresh: 0 dev.igb.1.interrupts.tx_queue_min_thresh: 0 dev.igb.1.interrupts.tx_queue_empty: 4315841 dev.igb.1.interrupts.tx_abs_timer: 0 dev.igb.1.interrupts.tx_pkt_timer: 4315904 dev.igb.1.interrupts.rx_abs_timer: 2921232 dev.igb.1.interrupts.rx_pkt_timer: 2921155 dev.igb.1.interrupts.asserts: 8803973 dev.igb.1.mac_stats.tso_ctx_fail: 0 dev.igb.1.mac_stats.tso_txd: 0 dev.igb.1.mac_stats.tx_frames_1024_1522: 3233544 dev.igb.1.mac_stats.tx_frames_512_1023: 62481 dev.igb.1.mac_stats.tx_frames_256_511: 72052 dev.igb.1.mac_stats.tx_frames_128_255: 119162 dev.igb.1.mac_stats.tx_frames_65_127: 781667 dev.igb.1.mac_stats.tx_frames_64: 46998 dev.igb.1.mac_stats.mcast_pkts_txd: 269918 dev.igb.1.mac_stats.bcast_pkts_txd: 118 dev.igb.1.mac_stats.good_pkts_txd: 4315904 dev.igb.1.mac_stats.total_pkts_txd: 4315904 dev.igb.1.mac_stats.total_octets_txd: 5014886629 dev.igb.1.mac_stats.good_octets_txd: 5014885349 dev.igb.1.mac_stats.total_octets_recvd: 344809463 dev.igb.1.mac_stats.good_octets_recvd: 344808248 dev.igb.1.mac_stats.rx_frames_1024_1522: 49390 dev.igb.1.mac_stats.rx_frames_512_1023: 61271 dev.igb.1.mac_stats.rx_frames_256_511: 60178 dev.igb.1.mac_stats.rx_frames_128_255: 132406 dev.igb.1.mac_stats.rx_frames_65_127: 2127900 dev.igb.1.mac_stats.rx_frames_64: 490087 dev.igb.1.mac_stats.mcast_pkts_recvd: 0 dev.igb.1.mac_stats.bcast_pkts_recvd: 4 dev.igb.1.mac_stats.good_pkts_recvd: 2921232 dev.igb.1.mac_stats.total_pkts_recvd: 2921232 dev.igb.1.mac_stats.mgmt_pkts_txd: 0 dev.igb.1.mac_stats.mgmt_pkts_drop: 0 dev.igb.1.mac_stats.mgmt_pkts_recvd: 0 dev.igb.1.mac_stats.unsupported_fc_recvd: 0 dev.igb.1.mac_stats.xoff_txd: 0 dev.igb.1.mac_stats.xoff_recvd: 0 dev.igb.1.mac_stats.xon_txd: 0 dev.igb.1.mac_stats.xon_recvd: 0 dev.igb.1.mac_stats.coll_ext_errs: 0 dev.igb.1.mac_stats.tx_no_crs: 0 dev.igb.1.mac_stats.alignment_errs: 0 dev.igb.1.mac_stats.crc_errs: 0 dev.igb.1.mac_stats.recv_errs: 0 dev.igb.1.mac_stats.recv_jabber: 0 dev.igb.1.mac_stats.recv_oversize: 0 dev.igb.1.mac_stats.recv_fragmented: 0 dev.igb.1.mac_stats.recv_undersize: 0 dev.igb.1.mac_stats.recv_no_buff: 0 dev.igb.1.mac_stats.recv_length_errors: 0 dev.igb.1.mac_stats.missed_packets: 0 dev.igb.1.mac_stats.defer_count: 0 dev.igb.1.mac_stats.sequence_errors: 0 dev.igb.1.mac_stats.symbol_errors: 0 dev.igb.1.mac_stats.collision_count: 0 dev.igb.1.mac_stats.late_coll: 0 dev.igb.1.mac_stats.multiple_coll: 0 dev.igb.1.mac_stats.single_coll: 0 dev.igb.1.mac_stats.excess_coll: 0 dev.igb.1.queue1.lro_flushed: 0 dev.igb.1.queue1.lro_queued: 0 dev.igb.1.queue1.rx_bytes: 152608531 dev.igb.1.queue1.rx_packets: 1226723 dev.igb.1.queue1.rxd_tail: 994 dev.igb.1.queue1.rxd_head: 995 dev.igb.1.queue1.tx_packets: 257 dev.igb.1.queue1.no_desc_avail: 0 dev.igb.1.queue1.txd_tail: 339 dev.igb.1.queue1.txd_head: 339 dev.igb.1.queue1.interrupt_rate: 76923 dev.igb.1.queue0.lro_flushed: 0 dev.igb.1.queue0.lro_queued: 0 dev.igb.1.queue0.rx_bytes: 180516276 dev.igb.1.queue0.rx_packets: 1694509 dev.igb.1.queue0.rxd_tail: 812 dev.igb.1.queue0.rxd_head: 813 dev.igb.1.queue0.tx_packets: 4315647 dev.igb.1.queue0.no_desc_avail: 0 dev.igb.1.queue0.txd_tail: 442 dev.igb.1.queue0.txd_head: 442 dev.igb.1.queue0.interrupt_rate: 90909 dev.igb.1.fc_low_water: 29480 dev.igb.1.fc_high_water: 29488 dev.igb.1.rx_buf_alloc: 34 dev.igb.1.tx_buf_alloc: 14 dev.igb.1.extended_int_mask: 2147484419 dev.igb.1.interrupt_mask: 4 dev.igb.1.rx_control: 67141658 dev.igb.1.device_control: 1087373896 dev.igb.1.watchdog_timeouts: 0 dev.igb.1.rx_overruns: 0 dev.igb.1.tx_dma_fail: 0 dev.igb.1.mbuf_defrag_fail: 0 dev.igb.1.link_irq: 2 dev.igb.1.dropped: 0 dev.igb.1.tx_processing_limit: -1 dev.igb.1.rx_processing_limit: 100 dev.igb.1.fc: 3 dev.igb.1.enable_aim: 1 dev.igb.1.nvm: -1 dev.igb.1.%parent: pci3 dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10a7 subvendor=0x8086 subdevice=0x10a7 class=0x020000 dev.igb.1.%location: slot=0 function=1 dbsf=pci0:3:0:1 dev.igb.1.%driver: igb dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k dev.igb.0.host.header_redir_missed: 0 dev.igb.0.host.serdes_violation_pkt: 0 dev.igb.0.host.length_errors: 0 dev.igb.0.host.tx_good_bytes: 702511124 dev.igb.0.host.rx_good_bytes: 8859910607 dev.igb.0.host.breaker_tx_pkt_drop: 0 dev.igb.0.host.tx_good_pkt: 389 dev.igb.0.host.breaker_rx_pkt_drop: 0 dev.igb.0.host.breaker_rx_pkts: 0 dev.igb.0.host.rx_pkt: 179 dev.igb.0.host.host_tx_pkt_discard: 0 dev.igb.0.host.breaker_tx_pkt: 0 dev.igb.0.interrupts.rx_overrun: 0 dev.igb.0.interrupts.rx_desc_min_thresh: 0 dev.igb.0.interrupts.tx_queue_min_thresh: 0 dev.igb.0.interrupts.tx_queue_empty: 8008878 dev.igb.0.interrupts.tx_abs_timer: 0 dev.igb.0.interrupts.tx_pkt_timer: 8009267 dev.igb.0.interrupts.rx_abs_timer: 9004187 dev.igb.0.interrupts.rx_pkt_timer: 9004008 dev.igb.0.interrupts.asserts: 18858568 dev.igb.0.mac_stats.tso_ctx_fail: 0 dev.igb.0.mac_stats.tso_txd: 0 dev.igb.0.mac_stats.tx_frames_1024_1522: 49679 dev.igb.0.mac_stats.tx_frames_512_1023: 59868 dev.igb.0.mac_stats.tx_frames_256_511: 65957 dev.igb.0.mac_stats.tx_frames_128_255: 117544 dev.igb.0.mac_stats.tx_frames_65_127: 4903787 dev.igb.0.mac_stats.tx_frames_64: 2812432 dev.igb.0.mac_stats.mcast_pkts_txd: 217 dev.igb.0.mac_stats.bcast_pkts_txd: 321 dev.igb.0.mac_stats.good_pkts_txd: 8009267 dev.igb.0.mac_stats.total_pkts_txd: 8009267 dev.igb.0.mac_stats.total_octets_txd: 702511679 dev.igb.0.mac_stats.good_octets_txd: 702510340 dev.igb.0.mac_stats.total_octets_recvd: 8859907035 dev.igb.0.mac_stats.good_octets_recvd: 8859915684 dev.igb.0.mac_stats.rx_frames_1024_1522: 5630206 dev.igb.0.mac_stats.rx_frames_512_1023: 67795 dev.igb.0.mac_stats.rx_frames_256_511: 155128 dev.igb.0.mac_stats.rx_frames_128_255: 445051 dev.igb.0.mac_stats.rx_frames_65_127: 765396 dev.igb.0.mac_stats.rx_frames_64: 1940609 dev.igb.0.mac_stats.mcast_pkts_recvd: 218995 dev.igb.0.mac_stats.bcast_pkts_recvd: 47673 dev.igb.0.mac_stats.good_pkts_recvd: 9004185 dev.igb.0.mac_stats.total_pkts_recvd: 9004224 dev.igb.0.mac_stats.mgmt_pkts_txd: 0 dev.igb.0.mac_stats.mgmt_pkts_drop: 0 dev.igb.0.mac_stats.mgmt_pkts_recvd: 0 dev.igb.0.mac_stats.unsupported_fc_recvd: 0 dev.igb.0.mac_stats.xoff_txd: 0 dev.igb.0.mac_stats.xoff_recvd: 1 dev.igb.0.mac_stats.xon_txd: 0 dev.igb.0.mac_stats.xon_recvd: 1 dev.igb.0.mac_stats.coll_ext_errs: 0 dev.igb.0.mac_stats.tx_no_crs: 0 dev.igb.0.mac_stats.alignment_errs: 0 dev.igb.0.mac_stats.crc_errs: 0 dev.igb.0.mac_stats.recv_errs: 0 dev.igb.0.mac_stats.recv_jabber: 0 dev.igb.0.mac_stats.recv_oversize: 0 dev.igb.0.mac_stats.recv_fragmented: 0 dev.igb.0.mac_stats.recv_undersize: 0 dev.igb.0.mac_stats.recv_no_buff: 0 dev.igb.0.mac_stats.recv_length_errors: 0 dev.igb.0.mac_stats.missed_packets: 0 dev.igb.0.mac_stats.defer_count: 0 dev.igb.0.mac_stats.sequence_errors: 0 dev.igb.0.mac_stats.symbol_errors: 0 dev.igb.0.mac_stats.collision_count: 0 dev.igb.0.mac_stats.late_coll: 0 dev.igb.0.mac_stats.multiple_coll: 0 dev.igb.0.mac_stats.single_coll: 0 dev.igb.0.mac_stats.excess_coll: 0 dev.igb.0.queue1.lro_flushed: 0 dev.igb.0.queue1.lro_queued: 0 dev.igb.0.queue1.rx_bytes: 0 dev.igb.0.queue1.rx_packets: 2432 dev.igb.0.queue1.rxd_tail: 35 dev.igb.0.queue1.rxd_head: 36 dev.igb.0.queue1.tx_packets: 1 dev.igb.0.queue1.no_desc_avail: 0 dev.igb.0.queue1.txd_tail: 0 dev.igb.0.queue1.txd_head: 0 dev.igb.0.queue1.interrupt_rate: 16129 dev.igb.0.queue0.lro_flushed: 0 dev.igb.0.queue0.lro_queued: 0 dev.igb.0.queue0.rx_bytes: 0 dev.igb.0.queue0.rx_packets: 7244 dev.igb.0.queue0.rxd_tail: 180 dev.igb.0.queue0.rxd_head: 181 dev.igb.0.queue0.tx_packets: 9386 dev.igb.0.queue0.no_desc_avail: 0 dev.igb.0.queue0.txd_tail: 755 dev.igb.0.queue0.txd_head: 755 dev.igb.0.queue0.interrupt_rate: 16129 dev.igb.0.fc_low_water: 29480 dev.igb.0.fc_high_water: 29488 dev.igb.0.rx_buf_alloc: 34 dev.igb.0.tx_buf_alloc: 14 dev.igb.0.extended_int_mask: 2147484419 dev.igb.0.interrupt_mask: 4 dev.igb.0.rx_control: 67141658 dev.igb.0.device_control: 1490027080 dev.igb.0.watchdog_timeouts: 0 dev.igb.0.rx_overruns: 0 dev.igb.0.tx_dma_fail: 0 dev.igb.0.mbuf_defrag_fail: 0 dev.igb.0.link_irq: 70 dev.igb.0.dropped: 0 dev.igb.0.tx_processing_limit: -1 dev.igb.0.rx_processing_limit: 100 dev.igb.0.fc: 3 dev.igb.0.enable_aim: 1 dev.igb.0.nvm: -1 dev.igb.0.%parent: pci3 dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10a7 subvendor=0x8086 subdevice=0x10a7 class=0x020000 dev.igb.0.%location: slot=0 function=0 dbsf=pci0:3:0:0 dev.igb.0.%driver: igb dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k dev.igb.%parent: Shell Output - sysctl -a | grep netmap device netmap dev.netmap.ixl_rx_miss_bufs: 0 dev.netmap.ixl_rx_miss: 0 dev.netmap.iflib_rx_miss_bufs: 0 dev.netmap.iflib_rx_miss: 0 dev.netmap.iflib_crcstrip: 1 dev.netmap.bridge_batch: 1024 dev.netmap.default_pipes: 0 dev.netmap.priv_buf_num: 4098 dev.netmap.priv_buf_size: 2048 dev.netmap.buf_curr_num: 163840 dev.netmap.buf_num: 163840 dev.netmap.buf_curr_size: 4096 dev.netmap.buf_size: 4096 dev.netmap.priv_ring_num: 4 dev.netmap.priv_ring_size: 20480 dev.netmap.ring_curr_num: 200 dev.netmap.ring_num: 200 dev.netmap.ring_curr_size: 36864 dev.netmap.ring_size: 36864 dev.netmap.priv_if_num: 1 dev.netmap.priv_if_size: 1024 dev.netmap.if_curr_num: 100 dev.netmap.if_num: 100 dev.netmap.if_curr_size: 1024 dev.netmap.if_size: 1024 dev.netmap.generic_rings: 1 dev.netmap.generic_ringsize: 1024 dev.netmap.generic_mit: 100000 dev.netmap.admode: 0 dev.netmap.fwd: 0 dev.netmap.flags: 0 dev.netmap.adaptive_io: 0 dev.netmap.txsync_retry: 2 dev.netmap.no_pendintr: 1 dev.netmap.mitigate: 1 dev.netmap.no_timestamp: 0 dev.netmap.verbose: 0 dev.netmap.ix_rx_miss_bufs: 0 dev.netmap.ix_rx_miss: 0 dev.netmap.ix_crcstrip: 0Every couple of days I get one or two netmap bad packet alert even after increasing this - netmap.buf_size: 4096. I run both Suricata and Snort on Wan and Lan however, I only enable blocking on Suricata WAN...all else are disabled. I have 8GB RAM however, I can only use 6GB as a failed processor killed a row/channel in my HP Pavilion 6242n trash find I converted into a pfSense firewall.
-
Under System / Advanced / Networking, is "Allow IPv6" checked?
And how many CPU cores? Is hyperthreading enabled?
-
@boobletins said in Suricata InLine with igb NICs:
Under System / Advanced / Networking, is "Allow IPv6" checked?
And how many CPU cores? Is hyperthreading enabled?
Yes...allowed IPv6 checked...CPU Type AMD Athlon(tm) 64 X2 Dual Core Processor 4800+
2 CPUs: 1 package(s) x 2 core(s)
AES-NI CPU Crypto: NoNo sure where to check for hyperthread...now I will disable IPv6...thought I did.
-
So here are some initial suggestions. Please keep in mind that I've been working on this for ~1 week (in other words: not long), and I'm not a FreeBSD, pfSense, or Suricata expert.
Start by making a backup of your configuration.
Do these first:
My understanding is that flow control should be off on any netmap interface. You have bi-directional flow control enabled:dev.igb.0.fc: 3Disable flow control on all active interfaces using system tunables. Set dev.igb.0.fc=0 (and dev.igb.1.fc=0)
Actively set energy efficient ethernet to disabled:
dev.igb.0.eee_disabled=1Actively force IPv6_TXCSUM6 off by adding the following to config.xml in a shellcmd tag:
ifconfig igb0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso(see above in this thread for a link on where/how to do that).
Edit:
To be clear: anywhere I have a command that says "igb0" or "igb.0" you will want to duplicate that for igb1 and any other interface you're running netmap on.So you will need 2 shellcmd lines in config.xml, and two new system tunables for flow control, etc
Consider changing later:
Set rx processing limit:
dev.igb.0.rx_processing_limit: -1It looks like your txd and rxd are both set to 1024 currently, I suggest you move those to 4096:
hw.igb.txd=4096
hw.igb.rxd=4096By changing your txd and rxd we may need to revisit your netmap buf/ring (memory settings).
We may also revisit your interrupt and queue settings.
-
It would be great if you could let me know what happens after the initial changes -- if you continue to get netmap errors or not.
If you do, don't jump right to the 2nd section of changes, we should verify that the changes we made above took properly. I learned the hard way that I was putting some settings in the wrong places.
-
boobletins...I apologize for not getting back sooner; other projects. I added the shellcmd line to the /cf/conf/config.xml file as you suggested. I re-enabled Suricata in Inline IPS Mode and restarted pfSense. I ran ifconfig against all four ethernet interfaces on my SuperMicro adapter and TXCSUM_IPV6 was not listed.
One thing I find interesting is in the Services > Suricata > Alerts tab, all of the text is now black in color when before making the above change it was all red in color in Inline IPS Mode. As well, there are no entries in the Blocks tab when before making the change it was automagically populated with over 300 blocked IP addresses in Inline IPS Mode. I don't know if this is normal or not. I didn't change any of the Suricata WAN Categories.
-
If you are running Suricata in inline mode, you will not see blocked IP addresses in the blocked tab, as any traffic that conforms to your "drop" rules is automatically intercepted and dropped (as opposed to initially logged, then ip banned as in Legacy mode).
The red text in the Alerts tab is letting you know that the traffic was indeed intercepted and dropped (since you don't have any information in the Block tab anymore).
That you are missing both blocks and red text means that either no traffic has conformed to your block rules yet, or something has gone wrong.
Double check which mode Suricata is running in. Then double check that you have some drop rules defined.
But originally the issue was netmap, yes? If so, have you see any netmap errors? Can you complete a speedtest with Suricata enabled in inline mode now?
-
I guess my "drop" rules are just based on the Suricata WAN Categories I selected/checked? When you checkmark what you want, Suricata applies those rules to the incoming data and drops it?
I'm definitely running in Inline IPS Mode and all of the Alerts entries are in black colored text. As well, all of the text I typed into the loader.conf.local file is commented out.
Yes, the original issue was many, many, netmap_grab_packets console/log messages. At the time of this post, I don't have any netmap errors, yet
. I can actually complete a speedtest in Inline IPS Mode now. -
So Suricata is complex and can do many things.
I suggest you read about SID management and check out the SID management and categories tabs.
If I had to guess, I would say that you have a minimal set of "drop" rules in place and just haven't had blocked traffic hit your server yet. When you do, you'll see the red text.
If you want to convert "alerts" to "drops," you'll typically do that in SID management.
-
Unfortunately, I just received several netmap_grap_packets errors:
netmap_grab_packets bad pkt at 465 len 2147
netmap_grab_packets bad pkt at 116 len 2154
netmap_grab_packets bad pkt at 39 len 2147
netmap_grab_packets bad pkt at 872 len 2147
netmap_grab_packets bad pkt at 860 len 2147
netmap_grab_packets bad pkt at 838 len 2147
netmap_grab_packets bad pkt at 777 len 2154Apparently the shellcmd didn't work after all.
-
Could you give me the same info I asked for from NollipfSense above?
The results of these commands:
ifconfig igb0 | grep CSUM
sysctl -a | grep igb
sysctl -a | grep netmap -
@newuser2pfsense said in Suricata InLine with igb NICs:
I guess my "drop" rules are just based on the Suricata WAN Categories I selected/checked? When you checkmark what you want, Suricata applies those rules to the incoming data and drops it?
I'm definitely running in Inline IPS Mode and all of the Alerts entries are in black colored text. As well, all of the text I typed into the loader.conf.local file is commented out.
Yes, the original issue was many, many, netmap_grab_packets console/log messages. At the time of this post, I don't have any netmap errors, yet
. I can actually complete a speedtest in Inline IPS Mode now.When you run with Inline IPS Mode enabled, you have to manually change rules you want to block traffic from ALERT to DROP. You can do that in two places: (1) on the RULES tab with the selected category displayed; (2) using SID Management keywords on the SID MGMT tab. Sounds like you do not have any DROP rules enabled if you are getting no red alert entries. Either that, or as @boobletins stated, your enabled DROP rules have not been triggered.
-
boobletins...Here is the output of the commands you requested.
ifconfig igbX | grep CSUM output -
for igb0: options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM> for igb1: options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM> for igb2 [not used]: options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM> for igb3 [WAN]: options=1000b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,NETMAP> for em0 [not used]: options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>sysctl -a | grep igb output -
device igb hw.igb.tx_process_limit: -1 hw.igb.rx_process_limit: 100 hw.igb.num_queues: 0 hw.igb.header_split: 0 hw.igb.max_interrupt_rate: 8000 hw.igb.enable_msix: 1 hw.igb.enable_aim: 1 hw.igb.txd: 1024 hw.igb.rxd: 1024 dev.igb.3.host.header_redir_missed: 0 dev.igb.3.host.serdes_violation_pkt: 0 dev.igb.3.host.length_errors: 3 dev.igb.3.host.tx_good_bytes: 213941 dev.igb.3.host.rx_good_bytes: 3967552 dev.igb.3.host.breaker_tx_pkt_drop: 0 dev.igb.3.host.tx_good_pkt: 0 dev.igb.3.host.breaker_rx_pkt_drop: 0 dev.igb.3.host.breaker_rx_pkts: 0 dev.igb.3.host.rx_pkt: 3 dev.igb.3.host.host_tx_pkt_discard: 0 dev.igb.3.host.breaker_tx_pkt: 0 dev.igb.3.interrupts.rx_overrun: 0 dev.igb.3.interrupts.rx_desc_min_thresh: 0 dev.igb.3.interrupts.tx_queue_min_thresh: 231 dev.igb.3.interrupts.tx_queue_empty: 2417 dev.igb.3.interrupts.tx_abs_timer: 0 dev.igb.3.interrupts.tx_pkt_timer: 0 dev.igb.3.interrupts.rx_abs_timer: 0 dev.igb.3.interrupts.rx_pkt_timer: 18194 dev.igb.3.interrupts.asserts: 22095 dev.igb.3.mac_stats.tso_ctx_fail: 0 dev.igb.3.mac_stats.tso_txd: 0 dev.igb.3.mac_stats.tx_frames_1024_1522: 3 dev.igb.3.mac_stats.tx_frames_512_1023: 16 dev.igb.3.mac_stats.tx_frames_256_511: 21 dev.igb.3.mac_stats.tx_frames_128_255: 90 dev.igb.3.mac_stats.tx_frames_65_127: 1502 dev.igb.3.mac_stats.tx_frames_64: 785 dev.igb.3.mac_stats.mcast_pkts_txd: 3 dev.igb.3.mac_stats.bcast_pkts_txd: 10 dev.igb.3.mac_stats.good_pkts_txd: 2417 dev.igb.3.mac_stats.total_pkts_txd: 2417 dev.igb.3.mac_stats.total_octets_txd: 213941 dev.igb.3.mac_stats.good_octets_txd: 213941 dev.igb.3.mac_stats.total_octets_recvd: 4000401 dev.igb.3.mac_stats.good_octets_recvd: 3967552 dev.igb.3.mac_stats.rx_frames_1024_1522: 1363 dev.igb.3.mac_stats.rx_frames_512_1023: 339 dev.igb.3.mac_stats.rx_frames_256_511: 1232 dev.igb.3.mac_stats.rx_frames_128_255: 1117 dev.igb.3.mac_stats.rx_frames_65_127: 6557 dev.igb.3.mac_stats.rx_frames_64: 7589 dev.igb.3.mac_stats.mcast_pkts_recvd: 11704 dev.igb.3.mac_stats.bcast_pkts_recvd: 3585 dev.igb.3.mac_stats.good_pkts_recvd: 18197 dev.igb.3.mac_stats.total_pkts_recvd: 18528 dev.igb.3.mac_stats.mgmt_pkts_txd: 0 dev.igb.3.mac_stats.mgmt_pkts_drop: 0 dev.igb.3.mac_stats.mgmt_pkts_recvd: 0 dev.igb.3.mac_stats.unsupported_fc_recvd: 0 dev.igb.3.mac_stats.xoff_txd: 0 dev.igb.3.mac_stats.xoff_recvd: 0 dev.igb.3.mac_stats.xon_txd: 0 dev.igb.3.mac_stats.xon_recvd: 0 dev.igb.3.mac_stats.coll_ext_errs: 0 dev.igb.3.mac_stats.tx_no_crs: 0 dev.igb.3.mac_stats.alignment_errs: 0 dev.igb.3.mac_stats.crc_errs: 0 dev.igb.3.mac_stats.recv_errs: 0 dev.igb.3.mac_stats.recv_jabber: 0 dev.igb.3.mac_stats.recv_oversize: 0 dev.igb.3.mac_stats.recv_fragmented: 0 dev.igb.3.mac_stats.recv_undersize: 0 dev.igb.3.mac_stats.recv_no_buff: 0 dev.igb.3.mac_stats.recv_length_errors: 0 dev.igb.3.mac_stats.missed_packets: 0 dev.igb.3.mac_stats.defer_count: 0 dev.igb.3.mac_stats.sequence_errors: 0 dev.igb.3.mac_stats.symbol_errors: 0 dev.igb.3.mac_stats.collision_count: 0 dev.igb.3.mac_stats.late_coll: 0 dev.igb.3.mac_stats.multiple_coll: 0 dev.igb.3.mac_stats.single_coll: 0 dev.igb.3.mac_stats.excess_coll: 0 dev.igb.3.queue7.lro_flushed: 0 dev.igb.3.queue7.lro_queued: 0 dev.igb.3.queue7.rx_bytes: 0 dev.igb.3.queue7.rx_packets: 129 dev.igb.3.queue7.rxd_tail: 561 dev.igb.3.queue7.rxd_head: 562 dev.igb.3.queue7.tx_packets: 0 dev.igb.3.queue7.no_desc_avail: 0 dev.igb.3.queue7.txd_tail: 0 dev.igb.3.queue7.txd_head: 0 dev.igb.3.queue7.interrupt_rate: 8000 dev.igb.3.queue6.lro_flushed: 0 dev.igb.3.queue6.lro_queued: 0 dev.igb.3.queue6.rx_bytes: 0 dev.igb.3.queue6.rx_packets: 30 dev.igb.3.queue6.rxd_tail: 902 dev.igb.3.queue6.rxd_head: 903 dev.igb.3.queue6.tx_packets: 0 dev.igb.3.queue6.no_desc_avail: 0 dev.igb.3.queue6.txd_tail: 0 dev.igb.3.queue6.txd_head: 0 dev.igb.3.queue6.interrupt_rate: 8000 dev.igb.3.queue5.lro_flushed: 0 dev.igb.3.queue5.lro_queued: 0 dev.igb.3.queue5.rx_bytes: 0 dev.igb.3.queue5.rx_packets: 30 dev.igb.3.queue5.rxd_tail: 344 dev.igb.3.queue5.rxd_head: 345 dev.igb.3.queue5.tx_packets: 0 dev.igb.3.queue5.no_desc_avail: 0 dev.igb.3.queue5.txd_tail: 0 dev.igb.3.queue5.txd_head: 0 dev.igb.3.queue5.interrupt_rate: 8000 dev.igb.3.queue4.lro_flushed: 0 dev.igb.3.queue4.lro_queued: 0 dev.igb.3.queue4.rx_bytes: 0 dev.igb.3.queue4.rx_packets: 87 dev.igb.3.queue4.rxd_tail: 524 dev.igb.3.queue4.rxd_head: 525 dev.igb.3.queue4.tx_packets: 0 dev.igb.3.queue4.no_desc_avail: 0 dev.igb.3.queue4.txd_tail: 0 dev.igb.3.queue4.txd_head: 0 dev.igb.3.queue4.interrupt_rate: 8000 dev.igb.3.queue3.lro_flushed: 0 dev.igb.3.queue3.lro_queued: 0 dev.igb.3.queue3.rx_bytes: 0 dev.igb.3.queue3.rx_packets: 29 dev.igb.3.queue3.rxd_tail: 453 dev.igb.3.queue3.rxd_head: 454 dev.igb.3.queue3.tx_packets: 0 dev.igb.3.queue3.no_desc_avail: 0 dev.igb.3.queue3.txd_tail: 0 dev.igb.3.queue3.txd_head: 0 dev.igb.3.queue3.interrupt_rate: 76923 dev.igb.3.queue2.lro_flushed: 0 dev.igb.3.queue2.lro_queued: 0 dev.igb.3.queue2.rx_bytes: 0 dev.igb.3.queue2.rx_packets: 28 dev.igb.3.queue2.rxd_tail: 876 dev.igb.3.queue2.rxd_head: 877 dev.igb.3.queue2.tx_packets: 0 dev.igb.3.queue2.no_desc_avail: 0 dev.igb.3.queue2.txd_tail: 0 dev.igb.3.queue2.txd_head: 0 dev.igb.3.queue2.interrupt_rate: 8000 dev.igb.3.queue1.lro_flushed: 0 dev.igb.3.queue1.lro_queued: 0 dev.igb.3.queue1.rx_bytes: 0 dev.igb.3.queue1.rx_packets: 17 dev.igb.3.queue1.rxd_tail: 482 dev.igb.3.queue1.rxd_head: 483 dev.igb.3.queue1.tx_packets: 0 dev.igb.3.queue1.no_desc_avail: 0 dev.igb.3.queue1.txd_tail: 0 dev.igb.3.queue1.txd_head: 0 dev.igb.3.queue1.interrupt_rate: 8000 dev.igb.3.queue0.lro_flushed: 0 dev.igb.3.queue0.lro_queued: 0 dev.igb.3.queue0.rx_bytes: 0 dev.igb.3.queue0.rx_packets: 142 dev.igb.3.queue0.rxd_tail: 253 dev.igb.3.queue0.rxd_head: 254 dev.igb.3.queue0.tx_packets: 353 dev.igb.3.queue0.no_desc_avail: 0 dev.igb.3.queue0.txd_tail: 18 dev.igb.3.queue0.txd_head: 18 dev.igb.3.queue0.interrupt_rate: 8000 dev.igb.3.fc_low_water: 33152 dev.igb.3.fc_high_water: 33168 dev.igb.3.rx_buf_alloc: 0 dev.igb.3.tx_buf_alloc: 0 dev.igb.3.extended_int_mask: 2147484159 dev.igb.3.interrupt_mask: 4 dev.igb.3.rx_control: 67141658 dev.igb.3.device_control: 1478230593 dev.igb.3.watchdog_timeouts: 0 dev.igb.3.rx_overruns: 0 dev.igb.3.tx_dma_fail: 0 dev.igb.3.mbuf_defrag_fail: 0 dev.igb.3.link_irq: 7 dev.igb.3.dropped: 0 dev.igb.3.eee_disabled: 0 dev.igb.3.dmac: 0 dev.igb.3.tx_processing_limit: -1 dev.igb.3.rx_processing_limit: 100 dev.igb.3.fc: 3 dev.igb.3.enable_aim: 1 dev.igb.3.nvm: -1 dev.igb.3.%parent: pci2 dev.igb.3.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1b12 class=0x020000 dev.igb.3.%location: slot=0 function=3 dbsf=pci0:2:0:3 dev.igb.3.%driver: igb dev.igb.3.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k dev.igb.2.host.header_redir_missed: 0 dev.igb.2.host.serdes_violation_pkt: 0 dev.igb.2.host.length_errors: 0 dev.igb.2.host.tx_good_bytes: 0 dev.igb.2.host.rx_good_bytes: 0 dev.igb.2.host.breaker_tx_pkt_drop: 0 dev.igb.2.host.tx_good_pkt: 0 dev.igb.2.host.breaker_rx_pkt_drop: 0 dev.igb.2.host.breaker_rx_pkts: 0 dev.igb.2.host.rx_pkt: 0 dev.igb.2.host.host_tx_pkt_discard: 0 dev.igb.2.host.breaker_tx_pkt: 0 dev.igb.2.interrupts.rx_overrun: 0 dev.igb.2.interrupts.rx_desc_min_thresh: 0 dev.igb.2.interrupts.tx_queue_min_thresh: 0 dev.igb.2.interrupts.tx_queue_empty: 0 dev.igb.2.interrupts.tx_abs_timer: 0 dev.igb.2.interrupts.tx_pkt_timer: 0 dev.igb.2.interrupts.rx_abs_timer: 0 dev.igb.2.interrupts.rx_pkt_timer: 0 dev.igb.2.interrupts.asserts: 0 dev.igb.2.mac_stats.tso_ctx_fail: 0 dev.igb.2.mac_stats.tso_txd: 0 dev.igb.2.mac_stats.tx_frames_1024_1522: 0 dev.igb.2.mac_stats.tx_frames_512_1023: 0 dev.igb.2.mac_stats.tx_frames_256_511: 0 dev.igb.2.mac_stats.tx_frames_128_255: 0 dev.igb.2.mac_stats.tx_frames_65_127: 0 dev.igb.2.mac_stats.tx_frames_64: 0 dev.igb.2.mac_stats.mcast_pkts_txd: 0 dev.igb.2.mac_stats.bcast_pkts_txd: 0 dev.igb.2.mac_stats.good_pkts_txd: 0 dev.igb.2.mac_stats.total_pkts_txd: 0 dev.igb.2.mac_stats.total_octets_txd: 0 dev.igb.2.mac_stats.good_octets_txd: 0 dev.igb.2.mac_stats.total_octets_recvd: 0 dev.igb.2.mac_stats.good_octets_recvd: 0 dev.igb.2.mac_stats.rx_frames_1024_1522: 0 dev.igb.2.mac_stats.rx_frames_512_1023: 0 dev.igb.2.mac_stats.rx_frames_256_511: 0 dev.igb.2.mac_stats.rx_frames_128_255: 0 dev.igb.2.mac_stats.rx_frames_65_127: 0 dev.igb.2.mac_stats.rx_frames_64: 0 dev.igb.2.mac_stats.mcast_pkts_recvd: 0 dev.igb.2.mac_stats.bcast_pkts_recvd: 0 dev.igb.2.mac_stats.good_pkts_recvd: 0 dev.igb.2.mac_stats.total_pkts_recvd: 0 dev.igb.2.mac_stats.mgmt_pkts_txd: 0 dev.igb.2.mac_stats.mgmt_pkts_drop: 0 dev.igb.2.mac_stats.mgmt_pkts_recvd: 0 dev.igb.2.mac_stats.unsupported_fc_recvd: 0 dev.igb.2.mac_stats.xoff_txd: 0 dev.igb.2.mac_stats.xoff_recvd: 0 dev.igb.2.mac_stats.xon_txd: 0 dev.igb.2.mac_stats.xon_recvd: 0 dev.igb.2.mac_stats.coll_ext_errs: 0 dev.igb.2.mac_stats.tx_no_crs: 0 dev.igb.2.mac_stats.alignment_errs: 0 dev.igb.2.mac_stats.crc_errs: 0 dev.igb.2.mac_stats.recv_errs: 0 dev.igb.2.mac_stats.recv_jabber: 0 dev.igb.2.mac_stats.recv_oversize: 0 dev.igb.2.mac_stats.recv_fragmented: 0 dev.igb.2.mac_stats.recv_undersize: 0 dev.igb.2.mac_stats.recv_no_buff: 0 dev.igb.2.mac_stats.recv_length_errors: 0 dev.igb.2.mac_stats.missed_packets: 0 dev.igb.2.mac_stats.defer_count: 0 dev.igb.2.mac_stats.sequence_errors: 0 dev.igb.2.mac_stats.symbol_errors: 0 dev.igb.2.mac_stats.collision_count: 0 dev.igb.2.mac_stats.late_coll: 0 dev.igb.2.mac_stats.multiple_coll: 0 dev.igb.2.mac_stats.single_coll: 0 dev.igb.2.mac_stats.excess_coll: 0 dev.igb.2.queue7.lro_flushed: 0 dev.igb.2.queue7.lro_queued: 0 dev.igb.2.queue7.rx_bytes: 0 dev.igb.2.queue7.rx_packets: 0 dev.igb.2.queue7.rxd_tail: 0 dev.igb.2.queue7.rxd_head: 0 dev.igb.2.queue7.tx_packets: 0 dev.igb.2.queue7.no_desc_avail: 0 dev.igb.2.queue7.txd_tail: 0 dev.igb.2.queue7.txd_head: 0 dev.igb.2.queue7.interrupt_rate: 0 dev.igb.2.queue6.lro_flushed: 0 dev.igb.2.queue6.lro_queued: 0 dev.igb.2.queue6.rx_bytes: 0 dev.igb.2.queue6.rx_packets: 0 dev.igb.2.queue6.rxd_tail: 0 dev.igb.2.queue6.rxd_head: 0 dev.igb.2.queue6.tx_packets: 0 dev.igb.2.queue6.no_desc_avail: 0 dev.igb.2.queue6.txd_tail: 0 dev.igb.2.queue6.txd_head: 0 dev.igb.2.queue6.interrupt_rate: 0 dev.igb.2.queue5.lro_flushed: 0 dev.igb.2.queue5.lro_queued: 0 dev.igb.2.queue5.rx_bytes: 0 dev.igb.2.queue5.rx_packets: 0 dev.igb.2.queue5.rxd_tail: 0 dev.igb.2.queue5.rxd_head: 0 dev.igb.2.queue5.tx_packets: 0 dev.igb.2.queue5.no_desc_avail: 0 dev.igb.2.queue5.txd_tail: 0 dev.igb.2.queue5.txd_head: 0 dev.igb.2.queue5.interrupt_rate: 0 dev.igb.2.queue4.lro_flushed: 0 dev.igb.2.queue4.lro_queued: 0 dev.igb.2.queue4.rx_bytes: 0 dev.igb.2.queue4.rx_packets: 0 dev.igb.2.queue4.rxd_tail: 0 dev.igb.2.queue4.rxd_head: 0 dev.igb.2.queue4.tx_packets: 0 dev.igb.2.queue4.no_desc_avail: 0 dev.igb.2.queue4.txd_tail: 0 dev.igb.2.queue4.txd_head: 0 dev.igb.2.queue4.interrupt_rate: 0 dev.igb.2.queue3.lro_flushed: 0 dev.igb.2.queue3.lro_queued: 0 dev.igb.2.queue3.rx_bytes: 0 dev.igb.2.queue3.rx_packets: 0 dev.igb.2.queue3.rxd_tail: 0 dev.igb.2.queue3.rxd_head: 0 dev.igb.2.queue3.tx_packets: 0 dev.igb.2.queue3.no_desc_avail: 0 dev.igb.2.queue3.txd_tail: 0 dev.igb.2.queue3.txd_head: 0 dev.igb.2.queue3.interrupt_rate: 0 dev.igb.2.queue2.lro_flushed: 0 dev.igb.2.queue2.lro_queued: 0 dev.igb.2.queue2.rx_bytes: 0 dev.igb.2.queue2.rx_packets: 0 dev.igb.2.queue2.rxd_tail: 0 dev.igb.2.queue2.rxd_head: 0 dev.igb.2.queue2.tx_packets: 0 dev.igb.2.queue2.no_desc_avail: 0 dev.igb.2.queue2.txd_tail: 0 dev.igb.2.queue2.txd_head: 0 dev.igb.2.queue2.interrupt_rate: 0 dev.igb.2.queue1.lro_flushed: 0 dev.igb.2.queue1.lro_queued: 0 dev.igb.2.queue1.rx_bytes: 0 dev.igb.2.queue1.rx_packets: 0 dev.igb.2.queue1.rxd_tail: 0 dev.igb.2.queue1.rxd_head: 0 dev.igb.2.queue1.tx_packets: 0 dev.igb.2.queue1.no_desc_avail: 0 dev.igb.2.queue1.txd_tail: 0 dev.igb.2.queue1.txd_head: 0 dev.igb.2.queue1.interrupt_rate: 0 dev.igb.2.queue0.lro_flushed: 0 dev.igb.2.queue0.lro_queued: 0 dev.igb.2.queue0.rx_bytes: 0 dev.igb.2.queue0.rx_packets: 0 dev.igb.2.queue0.rxd_tail: 0 dev.igb.2.queue0.rxd_head: 0 dev.igb.2.queue0.tx_packets: 0 dev.igb.2.queue0.no_desc_avail: 0 dev.igb.2.queue0.txd_tail: 0 dev.igb.2.queue0.txd_head: 0 dev.igb.2.queue0.interrupt_rate: 0 dev.igb.2.fc_low_water: 33152 dev.igb.2.fc_high_water: 33168 dev.igb.2.rx_buf_alloc: 0 dev.igb.2.tx_buf_alloc: 0 dev.igb.2.extended_int_mask: 2147483648 dev.igb.2.interrupt_mask: 0 dev.igb.2.rx_control: 0 dev.igb.2.device_control: 136053313 dev.igb.2.watchdog_timeouts: 0 dev.igb.2.rx_overruns: 0 dev.igb.2.tx_dma_fail: 0 dev.igb.2.mbuf_defrag_fail: 0 dev.igb.2.link_irq: 0 dev.igb.2.dropped: 0 dev.igb.2.eee_disabled: 0 dev.igb.2.dmac: 0 dev.igb.2.tx_processing_limit: -1 dev.igb.2.rx_processing_limit: 100 dev.igb.2.fc: 3 dev.igb.2.enable_aim: 1 dev.igb.2.nvm: -1 dev.igb.2.%parent: pci2 dev.igb.2.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1b12 class=0x020000 dev.igb.2.%location: slot=0 function=2 dbsf=pci0:2:0:2 dev.igb.2.%driver: igb dev.igb.2.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k dev.igb.1.host.header_redir_missed: 0 dev.igb.1.host.serdes_violation_pkt: 0 dev.igb.1.host.length_errors: 0 dev.igb.1.host.tx_good_bytes: 387524 dev.igb.1.host.rx_good_bytes: 58810 dev.igb.1.host.breaker_tx_pkt_drop: 0 dev.igb.1.host.tx_good_pkt: 1 dev.igb.1.host.breaker_rx_pkt_drop: 0 dev.igb.1.host.breaker_rx_pkts: 0 dev.igb.1.host.rx_pkt: 2 dev.igb.1.host.host_tx_pkt_discard: 0 dev.igb.1.host.breaker_tx_pkt: 0 dev.igb.1.interrupts.rx_overrun: 0 dev.igb.1.interrupts.rx_desc_min_thresh: 0 dev.igb.1.interrupts.tx_queue_min_thresh: 0 dev.igb.1.interrupts.tx_queue_empty: 584 dev.igb.1.interrupts.tx_abs_timer: 0 dev.igb.1.interrupts.tx_pkt_timer: 0 dev.igb.1.interrupts.rx_abs_timer: 0 dev.igb.1.interrupts.rx_pkt_timer: 381 dev.igb.1.interrupts.asserts: 3808 dev.igb.1.mac_stats.tso_ctx_fail: 0 dev.igb.1.mac_stats.tso_txd: 0 dev.igb.1.mac_stats.tx_frames_1024_1522: 213 dev.igb.1.mac_stats.tx_frames_512_1023: 25 dev.igb.1.mac_stats.tx_frames_256_511: 87 dev.igb.1.mac_stats.tx_frames_128_255: 9 dev.igb.1.mac_stats.tx_frames_65_127: 239 dev.igb.1.mac_stats.tx_frames_64: 12 dev.igb.1.mac_stats.mcast_pkts_txd: 7 dev.igb.1.mac_stats.bcast_pkts_txd: 5 dev.igb.1.mac_stats.good_pkts_txd: 585 dev.igb.1.mac_stats.total_pkts_txd: 585 dev.igb.1.mac_stats.total_octets_txd: 387524 dev.igb.1.mac_stats.good_octets_txd: 387524 dev.igb.1.mac_stats.total_octets_recvd: 70094 dev.igb.1.mac_stats.good_octets_recvd: 58810 dev.igb.1.mac_stats.rx_frames_1024_1522: 5 dev.igb.1.mac_stats.rx_frames_512_1023: 8 dev.igb.1.mac_stats.rx_frames_256_511: 17 dev.igb.1.mac_stats.rx_frames_128_255: 124 dev.igb.1.mac_stats.rx_frames_65_127: 217 dev.igb.1.mac_stats.rx_frames_64: 12 dev.igb.1.mac_stats.mcast_pkts_recvd: 20 dev.igb.1.mac_stats.bcast_pkts_recvd: 12 dev.igb.1.mac_stats.good_pkts_recvd: 383 dev.igb.1.mac_stats.total_pkts_recvd: 411 dev.igb.1.mac_stats.mgmt_pkts_txd: 0 dev.igb.1.mac_stats.mgmt_pkts_drop: 0 dev.igb.1.mac_stats.mgmt_pkts_recvd: 0 dev.igb.1.mac_stats.unsupported_fc_recvd: 0 dev.igb.1.mac_stats.xoff_txd: 0 dev.igb.1.mac_stats.xoff_recvd: 0 dev.igb.1.mac_stats.xon_txd: 0 dev.igb.1.mac_stats.xon_recvd: 0 dev.igb.1.mac_stats.coll_ext_errs: 0 dev.igb.1.mac_stats.tx_no_crs: 0 dev.igb.1.mac_stats.alignment_errs: 0 dev.igb.1.mac_stats.crc_errs: 0 dev.igb.1.mac_stats.recv_errs: 0 dev.igb.1.mac_stats.recv_jabber: 0 dev.igb.1.mac_stats.recv_oversize: 0 dev.igb.1.mac_stats.recv_fragmented: 0 dev.igb.1.mac_stats.recv_undersize: 0 dev.igb.1.mac_stats.recv_no_buff: 0 dev.igb.1.mac_stats.recv_length_errors: 0 dev.igb.1.mac_stats.missed_packets: 0 dev.igb.1.mac_stats.defer_count: 0 dev.igb.1.mac_stats.sequence_errors: 0 dev.igb.1.mac_stats.symbol_errors: 0 dev.igb.1.mac_stats.collision_count: 0 dev.igb.1.mac_stats.late_coll: 0 dev.igb.1.mac_stats.multiple_coll: 0 dev.igb.1.mac_stats.single_coll: 0 dev.igb.1.mac_stats.excess_coll: 0 dev.igb.1.queue7.lro_flushed: 0 dev.igb.1.queue7.lro_queued: 0 dev.igb.1.queue7.rx_bytes: 22234 dev.igb.1.queue7.rx_packets: 154 dev.igb.1.queue7.rxd_tail: 153 dev.igb.1.queue7.rxd_head: 154 dev.igb.1.queue7.tx_packets: 0 dev.igb.1.queue7.no_desc_avail: 0 dev.igb.1.queue7.txd_tail: 0 dev.igb.1.queue7.txd_head: 0 dev.igb.1.queue7.interrupt_rate: 62500 dev.igb.1.queue6.lro_flushed: 0 dev.igb.1.queue6.lro_queued: 0 dev.igb.1.queue6.rx_bytes: 4146 dev.igb.1.queue6.rx_packets: 22 dev.igb.1.queue6.rxd_tail: 21 dev.igb.1.queue6.rxd_head: 22 dev.igb.1.queue6.tx_packets: 0 dev.igb.1.queue6.no_desc_avail: 0 dev.igb.1.queue6.txd_tail: 0 dev.igb.1.queue6.txd_head: 0 dev.igb.1.queue6.interrupt_rate: 76923 dev.igb.1.queue5.lro_flushed: 0 dev.igb.1.queue5.lro_queued: 0 dev.igb.1.queue5.rx_bytes: 120 dev.igb.1.queue5.rx_packets: 2 dev.igb.1.queue5.rxd_tail: 1 dev.igb.1.queue5.rxd_head: 2 dev.igb.1.queue5.tx_packets: 0 dev.igb.1.queue5.no_desc_avail: 0 dev.igb.1.queue5.txd_tail: 0 dev.igb.1.queue5.txd_head: 0 dev.igb.1.queue5.interrupt_rate: 100000 dev.igb.1.queue4.lro_flushed: 0 dev.igb.1.queue4.lro_queued: 0 dev.igb.1.queue4.rx_bytes: 254 dev.igb.1.queue4.rx_packets: 4 dev.igb.1.queue4.rxd_tail: 3 dev.igb.1.queue4.rxd_head: 4 dev.igb.1.queue4.tx_packets: 0 dev.igb.1.queue4.no_desc_avail: 0 dev.igb.1.queue4.txd_tail: 0 dev.igb.1.queue4.txd_head: 0 dev.igb.1.queue4.interrupt_rate: 100000 dev.igb.1.queue3.lro_flushed: 0 dev.igb.1.queue3.lro_queued: 0 dev.igb.1.queue3.rx_bytes: 24026 dev.igb.1.queue3.rx_packets: 144 dev.igb.1.queue3.rxd_tail: 143 dev.igb.1.queue3.rxd_head: 144 dev.igb.1.queue3.tx_packets: 0 dev.igb.1.queue3.no_desc_avail: 0 dev.igb.1.queue3.txd_tail: 0 dev.igb.1.queue3.txd_head: 0 dev.igb.1.queue3.interrupt_rate: 6024 dev.igb.1.queue2.lro_flushed: 0 dev.igb.1.queue2.lro_queued: 0 dev.igb.1.queue2.rx_bytes: 3022 dev.igb.1.queue2.rx_packets: 14 dev.igb.1.queue2.rxd_tail: 13 dev.igb.1.queue2.rxd_head: 14 dev.igb.1.queue2.tx_packets: 0 dev.igb.1.queue2.no_desc_avail: 0 dev.igb.1.queue2.txd_tail: 0 dev.igb.1.queue2.txd_head: 0 dev.igb.1.queue2.interrupt_rate: 29411 dev.igb.1.queue1.lro_flushed: 0 dev.igb.1.queue1.lro_queued: 0 dev.igb.1.queue1.rx_bytes: 874 dev.igb.1.queue1.rx_packets: 7 dev.igb.1.queue1.rxd_tail: 6 dev.igb.1.queue1.rxd_head: 7 dev.igb.1.queue1.tx_packets: 0 dev.igb.1.queue1.no_desc_avail: 0 dev.igb.1.queue1.txd_tail: 0 dev.igb.1.queue1.txd_head: 0 dev.igb.1.queue1.interrupt_rate: 66666 dev.igb.1.queue0.lro_flushed: 0 dev.igb.1.queue0.lro_queued: 0 dev.igb.1.queue0.rx_bytes: 4193 dev.igb.1.queue0.rx_packets: 39 dev.igb.1.queue0.rxd_tail: 38 dev.igb.1.queue0.rxd_head: 39 dev.igb.1.queue0.tx_packets: 588 dev.igb.1.queue0.no_desc_avail: 0 dev.igb.1.queue0.txd_tail: 911 dev.igb.1.queue0.txd_head: 911 dev.igb.1.queue0.interrupt_rate: 90909 dev.igb.1.fc_low_water: 33152 dev.igb.1.fc_high_water: 33168 dev.igb.1.rx_buf_alloc: 0 dev.igb.1.tx_buf_alloc: 0 dev.igb.1.extended_int_mask: 2147484159 dev.igb.1.interrupt_mask: 4 dev.igb.1.rx_control: 67141634 dev.igb.1.device_control: 1478230593 dev.igb.1.watchdog_timeouts: 0 dev.igb.1.rx_overruns: 0 dev.igb.1.tx_dma_fail: 0 dev.igb.1.mbuf_defrag_fail: 0 dev.igb.1.link_irq: 4 dev.igb.1.dropped: 0 dev.igb.1.eee_disabled: 0 dev.igb.1.dmac: 0 dev.igb.1.tx_processing_limit: -1 dev.igb.1.rx_processing_limit: 100 dev.igb.1.fc: 3 dev.igb.1.enable_aim: 1 dev.igb.1.nvm: -1 dev.igb.1.%parent: pci2 dev.igb.1.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1b12 class=0x020000 dev.igb.1.%location: slot=0 function=1 dbsf=pci0:2:0:1 dev.igb.1.%driver: igb dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k dev.igb.0.wake: 0 dev.igb.0.host.header_redir_missed: 0 dev.igb.0.host.serdes_violation_pkt: 0 dev.igb.0.host.length_errors: 0 dev.igb.0.host.tx_good_bytes: 1710353 dev.igb.0.host.rx_good_bytes: 377365 dev.igb.0.host.breaker_tx_pkt_drop: 0 dev.igb.0.host.tx_good_pkt: 0 dev.igb.0.host.breaker_rx_pkt_drop: 0 dev.igb.0.host.breaker_rx_pkts: 0 dev.igb.0.host.rx_pkt: 0 dev.igb.0.host.host_tx_pkt_discard: 0 dev.igb.0.host.breaker_tx_pkt: 0 dev.igb.0.interrupts.rx_overrun: 0 dev.igb.0.interrupts.rx_desc_min_thresh: 0 dev.igb.0.interrupts.tx_queue_min_thresh: 23214 dev.igb.0.interrupts.tx_queue_empty: 1413 dev.igb.0.interrupts.tx_abs_timer: 0 dev.igb.0.interrupts.tx_pkt_timer: 0 dev.igb.0.interrupts.rx_abs_timer: 0 dev.igb.0.interrupts.rx_pkt_timer: 4886 dev.igb.0.interrupts.asserts: 8193 dev.igb.0.mac_stats.tso_ctx_fail: 0 dev.igb.0.mac_stats.tso_txd: 0 dev.igb.0.mac_stats.tx_frames_1024_1522: 1093 dev.igb.0.mac_stats.tx_frames_512_1023: 30 dev.igb.0.mac_stats.tx_frames_256_511: 30 dev.igb.0.mac_stats.tx_frames_128_255: 10 dev.igb.0.mac_stats.tx_frames_65_127: 42 dev.igb.0.mac_stats.tx_frames_64: 208 dev.igb.0.mac_stats.mcast_pkts_txd: 1 dev.igb.0.mac_stats.bcast_pkts_txd: 10 dev.igb.0.mac_stats.good_pkts_txd: 1413 dev.igb.0.mac_stats.total_pkts_txd: 1413 dev.igb.0.mac_stats.total_octets_txd: 1710353 dev.igb.0.mac_stats.good_octets_txd: 1710353 dev.igb.0.mac_stats.total_octets_recvd: 384601 dev.igb.0.mac_stats.good_octets_recvd: 377365 dev.igb.0.mac_stats.rx_frames_1024_1522: 3 dev.igb.0.mac_stats.rx_frames_512_1023: 1 dev.igb.0.mac_stats.rx_frames_256_511: 22 dev.igb.0.mac_stats.rx_frames_128_255: 385 dev.igb.0.mac_stats.rx_frames_65_127: 1331 dev.igb.0.mac_stats.rx_frames_64: 3144 dev.igb.0.mac_stats.mcast_pkts_recvd: 412 dev.igb.0.mac_stats.bcast_pkts_recvd: 2902 dev.igb.0.mac_stats.good_pkts_recvd: 4886 dev.igb.0.mac_stats.total_pkts_recvd: 4902 dev.igb.0.mac_stats.mgmt_pkts_txd: 0 dev.igb.0.mac_stats.mgmt_pkts_drop: 0 dev.igb.0.mac_stats.mgmt_pkts_recvd: 0 dev.igb.0.mac_stats.unsupported_fc_recvd: 0 dev.igb.0.mac_stats.xoff_txd: 0 dev.igb.0.mac_stats.xoff_recvd: 0 dev.igb.0.mac_stats.xon_txd: 0 dev.igb.0.mac_stats.xon_recvd: 0 dev.igb.0.mac_stats.coll_ext_errs: 0 dev.igb.0.mac_stats.tx_no_crs: 0 dev.igb.0.mac_stats.alignment_errs: 0 dev.igb.0.mac_stats.crc_errs: 0 dev.igb.0.mac_stats.recv_errs: 0 dev.igb.0.mac_stats.recv_jabber: 0 dev.igb.0.mac_stats.recv_oversize: 0 dev.igb.0.mac_stats.recv_fragmented: 0 dev.igb.0.mac_stats.recv_undersize: 0 dev.igb.0.mac_stats.recv_no_buff: 0 dev.igb.0.mac_stats.recv_length_errors: 0 dev.igb.0.mac_stats.missed_packets: 0 dev.igb.0.mac_stats.defer_count: 0 dev.igb.0.mac_stats.sequence_errors: 0 dev.igb.0.mac_stats.symbol_errors: 0 dev.igb.0.mac_stats.collision_count: 0 dev.igb.0.mac_stats.late_coll: 0 dev.igb.0.mac_stats.multiple_coll: 0 dev.igb.0.mac_stats.single_coll: 0 dev.igb.0.mac_stats.excess_coll: 0 dev.igb.0.queue7.lro_flushed: 0 dev.igb.0.queue7.lro_queued: 0 dev.igb.0.queue7.rx_bytes: 61172 dev.igb.0.queue7.rx_packets: 437 dev.igb.0.queue7.rxd_tail: 436 dev.igb.0.queue7.rxd_head: 437 dev.igb.0.queue7.tx_packets: 0 dev.igb.0.queue7.no_desc_avail: 0 dev.igb.0.queue7.txd_tail: 0 dev.igb.0.queue7.txd_head: 0 dev.igb.0.queue7.interrupt_rate: 50000 dev.igb.0.queue6.lro_flushed: 0 dev.igb.0.queue6.lro_queued: 0 dev.igb.0.queue6.rx_bytes: 6689 dev.igb.0.queue6.rx_packets: 78 dev.igb.0.queue6.rxd_tail: 77 dev.igb.0.queue6.rxd_head: 78 dev.igb.0.queue6.tx_packets: 0 dev.igb.0.queue6.no_desc_avail: 0 dev.igb.0.queue6.txd_tail: 0 dev.igb.0.queue6.txd_head: 0 dev.igb.0.queue6.interrupt_rate: 90909 dev.igb.0.queue5.lro_flushed: 0 dev.igb.0.queue5.lro_queued: 0 dev.igb.0.queue5.rx_bytes: 79329 dev.igb.0.queue5.rx_packets: 955 dev.igb.0.queue5.rxd_tail: 954 dev.igb.0.queue5.rxd_head: 955 dev.igb.0.queue5.tx_packets: 0 dev.igb.0.queue5.no_desc_avail: 0 dev.igb.0.queue5.txd_tail: 0 dev.igb.0.queue5.txd_head: 0 dev.igb.0.queue5.interrupt_rate: 90909 dev.igb.0.queue4.lro_flushed: 0 dev.igb.0.queue4.lro_queued: 0 dev.igb.0.queue4.rx_bytes: 2661 dev.igb.0.queue4.rx_packets: 39 dev.igb.0.queue4.rxd_tail: 38 dev.igb.0.queue4.rxd_head: 39 dev.igb.0.queue4.tx_packets: 0 dev.igb.0.queue4.no_desc_avail: 0 dev.igb.0.queue4.txd_tail: 0 dev.igb.0.queue4.txd_head: 0 dev.igb.0.queue4.interrupt_rate: 76923 dev.igb.0.queue3.lro_flushed: 0 dev.igb.0.queue3.lro_queued: 0 dev.igb.0.queue3.rx_bytes: 4055 dev.igb.0.queue3.rx_packets: 60 dev.igb.0.queue3.rxd_tail: 59 dev.igb.0.queue3.rxd_head: 60 dev.igb.0.queue3.tx_packets: 0 dev.igb.0.queue3.no_desc_avail: 0 dev.igb.0.queue3.txd_tail: 0 dev.igb.0.queue3.txd_head: 0 dev.igb.0.queue3.interrupt_rate: 90909 dev.igb.0.queue2.lro_flushed: 0 dev.igb.0.queue2.lro_queued: 0 dev.igb.0.queue2.rx_bytes: 7874 dev.igb.0.queue2.rx_packets: 107 dev.igb.0.queue2.rxd_tail: 106 dev.igb.0.queue2.rxd_head: 107 dev.igb.0.queue2.tx_packets: 0 dev.igb.0.queue2.no_desc_avail: 0 dev.igb.0.queue2.txd_tail: 0 dev.igb.0.queue2.txd_head: 0 dev.igb.0.queue2.interrupt_rate: 90909 dev.igb.0.queue1.lro_flushed: 0 dev.igb.0.queue1.lro_queued: 0 dev.igb.0.queue1.rx_bytes: 7373 dev.igb.0.queue1.rx_packets: 69 dev.igb.0.queue1.rxd_tail: 68 dev.igb.0.queue1.rxd_head: 69 dev.igb.0.queue1.tx_packets: 0 dev.igb.0.queue1.no_desc_avail: 0 dev.igb.0.queue1.txd_tail: 0 dev.igb.0.queue1.txd_head: 0 dev.igb.0.queue1.interrupt_rate: 90909 dev.igb.0.queue0.lro_flushed: 0 dev.igb.0.queue0.lro_queued: 0 dev.igb.0.queue0.rx_bytes: 189284 dev.igb.0.queue0.rx_packets: 3150 dev.igb.0.queue0.rxd_tail: 77 dev.igb.0.queue0.rxd_head: 78 dev.igb.0.queue0.tx_packets: 1413 dev.igb.0.queue0.no_desc_avail: 0 dev.igb.0.queue0.txd_tail: 393 dev.igb.0.queue0.txd_head: 393 dev.igb.0.queue0.interrupt_rate: 100000 dev.igb.0.fc_low_water: 33152 dev.igb.0.fc_high_water: 33168 dev.igb.0.rx_buf_alloc: 0 dev.igb.0.tx_buf_alloc: 0 dev.igb.0.extended_int_mask: 2147484159 dev.igb.0.interrupt_mask: 4 dev.igb.0.rx_control: 67141634 dev.igb.0.device_control: 1478230593 dev.igb.0.watchdog_timeouts: 0 dev.igb.0.rx_overruns: 0 dev.igb.0.tx_dma_fail: 0 dev.igb.0.mbuf_defrag_fail: 0 dev.igb.0.link_irq: 4 dev.igb.0.dropped: 0 dev.igb.0.eee_disabled: 0 dev.igb.0.dmac: 0 dev.igb.0.tx_processing_limit: -1 dev.igb.0.rx_processing_limit: 100 dev.igb.0.fc: 3 dev.igb.0.enable_aim: 1 dev.igb.0.nvm: -1 dev.igb.0.%parent: pci2 dev.igb.0.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1b12 class=0x020000 dev.igb.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 handle=\_SB_.PCI0.RP05.PXSX dev.igb.0.%driver: igb dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k dev.igb.%parent:sysctl -a | grep netmap output -
device netmap dev.netmap.ixl_rx_miss_bufs: 0 dev.netmap.ixl_rx_miss: 0 dev.netmap.iflib_rx_miss_bufs: 0 dev.netmap.iflib_rx_miss: 0 dev.netmap.iflib_crcstrip: 1 dev.netmap.bridge_batch: 1024 dev.netmap.default_pipes: 0 dev.netmap.priv_buf_num: 4098 dev.netmap.priv_buf_size: 2048 dev.netmap.buf_curr_num: 163840 dev.netmap.buf_num: 163840 dev.netmap.buf_curr_size: 2048 dev.netmap.buf_size: 2048 dev.netmap.priv_ring_num: 4 dev.netmap.priv_ring_size: 20480 dev.netmap.ring_curr_num: 200 dev.netmap.ring_num: 200 dev.netmap.ring_curr_size: 36864 dev.netmap.ring_size: 36864 dev.netmap.priv_if_num: 1 dev.netmap.priv_if_size: 1024 dev.netmap.if_curr_num: 100 dev.netmap.if_num: 100 dev.netmap.if_curr_size: 1024 dev.netmap.if_size: 1024 dev.netmap.generic_rings: 1 dev.netmap.generic_ringsize: 1024 dev.netmap.generic_mit: 100000 dev.netmap.admode: 0 dev.netmap.fwd: 0 dev.netmap.flags: 0 dev.netmap.adaptive_io: 0 dev.netmap.txsync_retry: 2 dev.netmap.no_pendintr: 1 dev.netmap.mitigate: 1 dev.netmap.no_timestamp: 0 dev.netmap.verbose: 0 dev.netmap.ix_rx_miss_bufs: 0 dev.netmap.ix_rx_miss: 0 dev.netmap.ix_crcstrip: 0 -
bmeeks...Thank you for the Suricata rules explanation. I did a little further digging, and even though I haven't been able to read all of it yet, I found a post of yours:
https://forum.netgate.com/topic/128480/how-automatic-sid-management-and-user-rule-overrides-work-in-snort-and-suricata -
@boobletins said in Suricata InLine with igb NICs:
So here are some initial suggestions. Please keep in mind that I've been working on this for ~1 week (in other words: not long), and I'm not a FreeBSD, pfSense, or Suricata expert.
Start by making a backup of your configuration.
Do these first:
My understanding is that flow control should be off on any netmap interface. You have bi-directional flow control enabled:dev.igb.0.fc: 3Disable flow control on all active interfaces using system tunables. Set dev.igb.0.fc=0 (and dev.igb.1.fc=0)
Actively set energy efficient ethernet to disabled:
dev.igb.0.eee_disabled=1Actively force IPv6_TXCSUM6 off by adding the following to config.xml in a shellcmd tag:
ifconfig igb0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso(see above in this thread for a link on where/how to do that).
Edit:
To be clear: anywhere I have a command that says "igb0" or "igb.0" you will want to duplicate that for igb1 and any other interface you're running netmap on.So you will need 2 shellcmd lines in config.xml, and two new system tunables for flow control, etc
Consider changing later:
Set rx processing limit:
dev.igb.0.rx_processing_limit: -1It looks like your txd and rxd are both set to 1024 currently, I suggest you move those to 4096:
hw.igb.txd=4096
hw.igb.rxd=4096By changing your txd and rxd we may need to revisit your netmap buf/ring (memory settings).
We may also revisit your interrupt and queue settings.
Boobletins, I will need to revisit later...currently, I am happy with just making adjustments to the buf_size:4096 and disable IPv6...haven't got any alert since and my Internet will be down for a while because of moving.
-
So you're running netmap/IPS mode on igb0 (LAN), igb1 (OPT?), and igb3 (WAN)?
What type of CPU is in the machine (# of cores?, is hyper-threading enabled)? How much RAM?
Are you saturating all 3 active interfaces? Or just 2?
Start by making a backup of your configuration.
First disable flow control (as discussed above):
You have the following on all igb interfaces which means bi-directional flow control is enabled.:dev.igb.0.fc: 3Change to fc=0 on all netmap interfaces in system tunables. This will take ethernet flow control out of the picture in favor of higher level flow control (TCP) which is less likely to mess with buffering and clog things up.
Let's look at what generates this particular netmap error:
From http://web.mit.edu/freebsd/head/sys/dev/netmap/netmap.c/* * put a copy of the buffers marked NS_FORWARD into an mbuf chain. * Take packets from hwcur to ring->head marked NS_FORWARD (or forced) * and pass them up. Drop remaining packets in the unlikely event * of an mbuf shortage. */ static void netmap_grab_packets(struct netmap_kring *kring, struct mbq *q, int force) { u_int const lim = kring->nkr_num_slots - 1; u_int const head = kring->ring->head; u_int n; struct netmap_adapter *na = kring->na; for (n = kring->nr_hwcur; n != head; n = nm_next(n, lim)) { struct mbuf *m; struct netmap_slot *slot = &kring->ring->slot[n]; if ((slot->flags & NS_FORWARD) == 0 && !force) continue; if (slot->len < 14 || slot->len > NETMAP_BUF_SIZE(na)) { RD(5, "bad pkt at %d len %d", n, slot->len); continue; } slot->flags &= ~NS_FORWARD; // XXX needed ? /* XXX TODO: adapt to the case of a multisegment packet */ m = m_devget(NMB(na, slot), slot->len, 0, na->ifp, NULL); if (m == NULL) break; mbq_enqueue(q, m); } }I'm no C expert, but as I read this code there are 2 ways to generate your error in netmap:
- a slot is of size less than 14
- a slot is of size greater than the netmap buffer can handle
I don't know what the magic number 14 represents, but let's assume it's some kind of minimum packet size we can't control. If that's the case, then the bad_pkt error is generated from packets that are actually bad.
That's not what you have. The error is telling us the current hwcur value (the first number - the slot number in the ring) and the length or size of the slot (eg #777 with len 2154).
So this is a memory issue. The error would be better off saying something like "dropped a packet because it was too short or too large!" -- but that would be useful to others and is thus verboten ;)
edited: Removed incorrect speculation. Skip to my latest post.
Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.