Well, I made a progress on incoming errors..
These errors accumulated rapidly and while doing no observable harm. But about a week ago situation became worse. On a month of uptime system freaked out, error counter went through the roof and WAN went down. I rebooted.. And it repeated 3 times during 1st or 2nd 24 hours of uptime since then (under load). And that's when I started playing with hardware and poking drivers with a stick. So, the results:
EXPI9301CT NIC itself is working fine, I tried swapping with the same exact model - same problems. Missed packets and "recv_no_buff" counters. Even on igb card, while this one is installed. So it should be driver (em) or this specific model.
Changing model, at first I used I217-LM (one of onboard NICs, em driver) for WAN passthrough. According to vSphere Client it has own pci lane, unlike other one, which seems to be interconnected through PCH. "That one should be superior," - I thought - "as long as they are pretty similar looking on Intel ARK". As a result, there was no incrementing error counters from the start on any interface, but during 2nd 24 hours of "stressing" the system with 80 mbps on average, WAN timed out for couple seconds (I actually experienced it), interface dropped 923 errors and continued functioning. Flimsy.. Although no errors on dev.igb. So, again should be driver (em) or this specific model (if momentary glitch was unrelated)
Then I switched WAN to another onboard NIC - I210 (igb driver). No incoming errors, no missed packets. Same on fiber card. I pushed roughly the same amount of traffic through system in 2 days which initially brought my WAN down after the month of runtime. System still stable. Missed packets and "recv_no_buff" at 0 on all NICs.
To sum up, I still don't know whether it's the specific model that causes issue during passthrough (while it is actually functional and now acts as a secondary LACP port for ESXi management in place of onboard one that is now passed to pfSense for WAN) or the em driver and there could be other em NICs having same issue with passthrough or even the 82574L chip has something to do in hardware. EXPI9301CT is a very old model meant for desktops, not virtual environments, although it is still present in Intel's 2016 catalogue. And also given that em dirver in FreeBSD man pages is dated October 11, 2011 while igb is of March 25, 2013, I just made a conclusion for myself that later is just more stable and Intel NICs are not bulletproof as I thought.
[image: XREfaHC.jpg]
I don't know what are these outgoing errors.. I recall having them all the time, even on bare metal, even on other platform. There is literally no info I managed to find about any possible causes of them or at least the meaning of these counters, so if someone may enlighten me, please do.. At least, they seem to not be as harmless.
$ sysctl dev.igb
dev.igb.%parent:
dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0
dev.igb.0.%driver: igb
dev.igb.0.%location: slot=0 function=0 handle=\_SB_.PCI0.PE40.S1F0
dev.igb.0.%pnpinfo: vendor=0x8086 device=0x1533 subvendor=0x15d9 subdevice=0x1533 class=0x020000
dev.igb.0.%parent: pci3
dev.igb.0.nvm: -1
dev.igb.0.enable_aim: 1
dev.igb.0.fc: 3
dev.igb.0.rx_processing_limit: 100
dev.igb.0.dmac: 0
dev.igb.0.eee_disabled: 0
dev.igb.0.link_irq: 0
dev.igb.0.dropped: 0
dev.igb.0.tx_dma_fail: 0
dev.igb.0.rx_overruns: 0
dev.igb.0.watchdog_timeouts: 0
dev.igb.0.device_control: 1478230593
dev.igb.0.rx_control: 71335938
dev.igb.0.interrupt_mask: 157
dev.igb.0.extended_int_mask: 2147483648
dev.igb.0.tx_buf_alloc: 0
dev.igb.0.rx_buf_alloc: 0
dev.igb.0.fc_high_water: 31328
dev.igb.0.fc_low_water: 31312
dev.igb.0.queue0.no_desc_avail: 0
dev.igb.0.queue0.tx_packets: 620869912
dev.igb.0.queue0.rx_packets: 618707893
dev.igb.0.queue0.rx_bytes: 851127332692
dev.igb.0.queue0.lro_queued: 0
dev.igb.0.queue0.lro_flushed: 0
dev.igb.0.mac_stats.excess_coll: 0
dev.igb.0.mac_stats.single_coll: 0
dev.igb.0.mac_stats.multiple_coll: 0
dev.igb.0.mac_stats.late_coll: 0
dev.igb.0.mac_stats.collision_count: 0
dev.igb.0.mac_stats.symbol_errors: 0
dev.igb.0.mac_stats.sequence_errors: 0
dev.igb.0.mac_stats.defer_count: 0
dev.igb.0.mac_stats.missed_packets: 0
dev.igb.0.mac_stats.recv_no_buff: 0
dev.igb.0.mac_stats.recv_undersize: 0
dev.igb.0.mac_stats.recv_fragmented: 0
dev.igb.0.mac_stats.recv_oversize: 0
dev.igb.0.mac_stats.recv_jabber: 0
dev.igb.0.mac_stats.recv_errs: 0
dev.igb.0.mac_stats.crc_errs: 0
dev.igb.0.mac_stats.alignment_errs: 0
dev.igb.0.mac_stats.coll_ext_errs: 0
dev.igb.0.mac_stats.xon_recvd: 0
dev.igb.0.mac_stats.xon_txd: 0
dev.igb.0.mac_stats.xoff_recvd: 0
dev.igb.0.mac_stats.xoff_txd: 0
dev.igb.0.mac_stats.total_pkts_recvd: 618715218
dev.igb.0.mac_stats.good_pkts_recvd: 618707618
dev.igb.0.mac_stats.bcast_pkts_recvd: 1428
dev.igb.0.mac_stats.mcast_pkts_recvd: 3090
dev.igb.0.mac_stats.rx_frames_64: 601133
dev.igb.0.mac_stats.rx_frames_65_127: 13638243
dev.igb.0.mac_stats.rx_frames_128_255: 10436929
dev.igb.0.mac_stats.rx_frames_256_511: 9534739
dev.igb.0.mac_stats.rx_frames_512_1023: 13436838
dev.igb.0.mac_stats.rx_frames_1024_1522: 571059736
dev.igb.0.mac_stats.good_octets_recvd: 853601812367
dev.igb.0.mac_stats.good_octets_txd: 71259681700
dev.igb.0.mac_stats.total_pkts_txd: 620869651
dev.igb.0.mac_stats.good_pkts_txd: 620869651
dev.igb.0.mac_stats.bcast_pkts_txd: 3
dev.igb.0.mac_stats.mcast_pkts_txd: 3
dev.igb.0.mac_stats.tx_frames_64: 3612420
dev.igb.0.mac_stats.tx_frames_65_127: 560906135
dev.igb.0.mac_stats.tx_frames_128_255: 28742303
dev.igb.0.mac_stats.tx_frames_256_511: 8907879
dev.igb.0.mac_stats.tx_frames_512_1023: 4722579
dev.igb.0.mac_stats.tx_frames_1024_1522: 13978335
dev.igb.0.mac_stats.tso_txd: 0
dev.igb.0.mac_stats.tso_ctx_fail: 0
dev.igb.0.interrupts.asserts: 965490770
dev.igb.0.interrupts.rx_pkt_timer: 618687254
dev.igb.0.interrupts.rx_abs_timer: 0
dev.igb.0.interrupts.tx_pkt_timer: 0
dev.igb.0.interrupts.tx_abs_timer: 0
dev.igb.0.interrupts.tx_queue_empty: 620853924
dev.igb.0.interrupts.tx_queue_min_thresh: 618707618
dev.igb.0.interrupts.rx_desc_min_thresh: 0
dev.igb.0.interrupts.rx_overrun: 0
dev.igb.0.host.breaker_tx_pkt: 0
dev.igb.0.host.host_tx_pkt_discard: 0
dev.igb.0.host.rx_pkt: 20364
dev.igb.0.host.breaker_rx_pkts: 0
dev.igb.0.host.breaker_rx_pkt_drop: 0
dev.igb.0.host.tx_good_pkt: 15727
dev.igb.0.host.breaker_tx_pkt_drop: 0
dev.igb.0.host.rx_good_bytes: 853601838827
dev.igb.0.host.tx_good_bytes: 71259681700
dev.igb.0.host.length_errors: 0
dev.igb.0.host.serdes_violation_pkt: 0
dev.igb.0.host.header_redir_missed: 0
dev.igb.0.wake: 0
dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0
dev.igb.1.%driver: igb
dev.igb.1.%location: slot=0 function=0 handle=\_SB_.PCI0.PE50.S1F0
dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10e6 subvendor=0x8086 subdevice=0xa02f class=0x020000
dev.igb.1.%parent: pci11
dev.igb.1.nvm: -1
dev.igb.1.enable_aim: 1
dev.igb.1.fc: 3
dev.igb.1.rx_processing_limit: 100
dev.igb.1.link_irq: 0
dev.igb.1.dropped: 0
dev.igb.1.tx_dma_fail: 0
dev.igb.1.rx_overruns: 0
dev.igb.1.watchdog_timeouts: 0
dev.igb.1.device_control: 1087117889
dev.igb.1.rx_control: 67141634
dev.igb.1.interrupt_mask: 157
dev.igb.1.extended_int_mask: 2147483648
dev.igb.1.tx_buf_alloc: 0
dev.igb.1.rx_buf_alloc: 0
dev.igb.1.fc_high_water: 58976
dev.igb.1.fc_low_water: 58960
dev.igb.1.queue0.no_desc_avail: 3
dev.igb.1.queue0.tx_packets: 1749306695
dev.igb.1.queue0.rx_packets: 631652753
dev.igb.1.queue0.rx_bytes: 827196373649
dev.igb.1.queue0.lro_queued: 0
dev.igb.1.queue0.lro_flushed: 0
dev.igb.1.mac_stats.excess_coll: 0
dev.igb.1.mac_stats.single_coll: 0
dev.igb.1.mac_stats.multiple_coll: 0
dev.igb.1.mac_stats.late_coll: 0
dev.igb.1.mac_stats.collision_count: 0
dev.igb.1.mac_stats.symbol_errors: 0
dev.igb.1.mac_stats.sequence_errors: 0
dev.igb.1.mac_stats.defer_count: 0
dev.igb.1.mac_stats.missed_packets: 0
dev.igb.1.mac_stats.recv_no_buff: 0
dev.igb.1.mac_stats.recv_undersize: 0
dev.igb.1.mac_stats.recv_fragmented: 0
dev.igb.1.mac_stats.recv_oversize: 0
dev.igb.1.mac_stats.recv_jabber: 0
dev.igb.1.mac_stats.recv_errs: 0
dev.igb.1.mac_stats.crc_errs: 0
dev.igb.1.mac_stats.alignment_errs: 0
dev.igb.1.mac_stats.coll_ext_errs: 0
dev.igb.1.mac_stats.xon_recvd: 0
dev.igb.1.mac_stats.xon_txd: 0
dev.igb.1.mac_stats.xoff_recvd: 0
dev.igb.1.mac_stats.xoff_txd: 0
dev.igb.1.mac_stats.total_pkts_recvd: 631666455
dev.igb.1.mac_stats.good_pkts_recvd: 631652747
dev.igb.1.mac_stats.bcast_pkts_recvd: 77781
dev.igb.1.mac_stats.mcast_pkts_recvd: 145425
dev.igb.1.mac_stats.rx_frames_64: 356
dev.igb.1.mac_stats.rx_frames_65_127: 81351672
dev.igb.1.mac_stats.rx_frames_128_255: 5213675
dev.igb.1.mac_stats.rx_frames_256_511: 1833018
dev.igb.1.mac_stats.rx_frames_512_1023: 1375058
dev.igb.1.mac_stats.rx_frames_1024_1522: 541878968
dev.igb.1.mac_stats.good_octets_recvd: 832249572479
dev.igb.1.mac_stats.good_octets_txd: 1817163097562
dev.igb.1.mac_stats.total_pkts_txd: 1749306409
dev.igb.1.mac_stats.good_pkts_txd: 1749306409
dev.igb.1.mac_stats.bcast_pkts_txd: 10921
dev.igb.1.mac_stats.mcast_pkts_txd: 408712
dev.igb.1.mac_stats.tx_frames_64: 7075834
dev.igb.1.mac_stats.tx_frames_65_127: 524157938
dev.igb.1.mac_stats.tx_frames_128_255: 17279963
dev.igb.1.mac_stats.tx_frames_256_511: 10565548
dev.igb.1.mac_stats.tx_frames_512_1023: 14796151
dev.igb.1.mac_stats.tx_frames_1024_1522: 1175430975
dev.igb.1.mac_stats.tso_txd: 0
dev.igb.1.mac_stats.tso_ctx_fail: 0
dev.igb.1.interrupts.asserts: 1372961031
dev.igb.1.interrupts.rx_pkt_timer: 631643467
dev.igb.1.interrupts.rx_abs_timer: 0
dev.igb.1.interrupts.tx_pkt_timer: 0
dev.igb.1.interrupts.tx_abs_timer: 631652747
dev.igb.1.interrupts.tx_queue_empty: 1749288623
dev.igb.1.interrupts.tx_queue_min_thresh: 0
dev.igb.1.interrupts.rx_desc_min_thresh: 0
dev.igb.1.interrupts.rx_overrun: 0
dev.igb.1.host.breaker_tx_pkt: 0
dev.igb.1.host.host_tx_pkt_discard: 0
dev.igb.1.host.rx_pkt: 9280
dev.igb.1.host.breaker_rx_pkts: 0
dev.igb.1.host.breaker_rx_pkt_drop: 0
dev.igb.1.host.tx_good_pkt: 17786
dev.igb.1.host.breaker_tx_pkt_drop: 0
dev.igb.1.host.rx_good_bytes: 832249624227
dev.igb.1.host.tx_good_bytes: 1817163097562
dev.igb.1.host.length_errors: 0
dev.igb.1.host.serdes_violation_pkt: 0
dev.igb.1.host.header_redir_missed: 0
dev.igb.1.wake: 0
dev.igb.2.%desc: Intel(R) PRO/1000 Network Connection version - 2.4.0
dev.igb.2.%driver: igb
dev.igb.2.%location: slot=0 function=1
dev.igb.2.%pnpinfo: vendor=0x8086 device=0x10e6 subvendor=0x8086 subdevice=0xa02f class=0x020000
dev.igb.2.%parent: pci11
dev.igb.2.nvm: -1
dev.igb.2.enable_aim: 1
dev.igb.2.fc: 3
dev.igb.2.rx_processing_limit: 100
dev.igb.2.link_irq: 0
dev.igb.2.dropped: 0
dev.igb.2.tx_dma_fail: 0
dev.igb.2.rx_overruns: 0
dev.igb.2.watchdog_timeouts: 0
dev.igb.2.device_control: 1087117889
dev.igb.2.rx_control: 67141634
dev.igb.2.interrupt_mask: 157
dev.igb.2.extended_int_mask: 2147483648
dev.igb.2.tx_buf_alloc: 0
dev.igb.2.rx_buf_alloc: 0
dev.igb.2.fc_high_water: 58976
dev.igb.2.fc_low_water: 58960
dev.igb.2.queue0.no_desc_avail: 0
dev.igb.2.queue0.tx_packets: 476889
dev.igb.2.queue0.rx_packets: 623282493
dev.igb.2.queue0.rx_bytes: 74446064995
dev.igb.2.queue0.lro_queued: 0
dev.igb.2.queue0.lro_flushed: 0
dev.igb.2.mac_stats.excess_coll: 0
dev.igb.2.mac_stats.single_coll: 0
dev.igb.2.mac_stats.multiple_coll: 0
dev.igb.2.mac_stats.late_coll: 0
dev.igb.2.mac_stats.collision_count: 0
dev.igb.2.mac_stats.symbol_errors: 0
dev.igb.2.mac_stats.sequence_errors: 0
dev.igb.2.mac_stats.defer_count: 0
dev.igb.2.mac_stats.missed_packets: 0
dev.igb.2.mac_stats.recv_no_buff: 0
dev.igb.2.mac_stats.recv_undersize: 0
dev.igb.2.mac_stats.recv_fragmented: 0
dev.igb.2.mac_stats.recv_oversize: 0
dev.igb.2.mac_stats.recv_jabber: 0
dev.igb.2.mac_stats.recv_errs: 0
dev.igb.2.mac_stats.crc_errs: 0
dev.igb.2.mac_stats.alignment_errs: 0
dev.igb.2.mac_stats.coll_ext_errs: 0
dev.igb.2.mac_stats.xon_recvd: 0
dev.igb.2.mac_stats.xon_txd: 0
dev.igb.2.mac_stats.xoff_recvd: 0
dev.igb.2.mac_stats.xoff_txd: 0
dev.igb.2.mac_stats.total_pkts_recvd: 697958370
dev.igb.2.mac_stats.good_pkts_recvd: 623282231
dev.igb.2.mac_stats.bcast_pkts_recvd: 129434
dev.igb.2.mac_stats.mcast_pkts_recvd: 17096
dev.igb.2.mac_stats.rx_frames_64: 1
dev.igb.2.mac_stats.rx_frames_65_127: 563028186
dev.igb.2.mac_stats.rx_frames_128_255: 28151786
dev.igb.2.mac_stats.rx_frames_256_511: 9427339
dev.igb.2.mac_stats.rx_frames_512_1023: 5069891
dev.igb.2.mac_stats.rx_frames_1024_1522: 17605028
dev.igb.2.mac_stats.good_octets_recvd: 79432278514
dev.igb.2.mac_stats.good_octets_txd: 52903487
dev.igb.2.mac_stats.total_pkts_txd: 476888
dev.igb.2.mac_stats.good_pkts_txd: 476888
dev.igb.2.mac_stats.bcast_pkts_txd: 11
dev.igb.2.mac_stats.mcast_pkts_txd: 165656
dev.igb.2.mac_stats.tx_frames_64: 7
dev.igb.2.mac_stats.tx_frames_65_127: 300650
dev.igb.2.mac_stats.tx_frames_128_255: 172846
dev.igb.2.mac_stats.tx_frames_256_511: 3385
dev.igb.2.mac_stats.tx_frames_512_1023: 0
dev.igb.2.mac_stats.tx_frames_1024_1522: 0
dev.igb.2.mac_stats.tso_txd: 0
dev.igb.2.mac_stats.tso_ctx_fail: 0
dev.igb.2.interrupts.asserts: 591617753
dev.igb.2.interrupts.rx_pkt_timer: 623273921
dev.igb.2.interrupts.rx_abs_timer: 0
dev.igb.2.interrupts.tx_pkt_timer: 0
dev.igb.2.interrupts.tx_abs_timer: 623282231
dev.igb.2.interrupts.tx_queue_empty: 476827
dev.igb.2.interrupts.tx_queue_min_thresh: 0
dev.igb.2.interrupts.rx_desc_min_thresh: 0
dev.igb.2.interrupts.rx_overrun: 0
dev.igb.2.host.breaker_tx_pkt: 0
dev.igb.2.host.host_tx_pkt_discard: 0
dev.igb.2.host.rx_pkt: 8310
dev.igb.2.host.breaker_rx_pkts: 0
dev.igb.2.host.breaker_rx_pkt_drop: 0
dev.igb.2.host.tx_good_pkt: 61
dev.igb.2.host.breaker_tx_pkt_drop: 0
dev.igb.2.host.rx_good_bytes: 79432278736
dev.igb.2.host.tx_good_bytes: 52903487
dev.igb.2.host.length_errors: 0
dev.igb.2.host.serdes_violation_pkt: 0
dev.igb.2.host.header_redir_missed: 0