Interface errors, missed packets and rec overruns
-
I'm slowly racking up interfaced errors on my CE 2.7.2 home box and I'm not sure why. It's an old firewall appliance box, a Lanner FW-7541C. It has an Intel D525 CPU, 1 Intel 82574L and 5 Intel 82583V interfaces, a SanDisk SSD storage device and 4GB of RAM. I keep missing a small number of packets. Like the majority are getting through fine and I'm not seeing any trouble, but the errors keep slowly ticking up. I have hardware offload enabled, with large send/recv disabled per the documentation recommendations. I know the D525 has all the processing power of a wet noodle, but I'm not trying to do anything processor intensive. Basically just basic firewall duty and that's it. I've tried turning hyper-threading on and off. I've tried moving the WAN from a direct line to the pfsense to passing through a switch. I've tried the 82583V's instead of the 84574L. I just don't get it, my internet is only like 300-350Mbit, I think it should be able to keep up with that, right? Is there any tweak I should try?
sysctl dev.em.0 output
dev.em.0.interrupts.rx_overrun: 3 dev.em.0.interrupts.rx_desc_min_thresh: 0 dev.em.0.interrupts.tx_queue_min_thresh: 1 dev.em.0.interrupts.tx_queue_empty: 0 dev.em.0.interrupts.tx_abs_timer: 2 dev.em.0.interrupts.tx_pkt_timer: 1 dev.em.0.interrupts.rx_abs_timer: 0 dev.em.0.interrupts.rx_pkt_timer: 32 dev.em.0.interrupts.asserts: 118670 dev.em.0.mac_stats.tso_ctx_fail: 0 dev.em.0.mac_stats.tso_txd: 0 dev.em.0.mac_stats.tx_frames_1024_1522: 181167932 dev.em.0.mac_stats.tx_frames_512_1023: 7667118 dev.em.0.mac_stats.tx_frames_256_511: 7806568 dev.em.0.mac_stats.tx_frames_128_255: 14078839 dev.em.0.mac_stats.tx_frames_65_127: 54260108 dev.em.0.mac_stats.tx_frames_64: 48427909 dev.em.0.mac_stats.mcast_pkts_txd: 1884495 dev.em.0.mac_stats.bcast_pkts_txd: 45661 dev.em.0.mac_stats.good_pkts_txd: 313408474 dev.em.0.mac_stats.total_pkts_txd: 313436934 dev.em.0.mac_stats.good_octets_txd: 285602451945 dev.em.0.mac_stats.good_octets_recvd: 287772957030 dev.em.0.mac_stats.rx_frames_1024_1522: 182980173 dev.em.0.mac_stats.rx_frames_512_1023: 7219728 dev.em.0.mac_stats.rx_frames_256_511: 7331488 dev.em.0.mac_stats.rx_frames_128_255: 12598599 dev.em.0.mac_stats.rx_frames_65_127: 98395148 dev.em.0.mac_stats.rx_frames_64: 4681295 dev.em.0.mac_stats.mcast_pkts_recvd: 286677 dev.em.0.mac_stats.bcast_pkts_recvd: 2006878 dev.em.0.mac_stats.good_pkts_recvd: 313206431 dev.em.0.mac_stats.total_pkts_recvd: 313284437 dev.em.0.mac_stats.xoff_txd: 25803 dev.em.0.mac_stats.xoff_recvd: 22665 dev.em.0.mac_stats.xon_txd: 2657 dev.em.0.mac_stats.xon_recvd: 22305 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_no_buff: 20674810 dev.em.0.mac_stats.missed_packets: 23148 dev.em.0.mac_stats.defer_count: 0 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.queue_rx_1.rx_irq: 0 dev.em.0.queue_rx_1.rxd_tail: 802 dev.em.0.queue_rx_1.rxd_head: 803 dev.em.0.queue_rx_0.rx_irq: 0 dev.em.0.queue_rx_0.rxd_tail: 616 dev.em.0.queue_rx_0.rxd_head: 617 dev.em.0.queue_tx_1.tx_irq: 0 dev.em.0.queue_tx_1.txd_tail: 507 dev.em.0.queue_tx_1.txd_head: 510 dev.em.0.queue_tx_0.tx_irq: 0 dev.em.0.queue_tx_0.txd_tail: 543 dev.em.0.queue_tx_0.txd_head: 543 dev.em.0.fc_low_water: 16932 dev.em.0.fc_high_water: 18432 dev.em.0.rx_control: 67403778 dev.em.0.device_control: 1477444168 dev.em.0.watchdog_timeouts: 0 dev.em.0.rx_overruns: 81652 dev.em.0.link_irq: 123229 dev.em.0.dropped: 0 dev.em.0.eee_control: 1 dev.em.0.itr: 488 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_int_delay: 66 dev.em.0.rx_int_delay: 0 dev.em.0.rs_dump: 0 dev.em.0.reg_dump: General Registers CTRL 58100248 STATUS 00080783 CTRL_EXT 80580000 Interrupt Registers ICR 80000001 RX Registers RCTL 04048002 RDLEN 00004000 RDH 00000269 RDT 00000268 RXDCTL 01050420 RDBAL 0461a000 RDBAH 00000000 TX Registers TCTL 3103f0fa TDBAL 04600000 TDBAH 00000000 TDLEN 00004000 TDH 0000021f TDT 0000021f TXDCTL 0341011f TDFH 00000b28 TDFT 00000b28 TDFHS 00000b28 TDFPC 00000000 dev.em.0.fc: 3 dev.em.0.debug: -1 dev.em.0.fw_version: EEPROM V1.9-0 dev.em.0.nvm: -1 dev.em.0.iflib.rxq1.rxq_fl0.buf_size: 2048 dev.em.0.iflib.rxq1.rxq_fl0.credits: 1023 dev.em.0.iflib.rxq1.rxq_fl0.cidx: 818 dev.em.0.iflib.rxq1.rxq_fl0.pidx: 817 dev.em.0.iflib.rxq1.cpu: 2 dev.em.0.iflib.rxq0.rxq_fl0.buf_size: 2048 dev.em.0.iflib.rxq0.rxq_fl0.credits: 1023 dev.em.0.iflib.rxq0.rxq_fl0.cidx: 617 dev.em.0.iflib.rxq0.rxq_fl0.pidx: 616 dev.em.0.iflib.rxq0.cpu: 0 dev.em.0.iflib.txq1.r_abdications: 0 dev.em.0.iflib.txq1.r_restarts: 0 dev.em.0.iflib.txq1.r_stalls: 0 dev.em.0.iflib.txq1.r_starts: 148037804 dev.em.0.iflib.txq1.r_drops: 0 dev.em.0.iflib.txq1.r_enqueues: 152658558 dev.em.0.iflib.txq1.ring_state: pidx_head: 0640 pidx_tail: 0640 cidx: 0640 state: IDLE dev.em.0.iflib.txq1.txq_cleaned: 171150878 dev.em.0.iflib.txq1.txq_processed: 171150918 dev.em.0.iflib.txq1.txq_in_use: 45 dev.em.0.iflib.txq1.txq_cidx_processed: 587 dev.em.0.iflib.txq1.txq_cidx: 550 dev.em.0.iflib.txq1.txq_pidx: 592 dev.em.0.iflib.txq1.no_tx_dma_setup: 0 dev.em.0.iflib.txq1.txd_encap_efbig: 0 dev.em.0.iflib.txq1.tx_map_failed: 0 dev.em.0.iflib.txq1.no_desc_avail: 0 dev.em.0.iflib.txq1.mbuf_defrag_failed: 0 dev.em.0.iflib.txq1.m_pullups: 12029099 dev.em.0.iflib.txq1.mbuf_defrag: 0 dev.em.0.iflib.txq1.cpu: 2 dev.em.0.iflib.txq0.r_abdications: 0 dev.em.0.iflib.txq0.r_restarts: 91 dev.em.0.iflib.txq0.r_stalls: 91 dev.em.0.iflib.txq0.r_starts: 156570070 dev.em.0.iflib.txq0.r_drops: 0 dev.em.0.iflib.txq0.r_enqueues: 160858354 dev.em.0.iflib.txq0.ring_state: pidx_head: 0242 pidx_tail: 0242 cidx: 0242 state: IDLE dev.em.0.iflib.txq0.txq_cleaned: 177709558 dev.em.0.iflib.txq0.txq_processed: 177709598 dev.em.0.iflib.txq0.txq_in_use: 41 dev.em.0.iflib.txq0.txq_cidx_processed: 542 dev.em.0.iflib.txq0.txq_cidx: 502 dev.em.0.iflib.txq0.txq_pidx: 543 dev.em.0.iflib.txq0.no_tx_dma_setup: 0 dev.em.0.iflib.txq0.txd_encap_efbig: 0 dev.em.0.iflib.txq0.tx_map_failed: 0 dev.em.0.iflib.txq0.no_desc_avail: 0 dev.em.0.iflib.txq0.mbuf_defrag_failed: 0 dev.em.0.iflib.txq0.m_pullups: 12129578 dev.em.0.iflib.txq0.mbuf_defrag: 0 dev.em.0.iflib.txq0.cpu: 0 dev.em.0.iflib.override_nrxds: 0 dev.em.0.iflib.override_ntxds: 0 dev.em.0.iflib.use_logical_cores: 0 dev.em.0.iflib.separate_txrx: 0 dev.em.0.iflib.core_offset: 0 dev.em.0.iflib.tx_abdicate: 0 dev.em.0.iflib.rx_budget: 0 dev.em.0.iflib.disable_msix: 0 dev.em.0.iflib.override_qs_enable: 0 dev.em.0.iflib.override_nrxqs: 0 dev.em.0.iflib.override_ntxqs: 0 dev.em.0.iflib.driver_version: 7.7.8-fbsd dev.em.0.%parent: pci1 dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x8086 subdevice=0x0000 class=0x020000 dev.em.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 dev.em.0.%driver: em dev.em.0.%desc: Intel(R) Gigabit CT 82574L -
Oh, I forgot possibly the most interesting tidbit. My brother is experiencing the same behavior and his system has a Ryzen 3 2200G. CPU to spare on that machine with an old quad NIC Intel 82571. It only seems to happen on the WAN side. I had the same experience when my WAN was separate. (I currently have all my traffic VLANed into a single physical NIC).
Brother's sysctl dev.em.0 output
dev.em.0.interrupts.rx_overrun: 0 dev.em.0.interrupts.rx_desc_min_thresh: 0 dev.em.0.interrupts.tx_queue_min_thresh: 24 dev.em.0.interrupts.tx_queue_empty: 0 dev.em.0.interrupts.tx_abs_timer: 9824 dev.em.0.interrupts.tx_pkt_timer: 7967 dev.em.0.interrupts.rx_abs_timer: 0 dev.em.0.interrupts.rx_pkt_timer: 80527 dev.em.0.interrupts.asserts: 484449432 dev.em.0.mac_stats.tso_ctx_fail: 0 dev.em.0.mac_stats.tso_txd: 0 dev.em.0.mac_stats.tx_frames_1024_1522: 83478008 dev.em.0.mac_stats.tx_frames_512_1023: 619494980 dev.em.0.mac_stats.tx_frames_256_511: 4447683 dev.em.0.mac_stats.tx_frames_128_255: 26350772 dev.em.0.mac_stats.tx_frames_65_127: 90891554 dev.em.0.mac_stats.tx_frames_64: 42590904 dev.em.0.mac_stats.mcast_pkts_txd: 1192 dev.em.0.mac_stats.bcast_pkts_txd: 3346 dev.em.0.mac_stats.good_pkts_txd: 867253901 dev.em.0.mac_stats.total_pkts_txd: 867253901 dev.em.0.mac_stats.good_octets_txd: 564113139828 dev.em.0.mac_stats.good_octets_recvd: 600386896379 dev.em.0.mac_stats.rx_frames_1024_1522: 395679755 dev.em.0.mac_stats.rx_frames_512_1023: 17668892 dev.em.0.mac_stats.rx_frames_256_511: 7229265 dev.em.0.mac_stats.rx_frames_128_255: 158134130 dev.em.0.mac_stats.rx_frames_65_127: 48923871 dev.em.0.mac_stats.rx_frames_64: 0 dev.em.0.mac_stats.mcast_pkts_recvd: 6626 dev.em.0.mac_stats.bcast_pkts_recvd: 0 dev.em.0.mac_stats.good_pkts_recvd: 627635913 dev.em.0.mac_stats.total_pkts_recvd: 629974672 dev.em.0.mac_stats.xoff_txd: 0 dev.em.0.mac_stats.xoff_recvd: 1165746 dev.em.0.mac_stats.xon_txd: 0 dev.em.0.mac_stats.xon_recvd: 1165746 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_no_buff: 0 dev.em.0.mac_stats.missed_packets: 7310 dev.em.0.mac_stats.defer_count: 1161488 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.queue_rx_0.rx_irq: 0 dev.em.0.queue_rx_0.rxd_tail: 518 dev.em.0.queue_rx_0.rxd_head: 519 dev.em.0.queue_tx_0.tx_irq: 0 dev.em.0.queue_tx_0.txd_tail: 164 dev.em.0.queue_tx_0.txd_head: 164 dev.em.0.fc_low_water: 29220 dev.em.0.fc_high_water: 30720 dev.em.0.rx_control: 67403778 dev.em.0.device_control: 1209795137 dev.em.0.watchdog_timeouts: 0 dev.em.0.rx_overruns: 111 dev.em.0.link_irq: 0 dev.em.0.dropped: 0 dev.em.0.eee_control: 1 dev.em.0.itr: 488 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_int_delay: 66 dev.em.0.rx_int_delay: 0 dev.em.0.rs_dump: 0 dev.em.0.reg_dump: General Registers CTRL 481c0241 STATUS 00080387 CTRL_EXT 101400c0 Interrupt Registers ICR 00000000 RX Registers RCTL 04048002 RDLEN 00004000 RDH 00000207 RDT 00000206 RXDCTL 00010000 RDBAL b0034000 RDBAH 00000000 TX Registers TCTL 3103f0fa TDBAL b002c000 TDBAH 00000000 TDLEN 00004000 TDH 000000aa TDT 000000aa TXDCTL 0341011f TDFH 00001296 TDFT 00001298 TDFHS 00001296 TDFPC 00000000 dev.em.0.fc: 3 dev.em.0.debug: -1 dev.em.0.fw_version: EEPROM V5.12-2 dev.em.0.nvm: -1 dev.em.0.iflib.rxq0.rxq_fl0.buf_size: 2048 dev.em.0.iflib.rxq0.rxq_fl0.credits: 1023 dev.em.0.iflib.rxq0.rxq_fl0.cidx: 519 dev.em.0.iflib.rxq0.rxq_fl0.pidx: 518 dev.em.0.iflib.rxq0.cpu: 3 dev.em.0.iflib.txq0.r_abdications: 324 dev.em.0.iflib.txq0.r_restarts: 56590 dev.em.0.iflib.txq0.r_stalls: 56590 dev.em.0.iflib.txq0.r_starts: 858032115 dev.em.0.iflib.txq0.r_drops: 1072 dev.em.0.iflib.txq0.r_enqueues: 868117756 dev.em.0.iflib.txq0.ring_state: pidx_head: 1279 pidx_tail: 1279 cidx: 1279 state: IDLE dev.em.0.iflib.txq0.txq_cleaned: 1474843782 dev.em.0.iflib.txq0.txq_processed: 1474843822 dev.em.0.iflib.txq0.txq_in_use: 44 dev.em.0.iflib.txq0.txq_cidx_processed: 174 dev.em.0.iflib.txq0.txq_cidx: 134 dev.em.0.iflib.txq0.txq_pidx: 178 dev.em.0.iflib.txq0.no_tx_dma_setup: 0 dev.em.0.iflib.txq0.txd_encap_efbig: 0 dev.em.0.iflib.txq0.tx_map_failed: 0 dev.em.0.iflib.txq0.no_desc_avail: 0 dev.em.0.iflib.txq0.mbuf_defrag_failed: 0 dev.em.0.iflib.txq0.m_pullups: 602346602 dev.em.0.iflib.txq0.mbuf_defrag: 0 dev.em.0.iflib.txq0.cpu: 2 dev.em.0.iflib.override_nrxds: 0 dev.em.0.iflib.override_ntxds: 0 dev.em.0.iflib.use_logical_cores: 0 dev.em.0.iflib.separate_txrx: 0 dev.em.0.iflib.core_offset: 0 dev.em.0.iflib.tx_abdicate: 0 dev.em.0.iflib.rx_budget: 0 dev.em.0.iflib.disable_msix: 1 dev.em.0.iflib.override_qs_enable: 0 dev.em.0.iflib.override_nrxqs: 0 dev.em.0.iflib.override_ntxqs: 0 dev.em.0.iflib.driver_version: 7.7.8-fbsd dev.em.0.%parent: pci3 dev.em.0.%pnpinfo: vendor=0x8086 device=0x10bc subvendor=0x103c subdevice=0x704b class=0x020000 dev.em.0.%location: slot=0 function=0 dbsf=pci0:18:0:0 dev.em.0.%driver: em dev.em.0.%desc: Intel(R) PRO/1000 PT 82571EB/82571GB (Quad Copper) -
Looks like slightly different cause though. recv_no_buff vs defer_count
The first thing I'd try is reassigning the NICs to use a different one as WAN and see if the issue follows it.
Next I'd try a different flow-control setting and check the current negotiated value. That should prevent the other side overloading the receive buffers if both ends support it.
Steve
-
@stephenw10 said in Interface errors, missed packets and rec overruns:
Looks like slightly different cause though. recv_no_buff vs defer_count
I noticed that too, but I'm not sure what "defer" means here. I mean I'm also not sure what "recv_no_buff" means, but my educated guess is it received a packet but had no buffer space available to place it in.
@stephenw10 said in Interface errors, missed packets and rec overruns:
The first thing I'd try is reassigning the NICs to use a different one as WAN and see if the issue follows it.
I tried that already, the 82583V NICs all do the same thing.
@stephenw10 said in Interface errors, missed packets and rec overruns:
Next I'd try a different flow-control setting and check the current negotiated value.
So it's currently set to 3 for flow control. How do I check the negotiated value?
-
AFAIK
recv_no_buffimplies there are no available receive buffers. So potentially you could increase the buffers.I will say that I see some no_buff failures on a box here and don't see any connection issues:
[2.7.2-RELEASE][admin@xtm5.stevew.lan]/root: sysctl dev.em.0 | grep buf dev.em.0.mac_stats.recv_no_buff: 36 dev.em.0.iflib.rxq1.rxq_fl0.buf_size: 2048 dev.em.0.iflib.rxq0.rxq_fl0.buf_size: 2048 dev.em.0.iflib.txq1.mbuf_defrag_failed: 0 dev.em.0.iflib.txq1.mbuf_defrag: 0 dev.em.0.iflib.txq0.mbuf_defrag_failed: 0 dev.em.0.iflib.txq0.mbuf_defrag: 0Good question about seeing how it's linked though! em doesn't appear to report that. I'd try setting fc to 0 and see if that changes anything.
-
OK, I found some interesting things. Turns out old Intel datasheets have really thorough descriptions of what all these counters mean. Intel 82583V datasheet
Defer Count: This register counts defer events. A defer event occurs when the transmitter cannot immediately send a packet due to the medium being busy either because:
• Another device is transmitting
• The IPG timer has not expired
• Half-duplex deferral events
• Reception of XOFF frames
• The link is not up
This register only increments if transmits are enabled. The behavior of this counter is slightly different in the 82583V relative to previous devices. For the 82583V, this counter does not increment for streaming transmits that are deferred due to TX IPG.Receive No Buffers Count: This register counts the number of times that frames were received when there were no available buffers in host memory to store those frames (receive descriptor head and tail pointers were equal). The packet is still received if there is space in the FIFO. This register only increments if receives are enabled. This register does not increment when flow control packets are received.
Missed Packets Count: Counts the number of missed packets. Packets are missed when the receive FIFO has insufficient space to store the incoming packet. This could be caused because of too few buffers allocated, or because there is insufficient bandwidth on the IO bus. Events setting this counter cause RXO, the receiver overrun interrupt, to be set. This register does not increment if receives are not enabled. Note that these packets are also counted in the Total Packets Received register as well as in the Total Octets Received register.
-
So Defers are specifically transmits and won't ever be Missed packets. And Recv_No_Buff is the NIC has a frame, but the CPU has no buffer for it. Recv_no_buff can become missed_packets if the situation persists long enough.
I reconfigured things. Instead of shoving everything into em.0 in a router on a stick config, I changed em.0 to be the only WAN and all the LAN VLANs on em.1. This was weird as I can literally just watch the recv_no_buff incrementing on em.0, but em.1 which is seeing the same number of packets is not having any trouble.
Then I reconfigured things again, this time put the WAN on em.5 and the LAN VLANs on em.4. This setup has no immediate issues, but in the past it slowly accumulates errors just the same.
My brother's stats confuse me. Defers seem like they should mostly not happen on a full-duplex link, but maybe this is older hardware that increments this when XOFF is received. His defers count is very similar to the XOFF received count. More confusing is that he's getting missed packets, but without recv_no_buff. That doesn't seem like it should be possible.
-
Hmm, interesting. Are those all the exact same NIC chip? Different PCIe bus maybe?
-
@stephenw10 em.0 is a 82574L, but em.1-5 are 82583V's. According to pciconf I think they all have a direct x1 lane to the CPU.
My brother is using an old HP 4 port nic with 82571 chips. I'm starting to think the older 82571 just doesn't have the recv_no_buff register.
-
Actually I just found a bit in the 2014 errata update for the 82571EB that might explain my brother's missed packets. It might just be old crap.
- Missed RX Packets
Problem:
When the device operates with multiple-requests or Large Send enabled, there could be receive packet loss. When the Tx FIFO is full, the Tx flow may block the host DMA interface of the device. When the transmission of packets is prevented for a long time, due to capture effect or very long backoff in half-duplex, the transmit FIFO is filled and the fetch of Rx descriptors is prevented also. This will prevent the release of the packets from the Rx FIFO to the host, causing the Rx buffer to overflow and the loss of incoming packets. This is a temporary state that will be released once the transmit side is be able to empty the Tx packet buffer.Implication:
There could be some packet loss in the Rx path if the transmission of packets is prevented for a long time. Normally, if this occurs, these packets will be re-transmitted by upper-layer protocols.Workaround:
None -
Hmm the 82574 is extremely common. I would have expected that to work more reliably if anything. But there is a difference there so whatever is causing a problem on em0 the 82583V apparently doesn't suffer from it.
-
Of the 2 options I have on this box it's supposed to be the "better" one. It has MSI-X and dual tx/rx queues to the 82583V's MSI and single tx/rx queue.
Also, I definitely had the em.5/WAN em.4/LAN setup in the past and it would miss packets over time, but this time it's all good.
Only I've reconfigured this so many times and it's never worked as well as it finally is. Computers man, what the hell.
-
Digging this thread up from the dead as I've switched back to this hardware and I'm still having the missed packet issue.
current setup:
pfSense 2.8.1
WAN: em.5
LAN: em.4dev.em.5.interrupts.rx_overrun: 0 dev.em.5.interrupts.rx_desc_min_thresh: 0 dev.em.5.interrupts.tx_queue_min_thresh: 0 dev.em.5.interrupts.tx_queue_empty: 0 dev.em.5.interrupts.tx_abs_timer: 4822 dev.em.5.interrupts.tx_pkt_timer: 3727 dev.em.5.interrupts.rx_abs_timer: 0 dev.em.5.interrupts.rx_pkt_timer: 27942 dev.em.5.interrupts.asserts: 98134014 dev.em.5.mac_stats.tso_ctx_fail: 0 dev.em.5.mac_stats.tso_txd: 0 dev.em.5.mac_stats.tx_frames_1024_1522: 10326417 dev.em.5.mac_stats.tx_frames_512_1023: 2180150 dev.em.5.mac_stats.tx_frames_256_511: 3740187 dev.em.5.mac_stats.tx_frames_128_255: 25962359 dev.em.5.mac_stats.tx_frames_65_127: 43359078 dev.em.5.mac_stats.tx_frames_64: 8451116 dev.em.5.mac_stats.mcast_pkts_txd: 65241 dev.em.5.mac_stats.bcast_pkts_txd: 1300 dev.em.5.mac_stats.good_pkts_txd: 94019307 dev.em.5.mac_stats.total_pkts_txd: 94019307 dev.em.5.mac_stats.good_octets_txd: 26588727113 dev.em.5.mac_stats.good_octets_recvd: 273900578371 dev.em.5.mac_stats.rx_frames_1024_1522: 192181323 dev.em.5.mac_stats.rx_frames_512_1023: 7080305 dev.em.5.mac_stats.rx_frames_256_511: 4642562 dev.em.5.mac_stats.rx_frames_128_255: 5800214 dev.em.5.mac_stats.rx_frames_65_127: 30089561 dev.em.5.mac_stats.rx_frames_64: 0 dev.em.5.mac_stats.mcast_pkts_recvd: 2622 dev.em.5.mac_stats.bcast_pkts_recvd: 0 dev.em.5.mac_stats.good_pkts_recvd: 239793965 dev.em.5.mac_stats.total_pkts_recvd: 239841614 dev.em.5.mac_stats.mgmt_pkts_txd: 0 dev.em.5.mac_stats.mgmt_pkts_drop: 0 dev.em.5.mac_stats.mgmt_pkts_recvd: 0 dev.em.5.mac_stats.unsupported_fc_recvd: 0 dev.em.5.mac_stats.xoff_txd: 0 dev.em.5.mac_stats.xoff_recvd: 22527 dev.em.5.mac_stats.xon_txd: 0 dev.em.5.mac_stats.xon_recvd: 22527 dev.em.5.mac_stats.coll_ext_errs: 0 dev.em.5.mac_stats.alignment_errs: 0 dev.em.5.mac_stats.crc_errs: 0 dev.em.5.mac_stats.recv_errs: 0 dev.em.5.mac_stats.recv_jabber: 0 dev.em.5.mac_stats.recv_oversize: 0 dev.em.5.mac_stats.recv_fragmented: 0 dev.em.5.mac_stats.recv_undersize: 0 dev.em.5.mac_stats.recv_no_buff: 0 dev.em.5.mac_stats.recv_length_errors: 0 dev.em.5.mac_stats.missed_packets: 2607 dev.em.5.mac_stats.defer_count: 0 dev.em.5.mac_stats.sequence_errors: 0 dev.em.5.mac_stats.symbol_errors: 0 dev.em.5.mac_stats.collision_count: 0 dev.em.5.mac_stats.late_coll: 0 dev.em.5.mac_stats.multiple_coll: 0 dev.em.5.mac_stats.single_coll: 0 dev.em.5.mac_stats.excess_coll: 0 dev.em.5.queue_rx_0.rx_irq: 0 dev.em.5.queue_rx_0.rxd_tail: 817 dev.em.5.queue_rx_0.rxd_head: 818 dev.em.5.queue_rx_0.interrupt_rate: 20032 dev.em.5.queue_tx_0.tx_irq: 0 dev.em.5.queue_tx_0.txd_tail: 225 dev.em.5.queue_tx_0.txd_head: 225 dev.em.5.queue_tx_0.interrupt_rate: 20032 dev.em.5.fc_low_water: 16932 dev.em.5.fc_high_water: 18432 dev.em.5.rx_control: 67403802 dev.em.5.device_control: 1209008712 dev.em.5.watchdog_timeouts: 0 dev.em.5.rx_overruns: 106 dev.em.5.link_irq: 0 dev.em.5.dropped: 0 dev.em.5.eee_control: 1 dev.em.5.tx_abs_int_delay: 66 dev.em.5.rx_abs_int_delay: 66 dev.em.5.tx_int_delay: 66 dev.em.5.rx_int_delay: 0 dev.em.5.tso_tcp_flags_mask_last_segment: 0 dev.em.5.tso_tcp_flags_mask_middle_segment: 0 dev.em.5.tso_tcp_flags_mask_first_segment: 0 dev.em.5.rs_dump: 0 dev.em.5.reg_dump: General Registers CTRL 48100248 STATUS 00080783 CTRL_EXT 00580000 Interrupt Registers ICR 00000000 RX Registers RCTL 0404801a RDLEN 00004000 RDH 00000332 RDT 00000331 RXDCTL 00010000 RDBAL 0571e000 RDBAH 00000000 TX Registers TCTL 3103f0fa TDBAL 05716000 TDBAH 00000000 TDLEN 00004000 TDH 000000e1 TDT 000000e1 TXDCTL 0341011f TDFH 000010da TDFT 000010da TDFHS 000010da TDFPC 00000000 dev.em.5.fc: 3 dev.em.5.debug: -1 dev.em.5.fw_version: EEPROM V1.10-0 dev.em.5.enable_aim: 1 dev.em.5.nvm: -1 dev.em.5.iflib.rxq0.rxq_fl0.buf_size: 2048 dev.em.5.iflib.rxq0.rxq_fl0.credits: 1023 dev.em.5.iflib.rxq0.rxq_fl0.cidx: 818 dev.em.5.iflib.rxq0.rxq_fl0.pidx: 817 dev.em.5.iflib.rxq0.cpu: 3 dev.em.5.iflib.txq0.r_abdications: 0 dev.em.5.iflib.txq0.r_restarts: 1863 dev.em.5.iflib.txq0.r_stalls: 1863 dev.em.5.iflib.txq0.r_starts: 91543486 dev.em.5.iflib.txq0.r_drops: 0 dev.em.5.iflib.txq0.r_enqueues: 94158185 dev.em.5.iflib.txq0.ring_state: pidx_head: 1385 pidx_tail: 1385 cidx: 1385 state: IDLE dev.em.5.iflib.txq0.txq_cleaned: 114867383 dev.em.5.iflib.txq0.txq_processed: 114867423 dev.em.5.iflib.txq0.txq_in_use: 42 dev.em.5.iflib.txq0.txq_cidx_processed: 223 dev.em.5.iflib.txq0.txq_cidx: 183 dev.em.5.iflib.txq0.txq_pidx: 225 dev.em.5.iflib.txq0.no_tx_dma_setup: 0 dev.em.5.iflib.txq0.txd_encap_efbig: 0 dev.em.5.iflib.txq0.tx_map_failed: 0 dev.em.5.iflib.txq0.no_desc_avail: 0 dev.em.5.iflib.txq0.mbuf_defrag_failed: 0 dev.em.5.iflib.txq0.m_pullups: 16662937 dev.em.5.iflib.txq0.mbuf_defrag: 0 dev.em.5.iflib.txq0.cpu: 2 dev.em.5.iflib.override_nrxds: 0 dev.em.5.iflib.override_ntxds: 0 dev.em.5.iflib.allocated_msix_vectors: 1 dev.em.5.iflib.use_extra_msix_vectors: 0 dev.em.5.iflib.use_logical_cores: 0 dev.em.5.iflib.separate_txrx: 0 dev.em.5.iflib.core_offset: 0 dev.em.5.iflib.tx_abdicate: 0 dev.em.5.iflib.rx_budget: 0 dev.em.5.iflib.disable_msix: 1 dev.em.5.iflib.override_qs_enable: 0 dev.em.5.iflib.override_nrxqs: 0 dev.em.5.iflib.override_ntxqs: 0 dev.em.5.iflib.driver_version: 7.7.8-fbsd dev.em.5.%iommu: rid=0x700 dev.em.5.%parent: pci6 dev.em.5.%pnpinfo: vendor=0x8086 device=0x150c subvendor=0x8086 subdevice=0x0000 class=0x020000 dev.em.5.%location: slot=0 function=0 dbsf=pci0:7:0:0 dev.em.5.%driver: em dev.em.5.%desc: Intel(R) 82583VSo things that are different, I'm not seeing the "recv_no_buffer" errors any more since I switched to em5. I have missed packets and rx_overruns. I believe rx_overruns to be the number of distinct events where there was an overrun and missed packets is the total of missed packets per event. So I'm dropping ~20 packets each time I have an overrun.
Does anyone know how to increase the buffer size on the em driver?
-
It should be automatic but you can try setting: dev.em.5.iflib.override_nrxds
The other thing you could try is running the Intel-em-kmod driver instead.
Other than the counters increasing are you actually seeing an issue?
-
@stephenw10 said in Interface errors, missed packets and rec overruns:
It should be automatic but you can try setting: dev.em.5.iflib.override_nrxds
According to the FreeBSD iflib docs that would change the number of queues, not the queue size.
@stephenw10 said in Interface errors, missed packets and rec overruns:
The other thing you could try is running the Intel-em-kmod driver instead.
Is that just a command I need to run, or do I need to load files on the machine?
@stephenw10 said in Interface errors, missed packets and rec overruns:
Other than the counters increasing are you actually seeing an issue?
The overall missed packet rate is something like 0.001%, so no, it's working fine. I'm pretty sure the issue is my CPU is borderline, but I'm not sure how to "prove" it.
-
override_nrxqssets the number of queues.override_nrxdssets the number of descriptors which is effectively the queue size for iflib as I understand it.To use the alternate driver you need to load the pkg then add the loader line to load the module at boot:
[2.8.1-RELEASE][admin@xtm5.stevew.lan]/root: pkg search intel cpu-microcode-intel-20250211 Intel CPU microcode updates intel-em-kmod-7.7.8.1500029_1 Gigabit FreeBSD Base Drivers for Intel(R) Ethernet [2.8.1-RELEASE][admin@xtm5.stevew.lan]/root: pkg install intel-em-kmod Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... pfSense repository is up to date. All repositories are up to date. The following 1 package(s) will be affected (of 0 checked): New packages to be INSTALLED: intel-em-kmod: 7.7.8.1500029_1 [pfSense] Number of packages to be installed: 1 113 KiB to be downloaded. Proceed with this action? [y/N]: y [1/1] Fetching intel-em-kmod-7.7.8.1500029_1.pkg: 100% 113 KiB 115.5kB/s 00:01 Checking integrity... done (0 conflicting) [1/1] Installing intel-em-kmod-7.7.8.1500029_1... [1/1] Extracting intel-em-kmod-7.7.8.1500029_1: 100% ===== Message from intel-em-kmod-7.7.8.1500029_1: -- THIS PACKAGE INSTALLS THE NEWER VERSION OF THE SOFTWARE WHICH CAN CAUSE SYSTEM INSTABILITY WHILE USED. USE THE UPDATED VERSION ONLY IF YOU EXPERIENCE PROBLEMS WITH THE DRIVER PRESENT IN THE KERNEL DISTRIBUTION Usage: To load the updated version of the driver add this: if_em_updated_load="YES" to your /boot/loader.conf and reboot the machine. There's no need to recompile the GENERIC kernel without if_em driver After the reboot you may see this kind of messaged in the dmesg: module_register: module pci/em already exists! Module pci/em failed to register: 17 This is the side effect of the newer version of the driver overriding the older one and can be safely disregardedThen:
echo 'if_em_updated_load="YES"' >> /boot/loader.conf.localThen reboot.
However on the one device I have left that has em NICs I'm seeing an error trying to load it:
KLD file if_em_updated.ko - could not finalize loading
-
@stephenw10 said in Interface errors, missed packets and rec overruns:
override_nrxqs sets the number of queues. override_nrxds sets the number of descriptors which is effectively the queue size for iflib as I understand it.
My mistake there, you are correct. The stock rx descriptors is like 1024, so I'm going to try doubling that to 2048 and see what happens.
So I made a change after my last post where I turned off hyperthreading on the theory that it might reduce interrupt collisions. It does seem to have helped, I'm seeing fewer missed packets than previously. Effective error rate at the moment is half what it was last time, now 0.0005%.
I'm going to punt on the driver change for the moment, I think I just need a little more buffer to handle some intermittent bursty traffic from my ISP that the CPU can't quite keep up with.
-
Well that's interesting.
dev.em.5.iflib.rxq0.rxq_fl0.buf_size: 2048 dev.em.5.iflib.rxq0.rxq_fl0.credits: 2047Doubling the descriptors changed the "credits", not the buffer size. 1023 => 2047.
Privacy Policy · Cookie Policy