High interrupts on WAN/LAN interfaces?

[rmeskill]

@stephenw10 yeah, no dice on that.

The only thing I will say is I tried a USB NIC and that also was seeing packet loss, so at this point I can't fathom, besides a CPU issue as a whole with the Topton box, how this could be anything but the provider's problem. Unfortunately they insist there's no connectivity issues to their modem and the gateway I'm connecting to is upstream of that, so theoretically they should see an issue if there was one...

[stephenw10]

No difference at all with a much lower hwpstate_intel setting?

[rmeskill]

@stephenw10 I blew away my router entirely and rebuilt it from scratch, restoring some of the FW side of things but not the system tunables and other settings just so I could confirm I was starting with a clean config. Here's what I have:

And here's what we look like:

ANd here's what the system log looks like for gateways:

[stephenw10]

Hmm, and no dropped packets or errors in the sysctl output?

[rmeskill]

@stephenw10

[24.03-RELEASE][admin@pfSense.home.arpa]/root: sysctl -a | grep drop
kern.ipc.tls.ifnet.reset_dropped: 0
vfs.cache.stats.drops: 0
net.inet.ip.intr_queue_drops: 0
net.inet.ip.intr_direct_queue_drops: 0
net.inet.icmp.drop_redirect: 0
net.inet.tcp.rexmit_drop_options: 0
net.inet.tcp.drop_synfin: 1
net.pf.default_to_drop: 0
hw.cxgbe.ofld_cong_drop: 0
hw.cxgbe.cong_drop: 0
hw.cxgbe.nm_cong_drop: 1
hw.cxgbe.drop_pkts_with_l4_errors: 0
hw.cxgbe.drop_pkts_with_l3_errors: 0
hw.cxgbe.drop_pkts_with_l2_errors: 1
hw.cxgbe.drop_ip_fragments: 0
kstat.zfs.misc.fm.erpt-dropped: 0
dev.igc.3.dropped: 0
dev.igc.3.iflib.txq3.r_drops: 0
dev.igc.3.iflib.txq2.r_drops: 0
dev.igc.3.iflib.txq1.r_drops: 0
dev.igc.3.iflib.txq0.r_drops: 0
dev.igc.2.dropped: 0
dev.igc.2.iflib.txq3.r_drops: 0
dev.igc.2.iflib.txq2.r_drops: 0
dev.igc.2.iflib.txq1.r_drops: 0
dev.igc.2.iflib.txq0.r_drops: 0
dev.igc.1.dropped: 0
dev.igc.1.iflib.txq3.r_drops: 0
dev.igc.1.iflib.txq2.r_drops: 0
dev.igc.1.iflib.txq1.r_drops: 0
dev.igc.1.iflib.txq0.r_drops: 0
dev.igc.0.dropped: 0
dev.igc.0.iflib.txq3.r_drops: 0
dev.igc.0.iflib.txq2.r_drops: 0
dev.igc.0.iflib.txq1.r_drops: 0
dev.igc.0.iflib.txq0.r_drops: 0

[stephenw10]

Can you just show the full output from sysctl dev.igc ?

[rmeskill]

[24.03-RELEASE][admin@pfSense.home.arpa]/root: sysctl dev.igc.0
dev.igc.0.wake: 0
dev.igc.0.interrupts.rx_desc_min_thresh: 0
dev.igc.0.interrupts.asserts: 135425174
dev.igc.0.mac_stats.tso_txd: 0
dev.igc.0.mac_stats.tx_frames_1024_1522: 25790251
dev.igc.0.mac_stats.tx_frames_512_1023: 404292
dev.igc.0.mac_stats.tx_frames_256_511: 288528
dev.igc.0.mac_stats.tx_frames_128_255: 388062
dev.igc.0.mac_stats.tx_frames_65_127: 44147139
dev.igc.0.mac_stats.tx_frames_64: 7442638
dev.igc.0.mac_stats.mcast_pkts_txd: 0
dev.igc.0.mac_stats.bcast_pkts_txd: 3
dev.igc.0.mac_stats.good_pkts_txd: 78460910
dev.igc.0.mac_stats.total_pkts_txd: 78460910
dev.igc.0.mac_stats.good_octets_txd: 39919020372
dev.igc.0.mac_stats.good_octets_recvd: 135812019489
dev.igc.0.mac_stats.rx_frames_1024_1522: 83698227
dev.igc.0.mac_stats.rx_frames_512_1023: 26510709
dev.igc.0.mac_stats.rx_frames_256_511: 279480
dev.igc.0.mac_stats.rx_frames_128_255: 410790
dev.igc.0.mac_stats.rx_frames_65_127: 10680338
dev.igc.0.mac_stats.rx_frames_64: 26084811
dev.igc.0.mac_stats.mcast_pkts_recvd: 0
dev.igc.0.mac_stats.bcast_pkts_recvd: 24188678
dev.igc.0.mac_stats.good_pkts_recvd: 147664355
dev.igc.0.mac_stats.total_pkts_recvd: 147669598
dev.igc.0.mac_stats.xoff_txd: 0
dev.igc.0.mac_stats.xoff_recvd: 0
dev.igc.0.mac_stats.xon_txd: 0
dev.igc.0.mac_stats.xon_recvd: 0
dev.igc.0.mac_stats.alignment_errs: 0
dev.igc.0.mac_stats.crc_errs: 0
dev.igc.0.mac_stats.recv_errs: 0
dev.igc.0.mac_stats.recv_jabber: 0
dev.igc.0.mac_stats.recv_oversize: 0
dev.igc.0.mac_stats.recv_fragmented: 0
dev.igc.0.mac_stats.recv_undersize: 0
dev.igc.0.mac_stats.recv_no_buff: 0
dev.igc.0.mac_stats.missed_packets: 0
dev.igc.0.mac_stats.defer_count: 0
dev.igc.0.mac_stats.sequence_errors: 0
dev.igc.0.mac_stats.symbol_errors: 0
dev.igc.0.mac_stats.collision_count: 0
dev.igc.0.mac_stats.late_coll: 0
dev.igc.0.mac_stats.multiple_coll: 0
dev.igc.0.mac_stats.single_coll: 0
dev.igc.0.mac_stats.excess_coll: 0
dev.igc.0.queue_rx_3.rx_irq: 0
dev.igc.0.queue_rx_3.rxd_tail: 182
dev.igc.0.queue_rx_3.rxd_head: 183
dev.igc.0.queue_rx_2.rx_irq: 0
dev.igc.0.queue_rx_2.rxd_tail: 831
dev.igc.0.queue_rx_2.rxd_head: 832
dev.igc.0.queue_rx_1.rx_irq: 0
dev.igc.0.queue_rx_1.rxd_tail: 364
dev.igc.0.queue_rx_1.rxd_head: 365
dev.igc.0.queue_rx_0.rx_irq: 0
dev.igc.0.queue_rx_0.rxd_tail: 211
dev.igc.0.queue_rx_0.rxd_head: 212
dev.igc.0.queue_tx_3.tx_irq: 0
dev.igc.0.queue_tx_3.txd_tail: 562
dev.igc.0.queue_tx_3.txd_head: 562
dev.igc.0.queue_tx_2.tx_irq: 0
dev.igc.0.queue_tx_2.txd_tail: 643
dev.igc.0.queue_tx_2.txd_head: 643
dev.igc.0.queue_tx_1.tx_irq: 0
dev.igc.0.queue_tx_1.txd_tail: 732
dev.igc.0.queue_tx_1.txd_head: 732
dev.igc.0.queue_tx_0.tx_irq: 0
dev.igc.0.queue_tx_0.txd_tail: 535
dev.igc.0.queue_tx_0.txd_head: 535
dev.igc.0.fc_low_water: 32752
dev.igc.0.fc_high_water: 32768
dev.igc.0.rx_control: 71335938
dev.igc.0.device_control: 1478231617
dev.igc.0.watchdog_timeouts: 0
dev.igc.0.rx_overruns: 0
dev.igc.0.link_irq: 3
dev.igc.0.dropped: 0
dev.igc.0.eee_control: 1
dev.igc.0.itr: 488
dev.igc.0.tx_abs_int_delay: 66
dev.igc.0.rx_abs_int_delay: 66
dev.igc.0.tx_int_delay: 66
dev.igc.0.rx_int_delay: 0
dev.igc.0.rs_dump: 0
dev.igc.0.reg_dump: General Registers
	CTRL	 581c0641
	STATUS	 00380683
	CTRL_EXIT	 100000c0

Interrupt Registers
	ICR	 00000000

RX Registers
	RCTL	 04408002
	RDLEN	 00004000
	RDH	 000000d4
	RDT	 000000d3
	RXDCTL	 02040808
	RDBAL	 02b29000
	RDBAH	 00000000

TX Registers
	TCTL	 a503f0fa
	TDBAL	 02aeb000
	TDBAH	 00000000
	TDLEN	 00004000
	TDH	 00000217
	TDT	 00000217
	TXDCTL	 0201011f
	TDFH	 00000000
	TDFT	 00000000
	TDFHS	 00000000
	TDFPC	 00000000


dev.igc.0.fc: 3
dev.igc.0.debug: -1
dev.igc.0.nvm: -1
dev.igc.0.iflib.rxq3.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq3.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq3.rxq_fl0.cidx: 183
dev.igc.0.iflib.rxq3.rxq_fl0.pidx: 182
dev.igc.0.iflib.rxq3.cpu: 3
dev.igc.0.iflib.rxq2.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq2.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq2.rxq_fl0.cidx: 832
dev.igc.0.iflib.rxq2.rxq_fl0.pidx: 831
dev.igc.0.iflib.rxq2.cpu: 2
dev.igc.0.iflib.rxq1.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq1.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq1.rxq_fl0.cidx: 365
dev.igc.0.iflib.rxq1.rxq_fl0.pidx: 364
dev.igc.0.iflib.rxq1.cpu: 1
dev.igc.0.iflib.rxq0.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq0.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq0.rxq_fl0.cidx: 212
dev.igc.0.iflib.rxq0.rxq_fl0.pidx: 211
dev.igc.0.iflib.rxq0.cpu: 0
dev.igc.0.iflib.txq3.r_abdications: 0
dev.igc.0.iflib.txq3.r_restarts: 0
dev.igc.0.iflib.txq3.r_stalls: 0
dev.igc.0.iflib.txq3.r_starts: 24413002
dev.igc.0.iflib.txq3.r_drops: 0
dev.igc.0.iflib.txq3.r_enqueues: 24414191
dev.igc.0.iflib.txq3.ring_state: pidx_head: 2031 pidx_tail: 2031 cidx: 2031 state: IDLE
dev.igc.0.iflib.txq3.txq_cleaned: 24542728
dev.igc.0.iflib.txq3.txq_processed: 24542768
dev.igc.0.iflib.txq3.txq_in_use: 42
dev.igc.0.iflib.txq3.txq_cidx_processed: 560
dev.igc.0.iflib.txq3.txq_cidx: 520
dev.igc.0.iflib.txq3.txq_pidx: 562
dev.igc.0.iflib.txq3.no_tx_dma_setup: 0
dev.igc.0.iflib.txq3.txd_encap_efbig: 0
dev.igc.0.iflib.txq3.tx_map_failed: 0
dev.igc.0.iflib.txq3.no_desc_avail: 0
dev.igc.0.iflib.txq3.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq3.m_pullups: 0
dev.igc.0.iflib.txq3.mbuf_defrag: 0
dev.igc.0.iflib.txq3.cpu: 3
dev.igc.0.iflib.txq2.r_abdications: 0
dev.igc.0.iflib.txq2.r_restarts: 0
dev.igc.0.iflib.txq2.r_stalls: 0
dev.igc.0.iflib.txq2.r_starts: 14895304
dev.igc.0.iflib.txq2.r_drops: 0
dev.igc.0.iflib.txq2.r_enqueues: 14897134
dev.igc.0.iflib.txq2.ring_state: pidx_head: 2030 pidx_tail: 2030 cidx: 2030 state: IDLE
dev.igc.0.iflib.txq2.txq_cleaned: 15051354
dev.igc.0.iflib.txq2.txq_processed: 15051394
dev.igc.0.iflib.txq2.txq_in_use: 41
dev.igc.0.iflib.txq2.txq_cidx_processed: 642
dev.igc.0.iflib.txq2.txq_cidx: 602
dev.igc.0.iflib.txq2.txq_pidx: 643
dev.igc.0.iflib.txq2.no_tx_dma_setup: 0
dev.igc.0.iflib.txq2.txd_encap_efbig: 0
dev.igc.0.iflib.txq2.tx_map_failed: 0
dev.igc.0.iflib.txq2.no_desc_avail: 0
dev.igc.0.iflib.txq2.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq2.m_pullups: 0
dev.igc.0.iflib.txq2.mbuf_defrag: 0
dev.igc.0.iflib.txq2.cpu: 2
dev.igc.0.iflib.txq1.r_abdications: 0
dev.igc.0.iflib.txq1.r_restarts: 0
dev.igc.0.iflib.txq1.r_stalls: 0
dev.igc.0.iflib.txq1.r_starts: 17466940
dev.igc.0.iflib.txq1.r_drops: 0
dev.igc.0.iflib.txq1.r_enqueues: 17479699
dev.igc.0.iflib.txq1.ring_state: pidx_head: 0019 pidx_tail: 0019 cidx: 0019 state: IDLE
dev.igc.0.iflib.txq1.txq_cleaned: 17631922
dev.igc.0.iflib.txq1.txq_processed: 17631962
dev.igc.0.iflib.txq1.txq_in_use: 42
dev.igc.0.iflib.txq1.txq_cidx_processed: 730
dev.igc.0.iflib.txq1.txq_cidx: 690
dev.igc.0.iflib.txq1.txq_pidx: 732
dev.igc.0.iflib.txq1.no_tx_dma_setup: 0
dev.igc.0.iflib.txq1.txd_encap_efbig: 0
dev.igc.0.iflib.txq1.tx_map_failed: 0
dev.igc.0.iflib.txq1.no_desc_avail: 0
dev.igc.0.iflib.txq1.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq1.m_pullups: 0
dev.igc.0.iflib.txq1.mbuf_defrag: 0
dev.igc.0.iflib.txq1.cpu: 1
dev.igc.0.iflib.txq0.r_abdications: 0
dev.igc.0.iflib.txq0.r_restarts: 0
dev.igc.0.iflib.txq0.r_stalls: 0
dev.igc.0.iflib.txq0.r_starts: 21684406
dev.igc.0.iflib.txq0.r_drops: 0
dev.igc.0.iflib.txq0.r_enqueues: 21696943
dev.igc.0.iflib.txq0.ring_state: pidx_head: 0431 pidx_tail: 0431 cidx: 0431 state: IDLE
dev.igc.0.iflib.txq0.txq_cleaned: 21832174
dev.igc.0.iflib.txq0.txq_processed: 21832214
dev.igc.0.iflib.txq0.txq_in_use: 41
dev.igc.0.iflib.txq0.txq_cidx_processed: 534
dev.igc.0.iflib.txq0.txq_cidx: 494
dev.igc.0.iflib.txq0.txq_pidx: 535
dev.igc.0.iflib.txq0.no_tx_dma_setup: 0
dev.igc.0.iflib.txq0.txd_encap_efbig: 0
dev.igc.0.iflib.txq0.tx_map_failed: 0
dev.igc.0.iflib.txq0.no_desc_avail: 0
dev.igc.0.iflib.txq0.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq0.m_pullups: 0
dev.igc.0.iflib.txq0.mbuf_defrag: 0
dev.igc.0.iflib.txq0.cpu: 0
dev.igc.0.iflib.override_nrxds: 0
dev.igc.0.iflib.override_ntxds: 0
dev.igc.0.iflib.use_logical_cores: 0
dev.igc.0.iflib.separate_txrx: 0
dev.igc.0.iflib.core_offset: 0
dev.igc.0.iflib.tx_abdicate: 0
dev.igc.0.iflib.rx_budget: 0
dev.igc.0.iflib.disable_msix: 0
dev.igc.0.iflib.override_qs_enable: 0
dev.igc.0.iflib.override_nrxqs: 1
dev.igc.0.iflib.override_ntxqs: 0
dev.igc.0.iflib.driver_version: 1
dev.igc.0.%parent: pci2
dev.igc.0.%pnpinfo: vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000 class=0x020000
dev.igc.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 handle=\_SB_.PC00.RP05.PXSX
dev.igc.0.%driver: igc
dev.igc.0.%desc: Intel(R) Ethernet Controller I226-V

[rmeskill]

igc.0 is the WAN that is seeing the loss

[rmeskill]

Just to add to the curiosity, all loss cleared up approximately 2 hours ago and appears to have remained stably loss-free since then. I spoke to my provider again and they seem quite insistent there is nothing wrong with my circuit and they see no reason why this should be happening. However, they also confirmed the modem they have given me is quite dumb and 1) cannot hard-code interface speeds/duplex and 2) does not support a DMZ host, so if I moved it back to router mode I'd need to manually open each and every port.

[stephenw10]

@rmeskill said in High interrupts on WAN/LAN interfaces?:

dev.igc.0.mac_stats.good_pkts_recvd: 147664355
dev.igc.0.mac_stats.total_pkts_recvd: 147669598

Hmm, the only thing I see there is that. Implying ~5000 bad packets received. But no errors so probably just rejected. Not really a huge number out of 147M.

I see about the same percentages here:

dev.igc.3.mac_stats.good_pkts_recvd: 1208385
dev.igc.3.mac_stats.total_pkts_recvd: 1208642

And have no issues.

One difference I do see is no msix_vector values:

dev.igc.3.iflib.allocated_msix_vectors: 2
dev.igc.3.iflib.use_extra_msix_vectors: 0

Do you see that in the boot logs?

igc0: <Intel(R) Ethernet Controller I226-V> mem 0x67a00000-0x67afffff,0x67b00000-0x67b03fff at device 0.0 on pci2
igc0: Using 1024 TX descriptors and 1024 RX descriptors
igc0: Using 1 RX queues 1 TX queues
igc0: Using MSI-X interrupts with 2 vectors
igc0: Ethernet address: 00:08:a2:12:ec:d4
igc0: netmap queues/slots: TX 1/1024, RX 1/1024

[stephenw10]

Oh and note that the igc driver can only link using negotiated speed and duplex. It will fail to connect to a NIC that is actually using a fixed speed and does not negotiate.

[rmeskill]

@stephenw10

igc0: <Intel(R) Ethernet Controller I226-V> mem 0x80200000-0x802fffff,0x80300000-0x80303fff at device 0.0 on pci2
igc0: Using 1024 TX descriptors and 1024 RX descriptors
igc0: Using 4 RX queues 4 TX queues
igc0: Using MSI-X interrupts with 5 vectors
igc0: Ethernet address: 7c:2b:e1:13:d7:66
igc0: netmap queues/slots: TX 4/1024, RX 4/1024

[stephenw10]

Hmm, did you reboot after setting override_nrxqs to 1? You might need to set that as a loader value rather than a tunable

[rmeskill]

@stephenw10 I did, but now I've added it to loader value, rebooted, and it's up:

loss is back, though-was completely clean for about 3h, now back to 40-60% loss...

[rmeskill]

If I'm honest, I'm starting to just think my provider might, on top of having some issues in their network, be blocking ICMP packets sporadically. Because despite seeing 20-60%+ loss, I'm seeing quite low RTTs that are staying quite consistent:

The issue for me has been my secondary gateways (via Wireguard tunnels) are dropping because I have them set to drop based on loss, but perhaps I should instead change the down setting to latency and see how that treats me... There are still definitely gremlins in there somewhere, as sometimes I do crap out and throughput drops to basically zero, but it seems somewhat counterintuitive that I can be allegedly running loss of 20-60% but also see fine real-world performance.

[stephenw10]

Are you seeing the NIC actually using 1 queue now?

Ah, also check: sysctl hw.igc.max_interrupt_rate

I think we bumped that default to 20000

[rmeskill]

@stephenw10
Yes, actually seeing 1 queue now, but still loss
igc0: <Intel(R) Ethernet Controller I226-V> mem 0x80200000-0x802fffff,0x80300000-0x80303fff at device 0.0 on pci2
igc0: Using 1024 TX descriptors and 1024 RX descriptors
igc0: Using 1 RX queues 1 TX queues
igc0: Using MSI-X interrupts with 2 vectors
igc0: Ethernet address: 7c:2b:e1:13:d7:66
igc0: netmap queues/slots: TX 1/1024, RX 1/1024
Checked the interrupt rate and it was 20000, so no dice there

[stephenw10]

Hmm, well seems to be configured correctly. At least largely the same as the 4200 which works fine.

I guess I'd try to test the WAN with something else.

[rmeskill]

@stephenw10 Yeah, I'm out of ideas as well. I can definitely 'feel' the crap performance when my packet loss is upwards of 50%, even if it feels 'better' down around 20%. I might even just try turning off my provider router for an hour or something just to see if that'll do anything-I'm absolutely beyond guessing at this point... Second option is just to go with a 5G router from another provider, but not sure what sort of throughput I can get from that.

Actually, unrelated, but as long as I've got your attention: if I have a USB wireless adapter would it be possible to use that to connect to my neighbor's wireless and set that as a secondary gateway in the event mine has packet loss? I plugged it in and it clearly comes up in boot, but I can't see it anywhere in the GUI...
Root mount waiting for: usbus0
ugen0.2: <Realtek 802.11ac NIC> at usbus0

[rmeskill]

In a hail mary tried to shut off the provider router overnight just to see if resting it somehow (heat/saved setting/etc) would fix it, but on turning it back on I'm seeing 50% loss again. Called the provider again and they confirm they're seeing no issues, but they're going to send a NID/in-line test for 24h to see if they can spot anything.