High interrupts on WAN/LAN interfaces?
-
I went to try and turn eee on for the NICs and it completely hosed my system-had to go back and turn it off (which appears the default) via a direct console connection to be able to get things working again. I think the interrupt rates themselves are actually irrelevant, rather it's the packet loss that's the symptom and the issue. I have absolutely no idea why or what is correlated to the loss. I'm currently trying an external USB NIC to see if that has the same issues or not, setup in a GW Group configuration so it should automatically failover if the main i226-v NIC starts seeing loss...
-
@stephenw10 I did also move the dev.cpu.0.cx_lowest = C3, dev.hwpstate_intel.0.epp = 90 to = 100. But in the 5m since I added the USB NIC it seems that's also seeing loss. But I can confirm I have another (GLiNet) router plugged into this modem, running a similar setup, and it works fine with no loss, so the only consistency here is the Topton PFSense box...
-
Actually at this point it just looks like the PFSense system itself is completely hosed. I've no idea why, but now even the GUI loads awfully slowly from the local network and sometimes gives me a 503 error on reboot. I'm probably just going to blow it away and start fresh
-
Yup eee should be disabled.
The values for hwpstate_intel should be set lower for higher performance. Setting it to 100 means the most power saving / lowest performance. Try setting it to 50 or just disable that. The default value depends on what the BIOS passes though.
-
@stephenw10 as it is I just deleted it entirely, so it shouldn't be doing any limiting at all. I couldn't find any specific features in the BIOS pointing at this, though, so I'm just running on defaults now. Fully rebuilt PFSense and still having the same loss issues, though, but, as I've confirmed loss on a second router as well, I'm leaning back towards an issue with the provider instead...
-
The hwpstate_intel driver is enabled by default and should select 50 by default. That should be more than sufficient but as I said some of those devices have some very odd choices set in the BIOS by default. You should try setting a high performance value there like 30 and see if it makes any difference.
-
@stephenw10 I'll give 30 a go. The seeming irony to my situation is my RTTs to my gateway monitor IP aren't even bad, it's just I'm seeing enormous loss (on ICMP):
If I run a throughput test I sometimes see 200-400Mbps down, which is in line with what I'd expect, but the issue is tunnels are dropping and apps disconnecting -
Check the mac stats shown in:
sysctl dev.igc.0
or whichever NICs you are using. -
@stephenw10 What should I be looking for?
dev.igc.0.mac_stats.tso_txd: 0
dev.igc.0.mac_stats.tx_frames_1024_1522: 1344934
dev.igc.0.mac_stats.tx_frames_512_1023: 15901
dev.igc.0.mac_stats.tx_frames_256_511: 13890
dev.igc.0.mac_stats.tx_frames_128_255: 21666
dev.igc.0.mac_stats.tx_frames_65_127: 1302018
dev.igc.0.mac_stats.tx_frames_64: 680328
dev.igc.0.mac_stats.mcast_pkts_txd: 0
dev.igc.0.mac_stats.bcast_pkts_txd: 3
dev.igc.0.mac_stats.good_pkts_txd: 3378737
dev.igc.0.mac_stats.total_pkts_txd: 3378737
dev.igc.0.mac_stats.good_octets_txd: 2039654276
dev.igc.0.mac_stats.good_octets_recvd: 4277785341
dev.igc.0.mac_stats.rx_frames_1024_1522: 2895839
dev.igc.0.mac_stats.rx_frames_512_1023: 83944
dev.igc.0.mac_stats.rx_frames_256_511: 18340
dev.igc.0.mac_stats.rx_frames_128_255: 38581
dev.igc.0.mac_stats.rx_frames_65_127: 613592
dev.igc.0.mac_stats.rx_frames_64: 1816390
dev.igc.0.mac_stats.mcast_pkts_recvd: 0
dev.igc.0.mac_stats.bcast_pkts_recvd: 1759297
dev.igc.0.mac_stats.good_pkts_recvd: 5466687
dev.igc.0.mac_stats.total_pkts_recvd: 5467091
dev.igc.0.mac_stats.xoff_txd: 0
dev.igc.0.mac_stats.xoff_recvd: 0
dev.igc.0.mac_stats.xon_txd: 0
dev.igc.0.mac_stats.xon_recvd: 0
dev.igc.0.mac_stats.alignment_errs: 0
dev.igc.0.mac_stats.crc_errs: 0
dev.igc.0.mac_stats.recv_errs: 0
dev.igc.0.mac_stats.recv_jabber: 0
dev.igc.0.mac_stats.recv_oversize: 0
dev.igc.0.mac_stats.recv_fragmented: 0
dev.igc.0.mac_stats.recv_undersize: 0
dev.igc.0.mac_stats.recv_no_buff: 0
dev.igc.0.mac_stats.missed_packets: 0
dev.igc.0.mac_stats.defer_count: 0
dev.igc.0.mac_stats.sequence_errors: 0
dev.igc.0.mac_stats.symbol_errors: 0
dev.igc.0.mac_stats.collision_count: 0
dev.igc.0.mac_stats.late_coll: 0
dev.igc.0.mac_stats.multiple_coll: 0
dev.igc.0.mac_stats.single_coll: 0
dev.igc.0.mac_stats.excess_coll: 0igc.0 is my WAN, fwiw
-
Missed packets or errors. Check the other NICs in use. Also check the other sysctl values for errors.
Also check dev.igc.X.iflib.override_nrxqs. We had to set that 1 on the 4200 to prevent context switching issues.
-
[24.03-RELEASE][admin@pfSense.home.arpa]/root: sysctl -a | grep iflib.override_nrxqs
dev.igc.3.iflib.override_nrxqs: 0
dev.igc.2.iflib.override_nrxqs: 0
dev.igc.1.iflib.override_nrxqs: 0
dev.igc.0.iflib.override_nrxqs: 0 -
Mmm, try setting those to 1. You may need to add them as loader values to /boot/loader.conf.local.
-
@stephenw10 yeah, no dice on that.
The only thing I will say is I tried a USB NIC and that also was seeing packet loss, so at this point I can't fathom, besides a CPU issue as a whole with the Topton box, how this could be anything but the provider's problem. Unfortunately they insist there's no connectivity issues to their modem and the gateway I'm connecting to is upstream of that, so theoretically they should see an issue if there was one...
-
No difference at all with a much lower hwpstate_intel setting?
-
@stephenw10 I blew away my router entirely and rebuilt it from scratch, restoring some of the FW side of things but not the system tunables and other settings just so I could confirm I was starting with a clean config. Here's what I have:
And here's what we look like:
ANd here's what the system log looks like for gateways:
-
Hmm, and no dropped packets or errors in the sysctl output?
-
[24.03-RELEASE][admin@pfSense.home.arpa]/root: sysctl -a | grep drop
kern.ipc.tls.ifnet.reset_dropped: 0
vfs.cache.stats.drops: 0
net.inet.ip.intr_queue_drops: 0
net.inet.ip.intr_direct_queue_drops: 0
net.inet.icmp.drop_redirect: 0
net.inet.tcp.rexmit_drop_options: 0
net.inet.tcp.drop_synfin: 1
net.pf.default_to_drop: 0
hw.cxgbe.ofld_cong_drop: 0
hw.cxgbe.cong_drop: 0
hw.cxgbe.nm_cong_drop: 1
hw.cxgbe.drop_pkts_with_l4_errors: 0
hw.cxgbe.drop_pkts_with_l3_errors: 0
hw.cxgbe.drop_pkts_with_l2_errors: 1
hw.cxgbe.drop_ip_fragments: 0
kstat.zfs.misc.fm.erpt-dropped: 0
dev.igc.3.dropped: 0
dev.igc.3.iflib.txq3.r_drops: 0
dev.igc.3.iflib.txq2.r_drops: 0
dev.igc.3.iflib.txq1.r_drops: 0
dev.igc.3.iflib.txq0.r_drops: 0
dev.igc.2.dropped: 0
dev.igc.2.iflib.txq3.r_drops: 0
dev.igc.2.iflib.txq2.r_drops: 0
dev.igc.2.iflib.txq1.r_drops: 0
dev.igc.2.iflib.txq0.r_drops: 0
dev.igc.1.dropped: 0
dev.igc.1.iflib.txq3.r_drops: 0
dev.igc.1.iflib.txq2.r_drops: 0
dev.igc.1.iflib.txq1.r_drops: 0
dev.igc.1.iflib.txq0.r_drops: 0
dev.igc.0.dropped: 0
dev.igc.0.iflib.txq3.r_drops: 0
dev.igc.0.iflib.txq2.r_drops: 0
dev.igc.0.iflib.txq1.r_drops: 0
dev.igc.0.iflib.txq0.r_drops: 0 -
Can you just show the full output from
sysctl dev.igc
? -
[24.03-RELEASE][admin@pfSense.home.arpa]/root: sysctl dev.igc.0 dev.igc.0.wake: 0 dev.igc.0.interrupts.rx_desc_min_thresh: 0 dev.igc.0.interrupts.asserts: 135425174 dev.igc.0.mac_stats.tso_txd: 0 dev.igc.0.mac_stats.tx_frames_1024_1522: 25790251 dev.igc.0.mac_stats.tx_frames_512_1023: 404292 dev.igc.0.mac_stats.tx_frames_256_511: 288528 dev.igc.0.mac_stats.tx_frames_128_255: 388062 dev.igc.0.mac_stats.tx_frames_65_127: 44147139 dev.igc.0.mac_stats.tx_frames_64: 7442638 dev.igc.0.mac_stats.mcast_pkts_txd: 0 dev.igc.0.mac_stats.bcast_pkts_txd: 3 dev.igc.0.mac_stats.good_pkts_txd: 78460910 dev.igc.0.mac_stats.total_pkts_txd: 78460910 dev.igc.0.mac_stats.good_octets_txd: 39919020372 dev.igc.0.mac_stats.good_octets_recvd: 135812019489 dev.igc.0.mac_stats.rx_frames_1024_1522: 83698227 dev.igc.0.mac_stats.rx_frames_512_1023: 26510709 dev.igc.0.mac_stats.rx_frames_256_511: 279480 dev.igc.0.mac_stats.rx_frames_128_255: 410790 dev.igc.0.mac_stats.rx_frames_65_127: 10680338 dev.igc.0.mac_stats.rx_frames_64: 26084811 dev.igc.0.mac_stats.mcast_pkts_recvd: 0 dev.igc.0.mac_stats.bcast_pkts_recvd: 24188678 dev.igc.0.mac_stats.good_pkts_recvd: 147664355 dev.igc.0.mac_stats.total_pkts_recvd: 147669598 dev.igc.0.mac_stats.xoff_txd: 0 dev.igc.0.mac_stats.xoff_recvd: 0 dev.igc.0.mac_stats.xon_txd: 0 dev.igc.0.mac_stats.xon_recvd: 0 dev.igc.0.mac_stats.alignment_errs: 0 dev.igc.0.mac_stats.crc_errs: 0 dev.igc.0.mac_stats.recv_errs: 0 dev.igc.0.mac_stats.recv_jabber: 0 dev.igc.0.mac_stats.recv_oversize: 0 dev.igc.0.mac_stats.recv_fragmented: 0 dev.igc.0.mac_stats.recv_undersize: 0 dev.igc.0.mac_stats.recv_no_buff: 0 dev.igc.0.mac_stats.missed_packets: 0 dev.igc.0.mac_stats.defer_count: 0 dev.igc.0.mac_stats.sequence_errors: 0 dev.igc.0.mac_stats.symbol_errors: 0 dev.igc.0.mac_stats.collision_count: 0 dev.igc.0.mac_stats.late_coll: 0 dev.igc.0.mac_stats.multiple_coll: 0 dev.igc.0.mac_stats.single_coll: 0 dev.igc.0.mac_stats.excess_coll: 0 dev.igc.0.queue_rx_3.rx_irq: 0 dev.igc.0.queue_rx_3.rxd_tail: 182 dev.igc.0.queue_rx_3.rxd_head: 183 dev.igc.0.queue_rx_2.rx_irq: 0 dev.igc.0.queue_rx_2.rxd_tail: 831 dev.igc.0.queue_rx_2.rxd_head: 832 dev.igc.0.queue_rx_1.rx_irq: 0 dev.igc.0.queue_rx_1.rxd_tail: 364 dev.igc.0.queue_rx_1.rxd_head: 365 dev.igc.0.queue_rx_0.rx_irq: 0 dev.igc.0.queue_rx_0.rxd_tail: 211 dev.igc.0.queue_rx_0.rxd_head: 212 dev.igc.0.queue_tx_3.tx_irq: 0 dev.igc.0.queue_tx_3.txd_tail: 562 dev.igc.0.queue_tx_3.txd_head: 562 dev.igc.0.queue_tx_2.tx_irq: 0 dev.igc.0.queue_tx_2.txd_tail: 643 dev.igc.0.queue_tx_2.txd_head: 643 dev.igc.0.queue_tx_1.tx_irq: 0 dev.igc.0.queue_tx_1.txd_tail: 732 dev.igc.0.queue_tx_1.txd_head: 732 dev.igc.0.queue_tx_0.tx_irq: 0 dev.igc.0.queue_tx_0.txd_tail: 535 dev.igc.0.queue_tx_0.txd_head: 535 dev.igc.0.fc_low_water: 32752 dev.igc.0.fc_high_water: 32768 dev.igc.0.rx_control: 71335938 dev.igc.0.device_control: 1478231617 dev.igc.0.watchdog_timeouts: 0 dev.igc.0.rx_overruns: 0 dev.igc.0.link_irq: 3 dev.igc.0.dropped: 0 dev.igc.0.eee_control: 1 dev.igc.0.itr: 488 dev.igc.0.tx_abs_int_delay: 66 dev.igc.0.rx_abs_int_delay: 66 dev.igc.0.tx_int_delay: 66 dev.igc.0.rx_int_delay: 0 dev.igc.0.rs_dump: 0 dev.igc.0.reg_dump: General Registers CTRL 581c0641 STATUS 00380683 CTRL_EXIT 100000c0 Interrupt Registers ICR 00000000 RX Registers RCTL 04408002 RDLEN 00004000 RDH 000000d4 RDT 000000d3 RXDCTL 02040808 RDBAL 02b29000 RDBAH 00000000 TX Registers TCTL a503f0fa TDBAL 02aeb000 TDBAH 00000000 TDLEN 00004000 TDH 00000217 TDT 00000217 TXDCTL 0201011f TDFH 00000000 TDFT 00000000 TDFHS 00000000 TDFPC 00000000 dev.igc.0.fc: 3 dev.igc.0.debug: -1 dev.igc.0.nvm: -1 dev.igc.0.iflib.rxq3.rxq_fl0.buf_size: 2048 dev.igc.0.iflib.rxq3.rxq_fl0.credits: 1023 dev.igc.0.iflib.rxq3.rxq_fl0.cidx: 183 dev.igc.0.iflib.rxq3.rxq_fl0.pidx: 182 dev.igc.0.iflib.rxq3.cpu: 3 dev.igc.0.iflib.rxq2.rxq_fl0.buf_size: 2048 dev.igc.0.iflib.rxq2.rxq_fl0.credits: 1023 dev.igc.0.iflib.rxq2.rxq_fl0.cidx: 832 dev.igc.0.iflib.rxq2.rxq_fl0.pidx: 831 dev.igc.0.iflib.rxq2.cpu: 2 dev.igc.0.iflib.rxq1.rxq_fl0.buf_size: 2048 dev.igc.0.iflib.rxq1.rxq_fl0.credits: 1023 dev.igc.0.iflib.rxq1.rxq_fl0.cidx: 365 dev.igc.0.iflib.rxq1.rxq_fl0.pidx: 364 dev.igc.0.iflib.rxq1.cpu: 1 dev.igc.0.iflib.rxq0.rxq_fl0.buf_size: 2048 dev.igc.0.iflib.rxq0.rxq_fl0.credits: 1023 dev.igc.0.iflib.rxq0.rxq_fl0.cidx: 212 dev.igc.0.iflib.rxq0.rxq_fl0.pidx: 211 dev.igc.0.iflib.rxq0.cpu: 0 dev.igc.0.iflib.txq3.r_abdications: 0 dev.igc.0.iflib.txq3.r_restarts: 0 dev.igc.0.iflib.txq3.r_stalls: 0 dev.igc.0.iflib.txq3.r_starts: 24413002 dev.igc.0.iflib.txq3.r_drops: 0 dev.igc.0.iflib.txq3.r_enqueues: 24414191 dev.igc.0.iflib.txq3.ring_state: pidx_head: 2031 pidx_tail: 2031 cidx: 2031 state: IDLE dev.igc.0.iflib.txq3.txq_cleaned: 24542728 dev.igc.0.iflib.txq3.txq_processed: 24542768 dev.igc.0.iflib.txq3.txq_in_use: 42 dev.igc.0.iflib.txq3.txq_cidx_processed: 560 dev.igc.0.iflib.txq3.txq_cidx: 520 dev.igc.0.iflib.txq3.txq_pidx: 562 dev.igc.0.iflib.txq3.no_tx_dma_setup: 0 dev.igc.0.iflib.txq3.txd_encap_efbig: 0 dev.igc.0.iflib.txq3.tx_map_failed: 0 dev.igc.0.iflib.txq3.no_desc_avail: 0 dev.igc.0.iflib.txq3.mbuf_defrag_failed: 0 dev.igc.0.iflib.txq3.m_pullups: 0 dev.igc.0.iflib.txq3.mbuf_defrag: 0 dev.igc.0.iflib.txq3.cpu: 3 dev.igc.0.iflib.txq2.r_abdications: 0 dev.igc.0.iflib.txq2.r_restarts: 0 dev.igc.0.iflib.txq2.r_stalls: 0 dev.igc.0.iflib.txq2.r_starts: 14895304 dev.igc.0.iflib.txq2.r_drops: 0 dev.igc.0.iflib.txq2.r_enqueues: 14897134 dev.igc.0.iflib.txq2.ring_state: pidx_head: 2030 pidx_tail: 2030 cidx: 2030 state: IDLE dev.igc.0.iflib.txq2.txq_cleaned: 15051354 dev.igc.0.iflib.txq2.txq_processed: 15051394 dev.igc.0.iflib.txq2.txq_in_use: 41 dev.igc.0.iflib.txq2.txq_cidx_processed: 642 dev.igc.0.iflib.txq2.txq_cidx: 602 dev.igc.0.iflib.txq2.txq_pidx: 643 dev.igc.0.iflib.txq2.no_tx_dma_setup: 0 dev.igc.0.iflib.txq2.txd_encap_efbig: 0 dev.igc.0.iflib.txq2.tx_map_failed: 0 dev.igc.0.iflib.txq2.no_desc_avail: 0 dev.igc.0.iflib.txq2.mbuf_defrag_failed: 0 dev.igc.0.iflib.txq2.m_pullups: 0 dev.igc.0.iflib.txq2.mbuf_defrag: 0 dev.igc.0.iflib.txq2.cpu: 2 dev.igc.0.iflib.txq1.r_abdications: 0 dev.igc.0.iflib.txq1.r_restarts: 0 dev.igc.0.iflib.txq1.r_stalls: 0 dev.igc.0.iflib.txq1.r_starts: 17466940 dev.igc.0.iflib.txq1.r_drops: 0 dev.igc.0.iflib.txq1.r_enqueues: 17479699 dev.igc.0.iflib.txq1.ring_state: pidx_head: 0019 pidx_tail: 0019 cidx: 0019 state: IDLE dev.igc.0.iflib.txq1.txq_cleaned: 17631922 dev.igc.0.iflib.txq1.txq_processed: 17631962 dev.igc.0.iflib.txq1.txq_in_use: 42 dev.igc.0.iflib.txq1.txq_cidx_processed: 730 dev.igc.0.iflib.txq1.txq_cidx: 690 dev.igc.0.iflib.txq1.txq_pidx: 732 dev.igc.0.iflib.txq1.no_tx_dma_setup: 0 dev.igc.0.iflib.txq1.txd_encap_efbig: 0 dev.igc.0.iflib.txq1.tx_map_failed: 0 dev.igc.0.iflib.txq1.no_desc_avail: 0 dev.igc.0.iflib.txq1.mbuf_defrag_failed: 0 dev.igc.0.iflib.txq1.m_pullups: 0 dev.igc.0.iflib.txq1.mbuf_defrag: 0 dev.igc.0.iflib.txq1.cpu: 1 dev.igc.0.iflib.txq0.r_abdications: 0 dev.igc.0.iflib.txq0.r_restarts: 0 dev.igc.0.iflib.txq0.r_stalls: 0 dev.igc.0.iflib.txq0.r_starts: 21684406 dev.igc.0.iflib.txq0.r_drops: 0 dev.igc.0.iflib.txq0.r_enqueues: 21696943 dev.igc.0.iflib.txq0.ring_state: pidx_head: 0431 pidx_tail: 0431 cidx: 0431 state: IDLE dev.igc.0.iflib.txq0.txq_cleaned: 21832174 dev.igc.0.iflib.txq0.txq_processed: 21832214 dev.igc.0.iflib.txq0.txq_in_use: 41 dev.igc.0.iflib.txq0.txq_cidx_processed: 534 dev.igc.0.iflib.txq0.txq_cidx: 494 dev.igc.0.iflib.txq0.txq_pidx: 535 dev.igc.0.iflib.txq0.no_tx_dma_setup: 0 dev.igc.0.iflib.txq0.txd_encap_efbig: 0 dev.igc.0.iflib.txq0.tx_map_failed: 0 dev.igc.0.iflib.txq0.no_desc_avail: 0 dev.igc.0.iflib.txq0.mbuf_defrag_failed: 0 dev.igc.0.iflib.txq0.m_pullups: 0 dev.igc.0.iflib.txq0.mbuf_defrag: 0 dev.igc.0.iflib.txq0.cpu: 0 dev.igc.0.iflib.override_nrxds: 0 dev.igc.0.iflib.override_ntxds: 0 dev.igc.0.iflib.use_logical_cores: 0 dev.igc.0.iflib.separate_txrx: 0 dev.igc.0.iflib.core_offset: 0 dev.igc.0.iflib.tx_abdicate: 0 dev.igc.0.iflib.rx_budget: 0 dev.igc.0.iflib.disable_msix: 0 dev.igc.0.iflib.override_qs_enable: 0 dev.igc.0.iflib.override_nrxqs: 1 dev.igc.0.iflib.override_ntxqs: 0 dev.igc.0.iflib.driver_version: 1 dev.igc.0.%parent: pci2 dev.igc.0.%pnpinfo: vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000 class=0x020000 dev.igc.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 handle=\_SB_.PC00.RP05.PXSX dev.igc.0.%driver: igc dev.igc.0.%desc: Intel(R) Ethernet Controller I226-V
-
igc.0 is the WAN that is seeing the loss