What causes Wan "Error in"?

[LoGman]

--- EdgeRouter ER-4 WAN Statistics

[stephenw10]

Well likely they are errors passed from upstream then.

If you reassign ix3 as a new interface connected to something local and pass traffic over it do you see errors?

Steve

[LoGman]

@stephenw10

I'll test when I get home.

The lan1_backup igc0 interface worked just fine.

[stephenw10]

You mean showed no errors?

It's the ix driver/hardware specifically which now shows numerous error types there. If you can run ix3 via the same switch but with local traffic and see no errors that rules out a local hardware issue IMO.

Steve

[LoGman]

When I did boot the device I did get LAN Errors Out (4)

Here is this test

ix3 (Wan2_Local_test)
Test1 file 10.19 GiB copy file from IX3 to LAN > no errors in ix3.
Test2 file 10.19 GiB copy file from LAN to IX3 > no errors in ix3.

ix3 (Wan2_Local_test)
Test1 Speedtest.net > IX3 TO WAN : No Errors in ix3.
Test2 Speedtest.net > IX3 TO WAN : No Errors in ix3.
Test3 Speedtest.net > IX3 TO WAN : No Errors in ix3.
Test4 Speedtest.net > IX3 TO WAN : No Errors in ix3.

ix3 (Wan2_Local_test)
Test1 Rapidgator > WAN to IX3 : File transfers from Rapidgator 1.9 GB : no errors in ix3.
Test2 Rapidgator > WAN to IX3 : File transfers from Rapidgator 1.9 GB : no errors in ix3.
Test3 Rapidgator > WAN to IX3 : Multiple (2) File transfers from Rapidgator 1.9 GB : no errors in ix3.
Test4 Rapidgator > WAN to IX3 : Multiple (3) File transfers from Rapidgator 1.9 GB : no errors in ix3.

---

IX1 (LAN)
Test1 Speedtest.net > WAN to IX1 : No Errors in ix0.
Test2 Speedtest.net > WAN to IX1 : No Errors in ix0.
Test3 Speedtest.net > WAN to IX1 : No Errors in ix0.

IX1 (LAN)
Test1 Rapidgator > WAN to IX1 : File transfers from Rapidgator 1.9 GB : No Errors in ix0.
Test2 Rapidgator > WAN to IX1 : File transfers from Rapidgator 1.9 GB : (Total errors 6468) in ix0.
Test3 Rapidgator > WAN to IX1 : Multiple (2) File transfers from Rapidgator 1.9 GB : (Total errors 6468) in ix0.
Test4 Rapidgator > WAN to IX1 : Multiple (3) File transfers from Rapidgator 1.9 GB : (Total errors 6468) in ix0
Test5 Rapidgator > WAN to IX1 : Multiple (6) File transfers from Rapidgator 1.9 GB : (Total errors 6468) in ix0.
Test6 Speedtest.net > WAN to IX1 : (Total errors 6468) in ix0.
Test7 Rapidgator > WAN to IX1 : Multiple (6) File transfers from Rapidgator 1.9 GB : (Total errors 6468) in ix0.
Test8 Speedtest.net > WAN to IX1 : (Total errors 6468) in ix0.

• Before the test, I did do a Firmware upgrade.
• Current Firmware Version 02.01.00.05t-uc-14
• Latest Firmware Version 02.02.00.00t-uc-15

[WAN] sysctl dev.ix.0
dev.ix.0.wake: 0
dev.ix.0.dmac: 0
dev.ix.0.fw_version: eTrack 0x8000084b PHY FW V65535
dev.ix.0.advertise_speed: 0
dev.ix.0.fc: 3
dev.ix.0.mac_stats.tx_frames_1024_1522: 4600206
dev.ix.0.mac_stats.tx_frames_512_1023: 14245
dev.ix.0.mac_stats.tx_frames_256_511: 5897
dev.ix.0.mac_stats.tx_frames_128_255: 2233
dev.ix.0.mac_stats.tx_frames_65_127: 1388626
dev.ix.0.mac_stats.tx_frames_64: 4484671
dev.ix.0.mac_stats.management_pkts_txd: 0
dev.ix.0.mac_stats.mcast_pkts_txd: 6
dev.ix.0.mac_stats.bcast_pkts_txd: 7
dev.ix.0.mac_stats.good_pkts_txd: 10495878
dev.ix.0.mac_stats.total_pkts_txd: 10496057
dev.ix.0.mac_stats.good_octets_txd: 7391946984
dev.ix.0.mac_stats.checksum_errs: 1
dev.ix.0.mac_stats.management_pkts_drpd: 0
dev.ix.0.mac_stats.management_pkts_rcvd: 0
dev.ix.0.mac_stats.recv_jabberd: 0
dev.ix.0.mac_stats.recv_oversized: 0
dev.ix.0.mac_stats.recv_fragmented: 0
dev.ix.0.mac_stats.recv_undersized: 0
dev.ix.0.mac_stats.rx_frames_1024_1522: 47798581
dev.ix.0.mac_stats.rx_frames_512_1023: 51262
dev.ix.0.mac_stats.rx_frames_256_511: 260481
dev.ix.0.mac_stats.rx_frames_128_255: 7677
dev.ix.0.mac_stats.rx_frames_65_127: 35626
dev.ix.0.mac_stats.rx_frames_64: 765691
dev.ix.0.mac_stats.bcast_pkts_rcvd: 0
dev.ix.0.mac_stats.mcast_pkts_rcvd: 1712
dev.ix.0.mac_stats.good_pkts_rcvd: 48919318
dev.ix.0.mac_stats.total_pkts_rcvd: 48923624
dev.ix.0.mac_stats.good_octets_rcvd: 72687592172
dev.ix.0.mac_stats.total_octets_rcvd: 72690596777
dev.ix.0.mac_stats.xoff_recvd: 0
dev.ix.0.mac_stats.xoff_txd: 171
dev.ix.0.mac_stats.xon_recvd: 0
dev.ix.0.mac_stats.xon_txd: 8
dev.ix.0.mac_stats.rx_missed_packets: 6468
dev.ix.0.mac_stats.rec_len_errs: 0
dev.ix.0.mac_stats.remote_faults: 0
dev.ix.0.mac_stats.local_faults: 1
dev.ix.0.mac_stats.short_discards: 0
dev.ix.0.mac_stats.byte_errs: 0
dev.ix.0.mac_stats.ill_errs: 0
dev.ix.0.mac_stats.crc_errs: 0
dev.ix.0.mac_stats.rx_errs: 6469
dev.ix.0.queue3.rx_discarded: 0
dev.ix.0.queue3.rx_copies: 0
dev.ix.0.queue3.rx_bytes: 14657681956
dev.ix.0.queue3.rx_packets: 9943928
dev.ix.0.queue3.rxd_tail: 887
dev.ix.0.queue3.rxd_head: 888
dev.ix.0.queue3.irqs: 6312359
dev.ix.0.queue3.interrupt_rate: 31250
dev.ix.0.queue3.tx_packets: 2544993
dev.ix.0.queue3.tso_tx: 0
dev.ix.0.queue3.txd_tail: 722
dev.ix.0.queue3.txd_head: 722
dev.ix.0.queue2.rx_discarded: 0
dev.ix.0.queue2.rx_copies: 0
dev.ix.0.queue2.rx_bytes: 14259027898
dev.ix.0.queue2.rx_packets: 9701789
dev.ix.0.queue2.rxd_tail: 412
dev.ix.0.queue2.rxd_head: 413
dev.ix.0.queue2.irqs: 5898628
dev.ix.0.queue2.interrupt_rate: 31250
dev.ix.0.queue2.tx_packets: 2439291
dev.ix.0.queue2.tso_tx: 0
dev.ix.0.queue2.txd_tail: 257
dev.ix.0.queue2.txd_head: 257
dev.ix.0.queue1.rx_discarded: 0
dev.ix.0.queue1.rx_copies: 0
dev.ix.0.queue1.rx_bytes: 26193117954
dev.ix.0.queue1.rx_packets: 17522425
dev.ix.0.queue1.rxd_tail: 1784
dev.ix.0.queue1.rxd_head: 1785
dev.ix.0.queue1.irqs: 9031975
dev.ix.0.queue1.interrupt_rate: 31250
dev.ix.0.queue1.tx_packets: 2463416
dev.ix.0.queue1.tso_tx: 0
dev.ix.0.queue1.txd_tail: 1398
dev.ix.0.queue1.txd_head: 1398
dev.ix.0.queue0.rx_discarded: 0
dev.ix.0.queue0.rx_copies: 0
dev.ix.0.queue0.rx_bytes: 17372305467
dev.ix.0.queue0.rx_packets: 11744709
dev.ix.0.queue0.rxd_tail: 1476
dev.ix.0.queue0.rxd_head: 1477
dev.ix.0.queue0.irqs: 6601864
dev.ix.0.queue0.interrupt_rate: 31250
dev.ix.0.queue0.tx_packets: 3048180
dev.ix.0.queue0.tso_tx: 0
dev.ix.0.queue0.txd_tail: 1519
dev.ix.0.queue0.txd_head: 1519
dev.ix.0.link_irq: 3
dev.ix.0.watchdog_events: 0
dev.ix.0.dropped: 0
dev.ix.0.iflib.rxq3.rxq_fl0.buf_size: 2048
dev.ix.0.iflib.rxq3.rxq_fl0.credits: 2047
dev.ix.0.iflib.rxq3.rxq_fl0.cidx: 888
dev.ix.0.iflib.rxq3.rxq_fl0.pidx: 887
dev.ix.0.iflib.rxq3.cpu: 3
dev.ix.0.iflib.rxq2.rxq_fl0.buf_size: 2048
dev.ix.0.iflib.rxq2.rxq_fl0.credits: 2047
dev.ix.0.iflib.rxq2.rxq_fl0.cidx: 413
dev.ix.0.iflib.rxq2.rxq_fl0.pidx: 412
dev.ix.0.iflib.rxq2.cpu: 2
dev.ix.0.iflib.rxq1.rxq_fl0.buf_size: 2048
dev.ix.0.iflib.rxq1.rxq_fl0.credits: 2047
dev.ix.0.iflib.rxq1.rxq_fl0.cidx: 1785
dev.ix.0.iflib.rxq1.rxq_fl0.pidx: 1784
dev.ix.0.iflib.rxq1.cpu: 1
dev.ix.0.iflib.rxq0.rxq_fl0.buf_size: 2048
dev.ix.0.iflib.rxq0.rxq_fl0.credits: 2047
dev.ix.0.iflib.rxq0.rxq_fl0.cidx: 1477
dev.ix.0.iflib.rxq0.rxq_fl0.pidx: 1476
dev.ix.0.iflib.rxq0.cpu: 0
dev.ix.0.iflib.txq3.r_abdications: 0
dev.ix.0.iflib.txq3.r_restarts: 0
dev.ix.0.iflib.txq3.r_stalls: 0
dev.ix.0.iflib.txq3.r_starts: 2544992
dev.ix.0.iflib.txq3.r_drops: 0
dev.ix.0.iflib.txq3.r_enqueues: 2544994
dev.ix.0.iflib.txq3.ring_state: pidx_head: 1378 pidx_tail: 1378 cidx: 1378 state: IDLE
dev.ix.0.iflib.txq3.txq_cleaned: 5089966
dev.ix.0.iflib.txq3.txq_processed: 5089998
dev.ix.0.iflib.txq3.txq_in_use: 36
dev.ix.0.iflib.txq3.txq_cidx_processed: 718
dev.ix.0.iflib.txq3.txq_cidx: 686
dev.ix.0.iflib.txq3.txq_pidx: 722
dev.ix.0.iflib.txq3.no_tx_dma_setup: 0
dev.ix.0.iflib.txq3.txd_encap_efbig: 0
dev.ix.0.iflib.txq3.tx_map_failed: 0
dev.ix.0.iflib.txq3.no_desc_avail: 0
dev.ix.0.iflib.txq3.mbuf_defrag_failed: 0
dev.ix.0.iflib.txq3.m_pullups: 0
dev.ix.0.iflib.txq3.mbuf_defrag: 0
dev.ix.0.iflib.txq3.cpu: 3
dev.ix.0.iflib.txq2.r_abdications: 0
dev.ix.0.iflib.txq2.r_restarts: 0
dev.ix.0.iflib.txq2.r_stalls: 0
dev.ix.0.iflib.txq2.r_starts: 2439291
dev.ix.0.iflib.txq2.r_drops: 0
dev.ix.0.iflib.txq2.r_enqueues: 2439292
dev.ix.0.iflib.txq2.ring_state: pidx_head: 0124 pidx_tail: 0124 cidx: 0124 state: IDLE
dev.ix.0.iflib.txq2.txq_cleaned: 4878559
dev.ix.0.iflib.txq2.txq_processed: 4878591
dev.ix.0.iflib.txq2.txq_in_use: 34
dev.ix.0.iflib.txq2.txq_cidx_processed: 255
dev.ix.0.iflib.txq2.txq_cidx: 223
dev.ix.0.iflib.txq2.txq_pidx: 257
dev.ix.0.iflib.txq2.no_tx_dma_setup: 0
dev.ix.0.iflib.txq2.txd_encap_efbig: 0
dev.ix.0.iflib.txq2.tx_map_failed: 0
dev.ix.0.iflib.txq2.no_desc_avail: 0
dev.ix.0.iflib.txq2.mbuf_defrag_failed: 0
dev.ix.0.iflib.txq2.m_pullups: 0
dev.ix.0.iflib.txq2.mbuf_defrag: 0
dev.ix.0.iflib.txq2.cpu: 2
dev.ix.0.iflib.txq1.r_abdications: 0
dev.ix.0.iflib.txq1.r_restarts: 0
dev.ix.0.iflib.txq1.r_stalls: 0
dev.ix.0.iflib.txq1.r_starts: 2463414
dev.ix.0.iflib.txq1.r_drops: 0
dev.ix.0.iflib.txq1.r_enqueues: 2463416
dev.ix.0.iflib.txq1.ring_state: pidx_head: 1720 pidx_tail: 1720 cidx: 1720 state: IDLE
dev.ix.0.iflib.txq1.txq_cleaned: 4926804
dev.ix.0.iflib.txq1.txq_processed: 4926836
dev.ix.0.iflib.txq1.txq_in_use: 34
dev.ix.0.iflib.txq1.txq_cidx_processed: 1396
dev.ix.0.iflib.txq1.txq_cidx: 1364
dev.ix.0.iflib.txq1.txq_pidx: 1398
dev.ix.0.iflib.txq1.no_tx_dma_setup: 0
dev.ix.0.iflib.txq1.txd_encap_efbig: 0
dev.ix.0.iflib.txq1.tx_map_failed: 0
dev.ix.0.iflib.txq1.no_desc_avail: 0
dev.ix.0.iflib.txq1.mbuf_defrag_failed: 0
dev.ix.0.iflib.txq1.m_pullups: 0
dev.ix.0.iflib.txq1.mbuf_defrag: 0
dev.ix.0.iflib.txq1.cpu: 1
dev.ix.0.iflib.txq0.r_abdications: 0
dev.ix.0.iflib.txq0.r_restarts: 0
dev.ix.0.iflib.txq0.r_stalls: 0
dev.ix.0.iflib.txq0.r_starts: 3048128
dev.ix.0.iflib.txq0.r_drops: 0
dev.ix.0.iflib.txq0.r_enqueues: 3048182
dev.ix.0.iflib.txq0.ring_state: pidx_head: 0758 pidx_tail: 0758 cidx: 0758 state: IDLE
dev.ix.0.iflib.txq0.txq_cleaned: 6096331
dev.ix.0.iflib.txq0.txq_processed: 6096363
dev.ix.0.iflib.txq0.txq_in_use: 36
dev.ix.0.iflib.txq0.txq_cidx_processed: 1515
dev.ix.0.iflib.txq0.txq_cidx: 1483
dev.ix.0.iflib.txq0.txq_pidx: 1519
dev.ix.0.iflib.txq0.no_tx_dma_setup: 0
dev.ix.0.iflib.txq0.txd_encap_efbig: 0
dev.ix.0.iflib.txq0.tx_map_failed: 0
dev.ix.0.iflib.txq0.no_desc_avail: 0
dev.ix.0.iflib.txq0.mbuf_defrag_failed: 0
dev.ix.0.iflib.txq0.m_pullups: 5
dev.ix.0.iflib.txq0.mbuf_defrag: 0
dev.ix.0.iflib.txq0.cpu: 0
dev.ix.0.iflib.override_nrxds: 0
dev.ix.0.iflib.override_ntxds: 0
dev.ix.0.iflib.use_logical_cores: 0
dev.ix.0.iflib.separate_txrx: 0
dev.ix.0.iflib.core_offset: 0
dev.ix.0.iflib.tx_abdicate: 0
dev.ix.0.iflib.rx_budget: 0
dev.ix.0.iflib.disable_msix: 0
dev.ix.0.iflib.override_qs_enable: 0
dev.ix.0.iflib.override_nrxqs: 0
dev.ix.0.iflib.override_ntxqs: 0
dev.ix.0.iflib.driver_version: 4.0.1-k
dev.ix.0.%parent: pci9
dev.ix.0.%pnpinfo: vendor=0x8086 device=0x15c4 subvendor=0x8086 subdevice=0x0000 class=0x020000
dev.ix.0.%location: slot=0 function=0 dbsf=pci0:3:0:0 handle=\_SB_.PCI0.VRP0.LAN0
dev.ix.0.%driver: ix
dev.ix.0.%desc: Intel(R) X553 N (SFP+)

[LAN] sysctl dev.ix.1
dev.ix.1.wake: 0
dev.ix.1.dmac: 0
dev.ix.1.fw_version: eTrack 0x8000084b PHY FW V65535
dev.ix.1.advertise_speed: 0
dev.ix.1.fc: 3
dev.ix.1.mac_stats.tx_frames_1024_1522: 40426957
dev.ix.1.mac_stats.tx_frames_512_1023: 28376
dev.ix.1.mac_stats.tx_frames_256_511: 156704
dev.ix.1.mac_stats.tx_frames_128_255: 22587
dev.ix.1.mac_stats.tx_frames_65_127: 32949
dev.ix.1.mac_stats.tx_frames_64: 4167693
dev.ix.1.mac_stats.management_pkts_txd: 0
dev.ix.1.mac_stats.mcast_pkts_txd: 12
dev.ix.1.mac_stats.bcast_pkts_txd: 380
dev.ix.1.mac_stats.good_pkts_txd: 44835266
dev.ix.1.mac_stats.total_pkts_txd: 44835266
dev.ix.1.mac_stats.good_octets_txd: 61822014147
dev.ix.1.mac_stats.checksum_errs: 0
dev.ix.1.mac_stats.management_pkts_drpd: 0
dev.ix.1.mac_stats.management_pkts_rcvd: 0
dev.ix.1.mac_stats.recv_jabberd: 0
dev.ix.1.mac_stats.recv_oversized: 0
dev.ix.1.mac_stats.recv_fragmented: 0
dev.ix.1.mac_stats.recv_undersized: 0
dev.ix.1.mac_stats.rx_frames_1024_1522: 9837757
dev.ix.1.mac_stats.rx_frames_512_1023: 12512
dev.ix.1.mac_stats.rx_frames_256_511: 6327
dev.ix.1.mac_stats.rx_frames_128_255: 38048
dev.ix.1.mac_stats.rx_frames_65_127: 6861394
dev.ix.1.mac_stats.rx_frames_64: 0
dev.ix.1.mac_stats.bcast_pkts_rcvd: 27541
dev.ix.1.mac_stats.mcast_pkts_rcvd: 0
dev.ix.1.mac_stats.good_pkts_rcvd: 16756038
dev.ix.1.mac_stats.total_pkts_rcvd: 16758164
dev.ix.1.mac_stats.good_octets_rcvd: 15476141254
dev.ix.1.mac_stats.total_octets_rcvd: 15476424890
dev.ix.1.mac_stats.xoff_recvd: 0
dev.ix.1.mac_stats.xoff_txd: 0
dev.ix.1.mac_stats.xon_recvd: 0
dev.ix.1.mac_stats.xon_txd: 0
dev.ix.1.mac_stats.rx_missed_packets: 0
dev.ix.1.mac_stats.rec_len_errs: 0
dev.ix.1.mac_stats.remote_faults: 0
dev.ix.1.mac_stats.local_faults: 1
dev.ix.1.mac_stats.short_discards: 0
dev.ix.1.mac_stats.byte_errs: 0
dev.ix.1.mac_stats.ill_errs: 0
dev.ix.1.mac_stats.crc_errs: 0
dev.ix.1.mac_stats.rx_errs: 0
dev.ix.1.queue3.rx_discarded: 0
dev.ix.1.queue3.rx_copies: 0
dev.ix.1.queue3.rx_bytes: 733496387
dev.ix.1.queue3.rx_packets: 1669503
dev.ix.1.queue3.rxd_tail: 383
dev.ix.1.queue3.rxd_head: 384
dev.ix.1.queue3.irqs: 4088146
dev.ix.1.queue3.interrupt_rate: 31250
dev.ix.1.queue3.tx_packets: 5976290
dev.ix.1.queue3.tso_tx: 0
dev.ix.1.queue3.txd_tail: 672
dev.ix.1.queue3.txd_head: 675
dev.ix.1.queue2.rx_discarded: 0
dev.ix.1.queue2.rx_copies: 0
dev.ix.1.queue2.rx_bytes: 12421022105
dev.ix.1.queue2.rx_packets: 11729716
dev.ix.1.queue2.rxd_tail: 819
dev.ix.1.queue2.rxd_head: 820
dev.ix.1.queue2.irqs: 6154248
dev.ix.1.queue2.interrupt_rate: 31250
dev.ix.1.queue2.tx_packets: 4451810
dev.ix.1.queue2.tso_tx: 0
dev.ix.1.queue2.txd_tail: 827
dev.ix.1.queue2.txd_head: 827
dev.ix.1.queue1.rx_discarded: 0
dev.ix.1.queue1.rx_copies: 0
dev.ix.1.queue1.rx_bytes: 907332242
dev.ix.1.queue1.rx_packets: 1437841
dev.ix.1.queue1.rxd_tail: 144
dev.ix.1.queue1.rxd_head: 145
dev.ix.1.queue1.irqs: 6538205
dev.ix.1.queue1.interrupt_rate: 31250
dev.ix.1.queue1.tx_packets: 12982156
dev.ix.1.queue1.tso_tx: 0
dev.ix.1.queue1.txd_tail: 1066
dev.ix.1.queue1.txd_head: 1066
dev.ix.1.queue0.rx_discarded: 0
dev.ix.1.queue0.rx_copies: 0
dev.ix.1.queue0.rx_bytes: 1280243682
dev.ix.1.queue0.rx_packets: 1918994
dev.ix.1.queue0.rxd_tail: 17
dev.ix.1.queue0.rxd_head: 18
dev.ix.1.queue0.irqs: 10315196
dev.ix.1.queue0.interrupt_rate: 31250
dev.ix.1.queue0.tx_packets: 21425045
dev.ix.1.queue0.tso_tx: 0
dev.ix.1.queue0.txd_tail: 1334
dev.ix.1.queue0.txd_head: 1334
dev.ix.1.link_irq: 3
dev.ix.1.watchdog_events: 0
dev.ix.1.dropped: 0
dev.ix.1.iflib.rxq3.rxq_fl0.buf_size: 2048
dev.ix.1.iflib.rxq3.rxq_fl0.credits: 2047
dev.ix.1.iflib.rxq3.rxq_fl0.cidx: 398
dev.ix.1.iflib.rxq3.rxq_fl0.pidx: 397
dev.ix.1.iflib.rxq3.cpu: 3
dev.ix.1.iflib.rxq2.rxq_fl0.buf_size: 2048
dev.ix.1.iflib.rxq2.rxq_fl0.credits: 2047
dev.ix.1.iflib.rxq2.rxq_fl0.cidx: 820
dev.ix.1.iflib.rxq2.rxq_fl0.pidx: 819
dev.ix.1.iflib.rxq2.cpu: 2
dev.ix.1.iflib.rxq1.rxq_fl0.buf_size: 2048
dev.ix.1.iflib.rxq1.rxq_fl0.credits: 2047
dev.ix.1.iflib.rxq1.rxq_fl0.cidx: 145
dev.ix.1.iflib.rxq1.rxq_fl0.pidx: 144
dev.ix.1.iflib.rxq1.cpu: 1
dev.ix.1.iflib.rxq0.rxq_fl0.buf_size: 2048
dev.ix.1.iflib.rxq0.rxq_fl0.credits: 2047
dev.ix.1.iflib.rxq0.rxq_fl0.cidx: 18
dev.ix.1.iflib.rxq0.rxq_fl0.pidx: 17
dev.ix.1.iflib.rxq0.cpu: 0
dev.ix.1.iflib.txq3.r_abdications: 0
dev.ix.1.iflib.txq3.r_restarts: 0
dev.ix.1.iflib.txq3.r_stalls: 0
dev.ix.1.iflib.txq3.r_starts: 5976331
dev.ix.1.iflib.txq3.r_drops: 0
dev.ix.1.iflib.txq3.r_enqueues: 5976334
dev.ix.1.iflib.txq3.ring_state: pidx_head: 0272 pidx_tail: 0272 cidx: 0272 state: IDLE
dev.ix.1.iflib.txq3.txq_cleaned: 11954923
dev.ix.1.iflib.txq3.txq_processed: 11954955
dev.ix.1.iflib.txq3.txq_in_use: 37
dev.ix.1.iflib.txq3.txq_cidx_processed: 784
dev.ix.1.iflib.txq3.txq_cidx: 752
dev.ix.1.iflib.txq3.txq_pidx: 788
dev.ix.1.iflib.txq3.no_tx_dma_setup: 0
dev.ix.1.iflib.txq3.txd_encap_efbig: 0
dev.ix.1.iflib.txq3.tx_map_failed: 0
dev.ix.1.iflib.txq3.no_desc_avail: 0
dev.ix.1.iflib.txq3.mbuf_defrag_failed: 0
dev.ix.1.iflib.txq3.m_pullups: 0
dev.ix.1.iflib.txq3.mbuf_defrag: 0
dev.ix.1.iflib.txq3.cpu: 3
dev.ix.1.iflib.txq2.r_abdications: 0
dev.ix.1.iflib.txq2.r_restarts: 0
dev.ix.1.iflib.txq2.r_stalls: 0
dev.ix.1.iflib.txq2.r_starts: 4451800
dev.ix.1.iflib.txq2.r_drops: 0
dev.ix.1.iflib.txq2.r_enqueues: 4451810
dev.ix.1.iflib.txq2.ring_state: pidx_head: 1506 pidx_tail: 1506 cidx: 1506 state: IDLE
dev.ix.1.iflib.txq2.txq_cleaned: 8905495
dev.ix.1.iflib.txq2.txq_processed: 8905527
dev.ix.1.iflib.txq2.txq_in_use: 36
dev.ix.1.iflib.txq2.txq_cidx_processed: 823
dev.ix.1.iflib.txq2.txq_cidx: 791
dev.ix.1.iflib.txq2.txq_pidx: 827
dev.ix.1.iflib.txq2.no_tx_dma_setup: 0
dev.ix.1.iflib.txq2.txd_encap_efbig: 0
dev.ix.1.iflib.txq2.tx_map_failed: 0
dev.ix.1.iflib.txq2.no_desc_avail: 0
dev.ix.1.iflib.txq2.mbuf_defrag_failed: 0
dev.ix.1.iflib.txq2.m_pullups: 0
dev.ix.1.iflib.txq2.mbuf_defrag: 0
dev.ix.1.iflib.txq2.cpu: 2
dev.ix.1.iflib.txq1.r_abdications: 0
dev.ix.1.iflib.txq1.r_restarts: 0
dev.ix.1.iflib.txq1.r_stalls: 0
dev.ix.1.iflib.txq1.r_starts: 12982123
dev.ix.1.iflib.txq1.r_drops: 0
dev.ix.1.iflib.txq1.r_enqueues: 12982156
dev.ix.1.iflib.txq1.ring_state: pidx_head: 1932 pidx_tail: 1932 cidx: 1932 state: IDLE
dev.ix.1.iflib.txq1.txq_cleaned: 25965574
dev.ix.1.iflib.txq1.txq_processed: 25965606
dev.ix.1.iflib.txq1.txq_in_use: 36
dev.ix.1.iflib.txq1.txq_cidx_processed: 1062
dev.ix.1.iflib.txq1.txq_cidx: 1030
dev.ix.1.iflib.txq1.txq_pidx: 1066
dev.ix.1.iflib.txq1.no_tx_dma_setup: 0
dev.ix.1.iflib.txq1.txd_encap_efbig: 0
dev.ix.1.iflib.txq1.tx_map_failed: 0
dev.ix.1.iflib.txq1.no_desc_avail: 0
dev.ix.1.iflib.txq1.mbuf_defrag_failed: 0
dev.ix.1.iflib.txq1.m_pullups: 0
dev.ix.1.iflib.txq1.mbuf_defrag: 0
dev.ix.1.iflib.txq1.cpu: 1
dev.ix.1.iflib.txq0.r_abdications: 0
dev.ix.1.iflib.txq0.r_restarts: 0
dev.ix.1.iflib.txq0.r_stalls: 0
dev.ix.1.iflib.txq0.r_starts: 21424972
dev.ix.1.iflib.txq0.r_drops: 0
dev.ix.1.iflib.txq0.r_enqueues: 21425047
dev.ix.1.iflib.txq0.ring_state: pidx_head: 0919 pidx_tail: 0919 cidx: 0919 state: IDLE
dev.ix.1.iflib.txq0.txq_cleaned: 42851602
dev.ix.1.iflib.txq0.txq_processed: 42851634
dev.ix.1.iflib.txq0.txq_in_use: 36
dev.ix.1.iflib.txq0.txq_cidx_processed: 1330
dev.ix.1.iflib.txq0.txq_cidx: 1298
dev.ix.1.iflib.txq0.txq_pidx: 1334
dev.ix.1.iflib.txq0.no_tx_dma_setup: 0
dev.ix.1.iflib.txq0.txd_encap_efbig: 0
dev.ix.1.iflib.txq0.tx_map_failed: 0
dev.ix.1.iflib.txq0.no_desc_avail: 0
dev.ix.1.iflib.txq0.mbuf_defrag_failed: 0
dev.ix.1.iflib.txq0.m_pullups: 0
dev.ix.1.iflib.txq0.mbuf_defrag: 0
dev.ix.1.iflib.txq0.cpu: 0
dev.ix.1.iflib.override_nrxds: 0
dev.ix.1.iflib.override_ntxds: 0
dev.ix.1.iflib.use_logical_cores: 0
dev.ix.1.iflib.separate_txrx: 0
dev.ix.1.iflib.core_offset: 0
dev.ix.1.iflib.tx_abdicate: 0
dev.ix.1.iflib.rx_budget: 0
dev.ix.1.iflib.disable_msix: 0
dev.ix.1.iflib.override_qs_enable: 0
dev.ix.1.iflib.override_nrxqs: 0
dev.ix.1.iflib.override_ntxqs: 0
dev.ix.1.iflib.driver_version: 4.0.1-k
dev.ix.1.%parent: pci9
dev.ix.1.%pnpinfo: vendor=0x8086 device=0x15c4 subvendor=0x8086 subdevice=0x0000 class=0x020000
dev.ix.1.%location: slot=0 function=1 dbsf=pci0:3:0:1 handle=\_SB_.PCI0.VRP0.LAN1
dev.ix.1.%driver: ix
dev.ix.1.%desc: Intel(R) X553 N (SFP+)

igc0 (LAN1_Backup)
Test1 file 10.19 GiB copy file from igc0 to LAN > no errors in ix3.
Test2 file 10.19 GiB copy file from LAN to igc0 > no errors in ix3.
Test3 Speedtest.net > Wan : No Errors in igc0. | Wan Errors in 6471.
Test4 Speedtest.net > Wan : No Errors in igc0. | Wan Errors in 6471.
Test5 Rapidgator > WAN : Multiple (6) File transfer from Rapidgator 1.9 GB : (Total errors 6471) in ix0.
Test6 Rapidgator > WAN : Multiple (2) File transfer from Rapidgator 1 GB : (Total errors 6471) in ix0.
Test7 Speedtest.net > Wan : No Errors in igc0. | Wan Errors in 6471.
Test8 Speedtest.net > Wan : No Errors in igc0. | Wan Errors in 6471.

[stephenw10]

@logman said in What causes Wan "Error in"?:

dev.ix.0.mac_stats.rx_missed_packets: 6468

So that and 1 crc error. The old driver version would only have shown the crc error in the Rx errors count.

If I'm reading data correctly all of those errors were produced by a single test? And the test competed correctly?

rx missed packets like that is often because the local buffer is full and it cannot recive more data. At Gigabit speeds I wouldn't normally expect to see that on the WAN.
You might have a flow-control mismatch with the switch though.
Currently it shows: dev.ix.0.fc: 3
Which means flow control is active in and out:

[22.01-RELEASE][admin@6100.stevew.lan]/root: sysctl -d dev.ix.3.fc
dev.ix.3.fc: 
Set flow control mode using these values:
	0 - off
	1 - rx pause
	2 - tx pause
	3 - tx and rx pause

Is the switch port set that way?

Steve

[LoGman]

@stephenw10

Yeah, I did each test one by one.

I did Check Flow Control is Off on the switch

[stephenw10]

Then you should disable it in pfSense for that interface. Or enable it in the switch.
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#flow-control

Steve

[LoGman]

@stephenw10

I disabled it to see if the problem is fixed.

[MoonKnight]

@logman
Or maybe try it without your switch. Connect your computer directly to your pfSense LAN ports and run the tests again

[LoGman]

@moonknight
Already Done that.

---

@stephenw10

~5h

Only CRC errors

[WAN] sysctl dev.ix.0 | grep "miss\|errs"
dev.ix.0.mac_stats.checksum_errs: 12
dev.ix.0.mac_stats.rx_missed_packets: 6468
dev.ix.0.mac_stats.rec_len_errs: 0
dev.ix.0.mac_stats.byte_errs: 0
dev.ix.0.mac_stats.ill_errs: 0
dev.ix.0.mac_stats.crc_errs: 0
dev.ix.0.mac_stats.rx_errs: 6480

[stephenw10]

Ah, so good after the flow-control change?

[LoGman]

@stephenw10

Yes, when flow-control is disabled no anymore "missed packets" only "checksum errors".

[WAN] sysctl dev.ix.0 | grep "miss\|errs"
dev.ix.0.mac_stats.checksum_errs: 279
dev.ix.0.mac_stats.rx_missed_packets: 6468
dev.ix.0.mac_stats.rec_len_errs: 0
dev.ix.0.mac_stats.byte_errs: 0
dev.ix.0.mac_stats.ill_errs: 0
dev.ix.0.mac_stats.crc_errs: 0
dev.ix.0.mac_stats.rx_errs: 6747

[stephenw10]

Ok, so to be clear when it was logging missed packets flow control was on in pfSense but off in the switch?

That would make sense if pfSense was sending the 'stop sending' packets and the switch ignored it and kept sending.

Steve

[LoGman]

@stephenw10
Yeah, it was on as default.

[LoGman]

Why is Flow-control not persistent when set in "System Tunables"

Reboot and settings again wrong.

Is this the only way?

So the correct way is "file: /boot/loader.conf.local hw.ix.flow_control="0" ?

And this Is not the correct way? 
/boot/loader.conf.local:
dev.ix.0.fc="0"
dev.ix.1.fc="0"
dev.ix.2.fc="0"
dev.ix.3.fc="0"

[stephenw10]

You can set: hw.ix.flow_control=0 as a loader variable and it will apply the default to all ix NICs.