WAN out errors since 2.8 upgrade

[brookheather]

Since upgrading to CE2.8 I've noticed a large number of WAN out errors on my interfaces page. I'm pretty sure I didn't have any noticeable errors when running 2.7.2. I am on BT FTTP 500/75 and have tried changing the network card (from a dual i350 to a dual i226), changed the ethernet cable to a new one and rebooted the ONT - no difference. Here are my stats since I rebooted yesterday:

Uptime 1d 00:46:48
In/out packets 38248861/22401176 (38.88 GiB/13.79 GiB)
In/out packets (pass) 38248861/22401176 (38.88 GiB/13.79 GiB)
In/out packets (block) 25920/9 (1.81 MiB/440 B)
In/out errors 0/92115
Collisions 0

Is this number of WAN out errors to be expected? Has something changed in the way the errors are reported in 2.8? Are the WAN errors due to packets being rejected by the ONT?

For comparison here are my LAN stats on the same PCIe network card connected to my Unifi switch (the LAN traffic is lower as it doesn't include 9GB of WireGuard traffic):

In/out packets 11291330/27507985 (4.09 GiB/29.05 GiB)
In/out packets (pass) 11291330/27507985 (4.09 GiB/29.05 GiB)
In/out packets (block) 18976/20 (1.85 MiB/1 KiB)
In/out errors 0/0
Collisions 0

[stephenw10]

Check the mac stats in the sysctls to see what sort of errors those are.
For example: sysctl dev.igc.0

0.4% is not a huge error rate but more than I'd expect. What is that NIC physically connected to?

[brookheather]

@stephenw10 The NIC is physically connected to the OpenReach Huawei ONT (Gigabit connection) - the output from the command is:

dev.igc.0.interrupts.rx_desc_min_thresh: 0
dev.igc.0.interrupts.asserts: 43969407
dev.igc.0.mac_stats.tso_txd: 0
dev.igc.0.mac_stats.tx_frames_1024_1522: 9891015
dev.igc.0.mac_stats.tx_frames_512_1023: 748735
dev.igc.0.mac_stats.tx_frames_256_511: 1006813
dev.igc.0.mac_stats.tx_frames_128_255: 1742288
dev.igc.0.mac_stats.tx_frames_65_127: 10311937
dev.igc.0.mac_stats.tx_frames_64: 352282
dev.igc.0.mac_stats.mcast_pkts_txd: 0
dev.igc.0.mac_stats.bcast_pkts_txd: 9
dev.igc.0.mac_stats.good_pkts_txd: 24053070
dev.igc.0.mac_stats.total_pkts_txd: 24053070
dev.igc.0.mac_stats.good_octets_txd: 16003109611
dev.igc.0.mac_stats.good_octets_recvd: 48063405551
dev.igc.0.mac_stats.rx_frames_1024_1522: 32590421
dev.igc.0.mac_stats.rx_frames_512_1023: 1036421
dev.igc.0.mac_stats.rx_frames_256_511: 1184829
dev.igc.0.mac_stats.rx_frames_128_255: 2528507
dev.igc.0.mac_stats.rx_frames_65_127: 4555804
dev.igc.0.mac_stats.rx_frames_64: 337518
dev.igc.0.mac_stats.mcast_pkts_recvd: 2
dev.igc.0.mac_stats.bcast_pkts_recvd: 0
dev.igc.0.mac_stats.good_pkts_recvd: 42233500
dev.igc.0.mac_stats.total_pkts_recvd: 42233500
dev.igc.0.mac_stats.mgmt_pkts_txd: 0
dev.igc.0.mac_stats.mgmt_pkts_drop: 0
dev.igc.0.mac_stats.mgmt_pkts_recvd: 0
dev.igc.0.mac_stats.unsupported_fc_recvd: 0
dev.igc.0.mac_stats.xoff_txd: 0
dev.igc.0.mac_stats.xoff_recvd: 0
dev.igc.0.mac_stats.xon_txd: 0
dev.igc.0.mac_stats.xon_recvd: 0
dev.igc.0.mac_stats.alignment_errs: 0
dev.igc.0.mac_stats.crc_errs: 0
dev.igc.0.mac_stats.recv_errs: 0
dev.igc.0.mac_stats.recv_jabber: 0
dev.igc.0.mac_stats.recv_oversize: 0
dev.igc.0.mac_stats.recv_fragmented: 0
dev.igc.0.mac_stats.recv_undersize: 0
dev.igc.0.mac_stats.recv_no_buff: 0
dev.igc.0.mac_stats.recv_length_errors: 0
dev.igc.0.mac_stats.missed_packets: 0
dev.igc.0.mac_stats.defer_count: 0
dev.igc.0.mac_stats.sequence_errors: 0
dev.igc.0.mac_stats.symbol_errors: 0
dev.igc.0.mac_stats.collision_count: 0
dev.igc.0.mac_stats.late_coll: 0
dev.igc.0.mac_stats.multiple_coll: 0
dev.igc.0.mac_stats.single_coll: 0
dev.igc.0.mac_stats.excess_coll: 0
dev.igc.0.queue_rx_3.rx_irq: 0
dev.igc.0.queue_rx_3.rxd_tail: 128
dev.igc.0.queue_rx_3.rxd_head: 0
dev.igc.0.queue_rx_3.interrupt_rate: 20832
dev.igc.0.queue_rx_2.rx_irq: 0
dev.igc.0.queue_rx_2.rxd_tail: 1
dev.igc.0.queue_rx_2.rxd_head: 2
dev.igc.0.queue_rx_2.interrupt_rate: 20832
dev.igc.0.queue_rx_1.rx_irq: 0
dev.igc.0.queue_rx_1.rxd_tail: 128
dev.igc.0.queue_rx_1.rxd_head: 0
dev.igc.0.queue_rx_1.interrupt_rate: 20832
dev.igc.0.queue_rx_0.rx_irq: 0
dev.igc.0.queue_rx_0.rxd_tail: 690
dev.igc.0.queue_rx_0.rxd_head: 691
dev.igc.0.queue_rx_0.interrupt_rate: 4032
dev.igc.0.queue_tx_3.tx_irq: 0
dev.igc.0.queue_tx_3.txd_tail: 424
dev.igc.0.queue_tx_3.txd_head: 424
dev.igc.0.queue_tx_3.interrupt_rate: 20832
dev.igc.0.queue_tx_2.tx_irq: 0
dev.igc.0.queue_tx_2.txd_tail: 552
dev.igc.0.queue_tx_2.txd_head: 552
dev.igc.0.queue_tx_2.interrupt_rate: 20832
dev.igc.0.queue_tx_1.tx_irq: 0
dev.igc.0.queue_tx_1.txd_tail: 644
dev.igc.0.queue_tx_1.txd_head: 644
dev.igc.0.queue_tx_1.interrupt_rate: 20832
dev.igc.0.queue_tx_0.tx_irq: 0
dev.igc.0.queue_tx_0.txd_tail: 506
dev.igc.0.queue_tx_0.txd_head: 506
dev.igc.0.queue_tx_0.interrupt_rate: 4032
dev.igc.0.fc_low_water: 32752
dev.igc.0.fc_high_water: 32768
dev.igc.0.rx_control: 71335970
dev.igc.0.device_control: 1478231617
dev.igc.0.watchdog_timeouts: 0
dev.igc.0.rx_overruns: 0
dev.igc.0.link_irq: 5
dev.igc.0.dropped: 0
dev.igc.0.eee_control: 1
dev.igc.0.tso_tcp_flags_mask_last_segment: 3967
dev.igc.0.tso_tcp_flags_mask_middle_segment: 3958
dev.igc.0.tso_tcp_flags_mask_first_segment: 4086
dev.igc.0.dmac: 0
dev.igc.0.rs_dump: 0
dev.igc.0.reg_dump: General Registers
	CTRL	 581c0641
	STATUS	 00380683
	CTRL_EXIT	 100000c0

Interrupt Registers
	ICR	 00000000

RX Registers
	RCTL	 04408022
	RDLEN	 00004000
	RDH	 000002b3
	RDT	 000002b2
	RXDCTL	 02040808
	RDBAL	 02e9e000
	RDBAH	 00000000

TX Registers
	TCTL	 a503f0fa
	TDBAL	 02e60000
	TDBAH	 00000000
	TDLEN	 00004000
	TDH	 000001fa
	TDT	 000001fa
	TXDCTL	 0201011f
	TDFH	 00000000
	TDFT	 00000000
	TDFHS	 00000000
	TDFPC	 00000000


dev.igc.0.fc: 3
dev.igc.0.debug: -1
dev.igc.0.fw_version: EEPROM V2.23-0 eTrack 0x80000397
dev.igc.0.enable_aim: 1
dev.igc.0.nvm: -1
dev.igc.0.iflib.rxq3.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq3.rxq_fl0.credits: 128
dev.igc.0.iflib.rxq3.rxq_fl0.cidx: 0
dev.igc.0.iflib.rxq3.rxq_fl0.pidx: 128
dev.igc.0.iflib.rxq3.cpu: 3
dev.igc.0.iflib.rxq2.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq2.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq2.rxq_fl0.cidx: 2
dev.igc.0.iflib.rxq2.rxq_fl0.pidx: 1
dev.igc.0.iflib.rxq2.cpu: 2
dev.igc.0.iflib.rxq1.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq1.rxq_fl0.credits: 128
dev.igc.0.iflib.rxq1.rxq_fl0.cidx: 0
dev.igc.0.iflib.rxq1.rxq_fl0.pidx: 128
dev.igc.0.iflib.rxq1.cpu: 1
dev.igc.0.iflib.rxq0.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq0.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq0.rxq_fl0.cidx: 691
dev.igc.0.iflib.rxq0.rxq_fl0.pidx: 690
dev.igc.0.iflib.rxq0.cpu: 0
dev.igc.0.iflib.txq3.r_abdications: 2
dev.igc.0.iflib.txq3.r_restarts: 0
dev.igc.0.iflib.txq3.r_stalls: 0
dev.igc.0.iflib.txq3.r_starts: 8692668
dev.igc.0.iflib.txq3.r_drops: 0
dev.igc.0.iflib.txq3.r_enqueues: 8698710
dev.igc.0.iflib.txq3.ring_state: pidx_head: 0854 pidx_tail: 0854 cidx: 0854 state: IDLE
dev.igc.0.iflib.txq3.txq_cleaned: 20038013
dev.igc.0.iflib.txq3.txq_processed: 20038053
dev.igc.0.iflib.txq3.txq_in_use: 43
dev.igc.0.iflib.txq3.txq_cidx_processed: 421
dev.igc.0.iflib.txq3.txq_cidx: 381
dev.igc.0.iflib.txq3.txq_pidx: 424
dev.igc.0.iflib.txq3.no_tx_dma_setup: 0
dev.igc.0.iflib.txq3.txd_encap_efbig: 0
dev.igc.0.iflib.txq3.tx_map_failed: 0
dev.igc.0.iflib.txq3.no_desc_avail: 0
dev.igc.0.iflib.txq3.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq3.m_pullups: 3254948
dev.igc.0.iflib.txq3.mbuf_defrag: 0
dev.igc.0.iflib.txq3.cpu: 3
dev.igc.0.iflib.txq2.r_abdications: 0
dev.igc.0.iflib.txq2.r_restarts: 0
dev.igc.0.iflib.txq2.r_stalls: 0
dev.igc.0.iflib.txq2.r_starts: 3823423
dev.igc.0.iflib.txq2.r_drops: 0
dev.igc.0.iflib.txq2.r_enqueues: 3825109
dev.igc.0.iflib.txq2.ring_state: pidx_head: 1493 pidx_tail: 1493 cidx: 1493 state: IDLE
dev.igc.0.iflib.txq2.txq_cleaned: 7308797
dev.igc.0.iflib.txq2.txq_processed: 7308837
dev.igc.0.iflib.txq2.txq_in_use: 43
dev.igc.0.iflib.txq2.txq_cidx_processed: 549
dev.igc.0.iflib.txq2.txq_cidx: 509
dev.igc.0.iflib.txq2.txq_pidx: 552
dev.igc.0.iflib.txq2.no_tx_dma_setup: 0
dev.igc.0.iflib.txq2.txd_encap_efbig: 0
dev.igc.0.iflib.txq2.tx_map_failed: 0
dev.igc.0.iflib.txq2.no_desc_avail: 0
dev.igc.0.iflib.txq2.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq2.m_pullups: 2872129
dev.igc.0.iflib.txq2.mbuf_defrag: 0
dev.igc.0.iflib.txq2.cpu: 2
dev.igc.0.iflib.txq1.r_abdications: 0
dev.igc.0.iflib.txq1.r_restarts: 0
dev.igc.0.iflib.txq1.r_stalls: 0
dev.igc.0.iflib.txq1.r_starts: 3850397
dev.igc.0.iflib.txq1.r_drops: 0
dev.igc.0.iflib.txq1.r_enqueues: 3852590
dev.igc.0.iflib.txq1.ring_state: pidx_head: 0302 pidx_tail: 0302 cidx: 0302 state: IDLE
dev.igc.0.iflib.txq1.txq_cleaned: 7776856
dev.igc.0.iflib.txq1.txq_processed: 7776896
dev.igc.0.iflib.txq1.txq_in_use: 44
dev.igc.0.iflib.txq1.txq_cidx_processed: 640
dev.igc.0.iflib.txq1.txq_cidx: 600
dev.igc.0.iflib.txq1.txq_pidx: 644
dev.igc.0.iflib.txq1.no_tx_dma_setup: 0
dev.igc.0.iflib.txq1.txd_encap_efbig: 0
dev.igc.0.iflib.txq1.tx_map_failed: 0
dev.igc.0.iflib.txq1.no_desc_avail: 0
dev.igc.0.iflib.txq1.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq1.m_pullups: 2752996
dev.igc.0.iflib.txq1.mbuf_defrag: 0
dev.igc.0.iflib.txq1.cpu: 1
dev.igc.0.iflib.txq0.r_abdications: 0
dev.igc.0.iflib.txq0.r_restarts: 0
dev.igc.0.iflib.txq0.r_stalls: 0
dev.igc.0.iflib.txq0.r_starts: 7671245
dev.igc.0.iflib.txq0.r_drops: 0
dev.igc.0.iflib.txq0.r_enqueues: 7678564
dev.igc.0.iflib.txq0.ring_state: pidx_head: 0612 pidx_tail: 0612 cidx: 0612 state: IDLE
dev.igc.0.iflib.txq0.txq_cleaned: 19215824
dev.igc.0.iflib.txq0.txq_processed: 19215864
dev.igc.0.iflib.txq0.txq_in_use: 42
dev.igc.0.iflib.txq0.txq_cidx_processed: 504
dev.igc.0.iflib.txq0.txq_cidx: 464
dev.igc.0.iflib.txq0.txq_pidx: 506
dev.igc.0.iflib.txq0.no_tx_dma_setup: 0
dev.igc.0.iflib.txq0.txd_encap_efbig: 0
dev.igc.0.iflib.txq0.tx_map_failed: 0
dev.igc.0.iflib.txq0.no_desc_avail: 0
dev.igc.0.iflib.txq0.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq0.m_pullups: 2094876
dev.igc.0.iflib.txq0.mbuf_defrag: 0
dev.igc.0.iflib.txq0.cpu: 0
dev.igc.0.iflib.override_nrxds: 0
dev.igc.0.iflib.override_ntxds: 0
dev.igc.0.iflib.allocated_msix_vectors: 5
dev.igc.0.iflib.use_extra_msix_vectors: 0
dev.igc.0.iflib.use_logical_cores: 0
dev.igc.0.iflib.separate_txrx: 0
dev.igc.0.iflib.core_offset: 0
dev.igc.0.iflib.tx_abdicate: 0
dev.igc.0.iflib.rx_budget: 0
dev.igc.0.iflib.disable_msix: 0
dev.igc.0.iflib.override_qs_enable: 0
dev.igc.0.iflib.override_nrxqs: 0
dev.igc.0.iflib.override_ntxqs: 0
dev.igc.0.iflib.driver_version: 1
dev.igc.0.%iommu: rid=0x300
dev.igc.0.%parent: pci3
dev.igc.0.%pnpinfo: vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000 class=0x020000
dev.igc.0.%location: slot=0 function=0 dbsf=pci0:3:0:0
dev.igc.0.%driver: igc
dev.igc.0.%desc: Intel(R) Ethernet Controller I226-V

[stephenw10]

Hmm, nothing obvuiously wrong there. I assume igc0 is the WAN?

[brookheather]

@stephenw10 Yes igc0 is the WAN. Speedtests and latency are fine - I haven't observed any actual issues with the network which is why I was wondering whether these WAN out errors are actually real?

[stephenw10]

Hmm, curious. I'd expect to see some errors in the stats there. Is it linked at 1G?

[brookheather]

@stephenw10 Yes linked at 1G on both WAN and LAN.

[stephenw10]

Ah, wait is this still PPPoE? BT have yet to deem my area fit for fiber.

In which case the errors could be on pppoe0. Are you using the new if_pppoe in 2.8?

[brookheather]

@stephenw10 Yes it is PPPoE and I am using the new if_pppoe code.

[stephenw10]

Ah, interesting.

Try looking at netstat -i, check if the errors are shown there. And if that are only on pppoe0 and not igc0.

Do you still see errors if you switch back to the old pppoe driver?

[brookheather]

@stephenw10 Yes errors are only showing on pppoe0 - there are none on igc0. I haven't tried switching off if_pppoe unless it is actually causing a problem? Do you think it is just mis-reporting the errors?

Name         Mtu Network                   Address                                                Ipkts Ierrs Idrop     Opkts Oerrs  Coll
igc0        1508 <Link#1>                  xx:xx:xx:xx:xx:xx                                   43520768     0     0  24375770     0     0
pppoe0      1500 <Link#11>                 pppoe0                                              38313171     0     0  22268397 93436     0

[stephenw10]

It could be. Though I'm using here (stuck on FTTC) without any errors:

pppoe0     1500 <Link#20>                pppoe0                                           27001708     0     0   14339373     0     0
pppoe0        - fe80::%pppoe0/64         fe80::201:21ff:fe01:6775%pppoe0                         0     -     -          0     -     -
pppoe0        - x.x.13.9/32           9.13.x.x.dyn.plus.net                           458252     -     -          1     -     -
pppoe1     1500 <Link#22>                pppoe1                                            6970581     0     0    2942040     0     0
pppoe1        - fe80::%pppoe1/64         fe80::201:21ff:fe01:6775%pppoe1                        34     -     -          5     -     -
pppoe1        - x.x.86.29/32          hostxx-x-x-29.btcentralplus.com      44548     -     -          0     -     -

Do you see any errors on warning in the system logs?

The debug output from if_pppoe is unlikely to be helpful here. It will fill the logs! But you can enable it with ifconfig pppoe0 debug and disable it with ifconfig pppoe0 -debug.

[brookheather]

@stephenw10 I tried enabling the debug - there were just lots of lines with pppoe - no errors that I could see. I will try reverting if_pppoe later today and see if that makes any difference.

[brookheather]

@stephenw10 So I have switched back to the old PPPoE code and it works fine - no errors at all - I have run a number of speedtests and the error count is still zero so it looks like there is an issue with the new if_pppoe code.

Name         Mtu Network                   Address                                            Ipkts Ierrs Idrop    Opkts Oerrs  Coll
pppoe0      1500 <Link#9>                  pppoe0                                            916340     0     0   375667     0     0

[brookheather]

@stephenw10 This bug seems to already be reported here: redmine though I don't see any follow up?

[stephenw10]

Yup I see it. I'll reach out to the dev and let him know. Though I think he's at BSDCan next week,