X520 errors in 2.6 and 22.01?

cescofs

@stephenw10
do you have any idea how to solve? it's very strange only "errors in"

tom-ato

@stephenw10 That is a very real possibility. I'm fine with that answer if it is indeed what is happening.

stephenw10

@cescofs said in X520 errors in 2.6 and 22.01?:

@stephenw10
it's hard to admit ...

pfSense does nothing with the MAC stats, the driver updates them directly.
You could try the alternative driver in 2.5.2.

Steve

cescofs

Something must have happened with 2.6. Anyway, I connected the pfsense directly to my pc and.....zero errors!!!!

stephenw10

Ah, interesting. So before it was via a switch? I wonder what's different there then. Something enabled on that port? Flowcontrol mismatch maybe?

tom-ato

@stephenw10 That's an interesting data point. My pfSense box is connected to an Aruba S2500 and @cescofs is using an Aruba JL261A. Could very well be some weirdness with link negotiation?

cescofs

These are packets from the switch to the lan interface of pfsense. So there are no logs errors about them on the switch.
The only thing we know is that the lan interface has problems for a few of them.....

cescofs

@tom-ato
Did you find any solution?

tom-ato

@cescofs I have not. Checked my port settings on the switch side; flow control is disabled and the interfaces properly auto-negotiated. I can't really remove the switch from the equation since it's a core part of my home network. I haven't tried changing the driver but may do so if the total number of errors goes to an uncomfortable number, currently it's sitting at 1425 which is relatively low.

brachy33

Adding to this list, I have an Intel X550-T1 (WAN) in my Thinkserver ST50 connected to a Metronet ISP, Nokia-branded ONT with symmetrical 1Gb. The ONT, besides basic provisioning, is a media converter to copper ethernet. Have been running rock solid on 2.4/2.5 CE variants for years.

After fresh install to 2.6 CE, I too am getting errors on WAN IN only (intel x550-t1). Correlated this to sysctl dev.ix.0 and dev.ix.0.mac_stats.checksum_errs: 9. These 9 checksum errors match the WAN in errors. Of course the error count resets after reboot, hence the following code shows 0 chcksum errors for int ix0:

dev.ix.0.wake: 0
dev.ix.0.dmac: 0
dev.ix.0.fw_version: Option ROM V1-b1276-p0 eTrack 0x80000c66 PHY FW V523
dev.ix.0.advertise_speed: 3
dev.ix.0.fc: 0
dev.ix.0.mac_stats.tx_frames_1024_1522: 1036715
dev.ix.0.mac_stats.tx_frames_512_1023: 70741
dev.ix.0.mac_stats.tx_frames_256_511: 8205
dev.ix.0.mac_stats.tx_frames_128_255: 78459
dev.ix.0.mac_stats.tx_frames_65_127: 222418
dev.ix.0.mac_stats.tx_frames_64: 206348
dev.ix.0.mac_stats.management_pkts_txd: 0
dev.ix.0.mac_stats.mcast_pkts_txd: 0
dev.ix.0.mac_stats.bcast_pkts_txd: 3
dev.ix.0.mac_stats.good_pkts_txd: 1622886
dev.ix.0.mac_stats.total_pkts_txd: 1622886
dev.ix.0.mac_stats.good_octets_txd: 1605608173
dev.ix.0.mac_stats.checksum_errs: 0
dev.ix.0.mac_stats.management_pkts_drpd: 0
dev.ix.0.mac_stats.management_pkts_rcvd: 0
dev.ix.0.mac_stats.recv_jabberd: 0
dev.ix.0.mac_stats.recv_oversized: 0
dev.ix.0.mac_stats.recv_fragmented: 0
dev.ix.0.mac_stats.recv_undersized: 0
dev.ix.0.mac_stats.rx_frames_1024_1522: 1526478
dev.ix.0.mac_stats.rx_frames_512_1023: 86441
dev.ix.0.mac_stats.rx_frames_256_511: 15312
dev.ix.0.mac_stats.rx_frames_128_255: 46695
dev.ix.0.mac_stats.rx_frames_65_127: 68834
dev.ix.0.mac_stats.rx_frames_64: 406828
dev.ix.0.mac_stats.bcast_pkts_rcvd: 0
dev.ix.0.mac_stats.mcast_pkts_rcvd: 0
dev.ix.0.mac_stats.good_pkts_rcvd: 2150588
dev.ix.0.mac_stats.total_pkts_rcvd: 2150588
dev.ix.0.mac_stats.good_octets_rcvd: 2377508547
dev.ix.0.mac_stats.total_octets_rcvd: 2377508547
dev.ix.0.mac_stats.xoff_recvd: 0
dev.ix.0.mac_stats.xoff_txd: 0
dev.ix.0.mac_stats.xon_recvd: 0
dev.ix.0.mac_stats.xon_txd: 0
dev.ix.0.mac_stats.rx_missed_packets: 0
dev.ix.0.mac_stats.rec_len_errs: 0
dev.ix.0.mac_stats.remote_faults: 0
dev.ix.0.mac_stats.local_faults: 2
dev.ix.0.mac_stats.short_discards: 0
dev.ix.0.mac_stats.byte_errs: 0
dev.ix.0.mac_stats.ill_errs: 0
dev.ix.0.mac_stats.crc_errs: 0
dev.ix.0.mac_stats.rx_errs: 0
dev.ix.0.queue5.rx_discarded: 0
dev.ix.0.queue5.rx_copies: 0
dev.ix.0.queue5.rx_bytes: 415306225
dev.ix.0.queue5.rx_packets: 393080
dev.ix.0.queue5.rxd_tail: 1911
dev.ix.0.queue5.rxd_head: 1912
dev.ix.0.queue5.irqs: 424161
dev.ix.0.queue5.interrupt_rate: 31250
dev.ix.0.queue5.tx_packets: 482557
dev.ix.0.queue5.tso_tx: 0
dev.ix.0.queue5.txd_tail: 1718
dev.ix.0.queue5.txd_head: 1718
dev.ix.0.queue4.rx_discarded: 0
dev.ix.0.queue4.rx_copies: 0
dev.ix.0.queue4.rx_bytes: 436384894
dev.ix.0.queue4.rx_packets: 548017
dev.ix.0.queue4.rxd_tail: 1200
dev.ix.0.queue4.rxd_head: 1201
dev.ix.0.queue4.irqs: 483397
dev.ix.0.queue4.interrupt_rate: 31250
dev.ix.0.queue4.tx_packets: 38897
dev.ix.0.queue4.tso_tx: 0
dev.ix.0.queue4.txd_tail: 525
dev.ix.0.queue4.txd_head: 525
dev.ix.0.queue3.rx_discarded: 0
dev.ix.0.queue3.rx_copies: 0
dev.ix.0.queue3.rx_bytes: 78301815
dev.ix.0.queue3.rx_packets: 65498
dev.ix.0.queue3.rxd_tail: 2009
dev.ix.0.queue3.rxd_head: 2010
dev.ix.0.queue3.irqs: 126529
dev.ix.0.queue3.interrupt_rate: 31250
dev.ix.0.queue3.tx_packets: 81232
dev.ix.0.queue3.tso_tx: 0
dev.ix.0.queue3.txd_tail: 2009
dev.ix.0.queue3.txd_head: 2009
dev.ix.0.queue2.rx_discarded: 0
dev.ix.0.queue2.rx_copies: 0
dev.ix.0.queue2.rx_bytes: 332160295
dev.ix.0.queue2.rx_packets: 230804
dev.ix.0.queue2.rxd_tail: 1427
dev.ix.0.queue2.rxd_head: 1428
dev.ix.0.queue2.irqs: 492288
dev.ix.0.queue2.interrupt_rate: 31250
dev.ix.0.queue2.tx_packets: 576023
dev.ix.0.queue2.tso_tx: 0
dev.ix.0.queue2.txd_tail: 1066
dev.ix.0.queue2.txd_head: 1066
dev.ix.0.queue1.rx_discarded: 0
dev.ix.0.queue1.rx_copies: 0
dev.ix.0.queue1.rx_bytes: 665262155
dev.ix.0.queue1.rx_packets: 580499
dev.ix.0.queue1.rxd_tail: 914
dev.ix.0.queue1.rxd_head: 915
dev.ix.0.queue1.irqs: 529926
dev.ix.0.queue1.interrupt_rate: 31250
dev.ix.0.queue1.tx_packets: 216662
dev.ix.0.queue1.tso_tx: 0
dev.ix.0.queue1.txd_tail: 89
dev.ix.0.queue1.txd_head: 89
dev.ix.0.queue0.rx_discarded: 0
dev.ix.0.queue0.rx_copies: 0
dev.ix.0.queue0.rx_bytes: 441619846
dev.ix.0.queue0.rx_packets: 332787
dev.ix.0.queue0.rxd_tail: 1010
dev.ix.0.queue0.rxd_head: 1011
dev.ix.0.queue0.irqs: 352341
dev.ix.0.queue0.interrupt_rate: 31250
dev.ix.0.queue0.tx_packets: 227634
dev.ix.0.queue0.tso_tx: 0
dev.ix.0.queue0.txd_tail: 792
dev.ix.0.queue0.txd_head: 792
dev.ix.0.link_irq: 1
dev.ix.0.watchdog_events: 0
dev.ix.0.dropped: 0
dev.ix.0.iflib.rxq5.rxq_fl0.buf_size: 2048
dev.ix.0.iflib.rxq5.rxq_fl0.credits: 2047
dev.ix.0.iflib.rxq5.rxq_fl0.cidx: 1912
dev.ix.0.iflib.rxq5.rxq_fl0.pidx: 1911
dev.ix.0.iflib.rxq5.cpu: 5
dev.ix.0.iflib.rxq4.rxq_fl0.buf_size: 2048
dev.ix.0.iflib.rxq4.rxq_fl0.credits: 2047
dev.ix.0.iflib.rxq4.rxq_fl0.cidx: 1201
dev.ix.0.iflib.rxq4.rxq_fl0.pidx: 1200
dev.ix.0.iflib.rxq4.cpu: 4
dev.ix.0.iflib.rxq3.rxq_fl0.buf_size: 2048
dev.ix.0.iflib.rxq3.rxq_fl0.credits: 2047
dev.ix.0.iflib.rxq3.rxq_fl0.cidx: 2010
dev.ix.0.iflib.rxq3.rxq_fl0.pidx: 2009
dev.ix.0.iflib.rxq3.cpu: 3
dev.ix.0.iflib.rxq2.rxq_fl0.buf_size: 2048
dev.ix.0.iflib.rxq2.rxq_fl0.credits: 2047
dev.ix.0.iflib.rxq2.rxq_fl0.cidx: 1428
dev.ix.0.iflib.rxq2.rxq_fl0.pidx: 1427
dev.ix.0.iflib.rxq2.cpu: 2
dev.ix.0.iflib.rxq1.rxq_fl0.buf_size: 2048
dev.ix.0.iflib.rxq1.rxq_fl0.credits: 2047
dev.ix.0.iflib.rxq1.rxq_fl0.cidx: 915
dev.ix.0.iflib.rxq1.rxq_fl0.pidx: 914
dev.ix.0.iflib.rxq1.cpu: 1
dev.ix.0.iflib.rxq0.rxq_fl0.buf_size: 2048
dev.ix.0.iflib.rxq0.rxq_fl0.credits: 2047
dev.ix.0.iflib.rxq0.rxq_fl0.cidx: 1011
dev.ix.0.iflib.rxq0.rxq_fl0.pidx: 1010
dev.ix.0.iflib.rxq0.cpu: 0
dev.ix.0.iflib.txq5.r_abdications: 0
dev.ix.0.iflib.txq5.r_restarts: 0
dev.ix.0.iflib.txq5.r_stalls: 0
dev.ix.0.iflib.txq5.r_starts: 482557
dev.ix.0.iflib.txq5.r_drops: 0
dev.ix.0.iflib.txq5.r_enqueues: 482557
dev.ix.0.iflib.txq5.ring_state: pidx_head: 1277 pidx_tail: 1277 cidx: 1277 state: IDLE
dev.ix.0.iflib.txq5.txq_cleaned: 482965
dev.ix.0.iflib.txq5.txq_processed: 482997
dev.ix.0.iflib.txq5.txq_in_use: 33
dev.ix.0.iflib.txq5.txq_cidx_processed: 1717
dev.ix.0.iflib.txq5.txq_cidx: 1685
dev.ix.0.iflib.txq5.txq_pidx: 1718
dev.ix.0.iflib.txq5.no_tx_dma_setup: 0
dev.ix.0.iflib.txq5.txd_encap_efbig: 0
dev.ix.0.iflib.txq5.tx_map_failed: 0
dev.ix.0.iflib.txq5.no_desc_avail: 0
dev.ix.0.iflib.txq5.mbuf_defrag_failed: 0
dev.ix.0.iflib.txq5.m_pullups: 0
dev.ix.0.iflib.txq5.mbuf_defrag: 0
dev.ix.0.iflib.txq5.cpu: 5
dev.ix.0.iflib.txq4.r_abdications: 0
dev.ix.0.iflib.txq4.r_restarts: 0
dev.ix.0.iflib.txq4.r_stalls: 0
dev.ix.0.iflib.txq4.r_starts: 38897
dev.ix.0.iflib.txq4.r_drops: 0
dev.ix.0.iflib.txq4.r_enqueues: 38897
dev.ix.0.iflib.txq4.ring_state: pidx_head: 2033 pidx_tail: 2033 cidx: 2033 state: IDLE
dev.ix.0.iflib.txq4.txq_cleaned: 39404
dev.ix.0.iflib.txq4.txq_processed: 39436
dev.ix.0.iflib.txq4.txq_in_use: 33
dev.ix.0.iflib.txq4.txq_cidx_processed: 524
dev.ix.0.iflib.txq4.txq_cidx: 492
dev.ix.0.iflib.txq4.txq_pidx: 525
dev.ix.0.iflib.txq4.no_tx_dma_setup: 0
dev.ix.0.iflib.txq4.txd_encap_efbig: 0
dev.ix.0.iflib.txq4.tx_map_failed: 0
dev.ix.0.iflib.txq4.no_desc_avail: 0
dev.ix.0.iflib.txq4.mbuf_defrag_failed: 0
dev.ix.0.iflib.txq4.m_pullups: 0
dev.ix.0.iflib.txq4.mbuf_defrag: 0
dev.ix.0.iflib.txq4.cpu: 4
dev.ix.0.iflib.txq3.r_abdications: 0
dev.ix.0.iflib.txq3.r_restarts: 0
dev.ix.0.iflib.txq3.r_stalls: 0
dev.ix.0.iflib.txq3.r_starts: 81231
dev.ix.0.iflib.txq3.r_drops: 0
dev.ix.0.iflib.txq3.r_enqueues: 81232
dev.ix.0.iflib.txq3.ring_state: pidx_head: 1360 pidx_tail: 1360 cidx: 1360 state: IDLE
dev.ix.0.iflib.txq3.txq_cleaned: 81847
dev.ix.0.iflib.txq3.txq_processed: 81879
dev.ix.0.iflib.txq3.txq_in_use: 34
dev.ix.0.iflib.txq3.txq_cidx_processed: 2007
dev.ix.0.iflib.txq3.txq_cidx: 1975
dev.ix.0.iflib.txq3.txq_pidx: 2009
dev.ix.0.iflib.txq3.no_tx_dma_setup: 0
dev.ix.0.iflib.txq3.txd_encap_efbig: 0
dev.ix.0.iflib.txq3.tx_map_failed: 0
dev.ix.0.iflib.txq3.no_desc_avail: 0
dev.ix.0.iflib.txq3.mbuf_defrag_failed: 0
dev.ix.0.iflib.txq3.m_pullups: 0
dev.ix.0.iflib.txq3.mbuf_defrag: 0
dev.ix.0.iflib.txq3.cpu: 3
dev.ix.0.iflib.txq2.r_abdications: 0
dev.ix.0.iflib.txq2.r_restarts: 0
dev.ix.0.iflib.txq2.r_stalls: 0
dev.ix.0.iflib.txq2.r_starts: 576022
dev.ix.0.iflib.txq2.r_drops: 0
dev.ix.0.iflib.txq2.r_enqueues: 576023
dev.ix.0.iflib.txq2.ring_state: pidx_head: 0535 pidx_tail: 0535 cidx: 0535 state: IDLE
dev.ix.0.iflib.txq2.txq_cleaned: 576515
dev.ix.0.iflib.txq2.txq_processed: 576547
dev.ix.0.iflib.txq2.txq_in_use: 39
dev.ix.0.iflib.txq2.txq_cidx_processed: 1059
dev.ix.0.iflib.txq2.txq_cidx: 1027
dev.ix.0.iflib.txq2.txq_pidx: 1066
dev.ix.0.iflib.txq2.no_tx_dma_setup: 0
dev.ix.0.iflib.txq2.txd_encap_efbig: 0
dev.ix.0.iflib.txq2.tx_map_failed: 0
dev.ix.0.iflib.txq2.no_desc_avail: 0
dev.ix.0.iflib.txq2.mbuf_defrag_failed: 0
dev.ix.0.iflib.txq2.m_pullups: 0
dev.ix.0.iflib.txq2.mbuf_defrag: 0
dev.ix.0.iflib.txq2.cpu: 2
dev.ix.0.iflib.txq1.r_abdications: 0
dev.ix.0.iflib.txq1.r_restarts: 0
dev.ix.0.iflib.txq1.r_stalls: 0
dev.ix.0.iflib.txq1.r_starts: 216662
dev.ix.0.iflib.txq1.r_drops: 0
dev.ix.0.iflib.txq1.r_enqueues: 216662
dev.ix.0.iflib.txq1.ring_state: pidx_head: 1622 pidx_tail: 1622 cidx: 1622 state: IDLE
dev.ix.0.iflib.txq1.txq_cleaned: 217143
dev.ix.0.iflib.txq1.txq_processed: 217175
dev.ix.0.iflib.txq1.txq_in_use: 34
dev.ix.0.iflib.txq1.txq_cidx_processed: 87
dev.ix.0.iflib.txq1.txq_cidx: 55
dev.ix.0.iflib.txq1.txq_pidx: 89
dev.ix.0.iflib.txq1.no_tx_dma_setup: 0
dev.ix.0.iflib.txq1.txd_encap_efbig: 0
dev.ix.0.iflib.txq1.tx_map_failed: 0
dev.ix.0.iflib.txq1.no_desc_avail: 0
dev.ix.0.iflib.txq1.mbuf_defrag_failed: 0
dev.ix.0.iflib.txq1.m_pullups: 0
dev.ix.0.iflib.txq1.mbuf_defrag: 0
dev.ix.0.iflib.txq1.cpu: 1
dev.ix.0.iflib.txq0.r_abdications: 0
dev.ix.0.iflib.txq0.r_restarts: 0
dev.ix.0.iflib.txq0.r_stalls: 0
dev.ix.0.iflib.txq0.r_starts: 227620
dev.ix.0.iflib.txq0.r_drops: 0
dev.ix.0.iflib.txq0.r_enqueues: 227634
dev.ix.0.iflib.txq0.ring_state: pidx_head: 0306 pidx_tail: 0306 cidx: 0306 state: IDLE
dev.ix.0.iflib.txq0.txq_cleaned: 228086
dev.ix.0.iflib.txq0.txq_processed: 228118
dev.ix.0.iflib.txq0.txq_in_use: 34
dev.ix.0.iflib.txq0.txq_cidx_processed: 790
dev.ix.0.iflib.txq0.txq_cidx: 758
dev.ix.0.iflib.txq0.txq_pidx: 792
dev.ix.0.iflib.txq0.no_tx_dma_setup: 0
dev.ix.0.iflib.txq0.txd_encap_efbig: 0
dev.ix.0.iflib.txq0.tx_map_failed: 0
dev.ix.0.iflib.txq0.no_desc_avail: 0
dev.ix.0.iflib.txq0.mbuf_defrag_failed: 0
dev.ix.0.iflib.txq0.m_pullups: 0
dev.ix.0.iflib.txq0.mbuf_defrag: 0
dev.ix.0.iflib.txq0.cpu: 0
dev.ix.0.iflib.override_nrxds: 0
dev.ix.0.iflib.override_ntxds: 0
dev.ix.0.iflib.use_logical_cores: 0
dev.ix.0.iflib.separate_txrx: 0
dev.ix.0.iflib.core_offset: 0
dev.ix.0.iflib.tx_abdicate: 0
dev.ix.0.iflib.rx_budget: 0
dev.ix.0.iflib.disable_msix: 0
dev.ix.0.iflib.override_qs_enable: 0
dev.ix.0.iflib.override_nrxqs: 0
dev.ix.0.iflib.override_ntxqs: 0
dev.ix.0.iflib.driver_version: 4.0.1-k
dev.ix.0.%parent: pci2
dev.ix.0.%pnpinfo: vendor=0x8086 device=0x15d1 subvendor=0x8086 subdevice=0x0002 class=0x020000
dev.ix.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 handle=\_SB_.PCI0.RP09.PXSX
dev.ix.0.%driver: ix
dev.ix.0.%desc: Intel(R) X550-T1

So far on ix0 int:

Disabled FC
Checked disabled hardware checksum offload
Verified int is running latest Intel driver (believe it's the latest - 4.0.1-k)
Checked latest firmware via Intel bootable update utility
swapped to multiple, new CAT6 patch cords including Commscope and Panduit (no cheap cords)
Bought a new X550-T1 card in case my previous card was dying.
Had Metronet provision a new Nokia G240G ONT and powersupply to no affect.

Lastly, I found the following on Intel's Download Center and it seems it couldn't be more applicable to my situation (Intel NIC, ONT, cksum errors):

https://www.intel.com/content/www/us/en/download/19174/disabling-tcp-ipv6-checksum-offload-capability-with-intel-1-10-gbe-controllers.html

I've tried modifying INET6 settings via Shell command as per FreeBSD IFCONFIG:

-rxcsum, -txcsum, -rxcsum6, -txcsum6
	     If	the driver supports user-configurable checksum offloading,
	     disable receive (or transmit) checksum offloading on the inter-
	     face.  The	feature	can be turned off selectively per protocol
	     family.  Use -rxcsum6, -txcsum6 for ip6(4)	or -rxcsum, -txcsum
	     otherwise.	 These settings	may not	always be independent of each
	     other.

Still no changes and errors continue on WAN IN. To be fair, for ~100GiB data in I might have 6-10 errors. I have also now upgraded the 2.6CE to pfSense Plus ver 22.01 with yearly support package.

Apologies, not trying to hijack OP's thread but hoping
to share more data to the WAN ERRORS IN. Thanks.

tinfoilmatt

Reporting the same on 2.6 with X557 NICs, all 'mac_stats.checksum_errs':

stephenw10

The difference here is that the ix driver in 2.6 includes all input errors in the mac stats value not just crc and length errors:
https://github.com/pfsense/FreeBSD-src/commit/5574b12aac2bdfe66ec1d9564f932eeec9ac213c

So nothing has actually changed, you are just seeing the errors now that were previously not shown.

Also see: https://redmine.pfsense.org/issues/12904

Steve

Dredex

If we don't have access to the other end of the cable (switch/server) we will never know if this errors are relevant or not ....

cescofs

@stephenw10
I'm not convinced by your answer. I should have seen errors on all nics and in every direction.

stephenw10

Why? The errors could be on one NIC only. You just didn't see them reported in 2.5.2.

qfixxx

Just curious if anyone ever found anything that might explain the source of these errors?

simplepanda

Seeing something similar I think.

On my LAN interface (multiple VLANs via a 10G Intel/ix Interface) i get exactly 2 IN errors every 20 seconds.

It's 2 errors. It's every 20 seconds. Like clockwork. I can run gigabits of traffic through the firewall and it only ever increases by 2 input errors, only every 20 seconds.

What on earth could this be? Something on my LAN sending dodgy frames? Why every 20 seconds?

Is there anyway to get a pcap of the dropped / error frames?

stephenw10

Hmm, that's interesting.

You could probably catpure that by using a mirror port on the switch to reflect whatever it's sending.

Steve

simplepanda

@stephenw10 Yes, setting up to do that now. I assume it's my Ubiquiti switch. Cheap 10G switch = cheap 10G switch problems, I guess.

qfixxx

@simplepanda Curious what you see. I have a similar situation just not with fixed pattern. Mine is just a slow increment of ~50 errors every 24hrs on both WAN interfaces. I've got a USW-Flex-XG between the WAN NICs and the respective modems (AT&T and Spectrum).