X520 errors in 2.6 and 22.01?
-
@stephenw10
After disabling it, the errors were still there.
Since there are no errors in the output, the drivers should be excluded as a cause. -
@stephenw10 right after booting i had 83 errors. The current count grew a little every day for the last 5 days.
-
@tom-ato
it's the same for me -
I found this ....
link text -
Do you actually see the new driver loaded though:
[22.01-RELEASE][admin@6100-2.stevew.lan]/root: sysctl dev.ix.0.iflib.driver_version dev.ix.0.iflib.driver_version: 4.0.1-k
-
no. to find a solution I downloaded the latest version from the intel site for freebsd -
Ok, so it did load and you are still seeing the errors reported by the latest Intel driver.
So maybe that is the correct count and in fact the driver in 2.5.2 was misreporting them.
Steve
-
@stephenw10
it's hard to admit ... -
@stephenw10
do you have any idea how to solve? it's very strange only "errors in" -
@stephenw10 That is a very real possibility. I'm fine with that answer if it is indeed what is happening.
-
@cescofs said in X520 errors in 2.6 and 22.01?:
@stephenw10
it's hard to admit ...pfSense does nothing with the MAC stats, the driver updates them directly.
You could try the alternative driver in 2.5.2.Steve
-
Something must have happened with 2.6. Anyway, I connected the pfsense directly to my pc and.....zero errors!!!!
-
Ah, interesting. So before it was via a switch? I wonder what's different there then. Something enabled on that port? Flowcontrol mismatch maybe?
-
@stephenw10 That's an interesting data point. My pfSense box is connected to an Aruba S2500 and @cescofs is using an Aruba JL261A. Could very well be some weirdness with link negotiation?
-
These are packets from the switch to the lan interface of pfsense. So there are no logs errors about them on the switch.
The only thing we know is that the lan interface has problems for a few of them..... -
@tom-ato
Did you find any solution? -
@cescofs I have not. Checked my port settings on the switch side; flow control is disabled and the interfaces properly auto-negotiated. I can't really remove the switch from the equation since it's a core part of my home network. I haven't tried changing the driver but may do so if the total number of errors goes to an uncomfortable number, currently it's sitting at 1425 which is relatively low.
-
Adding to this list, I have an Intel X550-T1 (WAN) in my Thinkserver ST50 connected to a Metronet ISP, Nokia-branded ONT with symmetrical 1Gb. The ONT, besides basic provisioning, is a media converter to copper ethernet. Have been running rock solid on 2.4/2.5 CE variants for years.
After fresh install to 2.6 CE, I too am getting errors on WAN IN only (intel x550-t1). Correlated this to sysctl dev.ix.0 and dev.ix.0.mac_stats.checksum_errs: 9. These 9 checksum errors match the WAN in errors. Of course the error count resets after reboot, hence the following code shows 0 chcksum errors for int ix0:
dev.ix.0.wake: 0 dev.ix.0.dmac: 0 dev.ix.0.fw_version: Option ROM V1-b1276-p0 eTrack 0x80000c66 PHY FW V523 dev.ix.0.advertise_speed: 3 dev.ix.0.fc: 0 dev.ix.0.mac_stats.tx_frames_1024_1522: 1036715 dev.ix.0.mac_stats.tx_frames_512_1023: 70741 dev.ix.0.mac_stats.tx_frames_256_511: 8205 dev.ix.0.mac_stats.tx_frames_128_255: 78459 dev.ix.0.mac_stats.tx_frames_65_127: 222418 dev.ix.0.mac_stats.tx_frames_64: 206348 dev.ix.0.mac_stats.management_pkts_txd: 0 dev.ix.0.mac_stats.mcast_pkts_txd: 0 dev.ix.0.mac_stats.bcast_pkts_txd: 3 dev.ix.0.mac_stats.good_pkts_txd: 1622886 dev.ix.0.mac_stats.total_pkts_txd: 1622886 dev.ix.0.mac_stats.good_octets_txd: 1605608173 dev.ix.0.mac_stats.checksum_errs: 0 dev.ix.0.mac_stats.management_pkts_drpd: 0 dev.ix.0.mac_stats.management_pkts_rcvd: 0 dev.ix.0.mac_stats.recv_jabberd: 0 dev.ix.0.mac_stats.recv_oversized: 0 dev.ix.0.mac_stats.recv_fragmented: 0 dev.ix.0.mac_stats.recv_undersized: 0 dev.ix.0.mac_stats.rx_frames_1024_1522: 1526478 dev.ix.0.mac_stats.rx_frames_512_1023: 86441 dev.ix.0.mac_stats.rx_frames_256_511: 15312 dev.ix.0.mac_stats.rx_frames_128_255: 46695 dev.ix.0.mac_stats.rx_frames_65_127: 68834 dev.ix.0.mac_stats.rx_frames_64: 406828 dev.ix.0.mac_stats.bcast_pkts_rcvd: 0 dev.ix.0.mac_stats.mcast_pkts_rcvd: 0 dev.ix.0.mac_stats.good_pkts_rcvd: 2150588 dev.ix.0.mac_stats.total_pkts_rcvd: 2150588 dev.ix.0.mac_stats.good_octets_rcvd: 2377508547 dev.ix.0.mac_stats.total_octets_rcvd: 2377508547 dev.ix.0.mac_stats.xoff_recvd: 0 dev.ix.0.mac_stats.xoff_txd: 0 dev.ix.0.mac_stats.xon_recvd: 0 dev.ix.0.mac_stats.xon_txd: 0 dev.ix.0.mac_stats.rx_missed_packets: 0 dev.ix.0.mac_stats.rec_len_errs: 0 dev.ix.0.mac_stats.remote_faults: 0 dev.ix.0.mac_stats.local_faults: 2 dev.ix.0.mac_stats.short_discards: 0 dev.ix.0.mac_stats.byte_errs: 0 dev.ix.0.mac_stats.ill_errs: 0 dev.ix.0.mac_stats.crc_errs: 0 dev.ix.0.mac_stats.rx_errs: 0 dev.ix.0.queue5.rx_discarded: 0 dev.ix.0.queue5.rx_copies: 0 dev.ix.0.queue5.rx_bytes: 415306225 dev.ix.0.queue5.rx_packets: 393080 dev.ix.0.queue5.rxd_tail: 1911 dev.ix.0.queue5.rxd_head: 1912 dev.ix.0.queue5.irqs: 424161 dev.ix.0.queue5.interrupt_rate: 31250 dev.ix.0.queue5.tx_packets: 482557 dev.ix.0.queue5.tso_tx: 0 dev.ix.0.queue5.txd_tail: 1718 dev.ix.0.queue5.txd_head: 1718 dev.ix.0.queue4.rx_discarded: 0 dev.ix.0.queue4.rx_copies: 0 dev.ix.0.queue4.rx_bytes: 436384894 dev.ix.0.queue4.rx_packets: 548017 dev.ix.0.queue4.rxd_tail: 1200 dev.ix.0.queue4.rxd_head: 1201 dev.ix.0.queue4.irqs: 483397 dev.ix.0.queue4.interrupt_rate: 31250 dev.ix.0.queue4.tx_packets: 38897 dev.ix.0.queue4.tso_tx: 0 dev.ix.0.queue4.txd_tail: 525 dev.ix.0.queue4.txd_head: 525 dev.ix.0.queue3.rx_discarded: 0 dev.ix.0.queue3.rx_copies: 0 dev.ix.0.queue3.rx_bytes: 78301815 dev.ix.0.queue3.rx_packets: 65498 dev.ix.0.queue3.rxd_tail: 2009 dev.ix.0.queue3.rxd_head: 2010 dev.ix.0.queue3.irqs: 126529 dev.ix.0.queue3.interrupt_rate: 31250 dev.ix.0.queue3.tx_packets: 81232 dev.ix.0.queue3.tso_tx: 0 dev.ix.0.queue3.txd_tail: 2009 dev.ix.0.queue3.txd_head: 2009 dev.ix.0.queue2.rx_discarded: 0 dev.ix.0.queue2.rx_copies: 0 dev.ix.0.queue2.rx_bytes: 332160295 dev.ix.0.queue2.rx_packets: 230804 dev.ix.0.queue2.rxd_tail: 1427 dev.ix.0.queue2.rxd_head: 1428 dev.ix.0.queue2.irqs: 492288 dev.ix.0.queue2.interrupt_rate: 31250 dev.ix.0.queue2.tx_packets: 576023 dev.ix.0.queue2.tso_tx: 0 dev.ix.0.queue2.txd_tail: 1066 dev.ix.0.queue2.txd_head: 1066 dev.ix.0.queue1.rx_discarded: 0 dev.ix.0.queue1.rx_copies: 0 dev.ix.0.queue1.rx_bytes: 665262155 dev.ix.0.queue1.rx_packets: 580499 dev.ix.0.queue1.rxd_tail: 914 dev.ix.0.queue1.rxd_head: 915 dev.ix.0.queue1.irqs: 529926 dev.ix.0.queue1.interrupt_rate: 31250 dev.ix.0.queue1.tx_packets: 216662 dev.ix.0.queue1.tso_tx: 0 dev.ix.0.queue1.txd_tail: 89 dev.ix.0.queue1.txd_head: 89 dev.ix.0.queue0.rx_discarded: 0 dev.ix.0.queue0.rx_copies: 0 dev.ix.0.queue0.rx_bytes: 441619846 dev.ix.0.queue0.rx_packets: 332787 dev.ix.0.queue0.rxd_tail: 1010 dev.ix.0.queue0.rxd_head: 1011 dev.ix.0.queue0.irqs: 352341 dev.ix.0.queue0.interrupt_rate: 31250 dev.ix.0.queue0.tx_packets: 227634 dev.ix.0.queue0.tso_tx: 0 dev.ix.0.queue0.txd_tail: 792 dev.ix.0.queue0.txd_head: 792 dev.ix.0.link_irq: 1 dev.ix.0.watchdog_events: 0 dev.ix.0.dropped: 0 dev.ix.0.iflib.rxq5.rxq_fl0.buf_size: 2048 dev.ix.0.iflib.rxq5.rxq_fl0.credits: 2047 dev.ix.0.iflib.rxq5.rxq_fl0.cidx: 1912 dev.ix.0.iflib.rxq5.rxq_fl0.pidx: 1911 dev.ix.0.iflib.rxq5.cpu: 5 dev.ix.0.iflib.rxq4.rxq_fl0.buf_size: 2048 dev.ix.0.iflib.rxq4.rxq_fl0.credits: 2047 dev.ix.0.iflib.rxq4.rxq_fl0.cidx: 1201 dev.ix.0.iflib.rxq4.rxq_fl0.pidx: 1200 dev.ix.0.iflib.rxq4.cpu: 4 dev.ix.0.iflib.rxq3.rxq_fl0.buf_size: 2048 dev.ix.0.iflib.rxq3.rxq_fl0.credits: 2047 dev.ix.0.iflib.rxq3.rxq_fl0.cidx: 2010 dev.ix.0.iflib.rxq3.rxq_fl0.pidx: 2009 dev.ix.0.iflib.rxq3.cpu: 3 dev.ix.0.iflib.rxq2.rxq_fl0.buf_size: 2048 dev.ix.0.iflib.rxq2.rxq_fl0.credits: 2047 dev.ix.0.iflib.rxq2.rxq_fl0.cidx: 1428 dev.ix.0.iflib.rxq2.rxq_fl0.pidx: 1427 dev.ix.0.iflib.rxq2.cpu: 2 dev.ix.0.iflib.rxq1.rxq_fl0.buf_size: 2048 dev.ix.0.iflib.rxq1.rxq_fl0.credits: 2047 dev.ix.0.iflib.rxq1.rxq_fl0.cidx: 915 dev.ix.0.iflib.rxq1.rxq_fl0.pidx: 914 dev.ix.0.iflib.rxq1.cpu: 1 dev.ix.0.iflib.rxq0.rxq_fl0.buf_size: 2048 dev.ix.0.iflib.rxq0.rxq_fl0.credits: 2047 dev.ix.0.iflib.rxq0.rxq_fl0.cidx: 1011 dev.ix.0.iflib.rxq0.rxq_fl0.pidx: 1010 dev.ix.0.iflib.rxq0.cpu: 0 dev.ix.0.iflib.txq5.r_abdications: 0 dev.ix.0.iflib.txq5.r_restarts: 0 dev.ix.0.iflib.txq5.r_stalls: 0 dev.ix.0.iflib.txq5.r_starts: 482557 dev.ix.0.iflib.txq5.r_drops: 0 dev.ix.0.iflib.txq5.r_enqueues: 482557 dev.ix.0.iflib.txq5.ring_state: pidx_head: 1277 pidx_tail: 1277 cidx: 1277 state: IDLE dev.ix.0.iflib.txq5.txq_cleaned: 482965 dev.ix.0.iflib.txq5.txq_processed: 482997 dev.ix.0.iflib.txq5.txq_in_use: 33 dev.ix.0.iflib.txq5.txq_cidx_processed: 1717 dev.ix.0.iflib.txq5.txq_cidx: 1685 dev.ix.0.iflib.txq5.txq_pidx: 1718 dev.ix.0.iflib.txq5.no_tx_dma_setup: 0 dev.ix.0.iflib.txq5.txd_encap_efbig: 0 dev.ix.0.iflib.txq5.tx_map_failed: 0 dev.ix.0.iflib.txq5.no_desc_avail: 0 dev.ix.0.iflib.txq5.mbuf_defrag_failed: 0 dev.ix.0.iflib.txq5.m_pullups: 0 dev.ix.0.iflib.txq5.mbuf_defrag: 0 dev.ix.0.iflib.txq5.cpu: 5 dev.ix.0.iflib.txq4.r_abdications: 0 dev.ix.0.iflib.txq4.r_restarts: 0 dev.ix.0.iflib.txq4.r_stalls: 0 dev.ix.0.iflib.txq4.r_starts: 38897 dev.ix.0.iflib.txq4.r_drops: 0 dev.ix.0.iflib.txq4.r_enqueues: 38897 dev.ix.0.iflib.txq4.ring_state: pidx_head: 2033 pidx_tail: 2033 cidx: 2033 state: IDLE dev.ix.0.iflib.txq4.txq_cleaned: 39404 dev.ix.0.iflib.txq4.txq_processed: 39436 dev.ix.0.iflib.txq4.txq_in_use: 33 dev.ix.0.iflib.txq4.txq_cidx_processed: 524 dev.ix.0.iflib.txq4.txq_cidx: 492 dev.ix.0.iflib.txq4.txq_pidx: 525 dev.ix.0.iflib.txq4.no_tx_dma_setup: 0 dev.ix.0.iflib.txq4.txd_encap_efbig: 0 dev.ix.0.iflib.txq4.tx_map_failed: 0 dev.ix.0.iflib.txq4.no_desc_avail: 0 dev.ix.0.iflib.txq4.mbuf_defrag_failed: 0 dev.ix.0.iflib.txq4.m_pullups: 0 dev.ix.0.iflib.txq4.mbuf_defrag: 0 dev.ix.0.iflib.txq4.cpu: 4 dev.ix.0.iflib.txq3.r_abdications: 0 dev.ix.0.iflib.txq3.r_restarts: 0 dev.ix.0.iflib.txq3.r_stalls: 0 dev.ix.0.iflib.txq3.r_starts: 81231 dev.ix.0.iflib.txq3.r_drops: 0 dev.ix.0.iflib.txq3.r_enqueues: 81232 dev.ix.0.iflib.txq3.ring_state: pidx_head: 1360 pidx_tail: 1360 cidx: 1360 state: IDLE dev.ix.0.iflib.txq3.txq_cleaned: 81847 dev.ix.0.iflib.txq3.txq_processed: 81879 dev.ix.0.iflib.txq3.txq_in_use: 34 dev.ix.0.iflib.txq3.txq_cidx_processed: 2007 dev.ix.0.iflib.txq3.txq_cidx: 1975 dev.ix.0.iflib.txq3.txq_pidx: 2009 dev.ix.0.iflib.txq3.no_tx_dma_setup: 0 dev.ix.0.iflib.txq3.txd_encap_efbig: 0 dev.ix.0.iflib.txq3.tx_map_failed: 0 dev.ix.0.iflib.txq3.no_desc_avail: 0 dev.ix.0.iflib.txq3.mbuf_defrag_failed: 0 dev.ix.0.iflib.txq3.m_pullups: 0 dev.ix.0.iflib.txq3.mbuf_defrag: 0 dev.ix.0.iflib.txq3.cpu: 3 dev.ix.0.iflib.txq2.r_abdications: 0 dev.ix.0.iflib.txq2.r_restarts: 0 dev.ix.0.iflib.txq2.r_stalls: 0 dev.ix.0.iflib.txq2.r_starts: 576022 dev.ix.0.iflib.txq2.r_drops: 0 dev.ix.0.iflib.txq2.r_enqueues: 576023 dev.ix.0.iflib.txq2.ring_state: pidx_head: 0535 pidx_tail: 0535 cidx: 0535 state: IDLE dev.ix.0.iflib.txq2.txq_cleaned: 576515 dev.ix.0.iflib.txq2.txq_processed: 576547 dev.ix.0.iflib.txq2.txq_in_use: 39 dev.ix.0.iflib.txq2.txq_cidx_processed: 1059 dev.ix.0.iflib.txq2.txq_cidx: 1027 dev.ix.0.iflib.txq2.txq_pidx: 1066 dev.ix.0.iflib.txq2.no_tx_dma_setup: 0 dev.ix.0.iflib.txq2.txd_encap_efbig: 0 dev.ix.0.iflib.txq2.tx_map_failed: 0 dev.ix.0.iflib.txq2.no_desc_avail: 0 dev.ix.0.iflib.txq2.mbuf_defrag_failed: 0 dev.ix.0.iflib.txq2.m_pullups: 0 dev.ix.0.iflib.txq2.mbuf_defrag: 0 dev.ix.0.iflib.txq2.cpu: 2 dev.ix.0.iflib.txq1.r_abdications: 0 dev.ix.0.iflib.txq1.r_restarts: 0 dev.ix.0.iflib.txq1.r_stalls: 0 dev.ix.0.iflib.txq1.r_starts: 216662 dev.ix.0.iflib.txq1.r_drops: 0 dev.ix.0.iflib.txq1.r_enqueues: 216662 dev.ix.0.iflib.txq1.ring_state: pidx_head: 1622 pidx_tail: 1622 cidx: 1622 state: IDLE dev.ix.0.iflib.txq1.txq_cleaned: 217143 dev.ix.0.iflib.txq1.txq_processed: 217175 dev.ix.0.iflib.txq1.txq_in_use: 34 dev.ix.0.iflib.txq1.txq_cidx_processed: 87 dev.ix.0.iflib.txq1.txq_cidx: 55 dev.ix.0.iflib.txq1.txq_pidx: 89 dev.ix.0.iflib.txq1.no_tx_dma_setup: 0 dev.ix.0.iflib.txq1.txd_encap_efbig: 0 dev.ix.0.iflib.txq1.tx_map_failed: 0 dev.ix.0.iflib.txq1.no_desc_avail: 0 dev.ix.0.iflib.txq1.mbuf_defrag_failed: 0 dev.ix.0.iflib.txq1.m_pullups: 0 dev.ix.0.iflib.txq1.mbuf_defrag: 0 dev.ix.0.iflib.txq1.cpu: 1 dev.ix.0.iflib.txq0.r_abdications: 0 dev.ix.0.iflib.txq0.r_restarts: 0 dev.ix.0.iflib.txq0.r_stalls: 0 dev.ix.0.iflib.txq0.r_starts: 227620 dev.ix.0.iflib.txq0.r_drops: 0 dev.ix.0.iflib.txq0.r_enqueues: 227634 dev.ix.0.iflib.txq0.ring_state: pidx_head: 0306 pidx_tail: 0306 cidx: 0306 state: IDLE dev.ix.0.iflib.txq0.txq_cleaned: 228086 dev.ix.0.iflib.txq0.txq_processed: 228118 dev.ix.0.iflib.txq0.txq_in_use: 34 dev.ix.0.iflib.txq0.txq_cidx_processed: 790 dev.ix.0.iflib.txq0.txq_cidx: 758 dev.ix.0.iflib.txq0.txq_pidx: 792 dev.ix.0.iflib.txq0.no_tx_dma_setup: 0 dev.ix.0.iflib.txq0.txd_encap_efbig: 0 dev.ix.0.iflib.txq0.tx_map_failed: 0 dev.ix.0.iflib.txq0.no_desc_avail: 0 dev.ix.0.iflib.txq0.mbuf_defrag_failed: 0 dev.ix.0.iflib.txq0.m_pullups: 0 dev.ix.0.iflib.txq0.mbuf_defrag: 0 dev.ix.0.iflib.txq0.cpu: 0 dev.ix.0.iflib.override_nrxds: 0 dev.ix.0.iflib.override_ntxds: 0 dev.ix.0.iflib.use_logical_cores: 0 dev.ix.0.iflib.separate_txrx: 0 dev.ix.0.iflib.core_offset: 0 dev.ix.0.iflib.tx_abdicate: 0 dev.ix.0.iflib.rx_budget: 0 dev.ix.0.iflib.disable_msix: 0 dev.ix.0.iflib.override_qs_enable: 0 dev.ix.0.iflib.override_nrxqs: 0 dev.ix.0.iflib.override_ntxqs: 0 dev.ix.0.iflib.driver_version: 4.0.1-k dev.ix.0.%parent: pci2 dev.ix.0.%pnpinfo: vendor=0x8086 device=0x15d1 subvendor=0x8086 subdevice=0x0002 class=0x020000 dev.ix.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 handle=\_SB_.PCI0.RP09.PXSX dev.ix.0.%driver: ix dev.ix.0.%desc: Intel(R) X550-T1
So far on ix0 int:
- Disabled FC
- Checked disabled hardware checksum offload
- Verified int is running latest Intel driver (believe it's the latest - 4.0.1-k)
- Checked latest firmware via Intel bootable update utility
- swapped to multiple, new CAT6 patch cords including Commscope and Panduit (no cheap cords)
- Bought a new X550-T1 card in case my previous card was dying.
- Had Metronet provision a new Nokia G240G ONT and powersupply to no affect.
Lastly, I found the following on Intel's Download Center and it seems it couldn't be more applicable to my situation (Intel NIC, ONT, cksum errors):
https://www.intel.com/content/www/us/en/download/19174/disabling-tcp-ipv6-checksum-offload-capability-with-intel-1-10-gbe-controllers.html
I've tried modifying INET6 settings via Shell command as per FreeBSD IFCONFIG:
-rxcsum, -txcsum, -rxcsum6, -txcsum6 If the driver supports user-configurable checksum offloading, disable receive (or transmit) checksum offloading on the inter- face. The feature can be turned off selectively per protocol family. Use -rxcsum6, -txcsum6 for ip6(4) or -rxcsum, -txcsum otherwise. These settings may not always be independent of each other.
Still no changes and errors continue on WAN IN. To be fair, for ~100GiB data in I might have 6-10 errors. I have also now upgraded the 2.6CE to pfSense Plus ver 22.01 with yearly support package.
Apologies, not trying to hijack OP's thread but hoping
to share more data to the WAN ERRORS IN. Thanks. -
-
Reporting the same on 2.6 with X557 NICs, all 'mac_stats.checksum_errs':
-
The difference here is that the ix driver in 2.6 includes all input errors in the mac stats value not just crc and length errors:
https://github.com/pfsense/FreeBSD-src/commit/5574b12aac2bdfe66ec1d9564f932eeec9ac213cSo nothing has actually changed, you are just seeing the errors now that were previously not shown.
Also see: https://redmine.pfsense.org/issues/12904
Steve