Intel X710 Issues
-
I’ve recently changed from an Intel x540-t2 NIC to an Intel x710-t2 and changed my modem from an SB8200 to an MB8611 because I wanted to take advantage of Comcast’s upgraded download speed from 1Gbs to 1.2Gbs.
Now that I’ve made those changes, I’m having some issues with inconsistent download performance where before I was able to easily hit 940Mbs now, I'm averaging 500 - 700 Mbs.
Network performance within the LAN (either 10Gbs to 10Gbs or 1Gbs to 1Gbs) shows that there are no issues on the LAN side.
So, I’m trying to determine if this is an issue with Speedtest sites, the new modem, or the new NIC. Due to where and how the server is installed, switching back to the 540 card is a difficult, but not impossible task. So, I have saved that for a last resort.
I’m on pfSense 2.5 and the box boots fine. Here is some hopefully relevant diagnostic information:
ifconfig ixl0 ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: WAN1_COMCAST options=e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> ether xx:xx:xx:xx:xx:xx hwaddr xx:xx:xx:xx:xxx:xx inet6 xxxx::xxxx:xxxx:xxxx:xxxx%ixl0 prefixlen 64 scopeid 0x1 inet6 xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx prefixlen 128 inet xxx.xxx.xxx.xxx netmask 0xfffffc00 broadcast 255.255.255.255 media: Ethernet autoselect (2500Base-T <full-duplex,rxpause>) status: active nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
netstat -I ixl0 Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll ixl0 1500 <Link#1> xx:xx:xx:xx:xx:xx 66453192 0 4294966048 28719745 0 0 ixl0 - fe80::%ixl0/6 fe80::bc0c:edff:f 312244 - - 312572 - - ixl0 - xxx.xxx.xxx.xxx/22 c-xx-xx-xx-xxx.hsd 2336062 - - 312435 - - ixl0 - xxx:xxx:xxxx xxxx:xxx:xxxx:xx: 399298 - - 1476 - -
sysctl -a | grep ixl0 irq257: ixl0:aq:57 @cpu0(domain0): 3 irq258: ixl0:rxq0:59 @cpu1(domain0): 6050062 irq259: ixl0:rxq1:61 @cpu0(domain0): 6664740 irq260: ixl0:rxq2:63 @cpu3(domain0): 5706167 irq261: ixl0:rxq3:65 @cpu2(domain0): 7734602 irq262: ixl0:rxq4:67 @cpu5(domain0): 6781857 irq263: ixl0:rxq5:69 @cpu4(domain0): 7005564 irq264: ixl0:rxq6:71 @cpu7(domain0): 7371266 irq265: ixl0:rxq7:73 @cpu6(domain0): 6575920
dev.ixl.0.fw_version: fw 8.1.63299 api 1.12 nvm 8.10 etid 800093ea oem 1.267.0
sysctl -a | grep ixl\.0 dev.ixl.0.mac.xoff_recvd: 0 dev.ixl.0.mac.xoff_txd: 0 dev.ixl.0.mac.xon_recvd: 0 dev.ixl.0.mac.xon_txd: 0 dev.ixl.0.mac.tx_frames_big: 0 dev.ixl.0.mac.tx_frames_1024_1522: 6109929 dev.ixl.0.mac.tx_frames_512_1023: 1008041 dev.ixl.0.mac.tx_frames_256_511: 664973 dev.ixl.0.mac.tx_frames_128_255: 2485605 dev.ixl.0.mac.tx_frames_65_127: 17366263 dev.ixl.0.mac.tx_frames_64: 1134418 dev.ixl.0.mac.checksum_errors: 0 dev.ixl.0.mac.rx_jabber: 0 dev.ixl.0.mac.rx_oversized: 0 dev.ixl.0.mac.rx_fragmented: 0 dev.ixl.0.mac.rx_undersize: 0 dev.ixl.0.mac.rx_frames_big: 0 dev.ixl.0.mac.rx_frames_1024_1522: 48130916 dev.ixl.0.mac.rx_frames_512_1023: 1452552 dev.ixl.0.mac.rx_frames_256_511: 2019464 dev.ixl.0.mac.rx_frames_128_255: 3456185 dev.ixl.0.mac.rx_frames_65_127: 10139155 dev.ixl.0.mac.rx_frames_64: 1333971 dev.ixl.0.mac.rx_length_errors: 0 dev.ixl.0.mac.remote_faults: 0 dev.ixl.0.mac.local_faults: 1 dev.ixl.0.mac.illegal_bytes: 0 dev.ixl.0.mac.crc_errors: 0 dev.ixl.0.mac.bcast_pkts_txd: 3 dev.ixl.0.mac.mcast_pkts_txd: 5603 dev.ixl.0.mac.ucast_pkts_txd: 28763623 dev.ixl.0.mac.good_octets_txd: 11938925817 dev.ixl.0.mac.rx_discards: 0 dev.ixl.0.mac.bcast_pkts_rcvd: 618 dev.ixl.0.mac.mcast_pkts_rcvd: 59715 dev.ixl.0.mac.ucast_pkts_rcvd: 66471910 dev.ixl.0.mac.good_octets_rcvd: 74532507761 dev.ixl.0.pf.txq07.itr: 122 dev.ixl.0.pf.txq07.bytes: 1182878011 dev.ixl.0.pf.txq07.packets: 3151710 dev.ixl.0.pf.txq07.mss_too_small: 0 dev.ixl.0.pf.txq07.tso: 0 dev.ixl.0.pf.txq06.itr: 122 dev.ixl.0.pf.txq06.bytes: 3609386915 dev.ixl.0.pf.txq06.packets: 5592768 dev.ixl.0.pf.txq06.mss_too_small: 0 dev.ixl.0.pf.txq06.tso: 0 dev.ixl.0.pf.txq05.itr: 122 dev.ixl.0.pf.txq05.bytes: 1233392624 dev.ixl.0.pf.txq05.packets: 2926836 dev.ixl.0.pf.txq05.mss_too_small: 0 dev.ixl.0.pf.txq05.tso: 0 dev.ixl.0.pf.txq04.itr: 122 dev.ixl.0.pf.txq04.bytes: 1410591375 dev.ixl.0.pf.txq04.packets: 3697406 dev.ixl.0.pf.txq04.mss_too_small: 0 dev.ixl.0.pf.txq04.tso: 0 dev.ixl.0.pf.txq03.itr: 122 dev.ixl.0.pf.txq03.bytes: 1000037614 dev.ixl.0.pf.txq03.packets: 3065998 dev.ixl.0.pf.txq03.mss_too_small: 0 dev.ixl.0.pf.txq03.tso: 0 dev.ixl.0.pf.txq02.itr: 122 dev.ixl.0.pf.txq02.bytes: 1080802445 dev.ixl.0.pf.txq02.packets: 3203676 dev.ixl.0.pf.txq02.mss_too_small: 0 dev.ixl.0.pf.txq02.tso: 0 dev.ixl.0.pf.txq01.itr: 122 dev.ixl.0.pf.txq01.bytes: 1151535920 dev.ixl.0.pf.txq01.packets: 3471789 dev.ixl.0.pf.txq01.mss_too_small: 0 dev.ixl.0.pf.txq01.tso: 0 dev.ixl.0.pf.txq00.itr: 122 dev.ixl.0.pf.txq00.bytes: 1144568137 dev.ixl.0.pf.txq00.packets: 3653700 dev.ixl.0.pf.txq00.mss_too_small: 0 dev.ixl.0.pf.txq00.tso: 0 dev.ixl.0.pf.rxq07.itr: 62 dev.ixl.0.pf.rxq07.desc_err: 0 dev.ixl.0.pf.rxq07.bytes: 10949318034 dev.ixl.0.pf.rxq07.packets: 9205753 dev.ixl.0.pf.rxq07.irqs: 6577452 dev.ixl.0.pf.rxq06.itr: 62 dev.ixl.0.pf.rxq06.desc_err: 0 dev.ixl.0.pf.rxq06.bytes: 7017993874 dev.ixl.0.pf.rxq06.packets: 6838505 dev.ixl.0.pf.rxq06.irqs: 7374314 dev.ixl.0.pf.rxq05.itr: 62 dev.ixl.0.pf.rxq05.desc_err: 0 dev.ixl.0.pf.rxq05.bytes: 9879807442 dev.ixl.0.pf.rxq05.packets: 9544935 dev.ixl.0.pf.rxq05.irqs: 7008611 dev.ixl.0.pf.rxq04.itr: 62 dev.ixl.0.pf.rxq04.desc_err: 0 dev.ixl.0.pf.rxq04.bytes: 10768414672 dev.ixl.0.pf.rxq04.packets: 9050156 dev.ixl.0.pf.rxq04.irqs: 6792564 dev.ixl.0.pf.rxq03.itr: 62 dev.ixl.0.pf.rxq03.desc_err: 0 dev.ixl.0.pf.rxq03.bytes: 10168392259 dev.ixl.0.pf.rxq03.packets: 9566705 dev.ixl.0.pf.rxq03.irqs: 7737964 dev.ixl.0.pf.rxq02.itr: 62 dev.ixl.0.pf.rxq02.desc_err: 0 dev.ixl.0.pf.rxq02.bytes: 7264365837 dev.ixl.0.pf.rxq02.packets: 6417728 dev.ixl.0.pf.rxq02.irqs: 5709226 dev.ixl.0.pf.rxq01.itr: 62 dev.ixl.0.pf.rxq01.desc_err: 0 dev.ixl.0.pf.rxq01.bytes: 10484107392 dev.ixl.0.pf.rxq01.packets: 8927330 dev.ixl.0.pf.rxq01.irqs: 6666136 dev.ixl.0.pf.rxq00.itr: 62 dev.ixl.0.pf.rxq00.desc_err: 0 dev.ixl.0.pf.rxq00.bytes: 7732703517 dev.ixl.0.pf.rxq00.packets: 6969477 dev.ixl.0.pf.rxq00.irqs: 6053252 dev.ixl.0.pf.bcast_pkts_txd: 3 dev.ixl.0.pf.mcast_pkts_txd: 226 dev.ixl.0.pf.ucast_pkts_txd: 28763623 dev.ixl.0.pf.good_octets_txd: 11813187746 dev.ixl.0.pf.rx_discards: 4294966048 dev.ixl.0.pf.bcast_pkts_rcvd: 618 dev.ixl.0.pf.mcast_pkts_rcvd: 48470 dev.ixl.0.pf.ucast_pkts_rcvd: 66471682 dev.ixl.0.pf.good_octets_rcvd: 74531467550 dev.ixl.0.admin_irq: 3 dev.ixl.0.eee.rx_lpi_count: 0 dev.ixl.0.eee.tx_lpi_count: 0 dev.ixl.0.eee.rx_lpi_status: 0 dev.ixl.0.eee.tx_lpi_status: 0 dev.ixl.0.eee.enable: 1 dev.ixl.0.fw_lldp: 1 dev.ixl.0.dynamic_tx_itr: 0 dev.ixl.0.dynamic_rx_itr: 0 dev.ixl.0.rx_itr: 62 dev.ixl.0.tx_itr: 122 dev.ixl.0.unallocated_queues: 760 dev.ixl.0.fw_version: fw 8.1.63299 api 1.12 nvm 8.10 etid 800093ea oem 1.267.0 dev.ixl.0.current_speed: 2.5 Gbps dev.ixl.0.supported_speeds: 199 dev.ixl.0.advertise_speed: 199 dev.ixl.0.fc: 0 dev.ixl.0.iflib.rxq7.rxq_fl0.buf_size: 2048 dev.ixl.0.iflib.rxq7.rxq_fl0.credits: 1023 dev.ixl.0.iflib.rxq7.rxq_fl0.cidx: 1017 dev.ixl.0.iflib.rxq7.rxq_fl0.pidx: 1016 dev.ixl.0.iflib.rxq6.rxq_fl0.buf_size: 2048 dev.ixl.0.iflib.rxq6.rxq_fl0.credits: 1023 dev.ixl.0.iflib.rxq6.rxq_fl0.cidx: 233 dev.ixl.0.iflib.rxq6.rxq_fl0.pidx: 232 dev.ixl.0.iflib.rxq5.rxq_fl0.buf_size: 2048 dev.ixl.0.iflib.rxq5.rxq_fl0.credits: 1023 dev.ixl.0.iflib.rxq5.rxq_fl0.cidx: 231 dev.ixl.0.iflib.rxq5.rxq_fl0.pidx: 230 dev.ixl.0.iflib.rxq4.rxq_fl0.buf_size: 2048 dev.ixl.0.iflib.rxq4.rxq_fl0.credits: 1023 dev.ixl.0.iflib.rxq4.rxq_fl0.cidx: 44 dev.ixl.0.iflib.rxq4.rxq_fl0.pidx: 43 dev.ixl.0.iflib.rxq3.rxq_fl0.buf_size: 2048 dev.ixl.0.iflib.rxq3.rxq_fl0.credits: 1023 dev.ixl.0.iflib.rxq3.rxq_fl0.cidx: 497 dev.ixl.0.iflib.rxq3.rxq_fl0.pidx: 496 dev.ixl.0.iflib.rxq2.rxq_fl0.buf_size: 2048 dev.ixl.0.iflib.rxq2.rxq_fl0.credits: 1023 dev.ixl.0.iflib.rxq2.rxq_fl0.cidx: 320 dev.ixl.0.iflib.rxq2.rxq_fl0.pidx: 319 dev.ixl.0.iflib.rxq1.rxq_fl0.buf_size: 2048 dev.ixl.0.iflib.rxq1.rxq_fl0.credits: 1023 dev.ixl.0.iflib.rxq1.rxq_fl0.cidx: 98 dev.ixl.0.iflib.rxq1.rxq_fl0.pidx: 97 dev.ixl.0.iflib.rxq0.rxq_fl0.buf_size: 2048 dev.ixl.0.iflib.rxq0.rxq_fl0.credits: 1023 dev.ixl.0.iflib.rxq0.rxq_fl0.cidx: 133 dev.ixl.0.iflib.rxq0.rxq_fl0.pidx: 132 dev.ixl.0.iflib.txq7.r_abdications: 0 dev.ixl.0.iflib.txq7.r_restarts: 0 dev.ixl.0.iflib.txq7.r_stalls: 0 dev.ixl.0.iflib.txq7.r_starts: 3151699 dev.ixl.0.iflib.txq7.r_drops: 0 dev.ixl.0.iflib.txq7.r_enqueues: 3151710 dev.ixl.0.iflib.txq7.ring_state: pidx_head: 1886 pidx_tail: 1886 cidx: 1886 state: IDLE dev.ixl.0.iflib.txq7.txq_cleaned: 3153559 dev.ixl.0.iflib.txq7.txq_processed: 3153567 dev.ixl.0.iflib.txq7.txq_in_use: 8 dev.ixl.0.iflib.txq7.txq_cidx_processed: 671 dev.ixl.0.iflib.txq7.txq_cidx: 663 dev.ixl.0.iflib.txq7.txq_pidx: 671 dev.ixl.0.iflib.txq7.no_tx_dma_setup: 0 dev.ixl.0.iflib.txq7.txd_encap_efbig: 0 dev.ixl.0.iflib.txq7.tx_map_failed: 0 dev.ixl.0.iflib.txq7.no_desc_avail: 0 dev.ixl.0.iflib.txq7.mbuf_defrag_failed: 0 dev.ixl.0.iflib.txq7.m_pullups: 0 dev.ixl.0.iflib.txq7.mbuf_defrag: 0 dev.ixl.0.iflib.txq6.r_abdications: 0 dev.ixl.0.iflib.txq6.r_restarts: 0 dev.ixl.0.iflib.txq6.r_stalls: 0 dev.ixl.0.iflib.txq6.r_starts: 5592303 dev.ixl.0.iflib.txq6.r_drops: 0 dev.ixl.0.iflib.txq6.r_enqueues: 5592775 dev.ixl.0.iflib.txq6.ring_state: pidx_head: 1735 pidx_tail: 1735 cidx: 1735 state: IDLE dev.ixl.0.iflib.txq6.txq_cleaned: 5596751 dev.ixl.0.iflib.txq6.txq_processed: 5596759 dev.ixl.0.iflib.txq6.txq_in_use: 8 dev.ixl.0.iflib.txq6.txq_cidx_processed: 599 dev.ixl.0.iflib.txq6.txq_cidx: 591 dev.ixl.0.iflib.txq6.txq_pidx: 599 dev.ixl.0.iflib.txq6.no_tx_dma_setup: 0 dev.ixl.0.iflib.txq6.txd_encap_efbig: 0 dev.ixl.0.iflib.txq6.tx_map_failed: 0 dev.ixl.0.iflib.txq6.no_desc_avail: 0 dev.ixl.0.iflib.txq6.mbuf_defrag_failed: 0 dev.ixl.0.iflib.txq6.m_pullups: 0 dev.ixl.0.iflib.txq6.mbuf_defrag: 0 dev.ixl.0.iflib.txq5.r_abdications: 0 dev.ixl.0.iflib.txq5.r_restarts: 0 dev.ixl.0.iflib.txq5.r_stalls: 0 dev.ixl.0.iflib.txq5.r_starts: 2926826 dev.ixl.0.iflib.txq5.r_drops: 0 dev.ixl.0.iflib.txq5.r_enqueues: 2926837 dev.ixl.0.iflib.txq5.ring_state: pidx_head: 0245 pidx_tail: 0245 cidx: 0245 state: IDLE dev.ixl.0.iflib.txq5.txq_cleaned: 2934125 dev.ixl.0.iflib.txq5.txq_processed: 2934133 dev.ixl.0.iflib.txq5.txq_in_use: 8 dev.ixl.0.iflib.txq5.txq_cidx_processed: 373 dev.ixl.0.iflib.txq5.txq_cidx: 365 dev.ixl.0.iflib.txq5.txq_pidx: 373 dev.ixl.0.iflib.txq5.no_tx_dma_setup: 0 dev.ixl.0.iflib.txq5.txd_encap_efbig: 0 dev.ixl.0.iflib.txq5.tx_map_failed: 0 dev.ixl.0.iflib.txq5.no_desc_avail: 0 dev.ixl.0.iflib.txq5.mbuf_defrag_failed: 0 dev.ixl.0.iflib.txq5.m_pullups: 0 dev.ixl.0.iflib.txq5.mbuf_defrag: 0 dev.ixl.0.iflib.txq4.r_abdications: 0 dev.ixl.0.iflib.txq4.r_restarts: 0 dev.ixl.0.iflib.txq4.r_stalls: 0 dev.ixl.0.iflib.txq4.r_starts: 3697378 dev.ixl.0.iflib.txq4.r_drops: 0 dev.ixl.0.iflib.txq4.r_enqueues: 3697407 dev.ixl.0.iflib.txq4.ring_state: pidx_head: 0767 pidx_tail: 0767 cidx: 0767 state: IDLE dev.ixl.0.iflib.txq4.txq_cleaned: 3700282 dev.ixl.0.iflib.txq4.txq_processed: 3700290 dev.ixl.0.iflib.txq4.txq_in_use: 8 dev.ixl.0.iflib.txq4.txq_cidx_processed: 578 dev.ixl.0.iflib.txq4.txq_cidx: 570 dev.ixl.0.iflib.txq4.txq_pidx: 578 dev.ixl.0.iflib.txq4.no_tx_dma_setup: 0 dev.ixl.0.iflib.txq4.txd_encap_efbig: 0 dev.ixl.0.iflib.txq4.tx_map_failed: 0 dev.ixl.0.iflib.txq4.no_desc_avail: 0 dev.ixl.0.iflib.txq4.mbuf_defrag_failed: 0 dev.ixl.0.iflib.txq4.m_pullups: 0 dev.ixl.0.iflib.txq4.mbuf_defrag: 0 dev.ixl.0.iflib.txq3.r_abdications: 0 dev.ixl.0.iflib.txq3.r_restarts: 0 dev.ixl.0.iflib.txq3.r_stalls: 0 dev.ixl.0.iflib.txq3.r_starts: 3065983 dev.ixl.0.iflib.txq3.r_drops: 0 dev.ixl.0.iflib.txq3.r_enqueues: 3065998 dev.ixl.0.iflib.txq3.ring_state: pidx_head: 0142 pidx_tail: 0142 cidx: 0142 state: IDLE dev.ixl.0.iflib.txq3.txq_cleaned: 3068787 dev.ixl.0.iflib.txq3.txq_processed: 3068795 dev.ixl.0.iflib.txq3.txq_in_use: 8 dev.ixl.0.iflib.txq3.txq_cidx_processed: 891 dev.ixl.0.iflib.txq3.txq_cidx: 883 dev.ixl.0.iflib.txq3.txq_pidx: 891 dev.ixl.0.iflib.txq3.no_tx_dma_setup: 0 dev.ixl.0.iflib.txq3.txd_encap_efbig: 0 dev.ixl.0.iflib.txq3.tx_map_failed: 0 dev.ixl.0.iflib.txq3.no_desc_avail: 0 dev.ixl.0.iflib.txq3.mbuf_defrag_failed: 0 dev.ixl.0.iflib.txq3.m_pullups: 0 dev.ixl.0.iflib.txq3.mbuf_defrag: 0 dev.ixl.0.iflib.txq2.r_abdications: 0 dev.ixl.0.iflib.txq2.r_restarts: 0 dev.ixl.0.iflib.txq2.r_stalls: 0 dev.ixl.0.iflib.txq2.r_starts: 3203666 dev.ixl.0.iflib.txq2.r_drops: 0 dev.ixl.0.iflib.txq2.r_enqueues: 3203676 dev.ixl.0.iflib.txq2.ring_state: pidx_head: 0604 pidx_tail: 0604 cidx: 0604 state: IDLE dev.ixl.0.iflib.txq2.txq_cleaned: 3207578 dev.ixl.0.iflib.txq2.txq_processed: 3207586 dev.ixl.0.iflib.txq2.txq_in_use: 8 dev.ixl.0.iflib.txq2.txq_cidx_processed: 418 dev.ixl.0.iflib.txq2.txq_cidx: 410 dev.ixl.0.iflib.txq2.txq_pidx: 418 dev.ixl.0.iflib.txq2.no_tx_dma_setup: 0 dev.ixl.0.iflib.txq2.txd_encap_efbig: 0 dev.ixl.0.iflib.txq2.tx_map_failed: 0 dev.ixl.0.iflib.txq2.no_desc_avail: 0 dev.ixl.0.iflib.txq2.mbuf_defrag_failed: 0 dev.ixl.0.iflib.txq2.m_pullups: 0 dev.ixl.0.iflib.txq2.mbuf_defrag: 0 dev.ixl.0.iflib.txq1.r_abdications: 0 dev.ixl.0.iflib.txq1.r_restarts: 0 dev.ixl.0.iflib.txq1.r_stalls: 0 dev.ixl.0.iflib.txq1.r_starts: 3471783 dev.ixl.0.iflib.txq1.r_drops: 0 dev.ixl.0.iflib.txq1.r_enqueues: 3471789 dev.ixl.0.iflib.txq1.ring_state: pidx_head: 0429 pidx_tail: 0429 cidx: 0429 state: IDLE dev.ixl.0.iflib.txq1.txq_cleaned: 3476015 dev.ixl.0.iflib.txq1.txq_processed: 3476023 dev.ixl.0.iflib.txq1.txq_in_use: 8 dev.ixl.0.iflib.txq1.txq_cidx_processed: 567 dev.ixl.0.iflib.txq1.txq_cidx: 559 dev.ixl.0.iflib.txq1.txq_pidx: 567 dev.ixl.0.iflib.txq1.no_tx_dma_setup: 0 dev.ixl.0.iflib.txq1.txd_encap_efbig: 0 dev.ixl.0.iflib.txq1.tx_map_failed: 0 dev.ixl.0.iflib.txq1.no_desc_avail: 0 dev.ixl.0.iflib.txq1.mbuf_defrag_failed: 0 dev.ixl.0.iflib.txq1.m_pullups: 0 dev.ixl.0.iflib.txq1.mbuf_defrag: 0 dev.ixl.0.iflib.txq0.r_abdications: 0 dev.ixl.0.iflib.txq0.r_restarts: 0 dev.ixl.0.iflib.txq0.r_stalls: 0 dev.ixl.0.iflib.txq0.r_starts: 3626345 dev.ixl.0.iflib.txq0.r_drops: 0 dev.ixl.0.iflib.txq0.r_enqueues: 3653702 dev.ixl.0.iflib.txq0.ring_state: pidx_head: 0070 pidx_tail: 0070 cidx: 0070 state: IDLE dev.ixl.0.iflib.txq0.txq_cleaned: 3664449 dev.ixl.0.iflib.txq0.txq_processed: 3664457 dev.ixl.0.iflib.txq0.txq_in_use: 8 dev.ixl.0.iflib.txq0.txq_cidx_processed: 585 dev.ixl.0.iflib.txq0.txq_cidx: 577 dev.ixl.0.iflib.txq0.txq_pidx: 585 dev.ixl.0.iflib.txq0.no_tx_dma_setup: 0 dev.ixl.0.iflib.txq0.txd_encap_efbig: 0 dev.ixl.0.iflib.txq0.tx_map_failed: 0 dev.ixl.0.iflib.txq0.no_desc_avail: 0 dev.ixl.0.iflib.txq0.mbuf_defrag_failed: 0 dev.ixl.0.iflib.txq0.m_pullups: 19463 dev.ixl.0.iflib.txq0.mbuf_defrag: 0 dev.ixl.0.iflib.override_nrxds: 0 dev.ixl.0.iflib.override_ntxds: 0 dev.ixl.0.iflib.separate_txrx: 0 dev.ixl.0.iflib.core_offset: 0 dev.ixl.0.iflib.tx_abdicate: 0 dev.ixl.0.iflib.rx_budget: 0 dev.ixl.0.iflib.disable_msix: 0 dev.ixl.0.iflib.override_qs_enable: 0 dev.ixl.0.iflib.override_nrxqs: 0 dev.ixl.0.iflib.override_ntxqs: 0 dev.ixl.0.iflib.driver_version: 2.3.0-k dev.ixl.0.%parent: pci4 dev.ixl.0.%pnpinfo: vendor=0x8086 device=0x15ff subvendor=0x8086 subdevice=0x0003 class=0x020000 dev.ixl.0.%location: slot=0 function=0 dbsf=pci0:4:0:0 dev.ixl.0.%driver: ixl dev.ixl.0.%desc: Intel(R) Ethernet Controller X710 for 10GBASE-T - 2.3.0-k
The thing I’m looking most closely at is the rxpause. I don’t know where that came from and what’s causing it or even if it’s contributing.
Any thoughts?
-
@mloiterman Are you testing the speed through the firewall from a client?
-
X540-T2 can easily push what you need. The X710 have driver issues. We see that in the DC's where we have the T4's as our primary datahauler in the 1U servers.
-
Yes, I’m testing from a LAN side client.
The issue with the 540 is that it doesn’t do the NBASE-T so it cannot connect at 2.5, only 1 and 10. And no modem supports 10.
I also think there are driver issues at play. I have another modem, the Arris S33, that I’m going to try today to see if that makes a difference. After that, I guess I’ll revert back to the 540?
There is also the 550 which does support NBASE-T, but that card seemed to have issues with linking at 2.5/5 on FreeBSD. But maybe that’s been addressed in pfSense 2.5?
-
@mloiterman
try to disable RXCSUM,TXCSUM -
VmWare recommends disabling LRO and TSO both IPv4 and IPv6 and then a reboot.
I still kept getting hit by the PSOD and that stopped when I disabled jumbo frames on the attached adapters.
Got a performance hit, but better than PSOD.
-
I still kept getting hit by the PSOD and that stopped when I disabled jumbo frames on the attached adapters.
What is PSOD and how did you disable jump frames?
So far this morning, I have tried the following with no success - they all have no impact on the reduced download speeds:
- In System Tuneables I have manually added the following, but it was overwritten on reboot so maybe I'm not doing something correctly. It's for the ix driver, so it's probably not relevant anyway:
hw.ix.flow_control: 0
- In loader.conf, I added via Diagnostics -> Edit File. Again, this was overwritten on reboot as confirmed from checking in systctl and it shows "0" for both which is presumably the correct setting, but I wanted to try the reverse:
dev.ixl.1.fc="3" dev.ixl.0.fc="3"
- In Advanced -> Networking: Disable hardware checksum offload
- In Advanced -> Networking: Enabled Hardware TCP Segmentation Offloading
- In Advanced -> Networking: Enabled Hardware Large Receive Offloading
- In Advanced -> Networking: Enabled Hardware Large Receive Offloading
- In Advanced -> Networking: Disabled the ALTQ support for hn NICs
Nothing works.
How do I turn off the rxpause? Shouldn't that be from dev.ixl.1.fc? It's at 0 now, but yet the interface shows it enabled.
-
I've now upgrade the firmware on the X710 and replaced the modem with an Arris S33.
dev.ixl.0.fw_version: fw 8.2.64244 api 1.13 nvm 8.20 etid 8000a062 oem 1.267.0
Since upgrading the firmware, now the interface shows:
media: Ethernet autoselect (2500Base-T <full-duplex>)
but on speed tests the best I'm able to do is about 750 or 800.
So, something is still going on.
dev.ixl.0.iflib.driver_version: 2.3.0-k
But really strange and inconsistent download performance:
speedtest-cli --server 1776 Retrieving speedtest.net configuration... Testing from Comcast Cable (xxxxxxxxx)... Retrieving speedtest.net server list... Retrieving information for the selected server... Hosted by Comcast (Chicago, IL) [38.69 km]: 44.566 ms Testing download speed................................................................................ Download: 1140.00 Mbit/s Testing upload speed...................................................................................................... Upload: 42.13 Mbit/s [mike@arch ~]$ speedtest-cli --server 1776 Retrieving speedtest.net configuration... Testing from Comcast Cable (xxxxxxxxx)... Retrieving speedtest.net server list... Retrieving information for the selected server... Hosted by Comcast (Chicago, IL) [38.69 km]: 13.061 ms Testing download speed................................................................................ Download: 701.39 Mbit/s Testing upload speed..................................................................................................... .Upload: 35.68 Mbit/s
How can I add and use the newest intel driver?
-
@mloiterman said in Intel X710 Issues:
How can I add and use the newest intel driver?
It should already be in V2.5 ... I believe!
-
Looks like the intel driver is at 3.3.22...?
-
Definitely try disabling flow control. Those should be added as sysctls/system tunables for ixl:
https://www.freebsd.org/cgi/man.cgi?query=ixl#SYSCTL_PROCEDURESSteve
-
Appreciate the response.
But it seems like it's already disabled:
sysctl -a | grep dev.ixl.0.fc dev.ixl.0.fc: 0
Is there another place where I need to check?
-
dmesg | grep ^ixl ixl0: Link is up, 2.5 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: ixl0: link state changed to UP ixl1: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: N ixl1: link state changed to UP
So it seems like despite:
sysctl -a | grep dev.ixl.0.fc dev.ixl.0.fc: 0
Flow Control is N for ixl1 but empty for ixl0.
This is so frustrating - it makes no sense.
-
ifconfig still shows
rxpause, txpause
? -
No not after upgrading the firmware on the card.
I tried enabling flow control by setting it to sequential setting it to 0,1,2,3 through the tunables page. I could get it to turn on txpause but not rxpause which is wierd because initially, before upgrading firmware, I could not get rxpause to turn off.
I did not reboot or reboot the modem inbetween changing those settings though.
-
Flow control is negotiated so you won't see it in the ifconfig output if the other side doesn't support it. You can force it on or off though on some NICs.
Steve
-
OK, I got it figured out.
I had to enable flow control on my switch for the port connected to pfSense and had to enable full flow control for the X710 port connected to my switch with these systctl tunables in System/Advanced/System Tunables:
dev.ixl.1.fc
3
Enable flow controlNow I can get ~ 945Mbs for the machines connected with 1G NICs and ~1,200Mbs for the 10G NICs.
-
Ah, nice result. Interesting. I guess you are hitting something that requires flow-control to mitigate which is unusual.
Steve
-
Did you update the drivers after updating the NVM firmware? I just updated to 8.30 FW, but following your steps, I have been unable to acheive the same results.
-
Yes, I compiled the latest version of intel's driver against the kernel in 2.5.1.
I can help you with that if you need it, but it will have to wait until later this week.
-
@mloiterman I would be interested in the compile instructions as well as I have the same issue you have although I did manage a work around with the old firmware 8.10. I updated the firmware to 8.30 :-( . I could reverse the firmware update but I figured I would move forward through this problem :-) . Something is seriously wrong with this card and PFsense at the moment and I hope netgate finds the time to fix it. Thanks
-
I’ve updated to the 8.30 firmware and the 1.12.16 version of the driver and it still seems like I need to enable Flow Control on the LAN side port of the X710 and the switch port it’s connected to.
For anyone that is interested here’s the basic steps to build you’re own version of the Intel driver.
- Get yourself a FreeBSD system where you can download source files and compile the driver. I would suggest NOT using your pfSense system to do this kind of work.
- Once you’ve got your FreeBSD system setup, do a deep clone of the source repository. For 12.2-RELEASE:
# git clone -b releng/12.2 https://git.freebsd.org/src.git /usr/src
- Determine which version of pfSense your kernel was compiled against from here. For pfSense CE 2.5.1, for example, it’s 12.2-STABLE@f4d0bc6aa6b.
- On the system where you’ve cloned the source tree in step 2, check out the specific version of the source tree you’ve determined in step 3 by using the hash value at the end of the version string:
# cd /usr/src # git checkout f4d0bc6aa6b
-
Download the driver source files from Intel. For 1.12.16, they can be found here.
-
Untar the archive:
# tar zxf ixl-1.12.16.tar.gz
This will create the ixl-1.12.16 directory.
- To load the driver onto a running system:
# cd ixl-1.12.16/src # make # kldload ./if_ixl.ko
-
Did you copy this to /boot/kernel to make it persistant?
-
No, you place the kernel module in /boot/modules.
Make sure the module is set to root:wheel and the permissions are 555:
# chown root:wheel if_ixl.ko && chmod 555 if_ixl.ko
and then in /boot/loader.conf.local, which may not exist unless you create it, you'll need to add:
if_ixl_load="YES"
-
Thanks. Running into some issues in my scenario.
- IPv6 addresses for WAN and LAN are no longer populated
- 1Gbps hosts are still not reaching 942Mbps
- WAN Interface showing unknown speed
- The interface status for ixl1 does not reflect rxpause, txpause (driver loaded confirmed)
-
@lnguyen and I are working together on this project.
I have the x710-T2L,I compiled the 1.12.16 drivers in my FreeBSD 12.2 VM and got us to this point.
I have flow control which works but the speed isn't showing. (dev.ixl.1.fc=3)
I notice that it takes a bit for the IPv6 addresses to populate on startup. This is new behavior.What are we missing here?
-
I cannot say what might be causing your issues, but I can tell you that I enabled Flow Control on the LAN switch ports and enabled full Flow Control on the LAN port on the X710 card. I had Flow Control disabled on the WAN port of the X710 card.
My understanding is the Flow Control is negotiated and must be active on both sides of the connection.
This whole thing is still a bit of a mystery to me and it’s not really clear why it’s needed. I suspect it may be partially rooted in the fact that my switches, HP 2920 48 port with PoE+ are 1Gbs switches with a 10Gbs add in cards. NBASE-T is a newer standard and there maybe bugs or issues like this that are triggered when you have a mixed environment. That’s only speculation from me, however.
Please let me know what you find because I would like to eventually disable Flow Control.
-
Messing around last night yielded results.
- In the GUI, added System Tunable
dev.ixl.1.fc=3
- Reverted back to the shipped driver (iflib)
- Enabled Flow Control on the Netgear switch (XS724EM)
- Restarted
Not entirely sure why this works but there is significance when CE 2.5.1 adds the tunable vs
sysctl
In testing, ATV 4K will obtain (has to work up to it) 940 Mbps.
Speeds do not "snap" up to their values. Speedtest seems to 'slew' its way up to the final value.
I suspect there is a better way to handle this but is well above my understanding level. Perhaps priority flow control ?
- In the GUI, added System Tunable
-
IPv6 addresses did eventually populate. I have a Cisco switch that is not negotiating FC with the Intel X710-T4L ports. The default kernel drivers report the correct speed negotiated from ifconfig
-
I think this scenario is due to WAN speed >1Gbps overflowing 1Gbps connected LAN hosts.
-
Two other things:
I noticed performance was subjectively better when flow control for the X710 lan port to “1” which is rxpause. 3 worked too, but “seemed” less performing. Although, I have no objective data to see if that’s really true or not.
Performance seemed better when I Disabled hardware checksum offload.
-
Thanks,
- Made the change.
- I'll play around with the offload. I will see what impact that has on the 10 GbE machines.
-
2 year old thread. Same issue but with WAN 2x1Gbps LAGG instead of 2.5Gbps
Low throughput over lagg with 1gb clients -
I think using ethernet flow control to solve this issue is not an ideal workaround. I think looking at TCP flow control or packet buffer sizes in pfSense is the long term resolution here. Just not sure what to tune in pfSense to accommodate WAN connections greater than 1Gbps LAN connections.
-
Is there any update to this issue? I'm using the x710-t2l with FC=1 on lan and a netgear MS510TX multigig switch with global flow control since the switch doesnt offer individual port controls. I would like to turn off FC if possible.
-
-
@mloiterman Followed your procedure, but I also had to replace the file completely in boot/kernel otherwise it would not load the newly compiled driver.
Notice the up-to-date nvm and driver combination (latest as of this writing).
22.05-RELEASE (amd64) built on Wed Jun 22 18:56:13 UTC 2022 FreeBSD 12.3-STABLE
sysctl -a | grep dev.ixl.0
dev.ixl.0.fw_version: fw 9.101.71489 api 1.15 nvm 9.00 etid 8000cec3 oem 0.0.0 dev.ixl.0.current_speed: 10 Gbps dev.ixl.0.supported_speeds: 6 dev.ixl.0.advertise_speed: 6 dev.ixl.0.fc: 3 dev.ixl.0.%parent: pci5 dev.ixl.0.%pnpinfo: vendor=0x8086 device=0x1572 subvendor=0x8086 subdevice=0x0000 class=0x020000 dev.ixl.0.%location: slot=0 function=0 dbsf=pci0:5:0:0 handle=\_SB_.PCI0.BR3A.H000 dev.ixl.0.%driver: ixl dev.ixl.0.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.12.35
dmesg | grep ixl0
ixl0: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.12.35> mem 0xf8000000-0xf87fffff,0xf9008000-0xf900ffff irq 16 at device 0.0 on pci5 ixl0: using 1024 tx descriptors and 1024 rx descriptors ixl0: fw 9.101.71489 api 1.15 nvm 9.00 etid 8000cec3 oem 0.0.0
-
@rle said in Intel X710 Issues:
I also had to replace the file completely in boot/kernel otherwise it would not load the newly compiled driver.
You could have also added:
if_ixl_name="/boot/modules/if_ixl.ko"
To make it load the new module.
Steve
-