Intel X710 Issues

[mloiterman]

I’ve recently changed from an Intel x540-t2 NIC to an Intel x710-t2 and changed my modem from an SB8200 to an MB8611 because I wanted to take advantage of Comcast’s upgraded download speed from 1Gbs to 1.2Gbs.

Now that I’ve made those changes, I’m having some issues with inconsistent download performance where before I was able to easily hit 940Mbs now, I'm averaging 500 - 700 Mbs.

Network performance within the LAN (either 10Gbs to 10Gbs or 1Gbs to 1Gbs) shows that there are no issues on the LAN side.

So, I’m trying to determine if this is an issue with Speedtest sites, the new modem, or the new NIC. Due to where and how the server is installed, switching back to the 540 card is a difficult, but not impossible task. So, I have saved that for a last resort.

I’m on pfSense 2.5 and the box boots fine. Here is some hopefully relevant diagnostic information:

ifconfig ixl0
ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: WAN1_COMCAST
	options=e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	ether xx:xx:xx:xx:xx:xx
	hwaddr xx:xx:xx:xx:xxx:xx
	inet6 xxxx::xxxx:xxxx:xxxx:xxxx%ixl0 prefixlen 64 scopeid 0x1
	inet6 xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx prefixlen 128
	inet xxx.xxx.xxx.xxx netmask 0xfffffc00 broadcast 255.255.255.255
	media: Ethernet autoselect (2500Base-T <full-duplex,rxpause>)
	status: active
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

netstat -I ixl0
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ixl0   1500 <Link#1>      xx:xx:xx:xx:xx:xx 66453192     0 4294966048 28719745     0     0
ixl0      - fe80::%ixl0/6 fe80::bc0c:edff:f   312244     -     -   312572     -     -
ixl0      - xxx.xxx.xxx.xxx/22  c-xx-xx-xx-xxx.hsd  2336062     -     -   312435     -     -
ixl0      - xxx:xxx:xxxx xxxx:xxx:xxxx:xx:   399298     -     -     1476     -     -

sysctl -a | grep ixl0
irq257: ixl0:aq:57 @cpu0(domain0): 3
irq258: ixl0:rxq0:59 @cpu1(domain0): 6050062
irq259: ixl0:rxq1:61 @cpu0(domain0): 6664740
irq260: ixl0:rxq2:63 @cpu3(domain0): 5706167
irq261: ixl0:rxq3:65 @cpu2(domain0): 7734602
irq262: ixl0:rxq4:67 @cpu5(domain0): 6781857
irq263: ixl0:rxq5:69 @cpu4(domain0): 7005564
irq264: ixl0:rxq6:71 @cpu7(domain0): 7371266
irq265: ixl0:rxq7:73 @cpu6(domain0): 6575920

dev.ixl.0.fw_version: fw 8.1.63299 api 1.12 nvm 8.10 etid 800093ea oem 1.267.0

sysctl -a | grep ixl\.0
dev.ixl.0.mac.xoff_recvd: 0
dev.ixl.0.mac.xoff_txd: 0
dev.ixl.0.mac.xon_recvd: 0
dev.ixl.0.mac.xon_txd: 0
dev.ixl.0.mac.tx_frames_big: 0
dev.ixl.0.mac.tx_frames_1024_1522: 6109929
dev.ixl.0.mac.tx_frames_512_1023: 1008041
dev.ixl.0.mac.tx_frames_256_511: 664973
dev.ixl.0.mac.tx_frames_128_255: 2485605
dev.ixl.0.mac.tx_frames_65_127: 17366263
dev.ixl.0.mac.tx_frames_64: 1134418
dev.ixl.0.mac.checksum_errors: 0
dev.ixl.0.mac.rx_jabber: 0
dev.ixl.0.mac.rx_oversized: 0
dev.ixl.0.mac.rx_fragmented: 0
dev.ixl.0.mac.rx_undersize: 0
dev.ixl.0.mac.rx_frames_big: 0
dev.ixl.0.mac.rx_frames_1024_1522: 48130916
dev.ixl.0.mac.rx_frames_512_1023: 1452552
dev.ixl.0.mac.rx_frames_256_511: 2019464
dev.ixl.0.mac.rx_frames_128_255: 3456185
dev.ixl.0.mac.rx_frames_65_127: 10139155
dev.ixl.0.mac.rx_frames_64: 1333971
dev.ixl.0.mac.rx_length_errors: 0
dev.ixl.0.mac.remote_faults: 0
dev.ixl.0.mac.local_faults: 1
dev.ixl.0.mac.illegal_bytes: 0
dev.ixl.0.mac.crc_errors: 0
dev.ixl.0.mac.bcast_pkts_txd: 3
dev.ixl.0.mac.mcast_pkts_txd: 5603
dev.ixl.0.mac.ucast_pkts_txd: 28763623
dev.ixl.0.mac.good_octets_txd: 11938925817
dev.ixl.0.mac.rx_discards: 0
dev.ixl.0.mac.bcast_pkts_rcvd: 618
dev.ixl.0.mac.mcast_pkts_rcvd: 59715
dev.ixl.0.mac.ucast_pkts_rcvd: 66471910
dev.ixl.0.mac.good_octets_rcvd: 74532507761
dev.ixl.0.pf.txq07.itr: 122
dev.ixl.0.pf.txq07.bytes: 1182878011
dev.ixl.0.pf.txq07.packets: 3151710
dev.ixl.0.pf.txq07.mss_too_small: 0
dev.ixl.0.pf.txq07.tso: 0
dev.ixl.0.pf.txq06.itr: 122
dev.ixl.0.pf.txq06.bytes: 3609386915
dev.ixl.0.pf.txq06.packets: 5592768
dev.ixl.0.pf.txq06.mss_too_small: 0
dev.ixl.0.pf.txq06.tso: 0
dev.ixl.0.pf.txq05.itr: 122
dev.ixl.0.pf.txq05.bytes: 1233392624
dev.ixl.0.pf.txq05.packets: 2926836
dev.ixl.0.pf.txq05.mss_too_small: 0
dev.ixl.0.pf.txq05.tso: 0
dev.ixl.0.pf.txq04.itr: 122
dev.ixl.0.pf.txq04.bytes: 1410591375
dev.ixl.0.pf.txq04.packets: 3697406
dev.ixl.0.pf.txq04.mss_too_small: 0
dev.ixl.0.pf.txq04.tso: 0
dev.ixl.0.pf.txq03.itr: 122
dev.ixl.0.pf.txq03.bytes: 1000037614
dev.ixl.0.pf.txq03.packets: 3065998
dev.ixl.0.pf.txq03.mss_too_small: 0
dev.ixl.0.pf.txq03.tso: 0
dev.ixl.0.pf.txq02.itr: 122
dev.ixl.0.pf.txq02.bytes: 1080802445
dev.ixl.0.pf.txq02.packets: 3203676
dev.ixl.0.pf.txq02.mss_too_small: 0
dev.ixl.0.pf.txq02.tso: 0
dev.ixl.0.pf.txq01.itr: 122
dev.ixl.0.pf.txq01.bytes: 1151535920
dev.ixl.0.pf.txq01.packets: 3471789
dev.ixl.0.pf.txq01.mss_too_small: 0
dev.ixl.0.pf.txq01.tso: 0
dev.ixl.0.pf.txq00.itr: 122
dev.ixl.0.pf.txq00.bytes: 1144568137
dev.ixl.0.pf.txq00.packets: 3653700
dev.ixl.0.pf.txq00.mss_too_small: 0
dev.ixl.0.pf.txq00.tso: 0
dev.ixl.0.pf.rxq07.itr: 62
dev.ixl.0.pf.rxq07.desc_err: 0
dev.ixl.0.pf.rxq07.bytes: 10949318034
dev.ixl.0.pf.rxq07.packets: 9205753
dev.ixl.0.pf.rxq07.irqs: 6577452
dev.ixl.0.pf.rxq06.itr: 62
dev.ixl.0.pf.rxq06.desc_err: 0
dev.ixl.0.pf.rxq06.bytes: 7017993874
dev.ixl.0.pf.rxq06.packets: 6838505
dev.ixl.0.pf.rxq06.irqs: 7374314
dev.ixl.0.pf.rxq05.itr: 62
dev.ixl.0.pf.rxq05.desc_err: 0
dev.ixl.0.pf.rxq05.bytes: 9879807442
dev.ixl.0.pf.rxq05.packets: 9544935
dev.ixl.0.pf.rxq05.irqs: 7008611
dev.ixl.0.pf.rxq04.itr: 62
dev.ixl.0.pf.rxq04.desc_err: 0
dev.ixl.0.pf.rxq04.bytes: 10768414672
dev.ixl.0.pf.rxq04.packets: 9050156
dev.ixl.0.pf.rxq04.irqs: 6792564
dev.ixl.0.pf.rxq03.itr: 62
dev.ixl.0.pf.rxq03.desc_err: 0
dev.ixl.0.pf.rxq03.bytes: 10168392259
dev.ixl.0.pf.rxq03.packets: 9566705
dev.ixl.0.pf.rxq03.irqs: 7737964
dev.ixl.0.pf.rxq02.itr: 62
dev.ixl.0.pf.rxq02.desc_err: 0
dev.ixl.0.pf.rxq02.bytes: 7264365837
dev.ixl.0.pf.rxq02.packets: 6417728
dev.ixl.0.pf.rxq02.irqs: 5709226
dev.ixl.0.pf.rxq01.itr: 62
dev.ixl.0.pf.rxq01.desc_err: 0
dev.ixl.0.pf.rxq01.bytes: 10484107392
dev.ixl.0.pf.rxq01.packets: 8927330
dev.ixl.0.pf.rxq01.irqs: 6666136
dev.ixl.0.pf.rxq00.itr: 62
dev.ixl.0.pf.rxq00.desc_err: 0
dev.ixl.0.pf.rxq00.bytes: 7732703517
dev.ixl.0.pf.rxq00.packets: 6969477
dev.ixl.0.pf.rxq00.irqs: 6053252
dev.ixl.0.pf.bcast_pkts_txd: 3
dev.ixl.0.pf.mcast_pkts_txd: 226
dev.ixl.0.pf.ucast_pkts_txd: 28763623
dev.ixl.0.pf.good_octets_txd: 11813187746
dev.ixl.0.pf.rx_discards: 4294966048
dev.ixl.0.pf.bcast_pkts_rcvd: 618
dev.ixl.0.pf.mcast_pkts_rcvd: 48470
dev.ixl.0.pf.ucast_pkts_rcvd: 66471682
dev.ixl.0.pf.good_octets_rcvd: 74531467550
dev.ixl.0.admin_irq: 3
dev.ixl.0.eee.rx_lpi_count: 0
dev.ixl.0.eee.tx_lpi_count: 0
dev.ixl.0.eee.rx_lpi_status: 0
dev.ixl.0.eee.tx_lpi_status: 0
dev.ixl.0.eee.enable: 1
dev.ixl.0.fw_lldp: 1
dev.ixl.0.dynamic_tx_itr: 0
dev.ixl.0.dynamic_rx_itr: 0
dev.ixl.0.rx_itr: 62
dev.ixl.0.tx_itr: 122
dev.ixl.0.unallocated_queues: 760
dev.ixl.0.fw_version: fw 8.1.63299 api 1.12 nvm 8.10 etid 800093ea oem 1.267.0
dev.ixl.0.current_speed: 2.5 Gbps
dev.ixl.0.supported_speeds: 199
dev.ixl.0.advertise_speed: 199
dev.ixl.0.fc: 0
dev.ixl.0.iflib.rxq7.rxq_fl0.buf_size: 2048
dev.ixl.0.iflib.rxq7.rxq_fl0.credits: 1023
dev.ixl.0.iflib.rxq7.rxq_fl0.cidx: 1017
dev.ixl.0.iflib.rxq7.rxq_fl0.pidx: 1016
dev.ixl.0.iflib.rxq6.rxq_fl0.buf_size: 2048
dev.ixl.0.iflib.rxq6.rxq_fl0.credits: 1023
dev.ixl.0.iflib.rxq6.rxq_fl0.cidx: 233
dev.ixl.0.iflib.rxq6.rxq_fl0.pidx: 232
dev.ixl.0.iflib.rxq5.rxq_fl0.buf_size: 2048
dev.ixl.0.iflib.rxq5.rxq_fl0.credits: 1023
dev.ixl.0.iflib.rxq5.rxq_fl0.cidx: 231
dev.ixl.0.iflib.rxq5.rxq_fl0.pidx: 230
dev.ixl.0.iflib.rxq4.rxq_fl0.buf_size: 2048
dev.ixl.0.iflib.rxq4.rxq_fl0.credits: 1023
dev.ixl.0.iflib.rxq4.rxq_fl0.cidx: 44
dev.ixl.0.iflib.rxq4.rxq_fl0.pidx: 43
dev.ixl.0.iflib.rxq3.rxq_fl0.buf_size: 2048
dev.ixl.0.iflib.rxq3.rxq_fl0.credits: 1023
dev.ixl.0.iflib.rxq3.rxq_fl0.cidx: 497
dev.ixl.0.iflib.rxq3.rxq_fl0.pidx: 496
dev.ixl.0.iflib.rxq2.rxq_fl0.buf_size: 2048
dev.ixl.0.iflib.rxq2.rxq_fl0.credits: 1023
dev.ixl.0.iflib.rxq2.rxq_fl0.cidx: 320
dev.ixl.0.iflib.rxq2.rxq_fl0.pidx: 319
dev.ixl.0.iflib.rxq1.rxq_fl0.buf_size: 2048
dev.ixl.0.iflib.rxq1.rxq_fl0.credits: 1023
dev.ixl.0.iflib.rxq1.rxq_fl0.cidx: 98
dev.ixl.0.iflib.rxq1.rxq_fl0.pidx: 97
dev.ixl.0.iflib.rxq0.rxq_fl0.buf_size: 2048
dev.ixl.0.iflib.rxq0.rxq_fl0.credits: 1023
dev.ixl.0.iflib.rxq0.rxq_fl0.cidx: 133
dev.ixl.0.iflib.rxq0.rxq_fl0.pidx: 132
dev.ixl.0.iflib.txq7.r_abdications: 0
dev.ixl.0.iflib.txq7.r_restarts: 0
dev.ixl.0.iflib.txq7.r_stalls: 0
dev.ixl.0.iflib.txq7.r_starts: 3151699
dev.ixl.0.iflib.txq7.r_drops: 0
dev.ixl.0.iflib.txq7.r_enqueues: 3151710
dev.ixl.0.iflib.txq7.ring_state: pidx_head: 1886 pidx_tail: 1886 cidx: 1886 state: IDLE
dev.ixl.0.iflib.txq7.txq_cleaned: 3153559
dev.ixl.0.iflib.txq7.txq_processed: 3153567
dev.ixl.0.iflib.txq7.txq_in_use: 8
dev.ixl.0.iflib.txq7.txq_cidx_processed: 671
dev.ixl.0.iflib.txq7.txq_cidx: 663
dev.ixl.0.iflib.txq7.txq_pidx: 671
dev.ixl.0.iflib.txq7.no_tx_dma_setup: 0
dev.ixl.0.iflib.txq7.txd_encap_efbig: 0
dev.ixl.0.iflib.txq7.tx_map_failed: 0
dev.ixl.0.iflib.txq7.no_desc_avail: 0
dev.ixl.0.iflib.txq7.mbuf_defrag_failed: 0
dev.ixl.0.iflib.txq7.m_pullups: 0
dev.ixl.0.iflib.txq7.mbuf_defrag: 0
dev.ixl.0.iflib.txq6.r_abdications: 0
dev.ixl.0.iflib.txq6.r_restarts: 0
dev.ixl.0.iflib.txq6.r_stalls: 0
dev.ixl.0.iflib.txq6.r_starts: 5592303
dev.ixl.0.iflib.txq6.r_drops: 0
dev.ixl.0.iflib.txq6.r_enqueues: 5592775
dev.ixl.0.iflib.txq6.ring_state: pidx_head: 1735 pidx_tail: 1735 cidx: 1735 state: IDLE
dev.ixl.0.iflib.txq6.txq_cleaned: 5596751
dev.ixl.0.iflib.txq6.txq_processed: 5596759
dev.ixl.0.iflib.txq6.txq_in_use: 8
dev.ixl.0.iflib.txq6.txq_cidx_processed: 599
dev.ixl.0.iflib.txq6.txq_cidx: 591
dev.ixl.0.iflib.txq6.txq_pidx: 599
dev.ixl.0.iflib.txq6.no_tx_dma_setup: 0
dev.ixl.0.iflib.txq6.txd_encap_efbig: 0
dev.ixl.0.iflib.txq6.tx_map_failed: 0
dev.ixl.0.iflib.txq6.no_desc_avail: 0
dev.ixl.0.iflib.txq6.mbuf_defrag_failed: 0
dev.ixl.0.iflib.txq6.m_pullups: 0
dev.ixl.0.iflib.txq6.mbuf_defrag: 0
dev.ixl.0.iflib.txq5.r_abdications: 0
dev.ixl.0.iflib.txq5.r_restarts: 0
dev.ixl.0.iflib.txq5.r_stalls: 0
dev.ixl.0.iflib.txq5.r_starts: 2926826
dev.ixl.0.iflib.txq5.r_drops: 0
dev.ixl.0.iflib.txq5.r_enqueues: 2926837
dev.ixl.0.iflib.txq5.ring_state: pidx_head: 0245 pidx_tail: 0245 cidx: 0245 state: IDLE
dev.ixl.0.iflib.txq5.txq_cleaned: 2934125
dev.ixl.0.iflib.txq5.txq_processed: 2934133
dev.ixl.0.iflib.txq5.txq_in_use: 8
dev.ixl.0.iflib.txq5.txq_cidx_processed: 373
dev.ixl.0.iflib.txq5.txq_cidx: 365
dev.ixl.0.iflib.txq5.txq_pidx: 373
dev.ixl.0.iflib.txq5.no_tx_dma_setup: 0
dev.ixl.0.iflib.txq5.txd_encap_efbig: 0
dev.ixl.0.iflib.txq5.tx_map_failed: 0
dev.ixl.0.iflib.txq5.no_desc_avail: 0
dev.ixl.0.iflib.txq5.mbuf_defrag_failed: 0
dev.ixl.0.iflib.txq5.m_pullups: 0
dev.ixl.0.iflib.txq5.mbuf_defrag: 0
dev.ixl.0.iflib.txq4.r_abdications: 0
dev.ixl.0.iflib.txq4.r_restarts: 0
dev.ixl.0.iflib.txq4.r_stalls: 0
dev.ixl.0.iflib.txq4.r_starts: 3697378
dev.ixl.0.iflib.txq4.r_drops: 0
dev.ixl.0.iflib.txq4.r_enqueues: 3697407
dev.ixl.0.iflib.txq4.ring_state: pidx_head: 0767 pidx_tail: 0767 cidx: 0767 state: IDLE
dev.ixl.0.iflib.txq4.txq_cleaned: 3700282
dev.ixl.0.iflib.txq4.txq_processed: 3700290
dev.ixl.0.iflib.txq4.txq_in_use: 8
dev.ixl.0.iflib.txq4.txq_cidx_processed: 578
dev.ixl.0.iflib.txq4.txq_cidx: 570
dev.ixl.0.iflib.txq4.txq_pidx: 578
dev.ixl.0.iflib.txq4.no_tx_dma_setup: 0
dev.ixl.0.iflib.txq4.txd_encap_efbig: 0
dev.ixl.0.iflib.txq4.tx_map_failed: 0
dev.ixl.0.iflib.txq4.no_desc_avail: 0
dev.ixl.0.iflib.txq4.mbuf_defrag_failed: 0
dev.ixl.0.iflib.txq4.m_pullups: 0
dev.ixl.0.iflib.txq4.mbuf_defrag: 0
dev.ixl.0.iflib.txq3.r_abdications: 0
dev.ixl.0.iflib.txq3.r_restarts: 0
dev.ixl.0.iflib.txq3.r_stalls: 0
dev.ixl.0.iflib.txq3.r_starts: 3065983
dev.ixl.0.iflib.txq3.r_drops: 0
dev.ixl.0.iflib.txq3.r_enqueues: 3065998
dev.ixl.0.iflib.txq3.ring_state: pidx_head: 0142 pidx_tail: 0142 cidx: 0142 state: IDLE
dev.ixl.0.iflib.txq3.txq_cleaned: 3068787
dev.ixl.0.iflib.txq3.txq_processed: 3068795
dev.ixl.0.iflib.txq3.txq_in_use: 8
dev.ixl.0.iflib.txq3.txq_cidx_processed: 891
dev.ixl.0.iflib.txq3.txq_cidx: 883
dev.ixl.0.iflib.txq3.txq_pidx: 891
dev.ixl.0.iflib.txq3.no_tx_dma_setup: 0
dev.ixl.0.iflib.txq3.txd_encap_efbig: 0
dev.ixl.0.iflib.txq3.tx_map_failed: 0
dev.ixl.0.iflib.txq3.no_desc_avail: 0
dev.ixl.0.iflib.txq3.mbuf_defrag_failed: 0
dev.ixl.0.iflib.txq3.m_pullups: 0
dev.ixl.0.iflib.txq3.mbuf_defrag: 0
dev.ixl.0.iflib.txq2.r_abdications: 0
dev.ixl.0.iflib.txq2.r_restarts: 0
dev.ixl.0.iflib.txq2.r_stalls: 0
dev.ixl.0.iflib.txq2.r_starts: 3203666
dev.ixl.0.iflib.txq2.r_drops: 0
dev.ixl.0.iflib.txq2.r_enqueues: 3203676
dev.ixl.0.iflib.txq2.ring_state: pidx_head: 0604 pidx_tail: 0604 cidx: 0604 state: IDLE
dev.ixl.0.iflib.txq2.txq_cleaned: 3207578
dev.ixl.0.iflib.txq2.txq_processed: 3207586
dev.ixl.0.iflib.txq2.txq_in_use: 8
dev.ixl.0.iflib.txq2.txq_cidx_processed: 418
dev.ixl.0.iflib.txq2.txq_cidx: 410
dev.ixl.0.iflib.txq2.txq_pidx: 418
dev.ixl.0.iflib.txq2.no_tx_dma_setup: 0
dev.ixl.0.iflib.txq2.txd_encap_efbig: 0
dev.ixl.0.iflib.txq2.tx_map_failed: 0
dev.ixl.0.iflib.txq2.no_desc_avail: 0
dev.ixl.0.iflib.txq2.mbuf_defrag_failed: 0
dev.ixl.0.iflib.txq2.m_pullups: 0
dev.ixl.0.iflib.txq2.mbuf_defrag: 0
dev.ixl.0.iflib.txq1.r_abdications: 0
dev.ixl.0.iflib.txq1.r_restarts: 0
dev.ixl.0.iflib.txq1.r_stalls: 0
dev.ixl.0.iflib.txq1.r_starts: 3471783
dev.ixl.0.iflib.txq1.r_drops: 0
dev.ixl.0.iflib.txq1.r_enqueues: 3471789
dev.ixl.0.iflib.txq1.ring_state: pidx_head: 0429 pidx_tail: 0429 cidx: 0429 state: IDLE
dev.ixl.0.iflib.txq1.txq_cleaned: 3476015
dev.ixl.0.iflib.txq1.txq_processed: 3476023
dev.ixl.0.iflib.txq1.txq_in_use: 8
dev.ixl.0.iflib.txq1.txq_cidx_processed: 567
dev.ixl.0.iflib.txq1.txq_cidx: 559
dev.ixl.0.iflib.txq1.txq_pidx: 567
dev.ixl.0.iflib.txq1.no_tx_dma_setup: 0
dev.ixl.0.iflib.txq1.txd_encap_efbig: 0
dev.ixl.0.iflib.txq1.tx_map_failed: 0
dev.ixl.0.iflib.txq1.no_desc_avail: 0
dev.ixl.0.iflib.txq1.mbuf_defrag_failed: 0
dev.ixl.0.iflib.txq1.m_pullups: 0
dev.ixl.0.iflib.txq1.mbuf_defrag: 0
dev.ixl.0.iflib.txq0.r_abdications: 0
dev.ixl.0.iflib.txq0.r_restarts: 0
dev.ixl.0.iflib.txq0.r_stalls: 0
dev.ixl.0.iflib.txq0.r_starts: 3626345
dev.ixl.0.iflib.txq0.r_drops: 0
dev.ixl.0.iflib.txq0.r_enqueues: 3653702
dev.ixl.0.iflib.txq0.ring_state: pidx_head: 0070 pidx_tail: 0070 cidx: 0070 state: IDLE
dev.ixl.0.iflib.txq0.txq_cleaned: 3664449
dev.ixl.0.iflib.txq0.txq_processed: 3664457
dev.ixl.0.iflib.txq0.txq_in_use: 8
dev.ixl.0.iflib.txq0.txq_cidx_processed: 585
dev.ixl.0.iflib.txq0.txq_cidx: 577
dev.ixl.0.iflib.txq0.txq_pidx: 585
dev.ixl.0.iflib.txq0.no_tx_dma_setup: 0
dev.ixl.0.iflib.txq0.txd_encap_efbig: 0
dev.ixl.0.iflib.txq0.tx_map_failed: 0
dev.ixl.0.iflib.txq0.no_desc_avail: 0
dev.ixl.0.iflib.txq0.mbuf_defrag_failed: 0
dev.ixl.0.iflib.txq0.m_pullups: 19463
dev.ixl.0.iflib.txq0.mbuf_defrag: 0
dev.ixl.0.iflib.override_nrxds: 0
dev.ixl.0.iflib.override_ntxds: 0
dev.ixl.0.iflib.separate_txrx: 0
dev.ixl.0.iflib.core_offset: 0
dev.ixl.0.iflib.tx_abdicate: 0
dev.ixl.0.iflib.rx_budget: 0
dev.ixl.0.iflib.disable_msix: 0
dev.ixl.0.iflib.override_qs_enable: 0
dev.ixl.0.iflib.override_nrxqs: 0
dev.ixl.0.iflib.override_ntxqs: 0
dev.ixl.0.iflib.driver_version: 2.3.0-k
dev.ixl.0.%parent: pci4
dev.ixl.0.%pnpinfo: vendor=0x8086 device=0x15ff subvendor=0x8086 subdevice=0x0003 class=0x020000
dev.ixl.0.%location: slot=0 function=0 dbsf=pci0:4:0:0
dev.ixl.0.%driver: ixl
dev.ixl.0.%desc: Intel(R) Ethernet Controller X710 for 10GBASE-T - 2.3.0-k

The thing I’m looking most closely at is the rxpause. I don’t know where that came from and what’s causing it or even if it’s contributing.

Any thoughts?

[NollipfSense]

@mloiterman Are you testing the speed through the firewall from a client?

[Cool_Corona]

X540-T2 can easily push what you need. The X710 have driver issues. We see that in the DC's where we have the T4's as our primary datahauler in the 1U servers.

[mloiterman]

Yes, I’m testing from a LAN side client.

The issue with the 540 is that it doesn’t do the NBASE-T so it cannot connect at 2.5, only 1 and 10. And no modem supports 10.

I also think there are driver issues at play. I have another modem, the Arris S33, that I’m going to try today to see if that makes a difference. After that, I guess I’ll revert back to the 540?

There is also the 550 which does support NBASE-T, but that card seemed to have issues with linking at 2.5/5 on FreeBSD. But maybe that’s been addressed in pfSense 2.5?

[kiokoman]

@mloiterman
try to disable RXCSUM,TXCSUM

[Cool_Corona]

@mloiterman

VmWare recommends disabling LRO and TSO both IPv4 and IPv6 and then a reboot.

I still kept getting hit by the PSOD and that stopped when I disabled jumbo frames on the attached adapters.

Got a performance hit, but better than PSOD.

[mloiterman]

@cool_corona

I still kept getting hit by the PSOD and that stopped when I disabled jumbo frames on the attached adapters.

What is PSOD and how did you disable jump frames?

So far this morning, I have tried the following with no success - they all have no impact on the reduced download speeds:

• In System Tuneables I have manually added the following, but it was overwritten on reboot so maybe I'm not doing something correctly. It's for the ix driver, so it's probably not relevant anyway:

hw.ix.flow_control: 0

• In loader.conf, I added via Diagnostics -> Edit File. Again, this was overwritten on reboot as confirmed from checking in systctl and it shows "0" for both which is presumably the correct setting, but I wanted to try the reverse:

dev.ixl.1.fc="3"
dev.ixl.0.fc="3"

• In Advanced -> Networking: Disable hardware checksum offload
• In Advanced -> Networking: Enabled Hardware TCP Segmentation Offloading
• In Advanced -> Networking: Enabled Hardware Large Receive Offloading
• In Advanced -> Networking: Enabled Hardware Large Receive Offloading
• In Advanced -> Networking: Disabled the ALTQ support for hn NICs

Nothing works.

How do I turn off the rxpause? Shouldn't that be from dev.ixl.1.fc? It's at 0 now, but yet the interface shows it enabled.

[mloiterman]

@mloiterman

I've now upgrade the firmware on the X710 and replaced the modem with an Arris S33.

dev.ixl.0.fw_version: fw 8.2.64244 api 1.13 nvm 8.20 etid 8000a062 oem 1.267.0

Since upgrading the firmware, now the interface shows:

media: Ethernet autoselect (2500Base-T <full-duplex>)

but on speed tests the best I'm able to do is about 750 or 800.

So, something is still going on.

dev.ixl.0.iflib.driver_version: 2.3.0-k

But really strange and inconsistent download performance:

speedtest-cli --server 1776
Retrieving speedtest.net configuration...
Testing from Comcast Cable (xxxxxxxxx)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by Comcast (Chicago, IL) [38.69 km]: 44.566 ms
Testing download speed................................................................................
Download: 1140.00 Mbit/s
Testing upload speed......................................................................................................
Upload: 42.13 Mbit/s
[mike@arch ~]$ speedtest-cli --server 1776
Retrieving speedtest.net configuration...
Testing from Comcast Cable (xxxxxxxxx)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by Comcast (Chicago, IL) [38.69 km]: 13.061 ms
Testing download speed................................................................................
Download: 701.39 Mbit/s
Testing upload speed.....................................................................................................
.Upload: 35.68 Mbit/s

How can I add and use the newest intel driver?

[NollipfSense]

@mloiterman said in Intel X710 Issues:

How can I add and use the newest intel driver?

It should already be in V2.5 ... I believe!

[mloiterman]

@nollipfsense

Looks like the intel driver is at 3.3.22...?

[stephenw10]

Definitely try disabling flow control. Those should be added as sysctls/system tunables for ixl:
https://www.freebsd.org/cgi/man.cgi?query=ixl#SYSCTL_PROCEDURES

Steve

[mloiterman]

@stephenw10

Appreciate the response.

But it seems like it's already disabled:

sysctl -a | grep dev.ixl.0.fc
dev.ixl.0.fc: 0

Is there another place where I need to check?

[mloiterman]

dmesg | grep ^ixl
ixl0: Link is up, 2.5 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control:
ixl0: link state changed to UP
ixl1: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: N
ixl1: link state changed to UP

So it seems like despite:

sysctl -a | grep dev.ixl.0.fc
dev.ixl.0.fc: 0

Flow Control is N for ixl1 but empty for ixl0.

This is so frustrating - it makes no sense.

[stephenw10]

ifconfig still shows rxpause, txpause?

[mloiterman]

@stephenw10

No not after upgrading the firmware on the card.

I tried enabling flow control by setting it to sequential setting it to 0,1,2,3 through the tunables page. I could get it to turn on txpause but not rxpause which is wierd because initially, before upgrading firmware, I could not get rxpause to turn off.

I did not reboot or reboot the modem inbetween changing those settings though.

[stephenw10]

Flow control is negotiated so you won't see it in the ifconfig output if the other side doesn't support it. You can force it on or off though on some NICs.

Steve

[mloiterman]

@stephenw10

OK, I got it figured out.

I had to enable flow control on my switch for the port connected to pfSense and had to enable full flow control for the X710 port connected to my switch with these systctl tunables in System/Advanced/System Tunables:

dev.ixl.1.fc
3
Enable flow control

Now I can get ~ 945Mbs for the machines connected with 1G NICs and ~1,200Mbs for the 10G NICs.

[stephenw10]

Ah, nice result. Interesting. I guess you are hitting something that requires flow-control to mitigate which is unusual.

Steve

[lnguyen]

@mloiterman

Did you update the drivers after updating the NVM firmware? I just updated to 8.30 FW, but following your steps, I have been unable to acheive the same results.

[mloiterman]

@lnguyen

Yes, I compiled the latest version of intel's driver against the kernel in 2.5.1.

I can help you with that if you need it, but it will have to wait until later this week.