Intel X710 Issues
-
X540-T2 can easily push what you need. The X710 have driver issues. We see that in the DC's where we have the T4's as our primary datahauler in the 1U servers.
-
Yes, I’m testing from a LAN side client.
The issue with the 540 is that it doesn’t do the NBASE-T so it cannot connect at 2.5, only 1 and 10. And no modem supports 10.
I also think there are driver issues at play. I have another modem, the Arris S33, that I’m going to try today to see if that makes a difference. After that, I guess I’ll revert back to the 540?
There is also the 550 which does support NBASE-T, but that card seemed to have issues with linking at 2.5/5 on FreeBSD. But maybe that’s been addressed in pfSense 2.5?
-
@mloiterman
try to disable RXCSUM,TXCSUM -
VmWare recommends disabling LRO and TSO both IPv4 and IPv6 and then a reboot.
I still kept getting hit by the PSOD and that stopped when I disabled jumbo frames on the attached adapters.
Got a performance hit, but better than PSOD.
-
I still kept getting hit by the PSOD and that stopped when I disabled jumbo frames on the attached adapters.
What is PSOD and how did you disable jump frames?
So far this morning, I have tried the following with no success - they all have no impact on the reduced download speeds:
- In System Tuneables I have manually added the following, but it was overwritten on reboot so maybe I'm not doing something correctly. It's for the ix driver, so it's probably not relevant anyway:
hw.ix.flow_control: 0- In loader.conf, I added via Diagnostics -> Edit File. Again, this was overwritten on reboot as confirmed from checking in systctl and it shows "0" for both which is presumably the correct setting, but I wanted to try the reverse:
dev.ixl.1.fc="3" dev.ixl.0.fc="3"- In Advanced -> Networking: Disable hardware checksum offload
- In Advanced -> Networking: Enabled Hardware TCP Segmentation Offloading
- In Advanced -> Networking: Enabled Hardware Large Receive Offloading
- In Advanced -> Networking: Enabled Hardware Large Receive Offloading
- In Advanced -> Networking: Disabled the ALTQ support for hn NICs
Nothing works.
How do I turn off the rxpause? Shouldn't that be from dev.ixl.1.fc? It's at 0 now, but yet the interface shows it enabled.
-
I've now upgrade the firmware on the X710 and replaced the modem with an Arris S33.
dev.ixl.0.fw_version: fw 8.2.64244 api 1.13 nvm 8.20 etid 8000a062 oem 1.267.0Since upgrading the firmware, now the interface shows:
media: Ethernet autoselect (2500Base-T <full-duplex>)but on speed tests the best I'm able to do is about 750 or 800.
So, something is still going on.
dev.ixl.0.iflib.driver_version: 2.3.0-kBut really strange and inconsistent download performance:
speedtest-cli --server 1776 Retrieving speedtest.net configuration... Testing from Comcast Cable (xxxxxxxxx)... Retrieving speedtest.net server list... Retrieving information for the selected server... Hosted by Comcast (Chicago, IL) [38.69 km]: 44.566 ms Testing download speed................................................................................ Download: 1140.00 Mbit/s Testing upload speed...................................................................................................... Upload: 42.13 Mbit/s [mike@arch ~]$ speedtest-cli --server 1776 Retrieving speedtest.net configuration... Testing from Comcast Cable (xxxxxxxxx)... Retrieving speedtest.net server list... Retrieving information for the selected server... Hosted by Comcast (Chicago, IL) [38.69 km]: 13.061 ms Testing download speed................................................................................ Download: 701.39 Mbit/s Testing upload speed..................................................................................................... .Upload: 35.68 Mbit/sHow can I add and use the newest intel driver?
-
@mloiterman said in Intel X710 Issues:
How can I add and use the newest intel driver?
It should already be in V2.5 ... I believe!
-
Looks like the intel driver is at 3.3.22...?
-
Definitely try disabling flow control. Those should be added as sysctls/system tunables for ixl:
https://www.freebsd.org/cgi/man.cgi?query=ixl#SYSCTL_PROCEDURESSteve
-
Appreciate the response.
But it seems like it's already disabled:
sysctl -a | grep dev.ixl.0.fc dev.ixl.0.fc: 0Is there another place where I need to check?
-
dmesg | grep ^ixl ixl0: Link is up, 2.5 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: ixl0: link state changed to UP ixl1: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: N ixl1: link state changed to UPSo it seems like despite:
sysctl -a | grep dev.ixl.0.fc dev.ixl.0.fc: 0Flow Control is N for ixl1 but empty for ixl0.
This is so frustrating - it makes no sense.
-
ifconfig still shows
rxpause, txpause? -
No not after upgrading the firmware on the card.
I tried enabling flow control by setting it to sequential setting it to 0,1,2,3 through the tunables page. I could get it to turn on txpause but not rxpause which is wierd because initially, before upgrading firmware, I could not get rxpause to turn off.
I did not reboot or reboot the modem inbetween changing those settings though.
-
Flow control is negotiated so you won't see it in the ifconfig output if the other side doesn't support it. You can force it on or off though on some NICs.
Steve
-
OK, I got it figured out.
I had to enable flow control on my switch for the port connected to pfSense and had to enable full flow control for the X710 port connected to my switch with these systctl tunables in System/Advanced/System Tunables:
dev.ixl.1.fc
3
Enable flow controlNow I can get ~ 945Mbs for the machines connected with 1G NICs and ~1,200Mbs for the 10G NICs.
-
Ah, nice result. Interesting. I guess you are hitting something that requires flow-control to mitigate which is unusual.
Steve
-
Did you update the drivers after updating the NVM firmware? I just updated to 8.30 FW, but following your steps, I have been unable to acheive the same results.
-
Yes, I compiled the latest version of intel's driver against the kernel in 2.5.1.
I can help you with that if you need it, but it will have to wait until later this week.
-
@mloiterman I would be interested in the compile instructions as well as I have the same issue you have although I did manage a work around with the old firmware 8.10. I updated the firmware to 8.30 :-( . I could reverse the firmware update but I figured I would move forward through this problem :-) . Something is seriously wrong with this card and PFsense at the moment and I hope netgate finds the time to fix it. Thanks
-
I’ve updated to the 8.30 firmware and the 1.12.16 version of the driver and it still seems like I need to enable Flow Control on the LAN side port of the X710 and the switch port it’s connected to.
For anyone that is interested here’s the basic steps to build you’re own version of the Intel driver.
- Get yourself a FreeBSD system where you can download source files and compile the driver. I would suggest NOT using your pfSense system to do this kind of work.
- Once you’ve got your FreeBSD system setup, do a deep clone of the source repository. For 12.2-RELEASE:
# git clone -b releng/12.2 https://git.freebsd.org/src.git /usr/src- Determine which version of pfSense your kernel was compiled against from here. For pfSense CE 2.5.1, for example, it’s 12.2-STABLE@f4d0bc6aa6b.
- On the system where you’ve cloned the source tree in step 2, check out the specific version of the source tree you’ve determined in step 3 by using the hash value at the end of the version string:
# cd /usr/src # git checkout f4d0bc6aa6b-
Download the driver source files from Intel. For 1.12.16, they can be found here.
-
Untar the archive:
# tar zxf ixl-1.12.16.tar.gzThis will create the ixl-1.12.16 directory.
- To load the driver onto a running system:
# cd ixl-1.12.16/src # make # kldload ./if_ixl.ko
