Intel X710 Issues

lnguyen

Did you update the drivers after updating the NVM firmware? I just updated to 8.30 FW, but following your steps, I have been unable to acheive the same results.

mloiterman

@lnguyen

Yes, I compiled the latest version of intel's driver against the kernel in 2.5.1.

I can help you with that if you need it, but it will have to wait until later this week.

froboz

@mloiterman I would be interested in the compile instructions as well as I have the same issue you have although I did manage a work around with the old firmware 8.10. I updated the firmware to 8.30 :-( . I could reverse the firmware update but I figured I would move forward through this problem :-) . Something is seriously wrong with this card and PFsense at the moment and I hope netgate finds the time to fix it. Thanks

mloiterman

I’ve updated to the 8.30 firmware and the 1.12.16 version of the driver and it still seems like I need to enable Flow Control on the LAN side port of the X710 and the switch port it’s connected to.

For anyone that is interested here’s the basic steps to build you’re own version of the Intel driver.

Get yourself a FreeBSD system where you can download source files and compile the driver. I would suggest NOT using your pfSense system to do this kind of work.
Once you’ve got your FreeBSD system setup, do a deep clone of the source repository. For 12.2-RELEASE:

# git clone -b releng/12.2 https://git.freebsd.org/src.git /usr/src

Determine which version of pfSense your kernel was compiled against from here. For pfSense CE 2.5.1, for example, it’s 12.2-STABLE@f4d0bc6aa6b.
On the system where you’ve cloned the source tree in step 2, check out the specific version of the source tree you’ve determined in step 3 by using the hash value at the end of the version string:

# cd /usr/src
# git checkout f4d0bc6aa6b

Download the driver source files from Intel. For 1.12.16, they can be found here.
Untar the archive:

# tar zxf ixl-1.12.16.tar.gz

This will create the ixl-1.12.16 directory.

To load the driver onto a running system:

# cd ixl-1.12.16/src
# make
# kldload ./if_ixl.ko

lnguyen

@mloiterman

Did you copy this to /boot/kernel to make it persistant?

mloiterman

@lnguyen

No, you place the kernel module in /boot/modules.

Make sure the module is set to root:wheel and the permissions are 555:

# chown root:wheel if_ixl.ko && chmod 555 if_ixl.ko

and then in /boot/loader.conf.local, which may not exist unless you create it, you'll need to add:

if_ixl_load="YES"

lnguyen

@mloiterman

Thanks. Running into some issues in my scenario.

IPv6 addresses for WAN and LAN are no longer populated
1Gbps hosts are still not reaching 942Mbps
WAN Interface showing unknown speed
The interface status for ixl1 does not reflect rxpause, txpause (driver loaded confirmed)

ChuckLasher

@mloiterman

@lnguyen and I are working together on this project.
I have the x710-T2L,

I compiled the 1.12.16 drivers in my FreeBSD 12.2 VM and got us to this point.

I have flow control which works but the speed isn't showing. (dev.ixl.1.fc=3)
I notice that it takes a bit for the IPv6 addresses to populate on startup. This is new behavior.

What are we missing here?

mloiterman

@chucklasher

I cannot say what might be causing your issues, but I can tell you that I enabled Flow Control on the LAN switch ports and enabled full Flow Control on the LAN port on the X710 card. I had Flow Control disabled on the WAN port of the X710 card.

My understanding is the Flow Control is negotiated and must be active on both sides of the connection.

This whole thing is still a bit of a mystery to me and it’s not really clear why it’s needed. I suspect it may be partially rooted in the fact that my switches, HP 2920 48 port with PoE+ are 1Gbs switches with a 10Gbs add in cards. NBASE-T is a newer standard and there maybe bugs or issues like this that are triggered when you have a mixed environment. That’s only speculation from me, however.

Please let me know what you find because I would like to eventually disable Flow Control.

ChuckLasher

@mloiterman

Messing around last night yielded results.

In the GUI, added System Tunable dev.ixl.1.fc=3
Reverted back to the shipped driver (iflib)
Enabled Flow Control on the Netgear switch (XS724EM)
Restarted

Screenshot from 2021-05-25 11-16-25.png

Not entirely sure why this works but there is significance when CE 2.5.1 adds the tunable vs sysctl

In testing, ATV 4K will obtain (has to work up to it) 940 Mbps.

Speeds do not "snap" up to their values. Speedtest seems to 'slew' its way up to the final value.

I suspect there is a better way to handle this but is well above my understanding level. Perhaps priority flow control ?

lnguyen

@mloiterman

IPv6 addresses did eventually populate. I have a Cisco switch that is not negotiating FC with the Intel X710-T4L ports. The default kernel drivers report the correct speed negotiated from ifconfig

lnguyen

@stephenw10

I think this scenario is due to WAN speed >1Gbps overflowing 1Gbps connected LAN hosts.

mloiterman

@chucklasher

Two other things:

I noticed performance was subjectively better when flow control for the X710 lan port to “1” which is rxpause. 3 worked too, but “seemed” less performing. Although, I have no objective data to see if that’s really true or not.

Performance seemed better when I Disabled hardware checksum offload.

ChuckLasher

@mloiterman

Thanks,

Made the change.
I'll play around with the offload. I will see what impact that has on the 10 GbE machines.

lnguyen

2 year old thread. Same issue but with WAN 2x1Gbps LAGG instead of 2.5Gbps
Low throughput over lagg with 1gb clients

lnguyen

I think using ethernet flow control to solve this issue is not an ideal workaround. I think looking at TCP flow control or packet buffer sizes in pfSense is the long term resolution here. Just not sure what to tune in pfSense to accommodate WAN connections greater than 1Gbps LAN connections.

chiapuddingmaster

Is there any update to this issue? I'm using the x710-t2l with FC=1 on lan and a netgear MS510TX multigig switch with global flow control since the switch doesnt offer individual port controls. I would like to turn off FC if possible.

rle

@mloiterman Followed your procedure, but I also had to replace the file completely in boot/kernel otherwise it would not load the newly compiled driver.

Notice the up-to-date nvm and driver combination (latest as of this writing).

22.05-RELEASE (amd64)
built on Wed Jun 22 18:56:13 UTC 2022
FreeBSD 12.3-STABLE

sysctl -a | grep dev.ixl.0

dev.ixl.0.fw_version: fw 9.101.71489 api 1.15 nvm 9.00 etid 8000cec3 oem 0.0.0
dev.ixl.0.current_speed: 10 Gbps
dev.ixl.0.supported_speeds: 6
dev.ixl.0.advertise_speed: 6
dev.ixl.0.fc: 3
dev.ixl.0.%parent: pci5
dev.ixl.0.%pnpinfo: vendor=0x8086 device=0x1572 subvendor=0x8086 subdevice=0x0000 class=0x020000
dev.ixl.0.%location: slot=0 function=0 dbsf=pci0:5:0:0 handle=\_SB_.PCI0.BR3A.H000
dev.ixl.0.%driver: ixl
dev.ixl.0.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.12.35

dmesg | grep ixl0

ixl0: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.12.35> mem 0xf8000000-0xf87fffff,0xf9008000-0xf900ffff irq 16 at device 0.0 on pci5
ixl0: using 1024 tx descriptors and 1024 rx descriptors
ixl0: fw 9.101.71489 api 1.15 nvm 9.00 etid 8000cec3 oem 0.0.0

stephenw10

@rle said in Intel X710 Issues:

I also had to replace the file completely in boot/kernel otherwise it would not load the newly compiled driver.

You could have also added:

if_ixl_name="/boot/modules/if_ixl.ko"

To make it load the new module.

Steve