Intel X710 Issues

mloiterman

dmesg | grep ^ixl
ixl0: Link is up, 2.5 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control:
ixl0: link state changed to UP
ixl1: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: N
ixl1: link state changed to UP

So it seems like despite:

sysctl -a | grep dev.ixl.0.fc
dev.ixl.0.fc: 0

Flow Control is N for ixl1 but empty for ixl0.

This is so frustrating - it makes no sense.

stephenw10

ifconfig still shows rxpause, txpause?

mloiterman

@stephenw10

No not after upgrading the firmware on the card.

I tried enabling flow control by setting it to sequential setting it to 0,1,2,3 through the tunables page. I could get it to turn on txpause but not rxpause which is wierd because initially, before upgrading firmware, I could not get rxpause to turn off.

I did not reboot or reboot the modem inbetween changing those settings though.

stephenw10

Flow control is negotiated so you won't see it in the ifconfig output if the other side doesn't support it. You can force it on or off though on some NICs.

Steve

mloiterman

@stephenw10

OK, I got it figured out.

I had to enable flow control on my switch for the port connected to pfSense and had to enable full flow control for the X710 port connected to my switch with these systctl tunables in System/Advanced/System Tunables:

dev.ixl.1.fc
3
Enable flow control

Now I can get ~ 945Mbs for the machines connected with 1G NICs and ~1,200Mbs for the 10G NICs.

stephenw10

Ah, nice result. Interesting. I guess you are hitting something that requires flow-control to mitigate which is unusual.

Steve

lnguyen

@mloiterman

Did you update the drivers after updating the NVM firmware? I just updated to 8.30 FW, but following your steps, I have been unable to acheive the same results.

mloiterman

@lnguyen

Yes, I compiled the latest version of intel's driver against the kernel in 2.5.1.

I can help you with that if you need it, but it will have to wait until later this week.

froboz

@mloiterman I would be interested in the compile instructions as well as I have the same issue you have although I did manage a work around with the old firmware 8.10. I updated the firmware to 8.30 :-( . I could reverse the firmware update but I figured I would move forward through this problem :-) . Something is seriously wrong with this card and PFsense at the moment and I hope netgate finds the time to fix it. Thanks

mloiterman

I’ve updated to the 8.30 firmware and the 1.12.16 version of the driver and it still seems like I need to enable Flow Control on the LAN side port of the X710 and the switch port it’s connected to.

For anyone that is interested here’s the basic steps to build you’re own version of the Intel driver.

Get yourself a FreeBSD system where you can download source files and compile the driver. I would suggest NOT using your pfSense system to do this kind of work.
Once you’ve got your FreeBSD system setup, do a deep clone of the source repository. For 12.2-RELEASE:

# git clone -b releng/12.2 https://git.freebsd.org/src.git /usr/src

Determine which version of pfSense your kernel was compiled against from here. For pfSense CE 2.5.1, for example, it’s 12.2-STABLE@f4d0bc6aa6b.
On the system where you’ve cloned the source tree in step 2, check out the specific version of the source tree you’ve determined in step 3 by using the hash value at the end of the version string:

# cd /usr/src
# git checkout f4d0bc6aa6b

Download the driver source files from Intel. For 1.12.16, they can be found here.
Untar the archive:

# tar zxf ixl-1.12.16.tar.gz

This will create the ixl-1.12.16 directory.

To load the driver onto a running system:

# cd ixl-1.12.16/src
# make
# kldload ./if_ixl.ko

lnguyen

@mloiterman

Did you copy this to /boot/kernel to make it persistant?

mloiterman

@lnguyen

No, you place the kernel module in /boot/modules.

Make sure the module is set to root:wheel and the permissions are 555:

# chown root:wheel if_ixl.ko && chmod 555 if_ixl.ko

and then in /boot/loader.conf.local, which may not exist unless you create it, you'll need to add:

if_ixl_load="YES"

lnguyen

@mloiterman

Thanks. Running into some issues in my scenario.

IPv6 addresses for WAN and LAN are no longer populated
1Gbps hosts are still not reaching 942Mbps
WAN Interface showing unknown speed
The interface status for ixl1 does not reflect rxpause, txpause (driver loaded confirmed)

ChuckLasher

@mloiterman

@lnguyen and I are working together on this project.
I have the x710-T2L,

I compiled the 1.12.16 drivers in my FreeBSD 12.2 VM and got us to this point.

I have flow control which works but the speed isn't showing. (dev.ixl.1.fc=3)
I notice that it takes a bit for the IPv6 addresses to populate on startup. This is new behavior.

What are we missing here?

mloiterman

@chucklasher

I cannot say what might be causing your issues, but I can tell you that I enabled Flow Control on the LAN switch ports and enabled full Flow Control on the LAN port on the X710 card. I had Flow Control disabled on the WAN port of the X710 card.

My understanding is the Flow Control is negotiated and must be active on both sides of the connection.

This whole thing is still a bit of a mystery to me and it’s not really clear why it’s needed. I suspect it may be partially rooted in the fact that my switches, HP 2920 48 port with PoE+ are 1Gbs switches with a 10Gbs add in cards. NBASE-T is a newer standard and there maybe bugs or issues like this that are triggered when you have a mixed environment. That’s only speculation from me, however.

Please let me know what you find because I would like to eventually disable Flow Control.

ChuckLasher

@mloiterman

Messing around last night yielded results.

In the GUI, added System Tunable dev.ixl.1.fc=3
Reverted back to the shipped driver (iflib)
Enabled Flow Control on the Netgear switch (XS724EM)
Restarted

Screenshot from 2021-05-25 11-16-25.png

Not entirely sure why this works but there is significance when CE 2.5.1 adds the tunable vs sysctl

In testing, ATV 4K will obtain (has to work up to it) 940 Mbps.

Speeds do not "snap" up to their values. Speedtest seems to 'slew' its way up to the final value.

I suspect there is a better way to handle this but is well above my understanding level. Perhaps priority flow control ?

lnguyen

@mloiterman

IPv6 addresses did eventually populate. I have a Cisco switch that is not negotiating FC with the Intel X710-T4L ports. The default kernel drivers report the correct speed negotiated from ifconfig

lnguyen

@stephenw10

I think this scenario is due to WAN speed >1Gbps overflowing 1Gbps connected LAN hosts.

mloiterman

@chucklasher

Two other things:

I noticed performance was subjectively better when flow control for the X710 lan port to “1” which is rxpause. 3 worked too, but “seemed” less performing. Although, I have no objective data to see if that’s really true or not.

Performance seemed better when I Disabled hardware checksum offload.

ChuckLasher

@mloiterman

Thanks,

Made the change.
I'll play around with the offload. I will see what impact that has on the 10 GbE machines.