10gbe acting like 100mbe

froboz

Let me start by saying I don't normally post so take it easy you bunch of animals ;-) . My knowledge is networks+ certification and a few Cisco classes many many moons ago so not much but let me get to the issues.

I have a cable modem with 2.5gbe port and 1 gig internet service provisioned to 1.2 from xfinity
I am using the popular HP 290-p0043w as my hardware with a Intel - X710T2LBLK (dual 10/5/2.5 gbe ports)
I am using pfsense 2.5.0-DEVELOPMENT
My switch is 10gbe rj-45 QNAP QSW-M408-4C 10GbE Managed Switch

The issue: If I set the switch to auto negotiate speed it sets itself to 10gbe and flow control on but on pfsense it is just 10gbe with no note of any flow control and speed tests are showing about 100 mbps and I should be gettings 1.2gbps

Next if I set switch to force 5gbe then switch shows 5gbe with flow control on and pfsense shows 5000Base-T <full-duplex,rxpause> I assume the rxpause is some sort of flow control.. Now everything is working as it should. When testing with a 2.5gbe hooked to switch I get 1.2 gbps as I would expect.

So in summary if I set port to 10gbe it acts like 100mbps and if I force 5 gbe everything is as expected.

I know I should be using iperf and that might be my next step but any thoughts or ideas on this one and does someone have a similar setup hardware wise? Thanks!

stephenw10

The ixl drivers in pfSense 2.4.5 do not have any support for 5G or 2.5G.

If you are running 2.4.5 I'm quite surprised it links at 5GbE to the switch.

You can enable/disable flowcontrol using the appropriate sysctls or loader values:
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#flow-control

As far as I know the only way to establish a 2.5GbE link in 2.4.5 is to use one of the newer Realtek cards with an updated driver or one of the Broadcom cards that can be firmware hacked to do so.

Steve

stephenw10

Nope, I'm wrong. Only at base-T though:
https://github.com/pfsense/FreeBSD-src/blob/RELENG_2_4_5/sys/dev/ixl/ixl_pf_main.c#L1850

Which it looks like you're using.

If you run ifconfig ixl0 -vvvm does it show 2.5 or 5 as avaialable link speeds?

Steve

kiokoman

@stephenw10
I am using pfsense 2.5.0-DEVELOPMENT
wrong RELENG

stephenw10

Wrong on all counts then.

froboz

@stephenw10

ifconfig ixl0 -vvvm returns ifconfig: -vvvm: bad value

ifconfig results for both ports below. I left out ip and mac info

ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN
options=e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
media: Ethernet autoselect (2500Base-T <full-duplex,rxpause>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

ixl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN
options=e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
media: Ethernet autoselect (5000Base-T <full-duplex,rxpause>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

froboz

@stephenw10

Thanks for the link I think I will try and switch to auto and then manually set flow control to receive pause. The question is would the entry be hw.ix.flow_control=1 or hw.ixl1.flow_control=1 ? and can I just run that as a temp command to see if it works without editing the file?

stephenw10

Sorry got the switches is the wrong order, clearly not my best day!
Should be ifconfig -vvvm ixl0

The 700 series NIC uses the ixl(4) driver, ix is for earlier Intel NICs.

You should be able to set that value per NIC as a test at the command line using, for example:

[admin@7100.stevew.lan]/root: sysctl dev.ixl.0.fc=0
dev.ixl.0.fc: 1 -> 0

Yes, tested that works immediately.

Steve

froboz

@stephenw10 I tried the command sysctl dev.ixl.1.fc=1 and it worked the 10gbe was now showing rxpause but the bad new is my connection was still only about 100mbps. I am sure I pushing my luck with a beta of pfsense and a brand new switch from QNAP and a newer .. I think intel card. 5gbe is perfectly fine for me and so none of this is a big deal but I am still willing to troubleshoot if anyone finds this stuff interesting.

ifconfig -vvvm ixl1

ixl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN
options=e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>

media: Ethernet autoselect (5000Base-T <full-duplex,rxpause>)
status: active
supported media:
	media autoselect
	media 10Gbase-T
	media 5000Base-T
	media 2500Base-T
	media 1000baseT
	media 100baseTX
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

stephenw10

So with the switch set to 5G fixed the pfSense interface auto-negotiates to 5G and you see the full 1.2Gbps across it?

But if you set both ends to autoselect it tries to comes up at 10G and you see ~100Mbps?
That sounds a lot like a bad cable.

Check the interface status when it's not passing as expected, do you see errors or collisions?

Steve

froboz

@stephenw10 Yes, you are correct in your summary of the problem. I am using a cable matter 5 ft cat 6 550mhz 24 gauge cable currently between the switch and router but I will switch that out and I was thinking of also trying a different port on the switch in case that port is maybe bad. I will check for collisions or errors as well. I have to wait until after 5pm today but I will give it a go and report back what I find. Thanks for the help.

froboz

@stephenw10 To summarize switching cable and ports made no difference at all but .... I did discover that if I force the switch to 5gbe and turn off flow control also from the switch I get the same 100mbps problem. So everything is pointing to flow control being the issue. Is it possible that command sysctl dev.ixl.1.fc=1 made rxpause appear but did not actually turn it on? like display only?

Pictures below because who doesn't like photos:

So this is internet speed test with switch port forced to 5gbe flow control on:

Here is after switching physical ports and cable with port set to auto negotiate 10gbe also error checks I did

stephenw10

Seeing rxpause there does indicate flowcontrol is active. It often would show txpause there too though.
Usually it would be disabling flowcontrol that is required if any change is actually required. Flow control is enabled by default on most 10GbE connections.

There's probably not that much that can be done here. Setting the switch to 5G fixed seems like reasonable option unless there is something you are trying to achieve beyond that?

Steve

froboz

@stephenw10 No, not really trying to do much other than future proof. Maybe this will help someone down the road a bit that is running into this problem as everything is really stock here . Thanks again for the help!

EDIT: By stock I mean a fresh install with no add ons and very little setting changes.

lnguyen

@froboz

I have a XG-1537 with a X710-T4L running pfSense+ 21.02.2
If ixl0 is connected directly to the 2.5Gbps port on an Arris S33, sub 1Gbps (500Mbps) downstream performance is realized for for all LAN hosts with 1Gbps. 10Gbps LAN hosts are able to saturate the 1440Mbps downstream bandwidth provisioned by Comcast
If ixl0 is connected directly to the 1Gbps port on the Arris S33, both 1Gbps and 10Gbps LAN hosts realize 942Mbps downstream performance.

Flow control function seems defunct. Sysctl output shows its disabled:

sysctl dev.ixl.0.fc
dev.ixl.0.fc: 0

The UI and ifconfig states otherwise:

ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
        ether "REDACTED"
        inet6 "REDACTED"%ixl0 prefixlen 64 scopeid 0x3
        inet6 "REDACTED" prefixlen 128
        inet "REDACTED" netmask 0xfffffe00 broadcast 255.255.255.255
        media: Ethernet autoselect (2500Base-T <full-duplex,rxpause>)
        status: active
        supported media:
                media autoselect
                media 10Gbase-T
                media 5000Base-T
                media 2500Base-T
                media 1000baseT
                media 100baseTX
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

froboz

@lnguyen Thanks for the information! if I am understanding correctly you are experiencing the same problem I am and it points to flow control not being set correctly.

lnguyen

@froboz

Yes. I am trying to update the FW from 3/31/21 on Intel's Support page.

https://downloadcenter.intel.com/download/24769/Non-Volatile-Memory-NVM-Update-Utility-for-Intel-Ethernet-Network-Adapter-700-Series

There is another thread that another user got this working:

https://forum.netgate.com/topic/162333/intel-x710-issues/8?_=1620678897780