10gbe acting like 100mbe
-
Let me start by saying I don't normally post so take it easy you bunch of animals ;-) . My knowledge is networks+ certification and a few Cisco classes many many moons ago so not much but let me get to the issues.
I have a cable modem with 2.5gbe port and 1 gig internet service provisioned to 1.2 from xfinity
I am using the popular HP 290-p0043w as my hardware with a Intel - X710T2LBLK (dual 10/5/2.5 gbe ports)
I am using pfsense 2.5.0-DEVELOPMENT
My switch is 10gbe rj-45 QNAP QSW-M408-4C 10GbE Managed SwitchThe issue: If I set the switch to auto negotiate speed it sets itself to 10gbe and flow control on but on pfsense it is just 10gbe with no note of any flow control and speed tests are showing about 100 mbps and I should be gettings 1.2gbps
Next if I set switch to force 5gbe then switch shows 5gbe with flow control on and pfsense shows 5000Base-T <full-duplex,rxpause> I assume the rxpause is some sort of flow control.. Now everything is working as it should. When testing with a 2.5gbe hooked to switch I get 1.2 gbps as I would expect.
So in summary if I set port to 10gbe it acts like 100mbps and if I force 5 gbe everything is as expected.
I know I should be using iperf and that might be my next step but any thoughts or ideas on this one and does someone have a similar setup hardware wise? Thanks!
-
The ixl drivers in pfSense 2.4.5 do not have any support for 5G or 2.5G.
If you are running 2.4.5 I'm quite surprised it links at 5GbE to the switch.
You can enable/disable flowcontrol using the appropriate sysctls or loader values:
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#flow-controlAs far as I know the only way to establish a 2.5GbE link in 2.4.5 is to use one of the newer Realtek cards with an updated driver or one of the Broadcom cards that can be firmware hacked to do so.
Steve
-
Nope, I'm wrong. Only at base-T though:
https://github.com/pfsense/FreeBSD-src/blob/RELENG_2_4_5/sys/dev/ixl/ixl_pf_main.c#L1850Which it looks like you're using.
If you run
ifconfig ixl0 -vvvmdoes it show 2.5 or 5 as avaialable link speeds?Steve
-
@stephenw10
I am using pfsense 2.5.0-DEVELOPMENT
wrong RELENG
-
Wrong on all counts then.
-
ifconfig ixl0 -vvvm returns ifconfig: -vvvm: bad value
ifconfig results for both ports below. I left out ip and mac info
ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN
options=e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
media: Ethernet autoselect (2500Base-T <full-duplex,rxpause>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>ixl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN
options=e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
media: Ethernet autoselect (5000Base-T <full-duplex,rxpause>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
-
Thanks for the link I think I will try and switch to auto and then manually set flow control to receive pause. The question is would the entry be hw.ix.flow_control=1 or hw.ixl1.flow_control=1 ? and can I just run that as a temp command to see if it works without editing the file?
-
Sorry got the switches is the wrong order, clearly not my best day!
Should beifconfig -vvvm ixl0The 700 series NIC uses the ixl(4) driver, ix is for earlier Intel NICs.
You should be able to set that value per NIC as a test at the command line using, for example:
[admin@7100.stevew.lan]/root: sysctl dev.ixl.0.fc=0 dev.ixl.0.fc: 1 -> 0Yes, tested that works immediately.
Steve
-
@stephenw10 I tried the command sysctl dev.ixl.1.fc=1 and it worked the 10gbe was now showing rxpause but the bad new is my connection was still only about 100mbps. I am sure I pushing my luck with a beta of pfsense and a brand new switch from QNAP and a newer .. I think intel card. 5gbe is perfectly fine for me and so none of this is a big deal but I am still willing to troubleshoot if anyone finds this stuff interesting.
ifconfig -vvvm ixl1
ixl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN
options=e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>media: Ethernet autoselect (5000Base-T <full-duplex,rxpause>) status: active supported media: media autoselect media 10Gbase-T media 5000Base-T media 2500Base-T media 1000baseT media 100baseTX nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
-
So with the switch set to 5G fixed the pfSense interface auto-negotiates to 5G and you see the full 1.2Gbps across it?
But if you set both ends to autoselect it tries to comes up at 10G and you see ~100Mbps?
That sounds a lot like a bad cable.Check the interface status when it's not passing as expected, do you see errors or collisions?
Steve
-
@stephenw10 Yes, you are correct in your summary of the problem. I am using a cable matter 5 ft cat 6 550mhz 24 gauge cable currently between the switch and router but I will switch that out and I was thinking of also trying a different port on the switch in case that port is maybe bad. I will check for collisions or errors as well. I have to wait until after 5pm today but I will give it a go and report back what I find. Thanks for the help.
-
@stephenw10 To summarize switching cable and ports made no difference at all but .... I did discover that if I force the switch to 5gbe and turn off flow control also from the switch I get the same 100mbps problem. So everything is pointing to flow control being the issue. Is it possible that command sysctl dev.ixl.1.fc=1 made rxpause appear but did not actually turn it on? like display only?
Pictures below because who doesn't like photos:
So this is internet speed test with switch port forced to 5gbe flow control on:
Here is after switching physical ports and cable with port set to auto negotiate 10gbe also error checks I did
-
Seeing rxpause there does indicate flowcontrol is active. It often would show txpause there too though.
Usually it would be disabling flowcontrol that is required if any change is actually required. Flow control is enabled by default on most 10GbE connections.There's probably not that much that can be done here. Setting the switch to 5G fixed seems like reasonable option unless there is something you are trying to achieve beyond that?
Steve
-
@stephenw10 No, not really trying to do much other than future proof. Maybe this will help someone down the road a bit that is running into this problem as everything is really stock here . Thanks again for the help!
EDIT: By stock I mean a fresh install with no add ons and very little setting changes.
