Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question
-
Here are the results, no speed issues when connected directly to the modem, I did expect this though, being I have no issues when PFSense LAN and client is also at 1Gb regardless of the WAN NIC at 2.5 or 1Gb.
In this test a Win11 Laptop with a 1Gb NIC was plugged into my Netgear Modems Yellow 2.5Gb WAN Port. The Laptop of course negotiated a 1Gb Link.
Also Waveform is always stable, its speedtest.net and fast.com that really highlight the issue.
-
Can you check if flow control is enabled on your windows machine?
Google / AI states:
To check Ethernet flow control settings on Windows 11 using the command line, open a command prompt as administrator and type "Get-NetAdapterQos"; this PowerShell command will display the Quality of Service (QoS) capabilities and configurations of your network adapter, including flow control settings if supported by your hardware.
I don't have a way to check.
Might you be able to please ? -
Get-NetAdapterQos -Name "*" | Where-Object -FilterScript { $_.Enabled }
Get-NetAdapterQos -Name "*" | Where-Object -FilterScript { $_.Enabled -Eq "False" }
Neither command is returning any results so far, still figuring this one out.
-
So the Intel Driver on the laptop I used shows both Rx & Tx Flow Control Enabled. Whether or not its actually working I guess is a different story. But It seems like Windows 11 with most Intel and Realtek NICs enable Flow Control by default. Now whether or not the Netgear modem and Xfinity ISP properly support it I guess is another story.
Side note, my PFSense WAN NIC also shows Rx & Tx FC established when connected to the same modem.
FC working or not 1Gb to 1Gb speeds are fine and have always been fine, I wouldn't think FC is needed in this situation and even if enabled I would not expect it to kick in, so I am kind of wondering how much this test would help.
-
@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
Isn't 3 the default value
It is but that could have been changed. Also @ChuckLasher is using ixl NICs not ix so there are also driver differences.
-
@ngr2001 If you can turn 802.3x FC off, we will know whether or not TCP Flow control works with your Windows PC directly connected to a DOCSIS network where the inbound from the coaxial is 2.35Gbps.
-
I'll give it a try in the am.
But 1 ? The modems port supports 2.5Gb and I pay for 2Gb service but when you plug a laptop with a 1Gb port Into that modem it will negotiate simply at 1Gb and give you a live ext IP. I guess you are thinking that if FC is disabled in windows and the speed test is poor then the isp supports and needs FC. However what If the speediest is fine, I guess what do the results mean in either case. I guess I'm hung up that there is no NAT on my end and we are live tapped in.
-
@ngr2001 If 802.3x FC is disabled and you still get 940/940, that means TCP Flow Control worked and it was able to tell the server to slow down so that the client doesn't end up with buffer overflow but still achieve its maximum potential. Which then points us back in the direction of pfSense. If it doesn't achieve 940/940, then we know TCP Flow Control doesn't work over DOCSIS and must rely on Layer 2 802.3x which is blah.....
-
TCP flow/congestion control always 'works', the fact that it just slows down rather than fails is evidence that it is. The issue here is that other flow control methods at lower layers adversely affect the TCP CC to drop the speed more than is required as I understand it.
In this situation though it appears that can only be the switch to pfSense link when it's overloaded by burst traffic. I can't see anything else that would explain it. It does feel like you might be able to some shaping in pfSense to mitigate it though.
-
Just ran a new test with Flow Control Hard Disabled in Windows.
The result was no difference in speed, but again I expect this. The modems port although being 2.5Gb ready can auto negotiate either 2.5Gb or 1Gb. In my test its negotiating 1Gb to the laptop thus I would expect this to be a very clean 1:1 connection with no speed mismatches to cause an any issues.
Flow Control Off:
Flow Control Off:
Flow Control Off:
-
@ngr2001 Thanks. This was very helpful to me to eliminate DOCSIS being the cause of any TCP Flow Control failure. This points us back to the pfSense and how its interacting with switches at 1GbE and 10GbE. I have been analyzing packet captures at the client. Next will be analyzing packet capture of the pfSense LAN interface
-
I'd love to test the problematic setup but with a different firewall in place like a Ubiquiti Dream Machine Pro Max. Have you or anyone else tried that, this would at least help pinpoint whether or not there is a potential problem on the PFSense side, worth a shot ?
-
@ngr2001 It's worth a try.
-
Mmm, that would be a good test. As long as it uses the same modules.
Another good test would be swapping out the modules/link with something correctly supports flow-control. But that would require a different NIC.
-
So I got the 3850 installed with just a basic config.
1st test:
qos queue-softmax-multiplier 1200 -- NOT SET YET
PF WAN = FC Enabled (by default) and Showing active
PF LAN = No FC (I assume a clean 3850 doesn't have RX and TX FC by default)PF WAN @2.5Gb, PF LAN @2.5Gb, Client @1Gb = Poor Speed test (~500Mbps) - Output drops detected on switchport
PF WAN @2.5Gb, PF LAN @2.5Gb, Client @2.5Gb = Normal speed test (~900Mbps)
PF WAN @2.5Gb, PF LAN @2.5Gb, Client @10Gb = Normal speed test (~900Mbps)
I would like to enable FC on the 3850 and test that next if anyone knows the command. after that I will add the QOS fix and test.
FYI 900Mbps is normal for me right now being I have a Codel Limiter capping the speed at 900.
-
@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
I would like to enable FC on the 3850 and test that next if anyone knows the command. after that I will add the QOS fix and test.
I mentioned this before in the threads, Cisco supports RX but not TX Pause. This is why I have been focusing on finding the root cause of why TCP Flow Control isn't able to do its job properly. I can't and don't like to rely on the last resort method of using L2 802.3x FC.
You need to use the global QOS setting I provided:
qos queue-softmax-multiplier 1200show interfaces tengigabitEthernet 1/0/48 capabilities | include Flowcontrol Flowcontrol: rx-(off,on,desired),tx-(none) show interface tenGigabitEthernet 1/0/48 flowcontrol Port Send FlowControl Receive FlowControl RxPause TxPause admin oper admin oper --------- -------- -------- -------- -------- ------- ------- Te1/0/48 Unsupp. Unsupp. on on 0 0FYI 900Mbps is normal for me right now being I have a Codel Limiter capping the speed at 900.
I assume this is just for testing?
-
Mmm, I would definitely try disabling Codel shaping at least as a test. That's just another factor that will be affecting TCP.
-
Codel is a hard requirement for me, its the only way I can get an A+ bufferbloat score, this setup is for competitive gaming.
That said, If seems like as soon as I allow the download speed to surpass 900Mbps even my client on 2.5Gb all the way around struggles to get an A+ score. Feels like everything I do to surpass 1Gb takes me 3 steps back. If I were to allow 1750Mbps in Codel should I expect clients at 1Gb to achieve a clean A+ bufferbloat score?
-
I would have expected it to. CoDel generally 'just works'. However it still replies on being the slowest part of the route. If you set the Limiter higher than some other link then all the buffering happens there and not locally where you can control it. So it implies there is some other 1G link in the route you're testing over.
But just as a test I would still try disabling it. I wouldn't expect it to make any difference to the test results other than for buffer-bloat etc. But if you suddenly stop seeing the packet-loss / 500M limiit for 1G clients that would be a pretty big clue!
-
I'll Give it a shot.
So, in the use case of having a 2Gb Service from the ISP and the Codel Limiter it set to 1750Mbps, is there anyway to prevent bufferbloat for the 1Gb clients or are they just SOL.
For example, is there any way to make a second Codel limiter set to 875Mbps that only applies to certain clients or clients that are at 1Gb ?
