Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question
-
What is going to be the main difference on say a $10K Ent Switch, simply a larger buffer 3GB+ and faster cpu ?
Would a super larger buffer be more or less the same type of solution of masking the issue ?
-
Would the test still matter if when in fact if I simply move the PF LAN NIC over to 1Gb my 1Gb clients get a full 940-980Mbps+ speedtest ?
-
This post is deleted! -
Finally found the command, was not documented anywhere in their manuals so frustrating.
Default was 1024 It requires a reload to kick in, I guess Ill try 4096 ?? or should we just max this out ??
Command to tweak:
system-max ip-qos-session 4096
-
So far I tried a value of 4096 and 8192 and there was no change in performance.
I think being this switch only has 2MB per ASIC may be a show stopper.
Patiently waiting for my 3850 to arrive.
-
@ngr2001 Yes it is still just masking the issue and introducing more bufferbloat. If we are talking about a switch with GB of buffers it would be a large chassis that would make zero sense.
-
@ngr2001 Yes it matters. If you can please test directly with a 1GbE client off the modem
-
Or with a 1G client connection to pfSense directly since we know a 10G client passes traffic as expected.
I still find it hard to imagine how a L2 switch cab affect traffic experiencing some TCP issue.
A mixed speed network like that has got to be how almost all ICX7250 were/are used.
There any more detailed explanation of this issue I read up on?
-
I work from home, so when I have a sec Ill try this test for you since you have been so helpful.
Please do note though.
My results so far:
PF WAN 1Gb, LAN 1Gb, Client 1Gb = No issues, 980Mbps Bench.
PF WAN 2.5Gb, LAN 1Gb, Client 1Gb = No issues, 980Mbps Bench.
PF WAN 2.5Gb, LAN 10Gb, Client 10Gb = No issues, 1980Mbps Bench.
PF WAN 2.5Gb, LAN 10Gb, Client 1Gb = Issues, 500Mbps Bench.
-
@stephenw10 The reason he doesn't see a problem with his 1GbE connection is because he has 802.3x FC enabled and it works on that switch port whereas it doesn't work on the 10GbE port. 802.3x FC is only a solution if you are willing to pause all traffic to the LAN and not just the one LAN client suffering from buffer overflow.
@ChuckLasher Seems to have 802.3x FC function on his 10GbE switch port that interfaces with his pfSense LAN port. Blunt and crude but it does allow his 1GbE LAN client to achieve 940Mbps while pausing the network for all other LAN clients.
This is why I keep stating that TCP Flow control is far superior but doesn't seem to be working over DOCSIS--at least I think. @stephenw10 Here is the link to the ChatGPT that can get you started:
https://chatgpt.com/share/67ab9776-0818-8005-810c-5c0403fe480d@ngr2001 This is why I am asking for you to test direct to the modem. I would like to know conclusively if you can achieve 940Mbps off of a standard DOCSIS 3.1 modem without 802.3x FC. I don't have my own modem to test this as I lease their XB8.
-
Now it clicks, makes perfect sense to me now. I got did out my spare laptop with an onboard NIC give me a few.
-
@ngr2001 Much easier to describe when you are in a war room or at least a WebEX war room session
-
Mmm, so a 10G SFP NIC in pfSense that allowed flow control to the 7250 (which is what I use) would also avoid this issue.
-
@stephenw10 Yes, if the 10GbE connection can negotiate 802.3x FC it will work around the issue with the caveat that all traffic will be paused in and out of the pfSense LAN interface everytime it receives a PAUSE frame.
-
I was slightly err'd in my post.
For clarity, this is my configuration (as reported by pfSense) with Ethernet flow control active.
The XB8 is connected at 2.5 G to ixl0 WAN 10G port,
The 10G switch is connected to ilx1 LAN 10G port.
In this configuration, with ethernet flow control enabled, isn't it true that all my 10G switch's traffic is paused when any 1G port it serves needs it. This seems to be the case in my testing
Therefore, I would like to know where the problem is; in the XB8 or the ixl0 port ?
Is it possible to determine this? -
Mmm, so if the upstream DOCSIS flow/congestion control is causing so much of an issue that TCP CC is shutting the window down you should be able to see that in a packet capture quite easily.
Though I'm not sure it is since here we only see an issue with the 10G to 1G 'bridge' happening in the switch.
Might be able to limit the burst rate from the pfSense LAN maybe....
-
@ChuckLasher said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
Therefore, I would like to know where the problem is; in the XB8 or the ixl0 port ?
Is it possible to determine this?Those interfaces only show rxpause not txpause. So if that's helping it can only be because the switch is telling the NIC to pause.
But you should also be able to tell be looking at any error counters. They will show errors in on whichever device is exhausting it's buffer.
-
How do I enable txpause ? ( Not an internals expert )
I'll grab whatever info you need.
My switch doesn't show any buffer error type info.
The best I can get is CRC errorsIn the netgear, I have this configured for ethernet flow control testing.
port number | name | port status | port speed | linked speed | flow control | max mtu
-
Two ways to enable both 802.3x RX and TX
-
sysctl dev.ixl.1.fc=3
-
Create a System Tunable Entry with the variable being
dev.ixl.1.fcand a value of3
-
-
Isn't 3 the default value ?
