Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question
-
@ngr2001 I have been thinking about this issue and discussing with other Network Engineering colleagues about this. When Comcast introduced the Gigabit Extra/Plus plans it was 1.2Gbps (provisioned 1440Mbps DS). This was the first time DOCSIS internet services surpassed the mainstream 1GbE LAN clients.
I had that plan for several years and never noticed an issue with buffer overflow on my 1GbE LAN clients--likely due to the buffer in my switch masking the broken TCP FC. However in December of 2022, when they released Gigabit x2 (provisioned as 2.35Gbps DS) I immediately experienced 500Mbps download/speedtest on 1GbE connected LAN clients-- yet full 2.35Gbps DS for 2.5GbE/5GbE/10GbE LAN clients.
I sent you a DM to a ChatGPT link. I asked it some questions to answer that might help you understand the way 802.3x FC works vs TCP FC. As well as how DOCSIS handles traffic congestion and its possible impact on TCP FC.
-
@ngr2001 Would you mind doing a test for me? Would you mind connecting a 1GbE client directly to your modem, reboot to get an IP and then test? If you still 500-800Mbps then it is not the pfSense, but if its 940/940Mbps, then its something in pfSense.
-
@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
If one wanted to throw stupid money at it, what is the proper solution appose to just masking the symptoms.
Are we talking $10K data center switches ?
You can go with their Gigabit Pro/x10 (up to 10Gbps symmetrical) service plan for $300/mo if you are within reach of their fiber nodes. This is basically the same as their metro ethernet services that is sold to business enterprises. In fact it is that team that gets it installed and provides ongoing support. I would have done that already if I was within reach of the fiber node in my neighborhood.
https://www.reddit.com/r/Comcast_Xfinity/comments/14t9bph/gigabit_pro_availability_inquiry/
-
I too have this problem.
I have pfSense as my edge.
HW configuration is as follows:
- ISP XB8 @ 2.5 GbE -> pfSense @ 10GbE
- pfSense (10G) -> Netgear XS724EM (10G)
- Netgear XS724EM -> Netgear GS110EMX (10G -> 1G) (for residual 1GbE devices)
Several 1GbE-only devices are in the 10GbE switch
The only solution I currently have is to use 802.3x Ethernet Flow Control on
the LAN side of the pfSense -AND- XS724EM "input" port.This is my "big hammer" approach.
I cannot afford enterprise grade equipment (I'm retired)
-
I couldn't figure out how, if you know the command I'm all ears.
-
Mmm, it's not obvious but I'd start by bumping the
ip-qos-sessionvalue.That appears to be a L3 parameter but.... easy to test.
-
What is going to be the main difference on say a $10K Ent Switch, simply a larger buffer 3GB+ and faster cpu ?
Would a super larger buffer be more or less the same type of solution of masking the issue ?
-
Would the test still matter if when in fact if I simply move the PF LAN NIC over to 1Gb my 1Gb clients get a full 940-980Mbps+ speedtest ?
-
This post is deleted! -
Finally found the command, was not documented anywhere in their manuals so frustrating.
Default was 1024 It requires a reload to kick in, I guess Ill try 4096 ?? or should we just max this out ??
Command to tweak:
system-max ip-qos-session 4096
-
So far I tried a value of 4096 and 8192 and there was no change in performance.
I think being this switch only has 2MB per ASIC may be a show stopper.
Patiently waiting for my 3850 to arrive.
-
@ngr2001 Yes it is still just masking the issue and introducing more bufferbloat. If we are talking about a switch with GB of buffers it would be a large chassis that would make zero sense.
-
@ngr2001 Yes it matters. If you can please test directly with a 1GbE client off the modem
-
Or with a 1G client connection to pfSense directly since we know a 10G client passes traffic as expected.
I still find it hard to imagine how a L2 switch cab affect traffic experiencing some TCP issue.
A mixed speed network like that has got to be how almost all ICX7250 were/are used.
There any more detailed explanation of this issue I read up on?
-
I work from home, so when I have a sec Ill try this test for you since you have been so helpful.
Please do note though.
My results so far:
PF WAN 1Gb, LAN 1Gb, Client 1Gb = No issues, 980Mbps Bench.
PF WAN 2.5Gb, LAN 1Gb, Client 1Gb = No issues, 980Mbps Bench.
PF WAN 2.5Gb, LAN 10Gb, Client 10Gb = No issues, 1980Mbps Bench.
PF WAN 2.5Gb, LAN 10Gb, Client 1Gb = Issues, 500Mbps Bench.
-
@stephenw10 The reason he doesn't see a problem with his 1GbE connection is because he has 802.3x FC enabled and it works on that switch port whereas it doesn't work on the 10GbE port. 802.3x FC is only a solution if you are willing to pause all traffic to the LAN and not just the one LAN client suffering from buffer overflow.
@ChuckLasher Seems to have 802.3x FC function on his 10GbE switch port that interfaces with his pfSense LAN port. Blunt and crude but it does allow his 1GbE LAN client to achieve 940Mbps while pausing the network for all other LAN clients.
This is why I keep stating that TCP Flow control is far superior but doesn't seem to be working over DOCSIS--at least I think. @stephenw10 Here is the link to the ChatGPT that can get you started:
https://chatgpt.com/share/67ab9776-0818-8005-810c-5c0403fe480d@ngr2001 This is why I am asking for you to test direct to the modem. I would like to know conclusively if you can achieve 940Mbps off of a standard DOCSIS 3.1 modem without 802.3x FC. I don't have my own modem to test this as I lease their XB8.
-
Now it clicks, makes perfect sense to me now. I got did out my spare laptop with an onboard NIC give me a few.
-
@ngr2001 Much easier to describe when you are in a war room or at least a WebEX war room session
-
Mmm, so a 10G SFP NIC in pfSense that allowed flow control to the 7250 (which is what I use) would also avoid this issue.
-
@stephenw10 Yes, if the 10GbE connection can negotiate 802.3x FC it will work around the issue with the caveat that all traffic will be paused in and out of the pfSense LAN interface everytime it receives a PAUSE frame.
