Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question
-
Here are some interesting continued results:
- After enabling "qos queue-softmax-multiplier 1200" on the 3850 I had what seemed like immediate alleviation of the performance issues. However I noticed that over time the performance started to degrade. Audio clicks and minor drops on zoom calls etc. I checked the switch logs and sure enough there were a significant amount of drops on my clients switch port. Oddly running a speedtest did not instant create drops, they were just accumulating at what seemed random.
PF WAN @2.5Gb, PF LAN @2.5Gb, Client @1Gb = Good Speed Test (~800-900Mbps) + Output Drops on Switchport over time.
This has me completely discouraged I am at the point where it does just not seem to make sense to keep messing with this, trying to achieve a WAN speed of 2Gb creates so many issues that its simply not worth it. So I started reverting everything back to 1Gb, i removed the "qos queue-softmax-multiplier 1200" being it really should not be needed and I wanted a clean baseline, this is what I encountered.
Cisco QOS Fix Removed:
FC disabled on WAN, LAN, and Switch.PF WAN @1Gb, PF LAN @2.5Gb, Client @1Gb = Bad Speed Test (~500-600Mbps) + Output Drops on Switchport
PF WAN @1Gb, PF LAN @1Gb, Client @2.5Gb = Output Drops on Switchport
There would be no point running the WAN at 2.5Gb and everything else at 1Gb so I simply did not test.
So in the end the Cisco QOS fix seems to be only a Band-Aid at best, drops will come back and they are random at best. VOIP traffic seems to take the biggest hit. I don't know if PF is the issue or if its these midgrade switches, but this was now been reproduced exactly across a Cisco 3650, Cisco 3850, and a Brocade ICX-7250.
My best solution seems like I should leave everything at 1Gb and call it a day which just kills me inside knowing I'm leaving 50% of my bandwidth behind.
-
@ngr2001 After your test of a computer directly off the modem, I performed a speedtest with the Comcast XB8 in router mode and the Cisco 3850
Comcast Node <--DOCSIS 2.35Gbps--> Comcast XB8 <--2.5GbE--> Cisco 3850
-
Comcast LAN Port 1 GbE: 940/360Mbps
-
Comcast LAN Port 2 GbE: 940/360Mbps
-
Comcast LAN Port 3 GbE: 940/360Mbps
-
Comcast LAN Port 4 2.5GbE <--> Cisco 5-Speed mGig Port 1/0/48 negotiated uplink @ 2.5GbE link
-
Cisco Port 1-36 GbE: 940/360Mbps
-
Cisco Port 37-47 10GbE: 2350/360Mbps
QOS setting removed. This tells me pfSense is the issue.
-
-
I feel like the issue is likely on the PF side too, but which of your results is the bad one, I'm not 100% sure how to interpret your results.
-
@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
Audio clicks and minor drops on zoom calls etc. I checked the switch logs and sure enough there were a significant amount of drops on my clients switch port. Oddly running a speedtest did not instant create drops, they were just accumulating at what seemed random.
Odd. I am always using VoIP and never experience this. Maybe it is the pfSense limiter?
-
Keep in mind the audio drops only happened when in a mixed speed mode of which PF does not seem to handle well.
0 issues when every connection is 1Gb
-
@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
I feel like the issue is likely on the PF side too, but which of your results is the bad one, I'm not 100% sure how to interpret your results.
WIth the Comcast XB8 acting as the edge WAN device doing NAT, there are no issues. Speedtest results were maxing out the 1GbE LAN clients and maxing out the services on 10GbE LAN clients
-
@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
Keep in mind the audio drops only happened when in a mixed speed mode of which PF does not seem to handle well.
0 issues when every connection is 1Gb
I have mixed speeds going on all the time. Just to give you an idea of my traffic usage on the account
-
But you run into the same problems I do correct ? or no ?
-
According to same app, I'm averaging 2TB per month.
-
@ngr2001 I do not run into any audio/video drop issues on Zoom, FaceTime, Signal or TeamSpeak, experience missed trigger clicks in Quake III, or any perceivable lag in other online console games on XBX or PS5 (when using XB8 in bridge mode <---> pfSense <---> Cisco switch with QOS global setting)
In the test scenario of removing pfSense from the setup and using the XB8 as the router (2.5GbE downlink to Cisco switch), 1GbE LAN clients achieve 940Mbps against Speedtest.net without having to use QOS global settings on the Cisco switch or Limiters in pfSense. 10GbE clients achieve 2.35Gbps against Speedtest.net as expected.
-
Ah ok, now I fully follow what you are saying, so yes that suggests the issue is directly related to PFSense.
My drop off issues were only the audio spectrum of zoom calls, teams was fine. It was minor, but definitely there though, and the dropped packets recorded on the switchport don't lie.
@stephenw10 what do you make of this and my testing which confirms the same.
-
That's with no flow control anywhere?
-
@stephenw10 The Cisco 3850 does not support sending pause frames, only receives them--therefore it can not tell the Comcast XB8 to pause when its buffer overflowing. Their answer is to use QOS setting to manage the frame buffer.
-
@stephenw10 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
That's with no flow control anywhere?
Regardless, with the pfSense out of the way and using the Comcast Xfinity XB8, TCP Flow Control via TCP Window Size update messages are sent by the 1GbE LAN client and received by the Ookla Speedtest servers and 940Mbps download test results are achieved. It is quite obvious pfSense is not up to the task of making sure these messages get to the sender so that the TCP stream is slowed down for a 1GbE LAN receiver.
-
Mmm, pfSense never sees those messages, unless you're proxying the traffic, so it's hard to see how it could have any effect there.
The one thing it can do is set an MSS value which only affects TCP traffic.
Either way it should be pretty obvious in a packet capture if the TCP windowing is significantly different.
So no flow control between the Comcast router and the switch?
-
@stephenw10 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
So no flow control between the Comcast router and the switch?
No FC in that scenario
-
I think even you saw some issues in your tests too right, is this something we can kick up as a potential performance bug and have the full team look into ?
-
No I've seen no such issues personally. But my local WAN here is <100M and I haven't been specifically looking.
What I did see was almost identical symptoms created by a bad MSS value. Which is why we tested that earlier.
Are you able to replicate it using iperf to a local server on the WAN side of pfSense?
If this was a pfSense issue I'd expect to see the same problems with a single 1G client connected directly to LAN and WAN at 2.5G. But as I understand it you do not.
-
You would 1Gb+ internet to see this issue for sure.
I am not 100% following the test you are asking, you want me to plug a workstation into the LAN nic of PFsense. If you can explain the test a little better ill give it a shot.
-
Yes. So remove the switch entirely. The bandwidth step-down from 2Gbs to 1Gbps is then all in pfSense.
I'm pretty sure you already ran that test though?
But it would be very interesting if you could replicate it with a local iperf server. So for example put it on a separate interface in pfSense, say OPT1. Then test to it from a client in LAN behind the switch. The packet path is essentially the same but without the modem/docsis link. The latency would obviously be lower. But I'd expect to see something similar if the OPT1 interface and server are connected at 10G. or even 2.5G.
Another interesting test would be to setup the switch in layer3 mode with the 10G and 1G clients on separate downstream subnets. When it's routing it will be using different buffering.
