Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question
-
@ngr2001 After your test of a computer directly off the modem, I performed a speedtest with the Comcast XB8 in router mode and the Cisco 3850
Comcast Node <--DOCSIS 2.35Gbps--> Comcast XB8 <--2.5GbE--> Cisco 3850
-
Comcast LAN Port 1 GbE: 940/360Mbps
-
Comcast LAN Port 2 GbE: 940/360Mbps
-
Comcast LAN Port 3 GbE: 940/360Mbps
-
Comcast LAN Port 4 2.5GbE <--> Cisco 5-Speed mGig Port 1/0/48 negotiated uplink @ 2.5GbE link
-
Cisco Port 1-36 GbE: 940/360Mbps
-
Cisco Port 37-47 10GbE: 2350/360Mbps
QOS setting removed. This tells me pfSense is the issue.
-
-
I feel like the issue is likely on the PF side too, but which of your results is the bad one, I'm not 100% sure how to interpret your results.
-
@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
Audio clicks and minor drops on zoom calls etc. I checked the switch logs and sure enough there were a significant amount of drops on my clients switch port. Oddly running a speedtest did not instant create drops, they were just accumulating at what seemed random.
Odd. I am always using VoIP and never experience this. Maybe it is the pfSense limiter?
-
Keep in mind the audio drops only happened when in a mixed speed mode of which PF does not seem to handle well.
0 issues when every connection is 1Gb
-
@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
I feel like the issue is likely on the PF side too, but which of your results is the bad one, I'm not 100% sure how to interpret your results.
WIth the Comcast XB8 acting as the edge WAN device doing NAT, there are no issues. Speedtest results were maxing out the 1GbE LAN clients and maxing out the services on 10GbE LAN clients
-
@ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
Keep in mind the audio drops only happened when in a mixed speed mode of which PF does not seem to handle well.
0 issues when every connection is 1Gb
I have mixed speeds going on all the time. Just to give you an idea of my traffic usage on the account
-
But you run into the same problems I do correct ? or no ?
-
According to same app, I'm averaging 2TB per month.
-
@ngr2001 I do not run into any audio/video drop issues on Zoom, FaceTime, Signal or TeamSpeak, experience missed trigger clicks in Quake III, or any perceivable lag in other online console games on XBX or PS5 (when using XB8 in bridge mode <---> pfSense <---> Cisco switch with QOS global setting)
In the test scenario of removing pfSense from the setup and using the XB8 as the router (2.5GbE downlink to Cisco switch), 1GbE LAN clients achieve 940Mbps against Speedtest.net without having to use QOS global settings on the Cisco switch or Limiters in pfSense. 10GbE clients achieve 2.35Gbps against Speedtest.net as expected.
-
Ah ok, now I fully follow what you are saying, so yes that suggests the issue is directly related to PFSense.
My drop off issues were only the audio spectrum of zoom calls, teams was fine. It was minor, but definitely there though, and the dropped packets recorded on the switchport don't lie.
@stephenw10 what do you make of this and my testing which confirms the same.
-
That's with no flow control anywhere?
-
@stephenw10 The Cisco 3850 does not support sending pause frames, only receives them--therefore it can not tell the Comcast XB8 to pause when its buffer overflowing. Their answer is to use QOS setting to manage the frame buffer.
-
@stephenw10 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
That's with no flow control anywhere?
Regardless, with the pfSense out of the way and using the Comcast Xfinity XB8, TCP Flow Control via TCP Window Size update messages are sent by the 1GbE LAN client and received by the Ookla Speedtest servers and 940Mbps download test results are achieved. It is quite obvious pfSense is not up to the task of making sure these messages get to the sender so that the TCP stream is slowed down for a 1GbE LAN receiver.
-
Mmm, pfSense never sees those messages, unless you're proxying the traffic, so it's hard to see how it could have any effect there.
The one thing it can do is set an MSS value which only affects TCP traffic.
Either way it should be pretty obvious in a packet capture if the TCP windowing is significantly different.
So no flow control between the Comcast router and the switch?
-
@stephenw10 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
So no flow control between the Comcast router and the switch?
No FC in that scenario
-
I think even you saw some issues in your tests too right, is this something we can kick up as a potential performance bug and have the full team look into ?
-
No I've seen no such issues personally. But my local WAN here is <100M and I haven't been specifically looking.
What I did see was almost identical symptoms created by a bad MSS value. Which is why we tested that earlier.
Are you able to replicate it using iperf to a local server on the WAN side of pfSense?
If this was a pfSense issue I'd expect to see the same problems with a single 1G client connected directly to LAN and WAN at 2.5G. But as I understand it you do not.
-
You would 1Gb+ internet to see this issue for sure.
I am not 100% following the test you are asking, you want me to plug a workstation into the LAN nic of PFsense. If you can explain the test a little better ill give it a shot.
-
Yes. So remove the switch entirely. The bandwidth step-down from 2Gbs to 1Gbps is then all in pfSense.
I'm pretty sure you already ran that test though?
But it would be very interesting if you could replicate it with a local iperf server. So for example put it on a separate interface in pfSense, say OPT1. Then test to it from a client in LAN behind the switch. The packet path is essentially the same but without the modem/docsis link. The latency would obviously be lower. But I'd expect to see something similar if the OPT1 interface and server are connected at 10G. or even 2.5G.
Another interesting test would be to setup the switch in layer3 mode with the 10G and 1G clients on separate downstream subnets. When it's routing it will be using different buffering.
-
To your point, I have already tested the direct step down, WAN 2.5Gb to client 1Gb
I didn't really follow what you were looking for on the other test.
I've reverted back to everything being set to 1Gb and performance is flawless I have to say. With mine and others who have reproduced this issue are we not ready for dev to take a closer look ?
