Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question
-
@stephenw10 The Cisco 3850 does not support sending pause frames, only receives them--therefore it can not tell the Comcast XB8 to pause when its buffer overflowing. Their answer is to use QOS setting to manage the frame buffer.
-
@stephenw10 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
That's with no flow control anywhere?
Regardless, with the pfSense out of the way and using the Comcast Xfinity XB8, TCP Flow Control via TCP Window Size update messages are sent by the 1GbE LAN client and received by the Ookla Speedtest servers and 940Mbps download test results are achieved. It is quite obvious pfSense is not up to the task of making sure these messages get to the sender so that the TCP stream is slowed down for a 1GbE LAN receiver.
-
Mmm, pfSense never sees those messages, unless you're proxying the traffic, so it's hard to see how it could have any effect there.
The one thing it can do is set an MSS value which only affects TCP traffic.
Either way it should be pretty obvious in a packet capture if the TCP windowing is significantly different.
So no flow control between the Comcast router and the switch?
-
@stephenw10 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:
So no flow control between the Comcast router and the switch?
No FC in that scenario
-
I think even you saw some issues in your tests too right, is this something we can kick up as a potential performance bug and have the full team look into ?
-
No I've seen no such issues personally. But my local WAN here is <100M and I haven't been specifically looking.
What I did see was almost identical symptoms created by a bad MSS value. Which is why we tested that earlier.
Are you able to replicate it using iperf to a local server on the WAN side of pfSense?
If this was a pfSense issue I'd expect to see the same problems with a single 1G client connected directly to LAN and WAN at 2.5G. But as I understand it you do not.
-
You would 1Gb+ internet to see this issue for sure.
I am not 100% following the test you are asking, you want me to plug a workstation into the LAN nic of PFsense. If you can explain the test a little better ill give it a shot.
-
Yes. So remove the switch entirely. The bandwidth step-down from 2Gbs to 1Gbps is then all in pfSense.
I'm pretty sure you already ran that test though?
But it would be very interesting if you could replicate it with a local iperf server. So for example put it on a separate interface in pfSense, say OPT1. Then test to it from a client in LAN behind the switch. The packet path is essentially the same but without the modem/docsis link. The latency would obviously be lower. But I'd expect to see something similar if the OPT1 interface and server are connected at 10G. or even 2.5G.
Another interesting test would be to setup the switch in layer3 mode with the 10G and 1G clients on separate downstream subnets. When it's routing it will be using different buffering.
-
To your point, I have already tested the direct step down, WAN 2.5Gb to client 1Gb
I didn't really follow what you were looking for on the other test.
I've reverted back to everything being set to 1Gb and performance is flawless I have to say. With mine and others who have reproduced this issue are we not ready for dev to take a closer look ?
-
I meant a local iperf server connected at 10 or 2.5G on a separate interface in pfSense would allow you test a bandwidth difference across pfSense whilst eliminating anything caused by the WAN side DOCSIS.
-
I'm also probably going to pull out the X550 Nic and go back to the X540 onboard NIC, only reason is that with the X550 the power consumption increased 20+ watts.
I was able to get it booted with the onboard nics re-enabled, total of 4 nics now. But that step already has nuked my NDI key. When I swapped to the X550 support gave me a new lic key but they said I could only do that once. Being I've had so much trouble here do you think they will reset once again when I go back to my original onboard NIC which should be my original NDI. I really dont need plus, just thought I would get faster updates to security patches, if that is not true I could go back to community edition and be happy ?
-
Is the NDI simply reverted to the old one?
Send it to me in chat and I'll check it.
-
Hmm, well now that I have 4 nics successfully enabled I could plug a client into IX3 or IX4 directly and try some speed tests. Not sure if i would need to configure any rules or settings to allow traffic to pass ?
-
If you put the server there you wouldn't need and rules, outbound traffic to it from a client would already be allowed. Unless you have policy routing in which case you'd need a bypass rule.
-
Wanted to give an update to all here. I recently was able to get Sonic Fiber 10Gbps Symmetrical. With this ISP connection connected to the same pfSense and Cisco and UniFi switches, there are no issues with 1GbE, 2.5GbE, GbE and 10GbE LAN clients getting the max speeds they are expected to achieve (940Mbps/2.35Gbps/4.7Gbps/9.4Gbps). 802.3x Ethernet Flow Control is not required as TCP Flow Control works. The root cause is clearly with how DOCSIS changes the TCP flow. Comcast for sure uses AQM. This is only an issue for customers subscribed to their 2100Mbps/300Mbps top tier plan and can not employ 802.3x Ethernet Flow Control as a last resort.
-
That is very interesting, thanks for the update. So we are officially blaming comcast for these issues then, that makes me feel better.
