10GB Lan causing strange performance issues, goes away when switched over to 1GB
-
@stephenw10 said in 10GB Lan causing strange performance issues, goes away when switched over to 1GB:
We'd be seeing floods of tickets. And pf would log that as blocked packets in the firewall logs (unless that has been disabled).
In terms of this. What is the percentage of pfSense users that have say 2/5/10Gbps Internet coming into the pfSense WAN interface and then having a mixture of 10GbE and 1GbE clients downstream. I would think the number is niche still. However I deployed several dozen of your appliances along with other security appliances in the fruit company. Connecting your appliances with 1GbE to the WAN would always resolve the issue. Connecting your appliances with 10GbE or greater always provided the same outcome for 1GbE clients downstream. I am not criticizing you but this is from over 3 decades of network engineering experience with all types of firewalls from Cisco, PAN, Fortinet and Netgate. They all can create the same issues.
-
Hmm, interesting. I'm not arguing that it doesn't happen more just that it's hard to see how it could. And consequently what could be done to address it.
An increasing number of users have access to faster WAN links and LAN side hardware. It's only goign to become more important.
-
@stephenw10 Exactly. I am not even saying it's not the AT&T or Comcast crappy gateways that are the culprit either. There are so many variables to these issues. Same thing happened when we saw GbE come into the market and most clients were FastEthernet 100Mbps.
-
@lnguyen This is great stuff, makes perfect sense now thanks for all the help.
A few follow-up questions.
The command below, am I correct that this should only be applied to the 10GB port on the cisco switch, should I leave the 1GB ports alone / stock ?
qos queue-softmax-multiplier 1200
- You mentioned that Catalyst 9300 suffers from the same problem, are there any 1U 48-Port Cisco switches that do not have this issue that you would recommend, perhaps not even Cisco ? You mentioned the above fix being a sledgehammer approach. I take that as this fix may not be ideal and perhaps keeping the lan simply on 1Gb would result in better overall performance and lower latency. This is just for my home network but as you can tell I have an obsession for speed.
Perhaps I should have also mentioned that on the pfSense side I do have a Codel limiter applied on the WAN NIC to alleviate bufferbloat.
My overall goal with this network is to have the ultimate lowest latencies for pc gaming purposes.
-
@ngr2001 said in 10GB Lan causing strange performance issues, goes away when switched over to 1GB:
The command below, am I correct that this should only be applied to the 10GB port on the cisco switch, should I leave the 1GB ports alone / stock ?
Its applied globally
@ngr2001 said in 10GB Lan causing strange performance issues, goes away when switched over to 1GB:
You mentioned that Catalyst 9300 suffers from the same problem, are there any 1U 48-Port Cisco switches that do not have this issue that you would recommend, perhaps not even Cisco ?
Not that I have seen. The large chassis with dedicated supervisors have huge buffers but I doubt you would be buying those.
@ngr2001 said in 10GB Lan causing strange performance issues, goes away when switched over to 1GB:
You mentioned the above fix being a sledgehammer approach.
Using L2 Ethernet flow control rather than TCP is a blunt sledgehammer because it impacts all frames flowing through that interface. TCP flow control is per client session--hence it is better.
@ngr2001 said in 10GB Lan causing strange performance issues, goes away when switched over to 1GB:
Perhaps I should have also mentioned that on the pfSense side I do have a Codel limiter applied on the WAN NIC to alleviate bufferbloat.
Try disabling it to see how it impacts what you are seeing.
-
Yeah traffic shaping could definitely be an issue.
-
I'll try both suggestions and report back, thank you.
In case anyone is interested this is my basic limiter setup to combat Bufferbloat, I would not give this setting up being I score a perfect A+ with it in place. If I disable I get wild swings in scores, I will disable for the sake of testing.
Other than that I have 4 VLANs using the Router on a stick method, everything else is hardened down, no plugins or other features running, other than I do have a QAT Crypto card installed and working.
-
Had some time to test again.
I applied the setting "qos queue-softmax-multiplier 1200" to my Cisco 3650 global config.
Like magic my speedtests while using the SPF+ 10GB port are now back to full speed. I am also monitoring my switch ports and so far no Output Drops.
I have to admit I am not 100% sure how the above fix works, I guess I am hung up on the fact that I have removed all QOS settings from my config yet this setting seems kind of tied to QOS. I guess in the background no matter what you do the switch still has some kind of intelligence built in for QOS purposes and this simply increases the buffer pool regardless if QOS is in use or not ?
No need to disable / test Codel Limiters on my pfSense as I see it, this issue was purely a Cisco Problem.
-
very happy with these results too.
-
Nice.
-
I'm looking into 10Gb 24-48 port low latency switches to replace my 3650.
Do you have any feedback on the Cisco nexus 3548-X. It's a full 48 port Sfp+ 10GB switch. Does it suffer from any of the same buffer or performance issues especially when mixing 1Gb and 10Gb clients.
-
-
I've read through the article, albeit quickly, I did not see any reference to "CTF" mode, quick google search comes back with nothing.
From what I have read the 3548-X has increased shared buffer pools and better hardware / architecture that results in super low latencies. I am not trading stocks but I am competing in E-Sports hence the quest for the lowest latency home network is my end game here, which is what led me to find the Nexus 3548-X.
What doesn't make sense to me is how cheap these switches can be bought for on Ebay. You can get a 48-Port 10GB 3548-X switch for $200, is there something I am seriously missing. I would wipe the switch, setup my trunk port and several Vlans and call it a day, my switch config is not overly complicated. The IP Base lic is also fine for me being I route my Vlan traffic on the pfSense side, so Layer 2 is all I need.
-
In addition, when it comes to "Normal" vs "Warp" mode, I think I would prefer to run in "Warp" mode, I do not see any down side to that setup for my use case. At any point in time I will only have 4-12 hardwired clients active, only 2-3 of which will have any kind of substantial traffic.
-
@ngr2001 said in 10GB Lan causing strange performance issues, goes away when switched over to 1GB:
What doesn't make sense to me is how cheap these switches can be bought for on Ebay.
Probably because of how loud and power hungry they are. Like most (all?) older 10G switches.
-
@lnguyen Switches are either store & forward or cut through forwarding. Its $200 because its an EOL DC switch. I deployed hundreds of these. It runs NexusOS but the command structure is very similar to that of Cisco Catalyst switches. Noise level and power draw will be higher than your current one. Its also all SFP+ ports.
-
According to the data sheets the 3650 uses 99W at idle and the 3854-X uses 112W at idle. In the read world my 3650 with light load is consuming 60W according to my UPS. Both have a similar PSU and FAN design I cant imagine the power or sound being any different.
That said, do you have any other recommendations, perhaps a Cisco 350X or 550X would be better suited for my needs. I would prefer to drop the my current 60W load to 20W if possible, but again I am chasing speed here.
-
I have nothing in mind. I couldn't run a switch like that here.
-
What are your thoughts on the Cisco 350X or 550X ?
-
@ngr2001 It’s Cisco’s lowest end gear. Don’t care for that product line in that channel of the market. There are other brands serving that channel that have better products
