Netgate 6100 10 gb Routing Throttled to 1 gb?
-
Hello Netgate Community,
I'll preface this with stating that although I'm fairly experienced with home networking, I'm very new to pfSense and have basically zero experience/knowledge regarding 10 gb hardware/networking. So, I have no idea if what I'm seeing is actually a problem, or if my expectations are just off base.
Here's the issue: I have my 6100 connected to a 10 gb managed switch via the WAN3 / ix0 SFP+ port using a DAC cable rated at 10 gb. Both the 6100 and the switch show that it's a 10 gb connection. However, when running iperf3 I'm getting far from 10 gb speeds. Specifically, any host on the switch connecting to the iperf3 server on the 6100 is limited to 1 gb or less. When going from the 6100 to an iperf3 server running on a host connected to the switch, it's at most 2 gb. I have a mix of 1 gb, 2.5 gb, and 10 gb hosts on the switch, and iperf3 between them always shows expected speeds (9.4 gb between my desktops, 2.37 gb between my desktops/laptop and wireless access points). So, the hosts on the switch definitely are not the bottleneck, nor is the switch.
Here's what I've tried:
- Swapping DAC cables. No change.
- Swapping to a different 10 gb port on the switch. No change.
- Connecting a host directly to the WAN4 / ix1 port on the 6100 via a DAC cable. Results in iperf3 speeds between host and 6100 of at most 2.5 gb. Speeds from this host to a 10 gb host on the switch top out at 2.2 gb.
- Disabling pfBlocker completely (I thought maybe excessive firewall rules could cause routing speeds to suffer). No change.
Any thoughts? Is this expected behavior? I wouldn't think so, since my understanding of its specs indicates the 6100 should have more than enough routing capacity to push 10 gb though an SFP+ port to a 10 gb switch. Is it a problem with the iperf3 server/client software on the 6100? Is it something else that I my limited knowledge/experience with 10 gb networking has never even heard of? Is it a pfSense limitation or configuration problem? Any thoughts/suggestions/insights would be greatly appreciated!
Cheers,
Brian
-
Have you tried iperf computer to computer? It may not run full speed on the 6100.
-
@andyrh said in Netgate 6100 10 gb Routing Throttled to 1 gb?:
Have you tried iperf computer to computer? It may not run full speed on the 6100.
Yep, I had one Linux machine hooked directly up to the 6100 and tested to another Linux machine on the switch running iperf3 as a server. Topped out at 2.2 gb.
-
How are you running iperf? Multiple steams? Both directions?
Are you just routing or NATing that connection too?
When you see 1G is it exactly 1G, like 941Mbps , implying a link issue somewhere?
If you are routing through the 6100 between the 10G and 2.5G ports you should see the 2.5G line rate in iperf3, 2.37Gbps.
Steve
-
@stephenw10 said in Netgate 6100 10 gb Routing Throttled to 1 gb?:
How are you running iperf? Multiple steams? Both directions?
Are you just routing or NATing that connection too?
When you see 1G is it exactly 1G, like 941Mbps , implying a link issue somewhere?
If you are routing through the 6100 between the 10G and 2.5G ports you should see the 2.5G line rate in iperf3, 2.37Gbps.
Steve
Ok, so I just did some more testing...When I go through the 6100 between the 10G and 2.5G ports, I see 2.37 in iPerf3. That's expected and makes total sense. Previously I've been running iperf unidirectional with one stream, which got me the 2.37 between the 10G and 2.5G, and also gets me 9.4G between hosts on the switch. However, I just tried it from a host on the switch to the iperf server on the router, in both directions, and with 10 streams got up 4.8G on both the sender and receiver. So, that's better...Right?
When I saw just 1G, it was actually exactly 941 Mbps every time for every transfer, and I noticed that the connection status in pfSense had changed from "10Gbase-Twinax <full-duplex,rxpause,txpause" to "unknown" but the switch still showed 10G. So, I unplugged the DAC cables on both the 6100 and the switch, plugged them back in, and now a single stream shows anywhere from 1.06G to 1.61G, and the like I stated above, multiple streams get up to 5G. Not sure what the issue was there, but unplugging the cable and plugging it back in seems to have fixed it.
I'm not sure what you mean by routing or NATing in this case...The adapter assignment on the router for the 10G WAN4 / ix0 port is 192.168.3.1. The switch is 192.168.3.2, and the host is 192.168.3.10. So, on the host (Ryzen 5950 running Linux) I'm just running "iperf3 -c 192.168.3.1 -d -P10."
Anyway, thanks for your help thus far! It seems it's definitely no longer stuck at 1G speeds, and although 5G isn't 10G, it's better than it was. Who know...Maybe more of your input, combined with input from the many other wise folks on here, can either explain why it's limited or actually get it to 10G.
Cheers, and thanks again!
Brian
-
@bpsantucci said in Netgate 6100 10 gb Routing Throttled to 1 gb?:
I'm not sure what you mean by routing or NATing in this case...
I mean if you are testing with a client on a 'LAN' and the server on a 'WAN' the auto outbound NAT rules will be NATing that out of the WAN interface. That requires additional processing limiting throughput.
pfSense determines what is a WAN interface in that situation by whether or not it has a gateway defined on it.The throughput is limited by the CPU in the 6100 in that setup. As long as you have pf enabled and firewall rules in place ~5G is what I would expect to see in iperf3.
Steve
-
@stephenw10 said in Netgate 6100 10 gb Routing Throttled to 1 gb?:
@bpsantucci said in Netgate 6100 10 gb Routing Throttled to 1 gb?:
I'm not sure what you mean by routing or NATing in this case...
I mean if you are testing with a client on a 'LAN' and the server on a 'WAN' the auto outbound NAT rules will be NATing that out of the WAN interface. That requires additional processing limiting throughput.
pfSense determines what is a WAN interface in that situation by whether or not it has a gateway defined on it.The throughput is limited by the CPU in the 6100 in that setup. As long as you have pf enabled and firewall rules in place ~5G is what I would expect to see in iperf3.
Steve
Got it...Never thought about it like that, but now I see how WAN/LAN NATing could create significant processing overhead. Thanks for the explanation!
