Netgate 6100 / 25.07 - any recipes / guidelines for optimising high speed LAN and WAN connections?
-
I have a Netgate 6100 unit running pfSense+ 25.07.
My LAN network is 10 GbE and the LAN connection to the 6100 is a 10 Gbit/s SFP+ uplink from my main switch.
My WAN connection is a 2.5 Gbits FTTH connection using PPPoE over VLAN 911.
I'm trying to optimise the throughput and latency of LAN <-> router, router <-> WAN and LAN <-> WAN (via router) traffic.
Things that I have done so far:
-
Switched to using the new PPPoE driver for the WAN. This definitely brought some useful improvements.
-
Disabled all hardware offloading and also hardware VLAN tagging/filtering/checksums (I do use VLANS).
-
Disabled ethernet flow control on all interfaces (I have never found it to be beneficial if everything else is properly configured).
-
On the WAN connection I have implemented a FQ_CODEL limiter based on the recipe in the pfSense documentation. This has significantly improved latency under heavy load without much impact on throughput.
One thing that puzzles me is that if I run an iperf3 speed test between a (high spec, 10 GbE connected) system in the LAN to iperf3 running on the router I can achieve 'only' ~3.9 Gbit/s download and less than 1.5 Gbit/s upload even though the underlying path is 10 Gbit/s all the way. If I run the same test between two LAN systems I can get 9.8 Gbit/s in both directions.
-
-
An iperf test to or from pfSense is usually limited by the pfSense hardware itself. In this case probably the single process running iperf on the C3558. You need to test through it between 10G interfaces to see the actual throughput. Though it won't be 10Gbps.
-
@stephenw10 For sure the NetGate 6100 is not super powerful hardware, but I'd still expect the upload and download tests to give similar results rather than such a disparity, even if the throughput is << 10 Gbit/s.
Sadly I only have on 10 Gbit/s upping to the router, shared by the main network (LAN) and 3 VLANS (all of which have very low traffic to/from the router). If I test between 10 GbE connected systems one on the main LAN and the other on one of the VLANs (so traffic is processed/routed by the router) then I get between 1.7 Gbit/s and 3.5 Gbit/s in each direction, so still a lot of variability but it seems like ~3.5 Gbit/s in and out concurrently is pretty much the limit for the hardware
Would you expect that I'd see better results with an 8200 appliance?
-
Yes the 8200 is faster and you would see better speeds. But it will still be limited in a single stream iperf test. The 8200 has 8 cores and 8 queue capable NICs so running 8 parallel streams through it should use everything it's got. But it still has better single core performance too.
-
@stephenw10 I should have been clear that my iperf3 test was a 4 stream test (as there are 4 cores in the 6100), which was why I was somewhat disappointed in the results.
-
Hmm, then I would expect to see better then 1.5G upload from a 6100. Though I would still check the per core CPU usage in that scenario because iperf was deliberately designed to be a single thread process even when run with multiple streams. But I'm not sure that applies to the current version.
-
Just a note of encouragement. My 7100, which I believe is the same CPU as the 6100, is able to pass symmetrical 2.5Gb to the internet running the speedtest net program. I am mostly stock on the settings. I observed the CPUs where nearing 100% and the test is a multithreaded test.
I do not have the equipment to try 10Gb VLAN to VLAN.
-
@AndyRH Were you running the Speedtest program on the router itself (if so how?) or on a host in the LAN?
-
There is a speedtest_cli pkg you can install at the command line. But a better test is using speedtest from a LAN side client so the firewall is only routing and filtering.
-
@ChrisJenk said in Netgate 6100 / 25.07 - any recipes / guidelines for optimising high speed LAN and WAN connections?:
Speedtest program on the router itself
No, I ran it on a Windows computer connected at 2.5Gb. I got full line speed up and down. I have since changed my internet to 1Gb so I only get 1.2Gb up and down now.
A while back a friend and I were building and testing a VPN tunnel between us, a 7100 and a 6100, we found a noticeable speed difference if we used iperf on pfSense vs a computer on each end. We only get in the 700Mb/s range and still iperf on pfSense really added a load and skewed the results at least 10%.
