What is the maximum throughput my setup can do ?
-
Hello.
I run pfSense on a Cisco UCS C210 M2 with 2x x5650 and 24G RAM... All in BIOS is set for throughput... When I do test with speedtest.net the maximum I get is only about 7.5 - 8 Gbit/s and CPU Usage never pass 5%... Seems machine can do more, but maybe Windows causing that maximum... How much do you think is the maximum bandwidth for this setup ? I do consider about getting better machine in the near future, but for now I want to be sure what can I expect from what i have :)
Thank you :)
-
Testing against speedtest for a 10G connection is probably not that accurate. Have you proved that you can get 10Gbps across that link?
Those CPUs are 6 cores and will appear as 12 if you have hyperthreading enabled so 24 total. That means that 5% CPU could be one core at 100%. Run
top -aSHduring the test to see the CPU usage per core breakdown.8Gbps is pretty good though for any pfSense install.
Steve
-
@stephenw10 said in What is the maximum throughput my setup can do ?:
Testing against speedtest for a 10G connection is probably not that accurate. Have you proved that you can get 10Gbps across that link?
Those CPUs are 6 cores and will appear as 12 if you have hyperthreading enabled so 24 total. That means that 5% CPU could be one core at 100%. Run
top -aSHduring the test to see the CPU usage per core breakdown.8Gbps is pretty good though for any pfSense install.
Steve
Thank you for answer. I have 2x 10G from separate ISP... During the test they are both loaded, not equal, sometimes one takes the load and sometimes another usually 5G by 3G and opposite... When i test with only one of them i get 8G as well, so i assume both are capable for such bandwidth... I mean there aren't any kind of shaper or such thing... I have disabled hyperthreading because i read somewhere it should be better without especially for router, firewall and any kind of traffic mover... I did test now like you say and
top -aSHshow one half of cores at 98%, 96%, 99%, etc... Other half remain at 100% idle... I also have NAT on this machine witch i think is more power hungry than direct routing without PF and NAT...Seems i had to find a way to do couple of tests simultaneously and then we'll see... I forgot to mention both 10G lines are on separate hardware ports to avoid bottleneck... LAN side going trough 2 ports as well...
-
Are you using ix NICs? I believe those would use 4 queues by default so if you have enough in bound and outbound connections I would expect to see enough queues available to load all the cores. We'd need to see the top output.
I would try to use iperf3 to test it if you can. You can set the number of processes etc.
Steve
-
@stephenw10 said in What is the maximum throughput my setup can do ?:
Are you using ix NICs? I believe those would use 4 queues by default so if you have enough in bound and outbound connections I would expect to see enough queues available to load all the cores. We'd need to see the top output.
I would try to use iperf3 to test it if you can. You can set the number of processes etc.
Steve
With iperf I get less result than speedtest.net even with multiple threads and UDP instead of TCP (TCP seems to be slower on iperf)... What about ix NICs ? I use QLogic Broadcom dual port and Chelsio S310E single port... Is there any option to choose how work is spread on CPUs ?
-
There may well be variables for those drivers that can be tweaked.
How many queues do you see for each driver?
Do you have multiple queues on cores that could be better spread?Mostly the defaults are pretty good at taking care of that but it's worth checking.
Steve
-
Check your ISP latency - this can impact what you measure from Internet iperf & speedtest sites.
If you want to measure your pfsense install properly put 10G iperf servers on both sides and try again. You can also do direct 10G between client-server to check the performance without pfsense.
-
I cannot complain about my ISP latency... For measuring with iperf I use servers from here : https://iperf.cc/
-
How are you actually running the test? You would need something internally connected with a >10G connection to test throughput really.
I suspect you are hitting a limit there since both WANs can pass at least 5Gbps that you've seen but not at the same time.Can you test each WAN directly from a client? Can you get 10Gbps to those iperf servers that way?
Steve
-
I want to test speeds when I go trough firewall instead of to firewall... That's why I do test to some points outside my network... Maybe I can setup another machine to put it on the other side of firewall, but I don't have second machine that fast as my main PC... My PC is with i7-5960x but it is only one I have with such power...
-
I agree with #stephenw10 in that you should confirm the WAN speeds first. Only then can you check if your PfSense can pass that level of traffic.
FYI, for reference, I did some testing on ESXi today using a couple of virtual switches inside:
Test 1: Windows 7 (4 core 4GB) --> Virtual switch --> Centos 7.5 (4 core 4GB)
Iperf3 gives me 13Gbps across the v-switch. (therefore the Iperf client & server Virt-machines have plenty of CPU & RAM.Test 2: Windows 7 (4 core 8GB) --> Virt-SW-1 --> PfSense (4 core 4GB) --> Virt-Sw-2 --> Centos 7.5 (4 core 4GB)
Iperf3 now only gives 2.5Gbps E2E through PfSense even with 4 Xeon & 8GB ram assigned to PfSense. Tried this with both E1000 and VMX3 virtual NICs but result is the same.Now I'm wondering what I need to tweak inside PF to get better throughput, or if this is a limitation of PF in ESXi environment ?
