Load Balancing multi-gigabit ISP connections?
-
Here's my client to pfsense IPERF test result:
c:\portable\iperf>iperf3 -c 172.27.7.7 -w 512k
Connecting to host 172.27.7.7, port 5201
[ 4] local 172.27.0.13 port 7988 connected to 172.27.7.7 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 661 MBytes 5.55 Gbits/sec
[ 4] 1.00-2.00 sec 648 MBytes 5.43 Gbits/sec
[ 4] 2.00-3.00 sec 701 MBytes 5.88 Gbits/sec
[ 4] 3.00-4.00 sec 660 MBytes 5.54 Gbits/sec
[ 4] 4.00-5.00 sec 740 MBytes 6.20 Gbits/sec
[ 4] 5.00-6.00 sec 713 MBytes 5.98 Gbits/sec
[ 4] 6.00-7.00 sec 676 MBytes 5.67 Gbits/sec
[ 4] 7.00-8.00 sec 661 MBytes 5.55 Gbits/sec
[ 4] 8.00-9.00 sec 754 MBytes 6.32 Gbits/sec
[ 4] 9.00-10.00 sec 682 MBytes 5.72 Gbits/sec
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 6.73 GBytes 5.78 Gbits/sec sender
[ 4] 0.00-10.00 sec 6.73 GBytes 5.78 Gbits/sec receiveriperf Done.
-
Also check the reverse with pc as server & pfsense client. Then try public iperf servers.
-
Iperf from pfsense to client:
[2.4.5-RELEASE][root@gateway]/root: iperf3 -c 172.27.0.13 -w 512k
Connecting to host 172.27.0.13, port 5201
[ 5] local 172.27.7.7 port 19800 connected to 172.27.0.13 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 265 MBytes 2.23 Gbits/sec 0 513 KBytes
[ 5] 1.00-2.00 sec 220 MBytes 1.85 Gbits/sec 0 513 KBytes
[ 5] 2.00-3.00 sec 283 MBytes 2.37 Gbits/sec 1 299 KBytes
[ 5] 3.00-4.00 sec 276 MBytes 2.31 Gbits/sec 0 500 KBytes
[ 5] 4.00-5.00 sec 257 MBytes 2.16 Gbits/sec 0 513 KBytes
[ 5] 5.00-6.00 sec 235 MBytes 1.98 Gbits/sec 0 513 KBytes
[ 5] 6.00-7.00 sec 251 MBytes 2.10 Gbits/sec 0 513 KBytes
[ 5] 7.00-8.00 sec 272 MBytes 2.28 Gbits/sec 0 513 KBytes
[ 5] 8.00-9.00 sec 256 MBytes 2.14 Gbits/sec 1 458 KBytes
[ 5] 9.00-10.00 sec 284 MBytes 2.38 Gbits/sec 0 513 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.54 GBytes 2.18 Gbits/sec 2 sender
[ 5] 0.00-10.00 sec 2.04 GBytes 1.76 Gbits/sec receiveriperf Done.
-
So when traffic flows downstream from Pfsense you are only getting roughly 2Gbps, although with the command you are using iperf is running single thread with one TCP stream.
Try using "-P 5" to run 5 streams simultaneously. Then run multiple iperf sessions in different shell or cmd windows, using "-p port" so each iperf server session uses a different TCP port. Do the same on the client side. This way each session should use a different cpu core.
You should be able to achieve 6Gbps in both directions given your upload test earlier.
Finally when doing external test using public servers, you need to run client locally, but use the "-R" switch to force download, direction.
If you play with iperf3 a bit, you can get a much better idea of what is happening. Remember without the switches, "iperf3 -c" sends traffic up to the server.
-
If I can achieve at least 1.5 Gbps combined internet speed, I will be happy for now.
-
You might also want to try multiple client PC's simultaneously. This should utilise the gateway group more evenly.
-
Load balancing does not aggregate links into one. It distributes states among the available outbound connections.
Please see this thread:
https://forum.netgate.com/topic/110595/4-wan-pfsense-not-loadbalancing-accurately/
Takeaway there is it is almost impossible to see load balancing working with any sort of speed test. You need to throw lots of users and lots of states at the mechanism for it to really show what it can do. Expectations are often inaccurate.
Based on the basic throughputs you posted before, I would set the weights of the various gateways to 1 for the 500Mbit and 2 for the 1000 Mbit connections. That will mean the gig circuit will get 2 states for every 1 given to the 500M gateways.
-
@Derelict Yes I know, hence why I suggested multiple PC's with multiple iperf3 sessions running so the PF state counts mount up..
-
It probably takes more states than you are generating to actually see maximum on all links.
-
Hi All!
Just to give an update to this, I moved my setup to a newer beefy server and I am now able to download upto 170Megabytes per seconds.
I did not do anything special, I just migrated PFSense to our new beefy server as a virtual machine and now I'm very happy as ever.
Thank you all for responses!
Consider this solved until 10Gbps is available in our location, that is to another milestone.
