Load Balancing multi-gigabit ISP connections?
-
@heper
Thanks for your response!I'm using download manager to download multiple segments from a test download.
When we still have like 100 to 200 Mbps link from each ISP we subscribed, we are able to saturate the combined bandwidth. Now that we have 3 500 Mbps and 1 1Gbps link, we cannot get past the 700 Mbps speed.
My computer and network hopefully is not the issue as I can copy at around 700Megabytes from my computer to our remote web server as in the image below [We are now on a 10Gbps network backbone]:
Below is the download test and putty session while testing it:
-
Is it loadbalancing at all? Or is it getting that 55MB/s of a single wan?
-
Yes, I believe it is getting the download from all links except that it cannot saturate the bandwidth of any of the available WAN links.
-
Are the fibres from different ISP's ?
If not, your ISP is probably giving you 4 fibres using the same upstream OLT to feed your 4 links !
-
I am subscribed to two different ISPs right now and in the process of adding our third ISP next year. Hopefully I'll be able to figure out what could be wrong or what needs to be tweaked in my setup.
-
I would suggest testing each WAN separately one at a time using Public iperf servers & a decent client PC connected directly to each ISP router.
For the load balancing you will either need a high spec client with 10Gig interface to PFsense, or multiple 1G client PC's. Either way you will need multiple iperf sessions, with each having multiple processes to properly work across the load balancer.
Also try a selection of internet speedtest sites through your gateway group, but make sure you use server that properly support multiple threads. I find https://speedsmart.net seems to work better than most, and doesn't clutter your browser with advertising.
-
Hi!
I've already tested each link and verified [I only used my download manager to test] that all the ISP connections are working when used stand-alone outside of pfsense.
My ultimate goal is to load balance all the connections and to saturate all the links when possible. It seems like the most I can get is 700 Mbps. If I can achieve at least 1.5Gbps combined, that would be great.
Could it be my hardware that's limiting it?
I installed it virtually before and thought that was the problem, then, I did bare metal install of PFSense which is my current setup right now and did not see any improvement when it comes to throughput.
Thanks everyone for your time trying to help.
-
@eap2018 Try doing upstream & downstream iperf tests from Client PC to PfSense. This would at least prove you can get >> 1G locally.
Also, do you know for certain that the download sites you are using can support >1Gig ? Many site limit individual connections to avoid server overloading.
I have used these Public Iperf in the past to test Docsis channel bonding for >> 3Gig.
-
Here's my client to pfsense IPERF test result:
c:\portable\iperf>iperf3 -c 172.27.7.7 -w 512k
Connecting to host 172.27.7.7, port 5201
[ 4] local 172.27.0.13 port 7988 connected to 172.27.7.7 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 661 MBytes 5.55 Gbits/sec
[ 4] 1.00-2.00 sec 648 MBytes 5.43 Gbits/sec
[ 4] 2.00-3.00 sec 701 MBytes 5.88 Gbits/sec
[ 4] 3.00-4.00 sec 660 MBytes 5.54 Gbits/sec
[ 4] 4.00-5.00 sec 740 MBytes 6.20 Gbits/sec
[ 4] 5.00-6.00 sec 713 MBytes 5.98 Gbits/sec
[ 4] 6.00-7.00 sec 676 MBytes 5.67 Gbits/sec
[ 4] 7.00-8.00 sec 661 MBytes 5.55 Gbits/sec
[ 4] 8.00-9.00 sec 754 MBytes 6.32 Gbits/sec
[ 4] 9.00-10.00 sec 682 MBytes 5.72 Gbits/sec
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 6.73 GBytes 5.78 Gbits/sec sender
[ 4] 0.00-10.00 sec 6.73 GBytes 5.78 Gbits/sec receiveriperf Done.
-
Also check the reverse with pc as server & pfsense client. Then try public iperf servers.
-
Iperf from pfsense to client:
[2.4.5-RELEASE][root@gateway]/root: iperf3 -c 172.27.0.13 -w 512k
Connecting to host 172.27.0.13, port 5201
[ 5] local 172.27.7.7 port 19800 connected to 172.27.0.13 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 265 MBytes 2.23 Gbits/sec 0 513 KBytes
[ 5] 1.00-2.00 sec 220 MBytes 1.85 Gbits/sec 0 513 KBytes
[ 5] 2.00-3.00 sec 283 MBytes 2.37 Gbits/sec 1 299 KBytes
[ 5] 3.00-4.00 sec 276 MBytes 2.31 Gbits/sec 0 500 KBytes
[ 5] 4.00-5.00 sec 257 MBytes 2.16 Gbits/sec 0 513 KBytes
[ 5] 5.00-6.00 sec 235 MBytes 1.98 Gbits/sec 0 513 KBytes
[ 5] 6.00-7.00 sec 251 MBytes 2.10 Gbits/sec 0 513 KBytes
[ 5] 7.00-8.00 sec 272 MBytes 2.28 Gbits/sec 0 513 KBytes
[ 5] 8.00-9.00 sec 256 MBytes 2.14 Gbits/sec 1 458 KBytes
[ 5] 9.00-10.00 sec 284 MBytes 2.38 Gbits/sec 0 513 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.54 GBytes 2.18 Gbits/sec 2 sender
[ 5] 0.00-10.00 sec 2.04 GBytes 1.76 Gbits/sec receiveriperf Done.
-
So when traffic flows downstream from Pfsense you are only getting roughly 2Gbps, although with the command you are using iperf is running single thread with one TCP stream.
Try using "-P 5" to run 5 streams simultaneously. Then run multiple iperf sessions in different shell or cmd windows, using "-p port" so each iperf server session uses a different TCP port. Do the same on the client side. This way each session should use a different cpu core.
You should be able to achieve 6Gbps in both directions given your upload test earlier.
Finally when doing external test using public servers, you need to run client locally, but use the "-R" switch to force download, direction.
If you play with iperf3 a bit, you can get a much better idea of what is happening. Remember without the switches, "iperf3 -c" sends traffic up to the server.
-
If I can achieve at least 1.5 Gbps combined internet speed, I will be happy for now.
-
You might also want to try multiple client PC's simultaneously. This should utilise the gateway group more evenly.
-
Load balancing does not aggregate links into one. It distributes states among the available outbound connections.
Please see this thread:
https://forum.netgate.com/topic/110595/4-wan-pfsense-not-loadbalancing-accurately/
Takeaway there is it is almost impossible to see load balancing working with any sort of speed test. You need to throw lots of users and lots of states at the mechanism for it to really show what it can do. Expectations are often inaccurate.
Based on the basic throughputs you posted before, I would set the weights of the various gateways to 1 for the 500Mbit and 2 for the 1000 Mbit connections. That will mean the gig circuit will get 2 states for every 1 given to the 500M gateways.
-
@Derelict Yes I know, hence why I suggested multiple PC's with multiple iperf3 sessions running so the PF state counts mount up..
-
It probably takes more states than you are generating to actually see maximum on all links.
-
Hi All!
Just to give an update to this, I moved my setup to a newer beefy server and I am now able to download upto 170Megabytes per seconds.
I did not do anything special, I just migrated PFSense to our new beefy server as a virtual machine and now I'm very happy as ever.
Thank you all for responses!
Consider this solved until 10Gbps is available in our location, that is to another milestone.
