Load Balancing multi-gigabit ISP connections?

eap2018 · Oct 12, 2020, 11:21 AM

Hi!

I am trying to load balance multiple fiber connections but I can't seem to achieve even 1 full Gigabit internet speed.

DNS Server Settings:
login-to-view

I'm connected alone to the LAN172 with 10Gbase-T connection.

The best speed I can get through my PFSENSE setup is 68 Megabytes per seconds but I can get more speed on a single 1000 Mbps link, directly connected via PPPoE.

The server specs are as follows:
CPU: Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz
4 CPUs: 1 package(s) x 4 core(s)

Memory: 32G

Network Cards:
1 x Intel I350-T4 [WAN connections]
1 x Intel X540-T2 dual-port 10GbE [LAN Connections]

I've already tried the following tweaks in /boot/loader.conf.local but no significant improvements except that it has increased the CPU usage

hw.igb.num_queues=1
hw.ix.num_queues=3
hw.pci.enable_msix=0
hw.pci.enable_msi=0
hw.pci.honor_msi_blacklist=0
net.isr.maxthreads=3
net.isr.numthreads=3
net.isr.dispatch=deferred
net.inet.tcp.tso=0
hw.ix.flow_control=0
net.isr.bindthreads=1

Any tweaks I should do to realize at least close to 2500 Mbps speed?

Thanks in advance!

heper · Oct 13, 2020, 3:21 AM

i suggest you remove everything from that list except net.isr.dispatch

you will never get more than 1GBit on a single stream.
What are you using to test your speed?
could you provide a screenshot of 'top' during a test?

eap2018 · Oct 13, 2020, 3:21 AM

@heper
Thanks for your response!

I'm using download manager to download multiple segments from a test download.

When we still have like 100 to 200 Mbps link from each ISP we subscribed, we are able to saturate the combined bandwidth. Now that we have 3 500 Mbps and 1 1Gbps link, we cannot get past the 700 Mbps speed.

My computer and network hopefully is not the issue as I can copy at around 700Megabytes from my computer to our remote web server as in the image below [We are now on a 10Gbps network backbone]:
login-to-view

Below is the download test and putty session while testing it:
login-to-view

heper · Oct 13, 2020, 6:11 AM

Is it loadbalancing at all? Or is it getting that 55MB/s of a single wan?

eap2018 · Oct 14, 2020, 1:14 AM

Yes, I believe it is getting the download from all links except that it cannot saturate the bandwidth of any of the available WAN links.
login-to-view

pwood999 · Oct 15, 2020, 2:41 PM

Are the fibres from different ISP's ?

If not, your ISP is probably giving you 4 fibres using the same upstream OLT to feed your 4 links !

eap2018 · Oct 16, 2020, 4:44 AM

I am subscribed to two different ISPs right now and in the process of adding our third ISP next year. Hopefully I'll be able to figure out what could be wrong or what needs to be tweaked in my setup.

pwood999 · Oct 16, 2020, 10:30 AM

I would suggest testing each WAN separately one at a time using Public iperf servers & a decent client PC connected directly to each ISP router.

For the load balancing you will either need a high spec client with 10Gig interface to PFsense, or multiple 1G client PC's. Either way you will need multiple iperf sessions, with each having multiple processes to properly work across the load balancer.

Also try a selection of internet speedtest sites through your gateway group, but make sure you use server that properly support multiple threads. I find https://speedsmart.net seems to work better than most, and doesn't clutter your browser with advertising.

eap2018 · Oct 19, 2020, 1:10 AM

Hi!

I've already tested each link and verified [I only used my download manager to test] that all the ISP connections are working when used stand-alone outside of pfsense.

My ultimate goal is to load balance all the connections and to saturate all the links when possible. It seems like the most I can get is 700 Mbps. If I can achieve at least 1.5Gbps combined, that would be great.

Could it be my hardware that's limiting it?

I installed it virtually before and thought that was the problem, then, I did bare metal install of PFSense which is my current setup right now and did not see any improvement when it comes to throughput.

Thanks everyone for your time trying to help.

pwood999 · Oct 19, 2020, 1:23 PM

@eap2018 Try doing upstream & downstream iperf tests from Client PC to PfSense. This would at least prove you can get >> 1G locally.

Also, do you know for certain that the download sites you are using can support >1Gig ? Many site limit individual connections to avoid server overloading.

I have used these Public Iperf in the past to test Docsis channel bonding for >> 3Gig.

eap2018 · Oct 22, 2020, 2:37 AM

Here's my client to pfsense IPERF test result:

![0_1603334209322_da738b45-7028-43be-8f9d-766c48d7e4e9-image.png](Uploading 100%)

c:\portable\iperf>iperf3 -c 172.27.7.7 -w 512k
Connecting to host 172.27.7.7, port 5201
[ 4] local 172.27.0.13 port 7988 connected to 172.27.7.7 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 661 MBytes 5.55 Gbits/sec
[ 4] 1.00-2.00 sec 648 MBytes 5.43 Gbits/sec
[ 4] 2.00-3.00 sec 701 MBytes 5.88 Gbits/sec
[ 4] 3.00-4.00 sec 660 MBytes 5.54 Gbits/sec
[ 4] 4.00-5.00 sec 740 MBytes 6.20 Gbits/sec
[ 4] 5.00-6.00 sec 713 MBytes 5.98 Gbits/sec
[ 4] 6.00-7.00 sec 676 MBytes 5.67 Gbits/sec
[ 4] 7.00-8.00 sec 661 MBytes 5.55 Gbits/sec
[ 4] 8.00-9.00 sec 754 MBytes 6.32 Gbits/sec
[ 4] 9.00-10.00 sec 682 MBytes 5.72 Gbits/sec

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 6.73 GBytes 5.78 Gbits/sec sender
[ 4] 0.00-10.00 sec 6.73 GBytes 5.78 Gbits/sec receiver

iperf Done.

pwood999 · Oct 22, 2020, 7:43 AM

Also check the reverse with pc as server & pfsense client. Then try public iperf servers.

eap2018 · Oct 22, 2020, 9:00 AM

Iperf from pfsense to client:

[2.4.5-RELEASE][root@gateway]/root: iperf3 -c 172.27.0.13 -w 512k
Connecting to host 172.27.0.13, port 5201
[ 5] local 172.27.7.7 port 19800 connected to 172.27.0.13 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 265 MBytes 2.23 Gbits/sec 0 513 KBytes
[ 5] 1.00-2.00 sec 220 MBytes 1.85 Gbits/sec 0 513 KBytes
[ 5] 2.00-3.00 sec 283 MBytes 2.37 Gbits/sec 1 299 KBytes
[ 5] 3.00-4.00 sec 276 MBytes 2.31 Gbits/sec 0 500 KBytes
[ 5] 4.00-5.00 sec 257 MBytes 2.16 Gbits/sec 0 513 KBytes
[ 5] 5.00-6.00 sec 235 MBytes 1.98 Gbits/sec 0 513 KBytes
[ 5] 6.00-7.00 sec 251 MBytes 2.10 Gbits/sec 0 513 KBytes
[ 5] 7.00-8.00 sec 272 MBytes 2.28 Gbits/sec 0 513 KBytes
[ 5] 8.00-9.00 sec 256 MBytes 2.14 Gbits/sec 1 458 KBytes
[ 5] 9.00-10.00 sec 284 MBytes 2.38 Gbits/sec 0 513 KBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.54 GBytes 2.18 Gbits/sec 2 sender
[ 5] 0.00-10.00 sec 2.04 GBytes 1.76 Gbits/sec receiver

iperf Done.

pwood999 · Oct 22, 2020, 9:43 AM

So when traffic flows downstream from Pfsense you are only getting roughly 2Gbps, although with the command you are using iperf is running single thread with one TCP stream.

Try using "-P 5" to run 5 streams simultaneously. Then run multiple iperf sessions in different shell or cmd windows, using "-p port" so each iperf server session uses a different TCP port. Do the same on the client side. This way each session should use a different cpu core.

You should be able to achieve 6Gbps in both directions given your upload test earlier.

Finally when doing external test using public servers, you need to run client locally, but use the "-R" switch to force download, direction.

If you play with iperf3 a bit, you can get a much better idea of what is happening. Remember without the switches, "iperf3 -c" sends traffic up to the server.

eap2018 · Oct 22, 2020, 10:24 AM

If I can achieve at least 1.5 Gbps combined internet speed, I will be happy for now.

pwood999 · Oct 22, 2020, 10:30 AM

You might also want to try multiple client PC's simultaneously. This should utilise the gateway group more evenly.

Derelict · Oct 22, 2020, 1:11 PM

Load balancing does not aggregate links into one. It distributes states among the available outbound connections.

Please see this thread:

https://forum.netgate.com/topic/110595/4-wan-pfsense-not-loadbalancing-accurately/

Takeaway there is it is almost impossible to see load balancing working with any sort of speed test. You need to throw lots of users and lots of states at the mechanism for it to really show what it can do. Expectations are often inaccurate.

Based on the basic throughputs you posted before, I would set the weights of the various gateways to 1 for the 500Mbit and 2 for the 1000 Mbit connections. That will mean the gig circuit will get 2 states for every 1 given to the 500M gateways.

pwood999 · Oct 22, 2020, 1:59 PM

@Derelict Yes I know, hence why I suggested multiple PC's with multiple iperf3 sessions running so the PF state counts mount up..

Derelict · Oct 22, 2020, 2:58 PM

It probably takes more states than you are generating to actually see maximum on all links.

eap2018 · Nov 6, 2020, 4:37 AM

Hi All!

Just to give an update to this, I moved my setup to a newer beefy server and I am now able to download upto 170Megabytes per seconds.

I did not do anything special, I just migrated PFSense to our new beefy server as a virtual machine and now I'm very happy as ever.

login-to-view

Thank you all for responses!

Consider this solved until 10Gbps is available in our location, that is to another milestone.