How to saturate 2 ISP links if ethernet ports are all 1Gbps
-
So I have two ISP's:
ISP1 - 800Mbps down/up
ISP2 - 600Mbps down/upMy pfsense box has 6 x 1Gbps ports and a load balance gateway group is setup. I have a LAG (LACP) that includes 2 of those ports and both are connected to a Ubiquiti 24 x 1Gbps ports switch. The load balancing works just fine and mutli-connection sessions from my clients get distributed to both ISP's but is only limited to around 950Mbps because of the physical port bandwidth limit.
One of my Windows desktop clients has two Realtek ethernet ports. 1 x 2.5Gbps and 1 x 1Gbps. Just for testing, I connected both ISP modems to each of those ports and did a test in speedtest.net. Surprisingly, it was able to reach the aggregate speed of 1.4Gbps (800 + 600 Mbps) without any special configuration. I'm assuming this is because the modems are on two different LAN networks so it provides two gateways to the client and that Windows automatically load balance between those two gateways or something. Is that even accurate?
With the same client connected to my Ubiquiti switch, they both are given IP's in the same network, of course, so not sure how this pans out. Do I need to configure my client to aggregate those two ports with the LACP protocol as well? Will this make multi-connection sessions reach 1.4Gbps Internet speed? I'm not sure how to do LAG with Realtek NIC's so I have to read up on this.
-
@kevindd992002 You could policy route some of the IP addresses or ports out the other interfaces.
-
@nogbadthebad said in How to saturate 2 ISP links if ethernet ports are all 1Gbps:
@kevindd992002 You could policy route some of the IP addresses or ports out the other interfaces.
But doesn't that fix the issue if it is between pfsense and the ISP modems? That is not my issue here because the load balance policy route I have for my LAN is working properly.
My problem is the bandwidth between the client and pfsense as it is limited to 1Gbps. So for example, when I do a multi connection download, it gets 600Mbs from ISP1 and 350Mbps from ISP2 (a total of 950Mbps to saturate a 1Gbps link).
-
@kevindd992002 Bandwidth between a client and the router will only flow over one of the two interlinks, regardless of the network card in the client and total bandwidth of the LACP uplink to the router, its the way LACP works.
You'd only see the benefit with multiple clients.
What exactly do you mean by "With the same client connected to my Ubiquiti switch, they both are given IP's in the same network"
-
@nogbadthebad said in How to saturate 2 ISP links if ethernet ports are all 1Gbps:
@kevindd992002 Bandwidth between a client and the router will only flow over one of the two interlinks, regardless of the network card in the client and total bandwidth of the LACP uplink to the router, its the way LACP works.
You'd only see the benefit with multiple clients.
That's what I understand about LACP too. However, won't mutiple-connection download try to use both NIC members in the team?
What exactly do you mean by "With the same client connected to my Ubiquiti switch, they both are given IP's in the same network"
What I mean by this is that with one flat subnet, when you connect both NICs of the client to that subnet (through the switch), they both get assigned IP's (and gateways) from the same subnet of course. So when this happens, the routing table of the client lists two exact same gateway IP's with the same metric but still only uses one when I do a multiple-connection download.
Since I have proven from my previous test that Windows load balances the two NIC's (no teaming involved) if they are on different subnets, what I did as a workaround was to create a new VLAN subnet in pfsense and in the switch and connected the 2nd NIC of that client to that new subnet. This worked as expected. The client now has two gateway IP's (192.168.110.1 and 192.168.111.1) with the same metric (25). So when I do a multi-connection download or speedtest, it now uses both NICs and saturates both connections.
However, I feel that this is a dirty workaround. It is working but I was expecting LACP to have the same effect, no?
-
@kevindd992002 Traffic from one host ( PC ) to another ( router ) will only flow over one link.
Suggest you look at the LACP documentation:-
https://docs.netgate.com/pfsense/en/latest/interfaces/lagg.html
"Traffic is balanced between all ports on the LAG, however, for communication between two single hosts it will only use one single port at a time because the client will only talk to one MAC address at a time. For multiple connections through multiple devices, this limitation effectively becomes irrelevant. The limitation is also not relevant for failover."
"Using a LAGG does not necessarily guarantee full throughput equal to the sum of all interfaces. In particular, a single flow will not exceed the throughput of a LAGG member interface. Traffic on a LAGG is hashed in such a way that flows between two hosts, such as this firewall and an upstream gateway, would only use a single link since the flow is between a single MAC address on each side.
In networks where many hosts communicate with different MAC addresses, the usage can approach the sum of all interfaces in the LAGG."
Your previous test is irrelevant if you connected the gateway devices directly to the 2.5 and 1 Gbps prots on the client.
