How to load balance multiple OpenVPN clients

MrHorizontal

A. Preliminary work
1. Firstly, read through part B to figure out what you have to find out, especially B.6 to figure out which servers you want to load balance against.
2. Then read through part A in How to create an OpenVPN client to a public OpenVPN provider about caveats choosing OpenVPN providers.
3. Create all the clients to each server individually by following the same howto.

B. Research
As a general overview about what this howto does, you must understand that since OpenVPN does a lot of 'magic' to ensure traffic flows to the VPN connection and for ease-of-use for clients connecting to OVPN servers, when it's in use on a routing platform like pfSense, it's absolute DEATH to a router, by properly screwing around with pfSense's routing table. The culprit of course is the 'redirect-gateway def1' command in OpenVPN that most servers will push to clients (and a very worthwhile command for single-connection OVPN WANs). If you followed the setup for an individual connection, right now your pfSense setup should be routing all traffic to the last connection that was brought up by OpenVPN, and pfSense itself is totally oblivious to what the heck is going on. The majority of this howto then, is getting OpenVPN and pfSense to play nice with each other, and informing pfSense about what to do with all these wonderful new connections.

The other important consideration with the load balanced set up is that traffic originating from the pfSense box will not go through the load balancer - only traffic from behind the router, accessed via the LAN interface will be. Take note when considering DNS, NTP and such like traffic that pfSense also handles (I mention this a little in section G below anyway).

Before we do this, you'll need to gather a few bits of information.

1. Once all the clients are created and connecting, pfSense should be routing the OpenVPN server that was last to connect, and maintaining connections to all the others, but not routing across all of them. In this howto, we figure out what to do to use pfSense's other features combined with them. My Status -> OpenVPN screen looks like this:

2. Next the routing table will be a mess, and with 4 connections it will look like this:

3. The important thing is that each OVPN client has its own gateway, and each connection and gateway are operating on a different subnet. We need to figure out what these gateways are, so pfSense can manage them. If you still have 'verb 5' in the advanced box of the client configuration, you will get the 'PUSH_REPLY' message in the log that will look similar to this:

openvpn[49520]: PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1,dhcp-option DNS 195.24.72.6,dhcp-option DNS 83.243.8.6,dhcp-option DNS 4.2.2.4,route 10.0.61.1,topology net30,ifconfig 10.0.61.54 10.0.61.53'

In this case since the connection is 10.0.61.50 and the gateway is 10.0.61.1, and the topology is net30, we can can also deduce the network is 10.0.61.0/24. Find the gateways with all the other connections as well and note them down. The absolutely critical bit of info you need here is the gateway and to be able to assign an arbitrary IP in the network. Notice how the server is pushing the controversial 'redirect-gateway def1' as well.
4. Now go back to VPN -> OpenVPN -> Client tab
5. Note down each connection, in the order they are in (which is usually the order you created them in). Each connection creates a new device called 'ovpncX', where 'X' is the number corresponding to the connection, which you'll need in order to assign the interfaces.
6. You also need to figure out how you want your load balancer to work. Find out which server has the lowest ping times, by pinging each and seeing the latency. In my case, Perfect Privacy's Amsterdam, Rotterdam and Steinsel servers are all very fast for me, but Erfurt has good throughput with slightly higher latency. So in my configuration, I load balance the first 3 as Tier 1 connections and Erfurt as Tier 2. Your setup will be different depending on what the best servers you're using are.

NB: Some servers use DNS round-robin to balance across multiple servers. In my case, Steinsel is a cluster of 8 machines, Amsterdam has 2, Rotterdam 4 and Erfurt 5. I'm afraid we can't use all of them and load balance between 19 servers, because each connection must have a different subnet, so I can only do it against 4. In any case, the provider has done this to increase capacity, but both their server and your pfSense box might begin to struggle under the processing weight of encrypting and compressing 19 tunnels, so be conscientious. 4 is fine as it's spreading your traffic over different servers rather than saturating one server while keeping CPU usage in check is what the ethos of load balancing is all about, so it's a good thing.

Now lets get on to load balancing OpenVPN.

C. Overriding the OpenVPN connection routing
As mentioned above 'redirect-gateway def1' is death to pfSense for multiple connections. We need to stop OpenVPN from doing this and behave properly.

1. Navigate to VPN -> OpenVPN -> Client tab.
2. In each client, click the 'e' button to edit it and go to the big 'Advanced' field. Remember, each command is separated by a ';' here:

Add 'route-nopull'. This instructs OpenVPN to set up the interface using ifconfig with the dynamic IP and do nothing else.
Add 'route 10.0.61.1', where '10.0.61.1' is the gateway as you found out in B.3 This instructs OpenVPN to set up the route to the gateway.
3. Press Save
4. Repeat for all the connections, changing the gateway IP in the 'route X.X.X.X' command appropriate for each connection as you discovered in B.3.

After this, your routing table in Diagnostics -> Routes should look something like this, each connection with a clearly explicit path and multiple gateways. If not, you'll have to read the OpenVPN docs and ensure it's looking like this:

D. Setting up the Interfaces
Now we need tell pfSense to assign these connections as Interfaces and their respective gateways.

NB: pfSense doesn't like dynamic OVPN connections and won't pick up the dynamic gateway when you assign an OVPN connection as an interface. Since pfSense obviously loads its interfaces before it brings up OpenVPN connections at boot time, you can assign the interface and provide it with a static IP address in the same subnet as the connection, but what matters is we set up the gateway for the LB and mapped to the interface so pfSense has a vague idea of what the heck is going on. When OpenVPN then fires up the connections, it will override this static IP address with the correct one, and functionality of pfSense is totally unaffected. Be warned however, that if something else is referencing this defined IP, it'll probably break since it's fake, so it's best to read the IP from the routing table instead.

1. Navigate to Interfaces -> (assign)
2. Click the '+' button for as many OVPN connections as you have created. pfSense should automatically select the next available interface, but ensure the table looks like this, with the 'ovpncX' Network port corresponding to the connection, and 'X' being the same number as the order you noted down in B.5.
3. Select Interfaces -> OPT1
4. Check 'Enable Interface'
5. Enter a proper description like 'VPNServerLocation', for example 'VPNSteinsel' for me.
6. Now pfSense doesn't really have support for using OVPN interfaces, so we have to bend the rules a bit to convince pfSense into using the OVPN connection.
7. Under 'Type', select 'Static', even though the IP is dynamically provided by the OVPN server.
8. Under 'IP Address', enter any arbitrary IP within the same subnet as the connection, BUT NOT THE GATEWAY IP! Since my connection's subnet is 10.0.61.0/24, and the gateway is 10.0.61.1, I put in 10.0.61.240. OpenVPN will override this, don't worry, but pfSense will throw an error if there's no IP address in a static interface.
9. Under 'Gateway', click the little link saying 'add a new one'
10. Keep 'Default Gateway' Empty.
11. Enter 'GWServerLocation' for the Gateway Name (in my case 'GWSteinsel').
12. Enter the gateway IP as you figured out in step 4, in this case 10.0.61.1.
13. Press 'Save Gateway' - the dialog box should disappear and 'GWSteinsel' should be selected.
14. Because the VPN connection is probably using a private IP range (in my case 10.0.0.0/8), leave 'Block private networks' unchecked.
15. Check 'Block bogon networks' and press Save.
16. Repeat with all the other connections, entering info specific to each connection in each interface.
17. Navigate to System -> Routing. The Gateways should look something like this:

E. Setting up the Load Balancer
With all your gateways and interfaces set up, we just need to define the group of them to load balance.

1. Navigate to System -> Routing -> Groups tab
2. Click the '+' button to add a new group.
3. Enter a good name for your group. I use 'EUGateways' for mine.
4. In Gateway Priority, select each gateway according to what you figure out in B.6 as the lowest latency tiers. In my case, GWAmsterdam, GWRotterdam and GWSteinsel are all Tier 1 and GWErfurt is Tier 2.
5. Under 'Trigger level', select 'Packet Loss or High Latency' and press Save.
6. Navigate to System -> Routing -> Settings
7. (Optional) Depending on the ping times you found in B.6, you may want to tailor the 'Latency thresholds' to be a little closer to the actual ones so the load balancer is quite responsive, but give a fair leeway of about 30-50% above the average ping times.

F. Setting up your Firewall Rules to use the Load Balancer

1. Navigate to Firewall -> Rules
2. Select 'LAN' from the 'Currently viewing' dropdown.
3. Click '+' to add a new rule:
4. Leave all options as default, except:

Under 'Protocol' select 'any'
(Optional) Under 'Description' enter 'Route traffic through VPN Load Balancer'
Under 'Advanced features' and 'Gateway', click the 'Advanced' button.
In the dropdown that appears, select the Gateway Group you named, in my case 'EUGateways'
5. Press Save.
6. Ensure this rule is below any other firewall rules you may have.
7. Apply changes.

G. Tidy up (all optional)

a. If you have latency-sensitive data that you don't mind being unencrypted such as NTP and VoIP/SIP add some firewall rules above the load balancer catch-all. Be sure to only allow the traffic on the service ports, ie 123 for NTP and 5060 for SIP.
b. It's advisable to keep 'sticky connections' with a load balancer, as connections may come from different IP's and some servers don't like this. Sticky connections are under System -> Advanced -> Miscellaneous tab
c. Encrypt DNS! The forwarding DNS servers can still see what you're doing, which is a fat lot of good since we're supposed to be being private from the likes of Google, OpenDNS and your ISP. Under System -> General setup you'll find a dropdown to select the gateway for each DNS server to use. Unfortunately the list doesn't contain the load balancer virtual gateway, but instead an individual OVPN interface, so select each DNS server using a different OVPN interface.
d. Because we didn't check 'Use private networks', you should add a rule allowing anything from the GW subnet of each OVPN connection to your pfSense box only and then a second rule blocking absolutely everything below it.
e. A lot of providers also have proxy servers, so you may want to consider putting a proxy server that strips headers, and use that resource as well for more acceleration. Be sure to use the OVPN gateways instead of the public addresses for the uplink peers if on the same box (but you need to ensure squid is set to be anonymous too!).

And that's it. You now have multiple OVPN connections being load balanced by pfSense, and you get a small glimpse into the sheer power this wonderful firewall has. Thank the wonderful devs for that!

I went a step further and added all UK and US subnets as aliases (beware of the 3000-network limit per alias), and then set up 2 further Load Balanced gateway groups, and routing all US traffic to load balanced US servers, and all UK traffic to load balanced UK servers. In either case, you just repeat instructions in this howto to set up even more OVPN interfaces, and load balance groups, and simply adjust your firewall rules accordingly.

Dev notes:
1. OpenVPN provides all of the dynamic IPs, gateways and so forth as environment variables when it sets up the routes. pfSense already executes the '/etc/rc.filter_configure' script after OVPN has done its thing, could this script be modified to automatically detect routing changes and set up a dynamic gateway when an ovpncX interface is assigned?
2. When an ovpncX interface is assigned, can you have a special 'Type' for 'OpenVPN' only, that allows for an IP address if configured and the dynamic gateway to be created, DHCP-style, but following the rules as specified in the VPN client config?
3. In this set up, the only issue that is otherwise a flawless pfSense performance, is the FTP proxy somehow panics pfSense, as noted here. I've sent the 'dt' and config.xml as would have been set up here to Ermal via PM, so I presume he's already on the case.
4. I think you're already on the case, but having load balancers care for different speed connections would be an important consideration.
5. For existing connections, can we have a place that shows the name of the device associated for the connection, ie Device: 'ovpnc1'.

ovpn_status.gif_thumb

routing_mess.gif_thumb

routing_sorted.gif_thumb

gateways_sorted.gif_thumb

eri--

If you find out how to get DNS info on openvpn up script than you will be set.
https://rcs.pfsense.org/projects/pfsense/repos/mainline/commits/8d964cea2ce977423f14cb24b4115a680c985ac7
This commit removes some of your headaches here.

winge

I almost followed the guideline above to have my openvpn connection as an interface. The descripancies I found was:

Change the "route-nopull" option to "route-noexec" and then edit the file /usr/local/sbin/ovpn-linkup.

Change the line:
/bin/echo $4 > /tmp/$1_router

to:
/bin/echo $route_vpn_gateway > /tmp/$1_router

This will give you the gateway supplied via the server push route-gateway as gateway for the interface and not the IP address of the interface.

Also when I added the interface I choose type as "none" and not "static". The gateway then "magically" appears under gateways without any problems when the openvpn-connection is setup.

Dev note:
Maybe the ovpn-linkup script should check if the environment variable route_vpn_gateway exists. If it does use it, otherwise don't set the $1_router at all? I can't see that setting it to the IP of the interface is meaningfull?

OyyoDams

Hi,

I have no problem connecting to one vpn provider using "redirect-gateway def1" so all my outbound traffic goes through this vpn.
But I'd like to connect to several vpn providers, and choose which one to use at a time using firewall rules.
In this case I definitely can't use "redirect-gateway def1" anymore.

But the problem is even if the connexion with the vpn provider is established, I can't make my traffic reach this provider with other configurations like "route-nopull" or "route-noexec" with the above tip. My firewall configuration is OK, I put the right outbound gateway, so I don't think the problem is here.
How can I configure several vpn providers, keep my main ISP as default gateway, have dynamic gateways for vpn providers (because they have several servers), and make my traffic go to these providers based on firewall rules ? How to configure vpn options exactly ? Muse I change any file like /usr/local/sbin/ovpn-linkup ?

Thanks.

xenta

Thank to this how to, I've configured 3 openVPN client, placed each one on a gateway group in the same tier to have load balancing.

With the dashboard's traffic graph I see that traffic pass in only one openVPN interface, it seem like load balancing not work at all.

Also on another one openVPN interface I see only DNS traffic with tcpdump. (since I set a DNS on this one)

What could be wrong with my configuration ?

eri--

@winge,

committed https://rcs.pfsense.org/projects/pfsense/repos/mainline/commits/7f2c8034fb5d31503ffb5921427a9be8f1eea0e3

xenta

@xenta:

Thank to this how to, I've configured 3 openVPN client, placed each one on a gateway group in the same tier to have load balancing.

With the dashboard's traffic graph I see that traffic pass in only one openVPN interface, it seem like load balancing not work at all.

Also on another one openVPN interface I see only DNS traffic with tcpdump. (since I set a DNS on this one)

What could be wrong with my configuration ?

I found that the OpenVPN client Interfaces have different netmask one with 255.255.255.255 and the others with 255.255.255.0. Does this possible problem ? There is something to do in this case ?

Sorry for my ignorance in OpenVPN…