OpenVPN Client Cascade

[John2893ax]

It makes no difference whether with these rules

or with these rules

In both cases the T1 client is down and I have no internet connection.

[stephenw10]

But it does prevent the T1 connection coming up in the wrong place?

It just fails entirely?

Steve

[John2893ax]

@stephenw10 said in OpenVPN Client Cascade:

But it does prevent the T1 connection coming up in the wrong place?

Yes.

It just fails entirely?

I do not know how to check it.

Under OpenVPN Log always says after reboot:

Oct 31 18:29:10 	openvpn 	28024 	write UDPv4: Permission denied (code=13)
Oct 31 18:29:10 	openvpn 	28024 	write UDPv4: Permission denied (code=13)
Oct 31 18:29:11 	openvpn 	28024 	write UDPv4: Permission denied (code=13)
Oct 31 18:29:11 	openvpn 	28024 	write UDPv4: Permission denied (code=13)
Oct 31 18:29:12 	openvpn 	28024 	write UDPv4: Permission denied (code=13)
Oct 31 18:29:12 	openvpn 	28024 	write UDPv4: Permission denied (code=13)
Oct 31 18:29:13 	openvpn 	28024 	write UDPv4: Permission denied (code=13) 

I have:
States on WAN-T3
States on T3-T2
States on T2-nothing

T3-T1 is no longer present.

I can T1 no longer restart manually.

[stephenw10]

Permission denied like that usually means the firewall is preventing it. Which either means it's trying to start on WAN or T3 and getting blocked or the rules are incorrectly blocking it on T2.

If it's the former I'm not really sure what we can do from there. There doesn't appear to be any way to set a specific outbound interface for it to use when that interface is dynamic.

Since you know the destination IPs here you could set a route to each on the correct client.
So try adding it as a remote network (/32). I.e. add the T1 server IP as a remote network on the T2 client. That should add a route to it and the T1 client should then use it.

Steve

[John2893ax]

When I add T1 server IP as a remote network on the T2 client, the following happens:

• After reboot T1 stays down.

• The message "write UDPv4: Permission denied (code=13)" remains.

But I can manually restart the OpenVPN client order T3->T2->T1. The internet connection is available after that.

If I add T1 server IP as remote network on the T2 client and if I add T2 server IP as remote network on the T3 client, the following happens:

• After reboot all 3 OpenVPN clients are online, but "Local Address" of T1 and T2 is set to (pending).

• The message "write UDPv4: Permission denied (code=13)" remains.

But the internet connection is not available. I have to restart the sequence T3->T2->T1 manually.

In both cases (after reboot and after manual restart) I see~:

States on WAN - T3
States on T3 - T2
States on T2 - T1

Also in both cases, I had to deactivate "Don't pull routes" in the OpenVPN client to make it work.

What else can I do?

[John2893ax]

@stephenw10

I still have some questions, just to get it right.

The specification of interfaces in the OpenVPN client secures the order, so that the traffic only goes via T1->T2->T3, right?

The specification of floating rules determines the routing order, right?

The specification of remote network(s) in the OpenVPN client additionally determines the routing order?

But there is no way to determine which client is started first and which client is started last, right? Because after a reboot the internet connection does not work, but after a manual client restart it works.
If this is the case, how could I solve it?
If pfSense offers no other possibilities to solve the problem, would a script be able to restart the order of the OpenVPN clients?

[John2893ax]

Edit:

@stephenw10 said in OpenVPN Client Cascade:

There is a chance it will just work fine since if, for example, tunnel 3 went down when it came back up it would restart tunnel 2 since that is running on it. That in turn would restart tunnel 1.

That did not work for me.

If T3 stopped, then T2 and T1 stopped, but restarting the clients did not work. Would it be possible that I would have to activate something extra for that?

[stephenw10]

Indeed the problem is that OpenVPN just chooses whatever source it wants .

So the firewall rules are to prevent it trying to use the wrong interface.

The remote networks add routes to the system when the clients connect for the client inside that which makes openvpn use it.

Steve

[John2893ax]

First of all, thanks for the great support @stewenw10!

What should such a script be able to do if the OpenVPN clients are to be started in the correct order?

I am still willing to pay 300 € for a script.

[stephenw10]

It's hard to say. It might take some work here because all the existing scripts in pfSense would be trying to bring up the connections normally. If they are defined normally at least.

The issue we need to overcome is preventing an inner tunnel client trying to start intil the outter tunnel is up. I have hoped they might do that anyway when they are on that interface but that doesn't appear to be the case.
Something like the code that prevents OpenVPN starting on a CARP VIP when it's not master for example.

Currently we are just blocking it from connecting by rejecting the states. Which should then allow it to come up when the outer tunnel connects and adds a route. But apparently that is insufficient. The client may need to restart to come up using the new route, which should happen anyway since the interface it's running on changes state but....

Steve

[John2893ax]

I have tried to create a flowchart.

Would a script with these functions be feasible?

[stephenw10]

You can have more than one OpenVPN client per CPU core. Potentially a lot more if the WAN bandwidth is low.

Most of the parts of script like that already exist to some extent. For example outer tunnel is dircetly on WAN, it doesn't try to come up until the WAN shows as UP. I had hoped the other tunnels would work similarly on the assigned interfaces but seemingly not.
The big problem you have making a separate script is that all the existing scripts are still going to be running and starting/stopping things in unhelpful ways. It's probably better to work with them as much as possible.

Steve

[John2893ax]

OpenVPN Client Cascade

About:
The advantage of a pfSense OpenVPN cascade is that you can combine it with different VPN providers. It is recommended to use more powerful hardware[1] for a cascade.
The OpenVPN cascade currently only works with static IPs.
Here is an example image with a tunnel in tunnel connection:

A connection "can" work better to different cities than a connection to several servers in one city.

If you like the script or want to have an extension added, get in touch with the devloper of the Script.

[John2893ax]

A Configuration for a manual OpenVPN cascade start.

Requirements:

• Don't activate "Don't pull routes" in the OpenVPN client.
• Make sure that first all OpenVPN clients are running correctly (Status/OpenVPN).

1 Interface Configuration (OpenVPN Client)

The idea behind this is that OpenVPN Client1 can only go online when OpenVPN Client2 is online. This is exactly the same with OpenVPN Client2. OpenVPN Client2 is only online when OpenVPN Client3 is online.

• First open OpenVPN Client1 and select the interface of OpenVPN Client2.

• VPN1 OpenVPN Client => VPN2Interface
• VPN2 OpenVPN Client => VPN3Interface
• VPN3 OpenVPN Client => WAN Interface

2 Remote Network(s) Configuration (OpenVPN Client)

The remote networks add routes to the system when the clients connect for the client inside that which makes openvpn use it.

Configuration:

• OpenVPN Client1 (VPN1) remains unchanged.
• Open OpenVPN Client2 (VPN2) and enter the IP of Client1 "85.17.28.145/32" in IPv4 Remote Network(s).
• Open OpenVPN Client3 (VPN3) and enter the IP of Client2 "82.199.134.162/32" in IPv4 Remote Network(s).

3 Firewall Floating Rules

The firewall floating rules are there to prevent it from trying to use the wrong interface.

Configuration:

• Create the first rule with these settings with the VPN1 IP as destination:

• Create the second rule for the IP of Client2 under destination.
• Finally, we need a third rule with OpenVPN Client3 interface and as destination the IP of VPN1.

4 Firewall Configuration

• Open the LAN rule and select VPN1 as gateway

5 Services/DNS Resolver Configuration

• As outgoing network interface we select VPN1

6 Firewall/NAT/Outbound Configuration

• In this example we select only VPN1 interface

7 Diagnostics/States Review

Now you can check all states to make sure that everything is correct. In this example the states should have the following sequence:

States on WAN Interface – VPN3 IP
States on VPN3 Interface – VPN2 IP
States on VPN2 Interface – VPN1 IP

WAN Interface:

VPN3 Interface:

VPN2 Interface:

8 Manual Start

Now you can stop all 3 VPN clients and start the sequence VPN3->VPN2->VPN1.

[1] https://docs.netgate.com/pfsense/en/latest/hardware/size.html#vpn-all-types

[John2893ax]

B Configuration for an automatic cascade start.

Original configuration can be found here:

https://github.com/ddowse/pf-tunnelactive

1 Interface Configuration (OpenVPN Client)

• Select "any" as interface in the OpenVPN client. Only at the last hop "wan" interface remains.
• Activate "Don't add/remove routes" everywhere except in the OpenVPN client that goes online.
• Add the following line to "Custom options" and change the IP for "NEXT_VPNSERVER_IP":

route-up "/root/pf-tunnelactive/addroute.sh NEXT_VPNSERVER_IP"

Example configuration:
VPN1:

• "Don't add/remove routes"
• Custom options: route-up command not necessary

VPN2:

• "Don't add/remove routes"
• Custom options: route-up "/root/pf-tunnelactive/addroute.sh 85.17.28.145"

VPN3:

• "Don't add/remove routes"
• Custom options: route-up "/root/pf-tunnelactive/addroute.sh 82.199.134.162"

• Make sure that first all OpenVPN clients are running correctly (Status/OpenVPN).

Please note that Firewall Rules are strictly optional but of course NAT Rules are mandatory.

2 Firewall Floating Rules

• Create a rule in “Firewall/Rules/Floating“
  o Action: Block
  o Interface: WAN
  o Address Family: IPv4
  o Protocol: Any
  o Source: LAN net (For example: Local Network)

3 Firewall LAN Rules

Important: Gateway configuration for LAN rules not necessary!

4 NAT configuration (Firewall/NAT/Outbound)

• Create a rule for each OpenVPN interface.
• Last 2 rules are also important

5 Script configuration

• Follow the steps under "Installation" and "Usage":
  https://github.com/ddowse/pf-tunnelactive

All other steps like restarting OpenVPN clients and monitoring are done by the script.

6 Optional: Shellcmd Package

If the script works, you can add this command to Shellscript Package:

nohup php /root/pf-tunnelactive/tunnelactive.php 10 3 >> /var/log/tunnelactive.log &

After that the script will be loaded on every restart.