• Cascades are used by many people. Just connect Router1-> Router2-> Router3-> in between. Why the others use cascades are reasons like day and night. My reason is because I am interested in the technology, even if I am not an expert. 😞

    Create the client for tunnel 2 and use the tunnel 3 interface to run it on. Assign the client 2 interface and create the tunnel 1 client on it.

    How to configure it? What happens when Tunnel1 goes offline? How is it restarted automatically?

    A security measure is still necessary.
    If Tunnel1 goes offline, a firewall rule could block everything. Exactly the same way you would need such a rule for Tunnel2->Tunnel3->Internet. So that the order is always correct.

    Maybe I could solve this with floating rules.

    Unfortunately I could only implement this for one tunnel.

    c92a561b-2199-4fc4-aecb-5333c4baf3dd-grafik.png

    386dddab-d31c-41d3-b15f-8133e8360e58-grafik.png

    2991fd55-7343-41a3-b322-625078a50aeb-grafik.png

    One problem with this rule (for Tunnel3) is that if Tunnel3 is online and Tunnel1 and Tunnel2 are offline, the traffic still flows through Tunnel3. Unfortunately I don't know how to create a firewall rule for this.

  • Netgate Administrator

    Running 3 tunnels inside each other is not the same as just having a route that goes through 3 routers.
    All ip traffic goes through multiple routers, that's how the internet works.
    A tunnel within a tunnel within a tunnel is a very different thing. I have never seen that (except by accident!).

    The cannot come up in the wrong order because, for example, tunnel 1 (the inner most tunnel) cannot connect until tunnel 2 is UP because it's running on the tunnel 2 interface.

    Tunnel 1 will continually try to establish as long as tunnel 2 is up.

    An outbound block rule with a gateway like that is invalid. The gateway will not be applied and it will simply block all.
    When you need to block there is traffic from the source subnet you're using.
    You could also remove the outbound NAT rule that allows traffic from the internal subnet you're using to use the WAN.

    Your policy routing rule on LAN should send traffic across the inner tunnel, tunnel 1 here.

    If you check 'Skip rules when gateway is down' in Sys > Adv > Misc that that will be omitted if Tunnel 1 goes down. If you don't have any other pass rules then traffic from LAN will be blocked when tunnel 1 is down.

    Also 'Kill Switch' is a bad term to use there, it implies something you need to manually activate which is the last thing you want. Fail Safe would be much better.

    Steve


  • @stephenw10 said in OpenVPN Client Cascade:

    Running 3 tunnels inside each other is not the same as just having a route that goes through 3 routers.
    All ip traffic goes through multiple routers, that's how the internet works.

    Is there a simple image diagram in Internet, like VPN-router1->VPN-router2->VPN router3->Internet to compare to a tunnel in tunnel connection for illustration?

    The cannot come up in the wrong order because, for example, tunnel 1 (the inner most tunnel) cannot connect until tunnel 2 is UP because it's running on the tunnel 2 interface.
    Tunnel 1 will continually try to establish as long as tunnel 2 is up.

    This makes sense, as you have explained. So the "protection" is already there, right?

    The question is only how to configure tunnel in tunnel. The theoretical explanation usually always looks different than the practical pfSense configuration.



  • @Pippin said in OpenVPN Client Cascade:

    Maybe this might help you:
    ...

    I already know the link, but I am looking for a pfSense solution.

    stephenw10 has already written that it could work:

    @stephenw10 said in OpenVPN Client Cascade:

    Create the client for tunnel 3, assign it as an interface. Create the client for tunnel 2 and use the tunnel 3 interface to run it on. Assign the client 2 interface and create the tunnel 1 client on it.

    Unfortunately I do not know how to configure this.


  • Maybe now I understand what stephenw10 meant:

    Create the client for tunnel 3, assign it as an interface. Create the client for tunnel 2 and use the tunnel 3 interface to run it on. Assign the client 2 interface and create the tunnel 1 client on it.

    I would probably have to configure these interfaces in the OpenVPN client:

    e98def09-4089-4524-a788-85d89ea7dc4c-grafik.png

    If I configure it like this:

    VPN-Client1 - Interface (VPN-Client2)
    VPN-Client2 - Interface (VPN-Client3)
    VPN-Client3 - Interface (WAN)

    Then it does not work properly. Probably I would have to activate additional options.

    For example:

    If VPN-Client3 goes offline, then VPN-Client1 and 2 go offline. The connection is closed.
    If VPN-Client3 is online and VPN-Client2 goes offline, then VPN-Client1 goes offline. The connection remains established via VPN-Client3.

    However, the connection should only work if VPN-Client 1,2 and 3 are online.

    Does anyone have an idea how this can be adapted?

  • Netgate Administrator

    That's correct behaviour for the tunnels. You are probably routing the traffic incorrectly.

    All of your unencrypted should go across tunnel 1.

    Only encrypted traffic from the tunnel 1 client goes across tunnel 2 and the double encrypted traffic from the tunnel 2 client goes across tunnel 3.

    You should not be routing anything from any internal interface to anywhere except the tunnel 1 gateway.

    Steve


  • Somehow I can't get rid of the feeling that basic settings are missing.

    Here is an example configuration with a VPN provider:

    8b8567e4-6b38-4495-8e18-ca355c1f2307-grafik.png

    a4a7e43f-1309-4529-99cb-7b28443e749f-grafik.png

    dfd7ee5d-af14-478b-bcc6-1c62d742f827-grafik.png

    c5abf135-8a81-4f99-a11a-d1dbede0c41f-grafik.png

    9085c2bb-6258-434b-8d7a-97790ba9debd-grafik.png

    ebebd0a2-e60e-4c18-a3e9-bb0c82e55513-grafik.png

    @stephenw10 said in OpenVPN Client Cascade:

    You should not be routing anything from any internal interface to anywhere except the tunnel 1 gateway.

    Can you show me an example of how the rule should look like?

    I have the following behavior now:

    If VPN-Client1 is online and the others are offline, then there is an internet connection, which should not be the case.

    56af5026-36dc-467f-b9e0-07ef5b894698-grafik.png

    The reason is probably that the local address is almost always set to (pending).

    If all 3 VPN-Clients are online and I restart them several times, then they are not set to (pending), but then the internet connection does not work either.

    In short, there is a leak at (pending), and with a local address the client connections are terminated as intended, only with 3 local addresses there is no internet connection.

  • Netgate Administrator

    That looks to be configured as I would expect. The routing from LAN is correct, all policy routed to the tunnel 1 gateway.

    I agree though tunnel 1 should not be able to come up until tunnel 2 is up becasue it is running on that interface.

    Maybe the interface assignments are wrong?

    Steve


  • @stephenw10 said in OpenVPN Client Cascade:

    Maybe the interface assignments are wrong?

    No, the interfaces seem to be correct.

    If you have Tunnel1 up after Tunnel2 is up, then I suspect that I am missing basic settings.

  • Netgate Administrator

    Can we see the assignments? I'm not sure how tunnel 1 can be UP when it doesn't have a local address because tunnel 2 is DOWN and that's what it's running on.

    Steve


  • @stephenw10 said in OpenVPN Client Cascade:

    Can we see the assignments?

    Of course.

    ebae4b91-42fc-4da7-8296-6d0d9245dae5-grafik.png

  • Netgate Administrator

    Hmm, check the OpenVPN logs. Check the state table. What is tunnel 1 actually running on if tunnel 2 is down?

    Steve


  • I cannot interpret the logs. Here are the logs, if only Tunnel1 is up:

    f6270601-f953-4fc1-be4e-e54d980cf13c-grafik.png
    5bf1c82a-1d7b-462a-9fa6-6bca2cebdc11-grafik.png
    c2cdf4de-5d20-484f-9b5a-8d4f6dc8f2aa-grafik.png

    4370b63c-83d6-4e08-8e52-63f67a269414-grafik.png

    49eb4748-e751-47f6-8d2b-9780c64c6899-grafik.png

  • Netgate Administrator

    It looks like you have the OpenVPN logging options set waaay too high. All of that is just the config it's using not the actual connection process. Set it logging level back to the defaults ot just get the logs covering the connection process.

    Though what I am seeing there is that the local side of the connection shows as not bound to an IP. That's probably what allows it to connect when T2 is down. Without a T2 address it probably omits the local statement from the config. You could check that in /var/etc/openvpn/client1.conf

    I'm not sure you can force that in pfSense. What you could do would be to add floating outbound block rules on WAN for the T1 and T2 server IPs so only T3 can connect directly.

    I still expect T2 and T3 to be trying to connect in that situation though and it appears they are not.

    If you can copy/paste the actual logs into replies using he code tags it's much, much easier to search than pictures on the logs.

    Steve


  • I have now set verbosity level "default" for all 3 servers.

    Though what I am seeing there is that the local side of the connection shows as not bound to an IP. That's probably what allows it to connect when T2 is down.

    Just for information.
    All 3 servers use the same "CA" and "Cert" certificate. With "Server host or address" I can also use amsterdam.vpn.com, then a server from Amsterdam1-5 is automatically selected.
    Instead of amsterdam.vpn.com, I can also specify the following:
    amsterdam1.vpn.com
    amsterdam2.vpn.com
    amsterdam3.vpn.com
    amsterdam4.vpn.com
    amsterdam5.vpn.com
    I'm not sure if this is the reason for conflicts.

    I will try to add other servers. For example, Basel, London or Paris and see if there are still conflicts.

    You could check that in /var/etc/openvpn/client1.conf

    Here is client1.conf:

    dev ovpnc1
    verb 1
    dev-type tun
    dev-node /dev/tun1
    writepid /var/run/openvpn_client1.pid
    #user nobody
    #group nobody
    script-security 3
    daemon
    inactive 604800
    ping 5
    ping-restart 120
    ping-timer-rem
    persist-tun
    persist-key
    proto udp4
    cipher AES-128-GCM
    auth SHA512
    up /usr/local/sbin/ovpn-linkup
    down /usr/local/sbin/ovpn-linkdown
    tls-client
    client
    nobind
    management /var/etc/openvpn/client1/sock unix
    remote 85.17.28.145 1149 udp4
    auth-user-pass /var/etc/openvpn/client1/up
    auth-retry nointeract
    capath /var/etc/openvpn/client1/ca
    cert /var/etc/openvpn/client1/cert 
    key /var/etc/openvpn/client1/key 
    tls-auth /var/etc/openvpn/client1/tls-auth 1
    ncp-disable
    comp-noadapt
    resolv-retry infinite
    route-nopull
    hand-window 120
    mute-replay-warnings
    persist-remote-ip
    reneg-sec 3600
    resolv-retry 60
    tls-cipher TLS-DHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256:TLS-DHE-RSA-WITH-CAMELLIA-256-CBC-SHA:TLS-DHE-RSA-WITH-AES-256-CBC-SHA:TLS-RSA-WITH-CAMELLIA-256-CBC-SHA:TLS-RSA-WITH-AES-256-CBC-SHA
    tls-timeout 5
    tun-mtu  1500
    fragment 1300
    mssfix
    remote-cert-tls server
    

    I'm not sure you can force that in pfSense. What you could do would be to add floating outbound block rules on WAN for the T1 and T2 server IPs so only T3 can connect directly.

    Let's assume for now that pfSense cannot force this. How exactly do I create the rules?

    If you can copy/paste the actual logs into replies using he code tags it's much, much easier to search than pictures on the logs.

    I can't add logs because the VPN provider is reported as spam.

  • Netgate Administrator

    Like this:

    Screenshot_2020-10-25 2220 stevew lan - Firewall Rules Floating Edit.png
    And the same the T2 server IP. Only the T3 server IP should be seen as an outbound connection on WAN.

    Steve


  • What you could do would be to add floating outbound block rules on WAN for the T1 and T2 server IPs so only T3 can connect directly.

    And the same the T2 server IP.

    Okay. I understood that I should only create two rules.

    Like these?

    Screenshot_2020-10-25 pfSense localdomain - Firewall Rules Floating.png

    T3 can still connect, but the others are (pending).

    Screenshot_2020-10-25 pfSense localdomain - Status OpenVPN.png

    Only the T3 server IP should be seen as an outbound connection on WAN.

    I don't know how this is meant.

  • Netgate Administrator

    I mean if you look at the states for port 1149 (:1149) on WAN you should only see the T3 client.

    Check the logs to see what the T2 client is doing. It should be trying to connect on the T3 client interface. If it isn't what is it doing? Did it error out trying to connect before T3 had connected and stop?

    Steve


  • Sorry for the stupid questions, but are floating rules for t1 and t2 correct?

    Only the T3 server IP should be seen as an outbound connection on WAN.

    Does this refer to floating t2 rule or to a new one with t3?

    I mean if you look at the states for port 1149 (:1149) on WAN you should only see the T3 client.

    Here is the output:

    3dba7c21-6206-4c70-ada4-2288e24ecd8b-grafik.png

    556d22d9-48e9-489c-a370-181c71989fb6-grafik.png

    Check the logs to see what the T2 client is doing. It should be trying to connect on the T3 client interface.

    At Diagnostics-> States?

    95b8dcea-f476-4c15-a01c-ff0e44da7a4e-grafik.png

    The same is also for T3.

    Does that mean that all 3 tunnels run parallel and not tunnel through tunnel?


  • Edit:

    Different city clients seem to work better than a group of clients from one city.

    I currently have the following servers:

    Screenshot_2020-10-29 pfSense localdomain - VPN OpenVPN Clients.png

    When I restart pfSense, I get completely different results than when I start the servers manually.

    1. the 2 floating rules work

    Screenshot_2020-10-29 pfSense localdomain - Firewall Rules Floating.png

    1. T1 and T2 shows only ICMP protocol.

    Screenshot_2020-10-29 pfSense localdomain - Diagnostics States States.png

    Screenshot_2020-10-29 pfSense localdomain - Diagnostics States States(1).png

    1. The traffic goes through T3, but there is no internet connection

    When I manually restart the clients, the servers do not start again.


  • Edit2:

    Maybe I have to set the routes manually in the OpenVPN client under "Remote Network(s)"?

    Like for example here?

    53c77d11-aa12-433b-91fb-15f4c08e5b25-grafik.png

    Screenshot_2020-10-29 pfSense localdomain - Diagnostics Routes.png

    But which settings would I then have to set for T3?

    677121fc-7c4f-449b-9f7a-dac83972f660-grafik.png

  • Netgate Administrator

    Did you set the direction as OUT on those floating rules?

    Did you reset the state table or reboot since you added them?

    You have at least one state on WAN that should have been rejected by that outbound floating rule.

    You should not see any states on WAN foe OpenVPN tunnels except tunnel 3.

    The fact they are up means they are not running tunnel-in-tunnel as you say.

    Steve


  • Did you set the direction as OUT on those floating rules?

    Yes.

    Did you reset the state table or reboot since you added them?

    I always enabled "Reset the firewall state table" under Diagnostics/States/Reset States and pressed [Reset]. In any case I assumed that it would work. Only with a reboot the floating rules and the start of the tunnel sequence worked simultaneously.

    You should not see any states on WAN foe OpenVPN tunnels except tunnel 3.

    Without remote network(s) entries in the OpenVPN client I see all 3 servers under States/WAN. So they are not running tunnel in tunnel.

    But I have to test again if with remote network(s) only the third server is visible.


  • OK. Whether with or without remote network(s) makes no difference.

    b3e92805-a2b8-4fd5-b461-43d9a72c9744-grafik.png

    The rules in the image currently do the following after a reboot:

    1. Under States/WAN I only see T3.

    Screenshot_2020-10-29 pfSense localdomain - Diagnostics States States.png

    1. Under States/T2 I see nothing.

    2. Under States/T3 I see T1 and T2. I do not know if this is correct.

    Screenshot_2020-10-29 pfSense localdomain - Diagnostics States States(1).png

    1. There is no internet connection with floating rules.

    Do I understand correctly that the rules must look like this?

    • Under States/WAN I should see T3.
    • Under States/T1 I should see T2.
    • Under States/T2 I should see T3.
    • Under States/T3 I should see WAN.

    Would that be correct?

  • Netgate Administrator

    No. You should see~:

    States on WAN - T3
    States on T3 - T2
    States on T2 - T1

    So you need a floating outbound reject rule on T3 to reject the T1 destination. It should only be able to create states on T2.

    Steve


  • @stephenw10 said in OpenVPN Client Cascade:

    So you need a floating outbound reject rule on T3 to reject the T1 destination

    Unfortunately I do not know how to configure this.
    Also I don't know which rule should be at the top and which at the bottom.

  • Netgate Administrator

    The ordering is not important since this will be the only floating rule applied to the T3 interface.

    Thinking about it you could just edit your existing floating rule for the T1 server destination and make it apply to WAN an T3 so can only ever open on T2.

    Steve


  • It makes no difference whether with these rules

    Screenshot_2020-10-30 pfSense localdomain - Firewall Rules Floating(1).png

    or with these rules

    Screenshot_2020-10-30 pfSense localdomain - Firewall Rules Floating.png

    In both cases the T1 client is down and I have no internet connection.

    Screenshot_2020-10-30 pfSense localdomain - Status OpenVPN.png

  • Netgate Administrator

    But it does prevent the T1 connection coming up in the wrong place?

    It just fails entirely?

    Steve


  • @stephenw10 said in OpenVPN Client Cascade:

    But it does prevent the T1 connection coming up in the wrong place?

    Yes.

    It just fails entirely?

    I do not know how to check it.

    Under OpenVPN Log always says after reboot:

    Oct 31 18:29:10 	openvpn 	28024 	write UDPv4: Permission denied (code=13)
    Oct 31 18:29:10 	openvpn 	28024 	write UDPv4: Permission denied (code=13)
    Oct 31 18:29:11 	openvpn 	28024 	write UDPv4: Permission denied (code=13)
    Oct 31 18:29:11 	openvpn 	28024 	write UDPv4: Permission denied (code=13)
    Oct 31 18:29:12 	openvpn 	28024 	write UDPv4: Permission denied (code=13)
    Oct 31 18:29:12 	openvpn 	28024 	write UDPv4: Permission denied (code=13)
    Oct 31 18:29:13 	openvpn 	28024 	write UDPv4: Permission denied (code=13) 
    

    I have:
    States on WAN-T3
    States on T3-T2
    States on T2-nothing

    T3-T1 is no longer present.

    I can T1 no longer restart manually.

  • Netgate Administrator

    Permission denied like that usually means the firewall is preventing it. Which either means it's trying to start on WAN or T3 and getting blocked or the rules are incorrectly blocking it on T2.

    If it's the former I'm not really sure what we can do from there. There doesn't appear to be any way to set a specific outbound interface for it to use when that interface is dynamic.

    Since you know the destination IPs here you could set a route to each on the correct client.
    So try adding it as a remote network (/32). I.e. add the T1 server IP as a remote network on the T2 client. That should add a route to it and the T1 client should then use it.

    Steve


  • When I add T1 server IP as a remote network on the T2 client, the following happens:

    • After reboot T1 stays down.

    Screenshot_2020-11-01 pfSense localdomain - Status OpenVPN.png

    • The message "write UDPv4: Permission denied (code=13)" remains.

    But I can manually restart the OpenVPN client order T3->T2->T1. The internet connection is available after that.

    If I add T1 server IP as remote network on the T2 client and if I add T2 server IP as remote network on the T3 client, the following happens:

    • After reboot all 3 OpenVPN clients are online, but "Local Address" of T1 and T2 is set to (pending).

    Screenshot_2020-11-01 pfSense localdomain - Status OpenVPN(1).png

    • The message "write UDPv4: Permission denied (code=13)" remains.

    But the internet connection is not available. I have to restart the sequence T3->T2->T1 manually.

    In both cases (after reboot and after manual restart) I see~:

    States on WAN - T3
    States on T3 - T2
    States on T2 - T1

    Also in both cases, I had to deactivate "Don't pull routes" in the OpenVPN client to make it work.

    What else can I do?


  • @stephenw10

    I still have some questions, just to get it right.

    The specification of interfaces in the OpenVPN client secures the order, so that the traffic only goes via T1->T2->T3, right?

    The specification of floating rules determines the routing order, right?

    The specification of remote network(s) in the OpenVPN client additionally determines the routing order?

    But there is no way to determine which client is started first and which client is started last, right? Because after a reboot the internet connection does not work, but after a manual client restart it works.
    If this is the case, how could I solve it?
    If pfSense offers no other possibilities to solve the problem, would a script be able to restart the order of the OpenVPN clients?


  • Edit:

    @stephenw10 said in OpenVPN Client Cascade:

    There is a chance it will just work fine since if, for example, tunnel 3 went down when it came back up it would restart tunnel 2 since that is running on it. That in turn would restart tunnel 1.

    That did not work for me.

    If T3 stopped, then T2 and T1 stopped, but restarting the clients did not work. Would it be possible that I would have to activate something extra for that?

  • Netgate Administrator

    Indeed the problem is that OpenVPN just chooses whatever source it wants .

    So the firewall rules are to prevent it trying to use the wrong interface.

    The remote networks add routes to the system when the clients connect for the client inside that which makes openvpn use it.

    Steve


  • First of all, thanks for the great support @stewenw10!

    What should such a script be able to do if the OpenVPN clients are to be started in the correct order?

    I am still willing to pay 300 € for a script.

  • Netgate Administrator

    It's hard to say. It might take some work here because all the existing scripts in pfSense would be trying to bring up the connections normally. If they are defined normally at least.

    The issue we need to overcome is preventing an inner tunnel client trying to start intil the outter tunnel is up. I have hoped they might do that anyway when they are on that interface but that doesn't appear to be the case.
    Something like the code that prevents OpenVPN starting on a CARP VIP when it's not master for example.

    Currently we are just blocking it from connecting by rejecting the states. Which should then allow it to come up when the outer tunnel connects and adds a route. But apparently that is insufficient. The client may need to restart to come up using the new route, which should happen anyway since the interface it's running on changes state but....

    Steve


  • I have tried to create a flowchart.

    Would a script with these functions be feasible?

    Flowchart.jpg

  • Netgate Administrator

    You can have more than one OpenVPN client per CPU core. Potentially a lot more if the WAN bandwidth is low.

    Most of the parts of script like that already exist to some extent. For example outer tunnel is dircetly on WAN, it doesn't try to come up until the WAN shows as UP. I had hoped the other tunnels would work similarly on the assigned interfaces but seemingly not.
    The big problem you have making a separate script is that all the existing scripts are still going to be running and starting/stopping things in unhelpful ways. It's probably better to work with them as much as possible.

    Steve