OpenVPN Client Cascade

stephenw10 · Oct 21, 2020, 1:03 AM

That looks to be configured as I would expect. The routing from LAN is correct, all policy routed to the tunnel 1 gateway.

I agree though tunnel 1 should not be able to come up until tunnel 2 is up becasue it is running on that interface.

Maybe the interface assignments are wrong?

Steve

John2893ax · Oct 23, 2020, 7:07 PM

@stephenw10 said in OpenVPN Client Cascade:

Maybe the interface assignments are wrong?

No, the interfaces seem to be correct.

If you have Tunnel1 up after Tunnel2 is up, then I suspect that I am missing basic settings.

stephenw10 · Oct 23, 2020, 9:27 PM

Can we see the assignments? I'm not sure how tunnel 1 can be UP when it doesn't have a local address because tunnel 2 is DOWN and that's what it's running on.

Steve

John2893ax · Oct 23, 2020, 10:46 PM

@stephenw10 said in OpenVPN Client Cascade:

Can we see the assignments?

Of course.

🔒 Log in to view

stephenw10 · Oct 24, 2020, 12:25 AM

Hmm, check the OpenVPN logs. Check the state table. What is tunnel 1 actually running on if tunnel 2 is down?

Steve

John2893ax · Oct 24, 2020, 8:12 PM

I cannot interpret the logs. Here are the logs, if only Tunnel1 is up:

🔒 Log in to view
🔒 Log in to view
🔒 Log in to view

🔒 Log in to view

stephenw10 · Oct 25, 2020, 12:48 PM

It looks like you have the OpenVPN logging options set waaay too high. All of that is just the config it's using not the actual connection process. Set it logging level back to the defaults ot just get the logs covering the connection process.

Though what I am seeing there is that the local side of the connection shows as not bound to an IP. That's probably what allows it to connect when T2 is down. Without a T2 address it probably omits the local statement from the config. You could check that in /var/etc/openvpn/client1.conf

I'm not sure you can force that in pfSense. What you could do would be to add floating outbound block rules on WAN for the T1 and T2 server IPs so only T3 can connect directly.

I still expect T2 and T3 to be trying to connect in that situation though and it appears they are not.

If you can copy/paste the actual logs into replies using he code tags it's much, much easier to search than pictures on the logs.

Steve

John2893ax · Oct 25, 2020, 1:56 PM

I have now set verbosity level "default" for all 3 servers.

Though what I am seeing there is that the local side of the connection shows as not bound to an IP. That's probably what allows it to connect when T2 is down.

Just for information.
All 3 servers use the same "CA" and "Cert" certificate. With "Server host or address" I can also use amsterdam.vpn.com, then a server from Amsterdam1-5 is automatically selected.
Instead of amsterdam.vpn.com, I can also specify the following:
amsterdam1.vpn.com
amsterdam2.vpn.com
amsterdam3.vpn.com
amsterdam4.vpn.com
amsterdam5.vpn.com
I'm not sure if this is the reason for conflicts.

I will try to add other servers. For example, Basel, London or Paris and see if there are still conflicts.

You could check that in /var/etc/openvpn/client1.conf

Here is client1.conf:

dev ovpnc1
verb 1
dev-type tun
dev-node /dev/tun1
writepid /var/run/openvpn_client1.pid
#user nobody
#group nobody
script-security 3
daemon
inactive 604800
ping 5
ping-restart 120
ping-timer-rem
persist-tun
persist-key
proto udp4
cipher AES-128-GCM
auth SHA512
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
tls-client
client
nobind
management /var/etc/openvpn/client1/sock unix
remote 85.17.28.145 1149 udp4
auth-user-pass /var/etc/openvpn/client1/up
auth-retry nointeract
capath /var/etc/openvpn/client1/ca
cert /var/etc/openvpn/client1/cert 
key /var/etc/openvpn/client1/key 
tls-auth /var/etc/openvpn/client1/tls-auth 1
ncp-disable
comp-noadapt
resolv-retry infinite
route-nopull
hand-window 120
mute-replay-warnings
persist-remote-ip
reneg-sec 3600
resolv-retry 60
tls-cipher TLS-DHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256:TLS-DHE-RSA-WITH-CAMELLIA-256-CBC-SHA:TLS-DHE-RSA-WITH-AES-256-CBC-SHA:TLS-RSA-WITH-CAMELLIA-256-CBC-SHA:TLS-RSA-WITH-AES-256-CBC-SHA
tls-timeout 5
tun-mtu  1500
fragment 1300
mssfix
remote-cert-tls server

I'm not sure you can force that in pfSense. What you could do would be to add floating outbound block rules on WAN for the T1 and T2 server IPs so only T3 can connect directly.

Let's assume for now that pfSense cannot force this. How exactly do I create the rules?

If you can copy/paste the actual logs into replies using he code tags it's much, much easier to search than pictures on the logs.

I can't add logs because the VPN provider is reported as spam.

stephenw10 · Oct 25, 2020, 2:16 PM

Like this:

🔒 Log in to view
And the same the T2 server IP. Only the T3 server IP should be seen as an outbound connection on WAN.

Steve

John2893ax · Oct 25, 2020, 6:26 PM

What you could do would be to add floating outbound block rules on WAN for the T1 and T2 server IPs so only T3 can connect directly.

And the same the T2 server IP.

Okay. I understood that I should only create two rules.

Like these?

🔒 Log in to view

T3 can still connect, but the others are (pending).

🔒 Log in to view

Only the T3 server IP should be seen as an outbound connection on WAN.

I don't know how this is meant.

stephenw10 · Oct 25, 2020, 8:04 PM

I mean if you look at the states for port 1149 (:1149) on WAN you should only see the T3 client.

Check the logs to see what the T2 client is doing. It should be trying to connect on the T3 client interface. If it isn't what is it doing? Did it error out trying to connect before T3 had connected and stop?

Steve

John2893ax · Oct 26, 2020, 7:14 PM

Sorry for the stupid questions, but are floating rules for t1 and t2 correct?

Only the T3 server IP should be seen as an outbound connection on WAN.

Does this refer to floating t2 rule or to a new one with t3?

I mean if you look at the states for port 1149 (:1149) on WAN you should only see the T3 client.

Here is the output:

🔒 Log in to view

Check the logs to see what the T2 client is doing. It should be trying to connect on the T3 client interface.

At Diagnostics-> States?

🔒 Log in to view

The same is also for T3.

Does that mean that all 3 tunnels run parallel and not tunnel through tunnel?

John2893ax · Oct 29, 2020, 1:11 AM

Edit:

Different city clients seem to work better than a group of clients from one city.

I currently have the following servers:

🔒 Log in to view

When I restart pfSense, I get completely different results than when I start the servers manually.

the 2 floating rules work

🔒 Log in to view

T1 and T2 shows only ICMP protocol.

🔒 Log in to view

The traffic goes through T3, but there is no internet connection

When I manually restart the clients, the servers do not start again.

John2893ax · Oct 29, 2020, 1:25 AM

Edit2:

Maybe I have to set the routes manually in the OpenVPN client under "Remote Network(s)"?

Like for example here?

🔒 Log in to view

But which settings would I then have to set for T3?

🔒 Log in to view

stephenw10 · Oct 29, 2020, 9:27 PM

Did you set the direction as OUT on those floating rules?

Did you reset the state table or reboot since you added them?

You have at least one state on WAN that should have been rejected by that outbound floating rule.

You should not see any states on WAN foe OpenVPN tunnels except tunnel 3.

The fact they are up means they are not running tunnel-in-tunnel as you say.

Steve

John2893ax · Oct 29, 2020, 10:34 PM

Did you set the direction as OUT on those floating rules?

Yes.

Did you reset the state table or reboot since you added them?

I always enabled "Reset the firewall state table" under Diagnostics/States/Reset States and pressed [Reset]. In any case I assumed that it would work. Only with a reboot the floating rules and the start of the tunnel sequence worked simultaneously.

You should not see any states on WAN foe OpenVPN tunnels except tunnel 3.

~~Without remote network(s) entries in the OpenVPN client I see all 3 servers under States/WAN~~. ~~So they are not running tunnel in tunnel.~~

But I have to test again if with remote network(s) only the third server is visible.

John2893ax · Oct 29, 2020, 10:50 PM

OK. Whether with or without remote network(s) makes no difference.

🔒 Log in to view

The rules in the image currently do the following after a reboot:

Under States/WAN I only see T3.

🔒 Log in to view

Under States/T2 I see nothing.
Under States/T3 I see T1 and T2. I do not know if this is correct.

🔒 Log in to view

There is no internet connection with floating rules.

Do I understand correctly that the rules must look like this?

Under States/WAN I should see T3.
Under States/T1 I should see T2.
Under States/T2 I should see T3.
Under States/T3 I should see WAN.

Would that be correct?

stephenw10 · Oct 30, 2020, 12:03 AM

No. You should see~:

States on WAN - T3
States on T3 - T2
States on T2 - T1

So you need a floating outbound reject rule on T3 to reject the T1 destination. It should only be able to create states on T2.

Steve

John2893ax · Oct 30, 2020, 3:10 AM

@stephenw10 said in OpenVPN Client Cascade:

So you need a floating outbound reject rule on T3 to reject the T1 destination

Unfortunately I do not know how to configure this.
Also I don't know which rule should be at the top and which at the bottom.

stephenw10 · Oct 30, 2020, 1:17 PM

The ordering is not important since this will be the only floating rule applied to the T3 interface.

Thinking about it you could just edit your existing floating rule for the T1 server destination and make it apply to WAN an T3 so can only ever open on T2.

Steve