Multiple OpenVPN clients non-functional

douglasg14b · Sep 4, 2017, 10:43 PM

TL;DR: All traffic on additional OpenVPN clients created after the first one times out.

I'm trying to setup multiple VPN clients to route different traffic though (ie. One for web traffic, one for torrenting, one for streaming). Let's ignore any firewall rule shenanigans for the time being, since I am treating every client the same for now. I have 2 interfaces, both are setup to allow all traffic before my WAN interface in the firewall rules, this works fine. Lets call them Int1 & Int2. Lets also call my OpenVPN clients VPN1 & VPN2

If I create an OpenVPN client VPN1 and assign Int1 to it, it connects just fine and works, traffic flows through…etc All is good. Similarly if I assign Int2 to VPN1, traffic flows fine, all is good.

If I create a 2nd OpenVPN client VPN2, and assign it to Int2, and disable the firewall rule for Int1 to force the traffic through Int2 & VPN2, all connections time out. Similarly if I disable VPN1 entirely, and assign either interface to VPN2 all traffic times out. If I delete VPN1, the issue persists for VPN2. If I delete all clients and recreate them, the 1st client always works, and any subsequent clients do not. VPN2's status will show as up, and with an IP. The Sent Bytes will slowly rise, but the Received Bytes will stay around 5 Bytes and not go up.

Sometimes VPN2's status will go to down, when that happens the following log entry only shows up at lvl 7 or higher:

TLS Warning: no data channel send key available: [key#0 state=S_PRE_START id=0 sid=00000000 00000000] [key#1 state=S_UNDEF id=0 sid=00000000 00000000] [key#2 state=S_UNDEF id=0 sid=00000000 00000000]

I have verified that the config files and all keys/CA's are exactly the same in var/etc/openvpn. When I delete VPN1 all config files for VPN2 continue to be called client2.*. If I then create a new client, it's config files are all named client1.*, and the new client will work.

It seems any client that has config files with anything past client1 does not function. Any idea why this is and how I can resolve it?

Example Config:

dev ovpnc2
verb 7
dev-type tun
dev-node /dev/tun2
writepid /var/run/openvpn_client2.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher AES-128-CBC
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
tls-client
client
nobind
management /var/etc/openvpn/client2.sock unix
remote west.usa.torguardvpnaccess.com 1912
auth-user-pass /var/etc/openvpn/client2.up
ca /var/etc/openvpn/client2.ca
cert /var/etc/openvpn/client2.cert
key /var/etc/openvpn/client2.key
tls-auth /var/etc/openvpn/client2.tls-auth 1
resolv-retry infinite
remote-cert-tls server

johnpoz · Sep 5, 2017, 12:50 PM

If your going to have more than 1 vpn client on pfsense you need to make sure your not pulling routes.. Since they will like to setup default route going out that vpn.

Once you do not pull routes you can then route the traffic you want out the vpn connections via policy routing.

douglasg14b · Sep 5, 2017, 3:43 PM

@johnpoz:

If your going to have more than 1 vpn client on pfsense you need to make sure your not pulling routes.. Since they will like to setup default route going out that vpn.

Once you do not pull routes you can then route the traffic you want out the vpn connections via policy routing.

johnpoz, can you explain that more for me? I'm fairly new at this and am not sure what "pulling routes" means/entails.

Would this also explain a single VPN client acting this way when it's anything other than client1 in it's config? (Create two clients, delete the first one, the 2nd one still doesn't work. Create a 3rd one, the 3rd one's config is named client1.* and it will work just fine)

johnpoz · Sep 5, 2017, 4:06 PM

In your vpn client config - there is a check box NOT to pull routes..

You could also have issues if your different vpn services are handing you the same network for each client.. When I get a chance I will fire up a client connection to one of my other vps running openvpn access server. I normally only maintain a client connection to one of them 24/7 But I can fire a couple of more to show you as example if needed..

BTW your not trying to create multiple connections to the same vpn service are you?

douglasg14b · Sep 6, 2017, 1:03 AM

@johnpoz:

In your vpn client config - there is a check box NOT to pull routes..

You could also have issues if your different vpn services are handing you the same network for each client.. When I get a chance I will fire up a client connection to one of my other vps running openvpn access server. I normally only maintain a client connection to one of them 24/7 But I can fire a couple of more to show you as example if needed..

BTW your not trying to create multiple connections to the same vpn service are you?

Gotcha, I have now checked that box, issue persists.

I want to note that I don't need both clients to be enabled to see this issue. I can create two clients both disabled upon creation, enable the 2nd one and the 2nd one will not work, but the 1st one will when enabled.

It shouldn't matter if it's to the same service in this instance since the clients are disabled anyways? But if it did, I am connecting to the same provider, but entirely different servers (One server is dedicated for streaming traffic).

johnpoz · Sep 6, 2017, 1:40 AM

"I am connecting to the same provider, but entirely different servers (One server is dedicated for streaming traffic)."

And your vpn service provides service to multiple connections from the same IP? I find that unlikely..

douglasg14b · Sep 6, 2017, 2:31 AM

@johnpoz:

"I am connecting to the same provider, but entirely different servers (One server is dedicated for streaming traffic)."

And your vpn service provides service to multiple connections from the same IP? I find that unlikely..

They do, I pay for it specifically. If I enable multiple clients they all connect and receive IPs from the provider.

I feel like we're getting hung up on issues that are not the root of the problem, and the main body of my posts isn't being read or understood…

The above is a side-issue, I don't need multiple active clients for this problem to occur. I mentioned that I can create two or more clients, all disabled upon creation. If I enable any client except for the first one they do not work, all traffic times out. I only have ONE active VPN connection during this scenario.

Edit: What do you need from me?

If I delete the first, working, client. All other clients still continue to be non-functional when enabled, all clients will have their original config names. If I create an additional client, it will have a config named client1.*, and it will work. This seems to point towards some issue in how PFsense or OpenVPN is handling additional clients?

Derelict · Sep 6, 2017, 2:29 AM

Yes, but we all (or at least many, many, many of us) have multiple OpenVPN clients configured and they work just fine. OpenVPN is routed using the routing table. One client being started should not affect other clients or other traffic unless there is something misconfigured.

You will need to provide more information.

You want to make sure that don't pull routes is enabled on ALL your client configs for now.

Connect to one, and post the Diagnostics > Routes (or netstat -rnfinet)

Connect to another, and post the Diagnostics > Routes (or netstat -rnfinet)

You'll probably want to sanitize public IPs. Everybody does. Please don't sanitize private, RFC1918 addresses. It is possible the VPN provider is screwing this up and we'll need to see that.

johnpoz · Sep 6, 2017, 2:49 AM

Here I just fired up my 2nd vpn client on pfsense - took all of couple of minutes to setup pointing to one of my other vps I have running openvpn-accessserver

As derelict says your going to have to provide more info.. I am quite sure there are many many people with multiple vpn clients setup on pfsense..

Notice my IP I get from vpn - I made sure the openvpn-as were using different networks for vpn clients. Or there would be overlap.. Maybe this what is happening to you. Since maybe they hand out the same networks to all clients because they don't think the same client will be making 2 connections?

Derelict · Sep 6, 2017, 2:51 AM

I have three. All work perfectly.

johnpoz · Sep 6, 2017, 2:55 AM

Well now I am going to have to fire up 2 more just so I have 4 ;) hehehee.. Couldn't let you have more vpns than me.. ROFL…

douglasg14b · Sep 6, 2017, 3:01 AM

First, thank you for your willingness to help thus far, I really do appreciate it.

@Derelict:

Yes, but we all (or at least many, many, many of us) have multiple OpenVPN clients configured and they work just fine.

Connect to one, and post the Diagnostics > Routes (or netstat -rnfinet)

Connect to another, and post the Diagnostics > Routes (or netstat -rnfinet)

That's my train of thought, I'm obviously not the only person trying to do this, and very few other people seem to have issues. A config issue or something is likely the cause, though I have dug in for a few days before posting here to try and rule out PEBKAC.

Please remember I only enable one client at a time, I'm not running multiple at once for these scenarios.

Here are the routing tables:

Note: Column 1 for VPN IP is the Remote Host, Column 2 is the Virtual Address

IPv4 Routes with Client 1 Connected:


Destination	Gateway		Flags	Use	Mtu	Netif	
default		[WAN IP]	UGS	695883	1500	bce1	
8.8.4.4		[WAN IP]	UGHS	0	1500	bce1	
8.8.8.8		[WAN IP]	UGHS	1	1500	bce1	
10.35.0.9	link#7	UH	11157	1500	ovpnc1	
10.35.0.10	link#7	UHS	0	16384	lo0	
[WAN IP]/23	link#2	U	365731	1500	bce1	
[WAN IP]	link#2	UHS	0	16384	lo0	
[VPN I{]	[VPN IP]	UGHS	751	1500	ovpnc1	
127.0.0.1	link#6	UH	17630565	16384	lo0	
192.168.2.0/24	link#1	U	73771556	1500	bce0	
192.168.2.1	link#1	UHS	204	16384	lo0

IPv4 Routes with Client 2 Connected:


Destination	Gateway		Flags	Use	Mtu	Netif	
default		[WAN IP]	UGS	716552	1500	bce1	
8.8.4.4		[WAN IP]	UGHS	0	1500	bce1	
8.8.8.8		[WAN IP]	UGHS	1	1500	bce1	
10.35.0.13	link#8	UH	125	1500	ovpnc2	
10.35.0.14	link#8	UHS	0	16384	lo0	
[WAN IP]/23	link#2	U	368396	1500	bce1	
[WAN IP]	link#2	UHS	0	16384	lo0	
[VPN IP]	[VPN IP]	UGHS	2	1500	ovpnc2	
127.0.0.1	link#6	UH	18127255	16384	lo0	
192.168.2.0/24	link#1	U	73864363	1500	bce0	
192.168.2.1	link#1	UHS	204	16384	lo0

johnpoz · Sep 6, 2017, 3:21 AM

Why are you hiding this?

[VPN I{] [VPN IP] UGHS 751 1500 ovpnc1

It wouldn't be public..
Note: Column 1 for VPN IP is the Remote Host, Column 2 is the Virtual Address

My conf files are client3.x and client4.x there are not client1.x in the var/etc/openvpn dir..

douglasg14b · Sep 6, 2017, 3:42 AM

@johnpoz:

Why are you hiding this?

[VPN I{] [VPN IP] UGHS 751 1500 ovpnc1

It wouldn't be public..
Note: Column 1 for VPN IP is the Remote Host, Column 2 is the Virtual Address

My conf files are client3.x and client4.x there are not client1.x in the var/etc/openvpn dir..

Oh, here then:

VPN 1:

VPN 2 (non working):

Derelict · Sep 6, 2017, 5:54 AM

Do you see how the tunnel addresses are in 10.35.0.X/?? for both ovpnc1 and ovpnc2? That certainly could be part of the problem, depending on the netmasks involved. It looks like they are using net30 which should be OK.

You also have 8.8.8.8 and 8.8.4.4 apparently assigned to the OpenVPN gateways and those change. That could also be part of the problem, depending on what is set to use those DNS server addresses, including the firewall itself.

I don't know what those routes for the 104.223.91 addresses are, but they are also host routes assigned to individual OpenVPN clients.

You do realize that you have to completely stop and start an OpenVPN client after assigning an interface to it, right?

Not sure what you're doing wrong…

If I …. all connections time out.

What, exactly, does that mean. All connections from where? To where? What find of connection? Do DNS lookups work? pings? is it just web browsing? Everything? Have you sent any traceroutes? where do they stop? Sorry but this is going to require some sleuthing.

douglasg14b · Sep 6, 2017, 5:25 PM

@Derelict:

Do you see how the tunnel addresses are in 10.35.0.X/?? for both ovpnc1 and ovpnc2? That certainly could be part of the problem, depending on the netmasks involved. It looks like they are using net30 which should be OK.

You also have 8.8.8.8 and 8.8.4.4 apparently assigned to the OpenVPN gateways and those change. That could also be part of the problem, depending on what is set to use those DNS server addresses, including the firewall itself.

I don't know what those routes for the 104.223.91 addresses are, but they are also host routes assigned to individual OpenVPN clients.

You do realize that you have to completely stop and start an OpenVPN client after assigning an interface to it, right?

Not sure what you're doing wrong…

If I …. all connections time out.

What, exactly, does that mean. All connections from where? To where? What find of connection? Do DNS lookups work? pings? is it just web browsing? Everything? Have you sent any traceroutes? where do they stop? Sorry but this is going to require some sleuthing.

I stop the clients before changing or assigning the interface, yes. As for timing out, I can't ping any addresses I've tried, DNS lookups fail (If I clear my local DNS cache), traceroutes timeout at every hop.

I've removed all clients, interfaces, and gateways and setup most of these things from scratch again to try and simplify things and remove the other "junk" that was a result of me testing. Here are my routes now (issue still exists)

Client 1 (working)

Client 2 (timeouts)

Derelict · Sep 6, 2017, 6:35 PM

What are your policy routing rules that send traffic over the openvpn circuits?

douglasg14b · Sep 8, 2017, 6:40 PM

@Derelict:

What are your policy routing rules that send traffic over the openvpn circuits?

The first two rules are evaluated first to allow certain devices or sites to use the WAN gateway instead of the VPN. This is mostly due to certain services restricting or blocking access. These work as expected.

The highlighted rule is the VPN rule, it should be a catch-all.

The next two rules are to try and prevent torrents from leaking over my WAN if for some reason the VPN is down.

The final rule is the WAN catch-all for IPv4.

When I try and use my 2nd VPN client I just go into the highlighted rule and switch it to the gateway for the 2nd VPN client for testing purpose. I can switch the interfaces for each client around in the OpenVPN config and the first one will always work and the 2nd not, regardless of interface & gateway.

Derelict · Sep 8, 2017, 8:34 PM

Sorry. I know that would work if I tried to duplicate it. No idea what you're doing wrong. But you're doing something wrong somewhere.

douglasg14b · Sep 9, 2017, 12:34 AM

@Derelict:

Sorry. I know that would work if I tried to duplicate it. No idea what you're doing wrong. But you're doing something wrong somewhere.

I've checked and referenced to the point where I'm going crazy, and others on here don't seem to know at this point either as my setup looks correct.

100% sure there isn't a bug in PFSense specific to my setup? By process of elimination how much is left? Any way I can try and diagnose this further to rule out additional items, or rule in others? Multiple VPN connections is a must-have for me, so I'm at a loss here…

Edit: Could this have anything to do with the difficult I have with assigning an interface to the OpenVPN client? When I try it would always tell me:

An IPv4 protocol was selected, but the selected interface has no IPv4 address.

Which always seemed like a catch-22 since it can't have an IPv4 address till I start the OpenVPN client up. This can be found on https://github.com/pfsense/pfsense/blob/master/src/usr/local/www/vpn_openvpn_server.php#L277 .