Multiple OpenVPN clients non-functional



  • TL;DR: All traffic on additional OpenVPN clients created after the first one times out.

    I'm trying to setup multiple VPN clients to route different traffic though (ie. One for web traffic, one for torrenting, one for streaming). Let's ignore any firewall rule shenanigans for the time being, since I am treating every client the same for now. I have 2 interfaces, both are setup to allow all traffic before my WAN interface in the firewall rules, this works fine. Lets call them Int1 & Int2. Lets also call my OpenVPN clients VPN1 & VPN2

    If I create an OpenVPN client VPN1 and assign Int1 to it, it connects just fine and works, traffic flows through…etc All is good. Similarly if I assign  Int2 to VPN1, traffic flows fine, all is good.

    If I create a 2nd OpenVPN client VPN2, and assign it to Int2, and disable the firewall rule for Int1 to force the traffic through Int2 & VPN2, all connections time out. Similarly if I disable VPN1 entirely, and assign either interface to VPN2 all traffic times out.  If I delete VPN1, the issue persists for VPN2. If I delete all clients and recreate them, the 1st client always works, and any subsequent clients do not. VPN2's status will show as up, and with an IP. The Sent Bytes will slowly rise, but the Received Bytes will stay around 5 Bytes and not go up.

    Sometimes VPN2's status will go to down, when that happens the following log entry only shows up at lvl 7 or higher:

    TLS Warning: no data channel send key available: [key#0 state=S_PRE_START id=0 sid=00000000 00000000] [key#1 state=S_UNDEF id=0 sid=00000000 00000000] [key#2 state=S_UNDEF id=0 sid=00000000 00000000]

    I have verified that the config files and all keys/CA's are exactly the same in var/etc/openvpn. When I delete VPN1 all config files for VPN2 continue to be called client2.*. If I then create a new client, it's config files are all named client1.*, and the new client will work.

    It seems any client that has config files with anything past client1 does not function. Any idea why this is and how I can resolve it?

    Example Config:

    dev ovpnc2
    verb 7
    dev-type tun
    dev-node /dev/tun2
    writepid /var/run/openvpn_client2.pid
    #user nobody
    #group nobody
    script-security 3
    daemon
    keepalive 10 60
    ping-timer-rem
    persist-tun
    persist-key
    proto udp
    cipher AES-128-CBC
    auth SHA256
    up /usr/local/sbin/ovpn-linkup
    down /usr/local/sbin/ovpn-linkdown
    tls-client
    client
    nobind
    management /var/etc/openvpn/client2.sock unix
    remote west.usa.torguardvpnaccess.com 1912
    auth-user-pass /var/etc/openvpn/client2.up
    ca /var/etc/openvpn/client2.ca
    cert /var/etc/openvpn/client2.cert
    key /var/etc/openvpn/client2.key
    tls-auth /var/etc/openvpn/client2.tls-auth 1
    resolv-retry infinite
    remote-cert-tls server


  • LAYER 8 Global Moderator

    If your going to have more than 1 vpn client on pfsense you need to make sure your not pulling routes.. Since they will like to setup default route going out that vpn.

    Once you do not pull routes you can then route the traffic you want out the vpn connections via policy routing.



  • @johnpoz:

    If your going to have more than 1 vpn client on pfsense you need to make sure your not pulling routes.. Since they will like to setup default route going out that vpn.

    Once you do not pull routes you can then route the traffic you want out the vpn connections via policy routing.

    johnpoz, can you explain that more for me? I'm fairly new at this and am not sure what "pulling routes" means/entails.

    Would this also explain a single VPN client acting this way when it's anything other than client1 in it's config? (Create two clients, delete the first one, the 2nd one still doesn't work. Create a 3rd one, the 3rd one's config is named client1.* and it will work just fine)


  • LAYER 8 Global Moderator

    In your vpn client config - there is a check box NOT to pull routes..

    You could also have issues if your different vpn services are handing you the same network for each client..  When I get a chance I will fire up a client connection to one of my other vps running openvpn access server.  I normally only maintain a client connection to one of them 24/7  But I can fire a couple of more to show you as example if needed..

    BTW your not trying to create multiple connections to the same vpn service are you?




  • @johnpoz:

    In your vpn client config - there is a check box NOT to pull routes..

    You could also have issues if your different vpn services are handing you the same network for each client..  When I get a chance I will fire up a client connection to one of my other vps running openvpn access server.  I normally only maintain a client connection to one of them 24/7  But I can fire a couple of more to show you as example if needed..

    BTW your not trying to create multiple connections to the same vpn service are you?

    Gotcha, I have now checked that box, issue persists.

    I want to note that I don't need both clients to be enabled to see this issue. I can create two clients both disabled upon creation, enable the 2nd one and the 2nd one will not work, but the 1st one will when enabled.

    It shouldn't matter if it's to the same service in this instance since the clients are disabled anyways? But if it did, I am connecting to the same provider, but entirely different servers (One server is dedicated for streaming traffic).


  • LAYER 8 Global Moderator

    "I am connecting to the same provider, but entirely different servers (One server is dedicated for streaming traffic)."

    And your vpn service provides service to multiple connections from the same IP?  I find that unlikely..



  • @johnpoz:

    "I am connecting to the same provider, but entirely different servers (One server is dedicated for streaming traffic)."

    And your vpn service provides service to multiple connections from the same IP?  I find that unlikely..

    They do, I pay for it specifically.  If I enable multiple clients they all connect and receive IPs from the provider.

    I feel like we're getting hung up on issues that are not the root of the problem, and the main body of my posts isn't being read or understood…

    The above is a side-issue, I don't need multiple active clients for this problem to occur. I mentioned that I can create two or more clients, all disabled upon creation. If I enable any client except for the first one they do not work, all traffic times out. I only have ONE active VPN connection during this scenario.

    Edit: What do you need from me?

    If I delete the first, working, client. All other clients still continue to be non-functional when enabled, all clients will have their original config names. If I create an additional client, it will have a config named client1.*, and it will work. This seems to point towards some issue in how PFsense or OpenVPN is handling additional clients?


  • LAYER 8 Netgate

    Yes, but we all (or at least many, many, many of us) have multiple OpenVPN clients configured and they work just fine. OpenVPN is routed using the routing table. One client being started should not affect other clients or other traffic unless there is something misconfigured.

    You will need to provide more information.

    You want to make sure that don't pull routes is enabled on ALL your client configs for now.

    Connect to one, and post the Diagnostics > Routes (or netstat -rnfinet)

    Connect to another, and post the Diagnostics > Routes (or netstat -rnfinet)

    You'll probably want to sanitize public IPs. Everybody does. Please don't sanitize private, RFC1918 addresses. It is possible the VPN provider is screwing this up and we'll need to see that.


  • LAYER 8 Global Moderator

    Here I just fired up my 2nd vpn client on pfsense - took all of couple of minutes to setup pointing to one of my other vps I have running openvpn-accessserver

    As derelict says your going to have to provide more info.. I am quite sure there are many many people with multiple vpn clients setup on pfsense..

    Notice my IP I get from vpn - I made sure the openvpn-as were using different networks for vpn clients.  Or there would be overlap.. Maybe this what is happening to you.  Since maybe they hand out the same networks to all clients because they don't think the same client will be making 2 connections?



  • LAYER 8 Netgate

    I have three. All work perfectly.


  • LAYER 8 Global Moderator

    Well now I am going to have to fire up 2 more just so I have 4 ;) hehehee.. Couldn't let you have more vpns than me.. ROFL…



  • First, thank you for your willingness to help thus far, I really do appreciate it.

    @Derelict:

    Yes, but we all (or at least many, many, many of us) have multiple OpenVPN clients configured and they work just fine.

    Connect to one, and post the Diagnostics > Routes (or netstat -rnfinet)

    Connect to another, and post the Diagnostics > Routes (or netstat -rnfinet)

    That's my train of thought, I'm obviously not the only person trying to do this, and very few other people seem to have issues. A config issue or something is likely the cause, though I have dug in for a few days before posting here to try and rule out PEBKAC.

    Please remember I only enable one client at a time, I'm not running multiple at once for these scenarios.

    Here are the routing tables:

    Note: Column 1 for VPN IP is the Remote Host, Column 2 is the Virtual Address

    IPv4 Routes with Client 1 Connected:

    
    Destination	Gateway		Flags	Use	Mtu	Netif	
    default		[WAN IP]	UGS	695883	1500	bce1	
    8.8.4.4		[WAN IP]	UGHS	0	1500	bce1	
    8.8.8.8		[WAN IP]	UGHS	1	1500	bce1	
    10.35.0.9	link#7	UH	11157	1500	ovpnc1	
    10.35.0.10	link#7	UHS	0	16384	lo0	
    [WAN IP]/23	link#2	U	365731	1500	bce1	
    [WAN IP]	link#2	UHS	0	16384	lo0	
    [VPN I{]	[VPN IP]	UGHS	751	1500	ovpnc1	
    127.0.0.1	link#6	UH	17630565	16384	lo0	
    192.168.2.0/24	link#1	U	73771556	1500	bce0	
    192.168.2.1	link#1	UHS	204	16384	lo0
    
    

    IPv4 Routes with Client 2 Connected:

    
    Destination	Gateway		Flags	Use	Mtu	Netif	
    default		[WAN IP]	UGS	716552	1500	bce1	
    8.8.4.4		[WAN IP]	UGHS	0	1500	bce1	
    8.8.8.8		[WAN IP]	UGHS	1	1500	bce1	
    10.35.0.13	link#8	UH	125	1500	ovpnc2	
    10.35.0.14	link#8	UHS	0	16384	lo0	
    [WAN IP]/23	link#2	U	368396	1500	bce1	
    [WAN IP]	link#2	UHS	0	16384	lo0	
    [VPN IP]	[VPN IP]	UGHS	2	1500	ovpnc2	
    127.0.0.1	link#6	UH	18127255	16384	lo0	
    192.168.2.0/24	link#1	U	73864363	1500	bce0	
    192.168.2.1	link#1	UHS	204	16384	lo0
    
    

  • LAYER 8 Global Moderator

    Why are you hiding this?

    [VPN I{] [VPN IP] UGHS 751 1500 ovpnc1

    It wouldn't be public..
    Note: Column 1 for VPN IP is the Remote Host, Column 2 is the Virtual Address

    My conf files are client3.x and client4.x there are not client1.x in the var/etc/openvpn dir..



  • @johnpoz:

    Why are you hiding this?

    [VPN I{] [VPN IP] UGHS 751 1500 ovpnc1

    It wouldn't be public..
    Note: Column 1 for VPN IP is the Remote Host, Column 2 is the Virtual Address

    My conf files are client3.x and client4.x there are not client1.x in the var/etc/openvpn dir..

    Oh, here then:

    VPN 1:

    VPN 2 (non working):


  • LAYER 8 Netgate

    Do you see how the tunnel addresses are in 10.35.0.X/?? for both ovpnc1 and ovpnc2? That certainly could be part of the problem, depending on the netmasks involved. It looks like they are using net30 which should be OK.

    You also have 8.8.8.8 and 8.8.4.4 apparently assigned to the OpenVPN gateways and those change. That could also be part of the problem, depending on what is set to use those DNS server addresses, including the firewall itself.

    I don't know what those routes for the 104.223.91 addresses are, but they are also host routes assigned to individual OpenVPN clients.

    You do realize that you have to completely stop and start an OpenVPN client after assigning an interface to it, right?

    Not sure what you're doing wrong…

    If I …. all connections time out.

    What, exactly, does that mean. All connections from where? To where? What find of connection? Do DNS lookups work? pings? is it just web browsing? Everything? Have you sent any traceroutes? where do they stop? Sorry but this is going to require some sleuthing.



  • @Derelict:

    Do you see how the tunnel addresses are in 10.35.0.X/?? for both ovpnc1 and ovpnc2? That certainly could be part of the problem, depending on the netmasks involved. It looks like they are using net30 which should be OK.

    You also have 8.8.8.8 and 8.8.4.4 apparently assigned to the OpenVPN gateways and those change. That could also be part of the problem, depending on what is set to use those DNS server addresses, including the firewall itself.

    I don't know what those routes for the 104.223.91 addresses are, but they are also host routes assigned to individual OpenVPN clients.

    You do realize that you have to completely stop and start an OpenVPN client after assigning an interface to it, right?

    Not sure what you're doing wrong…

    If I …. all connections time out.

    What, exactly, does that mean. All connections from where? To where? What find of connection? Do DNS lookups work? pings? is it just web browsing? Everything? Have you sent any traceroutes? where do they stop? Sorry but this is going to require some sleuthing.

    I stop the clients before changing or assigning the interface, yes. As for timing out, I can't ping any addresses I've tried, DNS lookups fail (If I clear my local DNS cache), traceroutes timeout at every hop.

    I've removed all clients, interfaces, and gateways and setup most of these things from scratch again to try and simplify things and remove the other "junk" that was a result of me testing. Here are my routes now (issue still exists)

    Client 1 (working)

    Client 2 (timeouts)


  • LAYER 8 Netgate

    What are your policy routing rules that send traffic over the openvpn circuits?



  • @Derelict:

    What are your policy routing rules that send traffic over the openvpn circuits?

    The first two rules are evaluated first to allow certain devices or sites to use the WAN gateway instead of the VPN. This is mostly due to certain services restricting or blocking access. These work as expected.

    The highlighted rule is the VPN rule, it should be a catch-all.

    The next two rules are to try and prevent torrents from leaking over my WAN if for some reason the VPN is down.

    The final rule is the WAN catch-all for IPv4.


    When I try and use my 2nd VPN client I just go into the highlighted rule and switch it to the gateway for the 2nd VPN client for testing purpose. I can switch the interfaces for each client around in the OpenVPN config and the first one will always work and the 2nd not, regardless of interface & gateway.


  • LAYER 8 Netgate

    Sorry. I know that would work if I tried to duplicate it. No idea what you're doing wrong. But you're doing something wrong somewhere.



  • @Derelict:

    Sorry. I know that would work if I tried to duplicate it. No idea what you're doing wrong. But you're doing something wrong somewhere.

    I've checked and referenced to the point where I'm going crazy, and others on here don't seem to know at this point either as my setup looks correct.

    100% sure there isn't a bug in PFSense specific to my setup? By process of elimination how much is left? Any way I can try and diagnose this further to rule out additional items, or rule in others? Multiple VPN connections is a must-have for me, so I'm at a loss here…

    Edit: Could this have anything to do with the difficult I have with assigning an interface to the OpenVPN client? When I try it would always tell me:

    An IPv4 protocol was selected, but the selected interface has no IPv4 address.

    Which always seemed like a catch-22 since it can't have an IPv4 address till I start the OpenVPN client up. This can be found on https://github.com/pfsense/pfsense/blob/master/src/usr/local/www/vpn_openvpn_server.php#L277 .


  • LAYER 8 Netgate

    You don't put an address on an OpenVPN assigned interface. They get Enable checked, a name, then None / None selected there. That's it.

    I have never seen that message when assigning an interface. What, exactly, are you doing when it happens?



  • @Derelict:

    You don't put an address on an OpenVPN assigned interface. They get Enable checked, a name, then None / None selected there. That's it.

    I have never seen that message when assigning an interface. What, exactly, are you doing when it happens?

    I get this error when creating or editing the OpenVPN client when I have an interface like the one below selected. I can just create a new client, enter any random gibberish into it and select an interface and that error will pop up. I get around it by commenting out that line of code.

    Here is an example OpenVPN client: (not embedded because long) http://i.imgur.com/znt5J08.png

    Interface is setup with none/none:


  • LAYER 8 Netgate

    Aha. I knew it had to be something.

    The interface you select there is not the assigned interface, it is the interface used to establish the VPN connection - usually WAN. Or maybe a gateway group if you are running multi-wan and you want it to switch if your tier 1 fails.

    So all of your OpenVPN client configurations will have WAN selected there because that is the interface used to ESTABLISH the VPN connection.


  • LAYER 8 Global Moderator

    Good catch Derelict..  Glad that is settled…



  • @Derelict:

    Aha. I knew it had to be something.

    The interface you select there is not the assigned interface, it is the interface used to establish the VPN connection - usually WAN. Or maybe a gateway group if you are running multi-wan and you want it to switch if your tier 1 fails.

    So all of your OpenVPN client configurations will have WAN selected there because that is the interface used to ESTABLISH the VPN connection.

    Sorry for being dull, but I'm gonna parrot what you said to make sure I understand.

    The WAN interface is used to establish the VPN connection, so regardless of what interface I have selected there (unless I have multi-wan) it will use the WAN interface?

    If that's the case, how do I set up routing rules for the VPN clients if they will all use the WAN interface? Do I always select  WAN in my OpenVPN client setup?


  • LAYER 8 Netgate

    You have to specify a real interface there. In the example you posted to imgur, when your firewall needs to connect to xxx.vpnaccess.com on server port xxxx it will use the interface specified there to source from for the TUNNEL PACKETS - the Outer tunnel packets. It has nothing to do with what traffic is sent through the tunnel itself. It has to be a real interface. What you have there is nonsense.

    Create all three OpenVPN clients. Tell them all to use the WAN interface.

    Go into Interfaces > (assign). Assign an interface to each ovpncx instance.

    Edit each interface, enable it, name it, leave the IPv4 and IPv6 configurations as None.

    Make sure you bounce each OpenVPN as this is a required step after initial interface assignment. Another Edit/Save with no changes of each OpenVPN client will do this, as will a Stop/Start of each service in Status > Services.

    Make sure outbound NAT is set for each new interface so traffic leaving that interface gets source translated to that particular tunnel address.

    You can then policy route whatever traffic you want over each OpenVPN at any time using the policy routing rules on the source network interface. You will have three OpenVPN gateways to choose from. One on each assigned interface.



  • @Derelict:

    You have to specify a real interface there. In the example you posted to imgur, when your firewall needs to connect to xxx.vpnaccess.com on server port xxxx it will use the interface specified there to source from for the TUNNEL PACKETS - the Outer tunnel packets. It has nothing to do with what traffic is sent through the tunnel itself. It has to be a real interface. What you have there is nonsense.

    Create all three OpenVPN clients. Tell them all to use the WAN interface.

    Go into Interfaces > (assign). Assign an interface to each ovpncx instance.

    Edit each interface, enable it, name it, leave the IPv4 and IPv6 configurations as None.

    Make sure you bounce each OpenVPN as this is a required step after initial interface assignment. Another Edit/Save with no changes of each OpenVPN client will do this, as will a Stop/Start of each service in Status > Services.

    Make sure outbound NAT is set for each new interface so traffic leaving that interface gets source translated to that particular tunnel address.

    You can then policy route whatever traffic you want over each OpenVPN at any time using the policy routing rules on the source network interface. You will have three OpenVPN gateways to choose from. One on each assigned interface.

    And we're working!

    Thanks a ton for sticking with me and helping me sort this out.


  • LAYER 8 Netgate

    Excellent to hear. Glad it's working.


  • LAYER 8 Netgate

    It is absolutely NOT possible to run 2 concurrent OpenVPN clients in 2.4.

    Absolute nonsense.



  • @sense678:

    @Derelict:

    It is absolutely NOT possible to run 2 concurrent OpenVPN clients in 2.4.

    Absolute nonsense.

    Great answer! At least wirh AirVPN this does not. I changed nothing configuration wise, just updated and nothing is working anymore. Then did everything from scratch, not working. I configured 20 times minimum 2 concurrent VPN on AirVPN with 2.x. No problems, never.
    And as said there are always hard ifconfig errors coming up, change just a simple setting in Routing of one VPNM and both go down (a monitoring IP for instance).
    What do you want to tell me with this significant sentence?

    If you can help help otherwise why bother?

    When I was testing 2.4 I was getting ifconfig errors too.  But, obviously it works for some people.

    I've always had Don't Pull Routes and Don't Add/Remove Routes enabled for my OVPN clients.

    The networks for the different VPNs I used did not overlap.

    Anyway, I went back to 2.3.x and am happy in the meantime.  Hopefully this will get figured out because I've seen several other people with the same problem.



  • @cosmoxl:

    @sense678:

    @Derelict:

    It is absolutely NOT possible to run 2 concurrent OpenVPN clients in 2.4.

    Absolute nonsense.

    Great answer! At least wirh AirVPN this does not. I changed nothing configuration wise, just updated and nothing is working anymore. Then did everything from scratch, not working. I configured 20 times minimum 2 concurrent VPN on AirVPN with 2.x. No problems, never.
    And as said there are always hard ifconfig errors coming up, change just a simple setting in Routing of one VPNM and both go down (a monitoring IP for instance).
    What do you want to tell me with this significant sentence?

    If you can help help otherwise why bother?

    When I was testing 2.4 I was getting ifconfig errors too.  But, obviously it works for some people.

    I've always had Don't Pull Routes and Don't Add/Remove Routes enabled for my OVPN clients.

    The networks for the different VPNs I used did not overlap.

    Anyway, I went back to 2.3.x and am happy in the meantime.  Hopefully this will get figured out because I've seen several other people with the same problem.

    Don't pull routes should be standard, yes. That was not the problem. If you had monitoring IPs in your Routing try to to remove them and just leave it blank. That did it for me. In pfsense 2.3.x this was no problem at all but in 2.4.1 it seems to. So no monitoring IPs and everything is working as expected now.

    Thank you for your answer.

    Cheers



  • @sense678:

    @cosmoxl:

    @sense678:

    @Derelict:

    It is absolutely NOT possible to run 2 concurrent OpenVPN clients in 2.4.

    Absolute nonsense.

    Great answer! At least wirh AirVPN this does not. I changed nothing configuration wise, just updated and nothing is working anymore. Then did everything from scratch, not working. I configured 20 times minimum 2 concurrent VPN on AirVPN with 2.x. No problems, never.
    And as said there are always hard ifconfig errors coming up, change just a simple setting in Routing of one VPNM and both go down (a monitoring IP for instance).
    What do you want to tell me with this significant sentence?

    If you can help help otherwise why bother?

    When I was testing 2.4 I was getting ifconfig errors too.  But, obviously it works for some people.

    I've always had Don't Pull Routes and Don't Add/Remove Routes enabled for my OVPN clients.

    The networks for the different VPNs I used did not overlap.

    Anyway, I went back to 2.3.x and am happy in the meantime.  Hopefully this will get figured out because I've seen several other people with the same problem.

    Don't pull routes should be standard, yes. That was not the problem. If you had monitoring IPs in your Routing try to to remove them and just leave it blank. That did it for me. In pfsense 2.3.x this was no problem at all but in 2.4.1 it seems to. So no monitoring IPs and everything is working as expected now.

    Thank you for your answer.

    Cheers

    Oh, is that it?  Thanks for that tip.  Yes, I do edit the IP for gateway monitoring as I like to see the RTT to the other side of the tunnel.  Stinks that won't work in 2.4.



  • Same with me or like i did. Deleting it solved ALL the problems there were with the gateways, the ifconfig problems and everything else.
    In a second :(


Log in to reply