routing bounces between vpn tunnels
-
I have two OpenVPN clients set up on my pfSense box, both work fine when only one is turned on, but when I turn them both on, the routing for the server side of the VPNs bounces between the two when accessed by a client on my client side network, when accessed form pfSense, the routing is fine.
I'll try to explain....
Desktop: 192.168.0.7
pfSense: 192.168.0.9
VPN1: 10.8.0.50 - 10.8.0.49 via ovpnc1
VPN2: 10.3.0.14 - 10.3.0.13 via ovpnc3
ovpnc2 doesn't exist.
Host on VPN1: 10.5.1.1
Host on VPN2: 10.6.1.1These are the routes, they look good and are stable, if I make the requests multiple times, I get the same answer every time.
[21.02-RELEASE][root@firewall.krynn.int]/root: route show 10.5.1.1 route to: hb.digi.lab destination: 10.5.1.0 mask: 255.255.255.0 gateway: 10.8.0.49 fib: 0 interface: ovpnc1 flags: <UP,GATEWAY,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 [21.02-RELEASE][root@firewall.krynn.int]/root: route show 10.6.1.1 route to: 10.6.1.1 destination: 10.6.1.0 mask: 255.255.255.0 gateway: 10.3.0.13 fib: 0 interface: ovpnc3 flags: <UP,GATEWAY,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0If I ping from my desktop, this is what I get:
desktop $ ping 10.6.1.1 PING 10.6.1.1 (10.6.1.1) 56(84) bytes of data. ^C --- 10.6.1.1 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2046ms desktop $ ping 10.6.1.1 PING 10.6.1.1 (10.6.1.1) 56(84) bytes of data. 64 bytes from 10.6.1.1: icmp_seq=1 ttl=63 time=12.8 ms 64 bytes from 10.6.1.1: icmp_seq=2 ttl=63 time=12.8 ms 64 bytes from 10.6.1.1: icmp_seq=3 ttl=63 time=12.7 ms ^C --- 10.6.1.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 12.721/12.752/12.773/0.022 ms desktop $ ping 10.6.1.1 PING 10.6.1.1 (10.6.1.1) 56(84) bytes of data. ^C --- 10.6.1.1 ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 1023ms desktop $ ping 10.6.1.1 PING 10.6.1.1 (10.6.1.1) 56(84) bytes of data. 64 bytes from 10.6.1.1: icmp_seq=1 ttl=63 time=12.8 ms 64 bytes from 10.6.1.1: icmp_seq=2 ttl=63 time=12.7 ms ^C --- 10.6.1.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1000ms rtt min/avg/max/mdev = 12.670/12.757/12.844/0.087 ms desktop $The first fails, second works, third fails, fourth works. This is the same for SSH connections and all other traffic.
This is the network traffic from ovpnc3 when I try to ping 10.6.1.1 like above, you can see it alternating between the correct tunnel and the wrong one.
11:51:45.661621 IP 10.8.0.50 > 10.6.1.1: ICMP echo request, id 51622, seq 1, length 64 11:51:46.692061 IP 10.8.0.50 > 10.6.1.1: ICMP echo request, id 51622, seq 2, length 64 11:51:47.720062 IP 10.8.0.50 > 10.6.1.1: ICMP echo request, id 51622, seq 3, length 64 11:51:48.744124 IP 10.8.0.50 > 10.6.1.1: ICMP echo request, id 51622, seq 4, length 64 11:51:49.764144 IP 10.8.0.50 > 10.6.1.1: ICMP echo request, id 51622, seq 5, length 64 11:51:50.788134 IP 10.8.0.50 > 10.6.1.1: ICMP echo request, id 51622, seq 6, length 64 11:51:51.969970 IP 10.3.0.14 > 10.6.1.1: ICMP echo request, id 35180, seq 1, length 64 11:51:51.982337 IP 10.6.1.1 > 10.3.0.14: ICMP echo reply, id 35180, seq 1, length 64 11:51:52.971522 IP 10.3.0.14 > 10.6.1.1: ICMP echo request, id 35180, seq 2, length 64 11:51:52.983773 IP 10.6.1.1 > 10.3.0.14: ICMP echo reply, id 35180, seq 2, length 64 11:51:53.973248 IP 10.3.0.14 > 10.6.1.1: ICMP echo request, id 35180, seq 3, length 64 11:51:53.985299 IP 10.6.1.1 > 10.3.0.14: ICMP echo reply, id 35180, seq 3, length 64 11:51:54.974232 IP 10.3.0.14 > 10.6.1.1: ICMP echo request, id 35180, seq 4, length 64 11:51:54.986547 IP 10.6.1.1 > 10.3.0.14: ICMP echo reply, id 35180, seq 4, length 64 11:52:03.493091 IP 10.8.0.50 > 10.6.1.1: ICMP echo request, id 61572, seq 1, length 64 11:52:04.516061 IP 10.8.0.50 > 10.6.1.1: ICMP echo request, id 61572, seq 2, length 64 11:52:05.540100 IP 10.8.0.50 > 10.6.1.1: ICMP echo request, id 61572, seq 3, length 64 11:52:06.564231 IP 10.8.0.50 > 10.6.1.1: ICMP echo request, id 61572, seq 4, length 64 11:52:08.110077 IP 10.3.0.14 > 10.6.1.1: ICMP echo request, id 46012, seq 1, length 64 11:52:08.122485 IP 10.6.1.1 > 10.3.0.14: ICMP echo reply, id 46012, seq 1, length 64 11:52:09.110866 IP 10.3.0.14 > 10.6.1.1: ICMP echo request, id 46012, seq 2, length 64 11:52:09.123225 IP 10.6.1.1 > 10.3.0.14: ICMP echo reply, id 46012, seq 2, length 64 11:52:10.112106 IP 10.3.0.14 > 10.6.1.1: ICMP echo request, id 46012, seq 3, length 64 11:52:10.124476 IP 10.6.1.1 > 10.3.0.14: ICMP echo reply, id 46012, seq 3, length 64Once an SSH connection is established, it is stable and works fine, it doesn't drop out or have issues.
What could I have done wrong or what should I look at to troubleshoot this?
-
@digininja99
Do you have defined interface groups?What are your outbound NAT rules?
-
@viragomann I think that might be it. When I added the second VPN I added a rule matching the first, but they are both on the OpenVPN interface. Do I need to create two new interfaces in the interface assignments tab and then use those instead?
If I do need to do that, do I need to change anything else when I do it? I remember messing with this a while ago and things broke so I rolled back to just having OpenVPN.
There are no interface groups defined.
-
@digininja99 said in routing bounces between vpn tunnels:
Do I need to create two new interfaces in the interface assignments tab and then use those instead?
If you want to do NAT, yes, then you have to assign a seperate interface.to each instance.
As far as I know, it is not necessary, when you route the traffic. If you have routes on each remote site pointing to your VPN IP, there would be no need for NAT.
But I ever used assigned interfaces for site2site OpenVPN instances myself. -
@viragomann All I want to be able to do is to directly access services running on the server side of the two VPNs, in this case, mostly web apps that are bound to their private IPs rather than public. Do I need NAT for that?
I don't want the server side being able to talk directly to hosts on the client side (pfSense side).
The network works fine with just one VPN enabled so routing in some way is working.
I just tried assigning two new interfaces and now I have OPT2 and OPT3 pointing at ovpnc1 and 3 but I don't know what to do with them now.
-
@digininja99
You to outbound NAT know (masquerading), that means that IPs of request packets is translated when it is going to the remote site.
However "OpenVPN" is indeed an interface group containing all OpenVPN instances. It is created automatically, when you set up an OpenVPN instance.
Since it's an interface group, you cannot use it for NAT.In an routing environment, the remote OpenVPN server also needs a route to your LAN. If that is given, there is no need for NAT and you can remove these rules.
If you don't want to touch the OpenVPN server, do NAT and use separate interfaces for both instances.
I just tried assigning two new interfaces and now I have OPT2 and OPT3 pointing at ovpnc1 and 3 but I don't know what to do with them now.
After assigning the network ports (ovpnc1,..) open the interfaces and enable them. You may also enter a friendly name, but no IP settings.
Then go back to Outbound NAT and edit the rules to change the interface to the proper value. -
@viragomann Things are now broken, but I think they are closer to what they should be.
I created the interfaces (forgot to enable them before), renamed them, and enabled them.
Set up the NAT rules
But now I have no route to the server, 10.254.254.1 is my default gateway.
route to: 10.5.1.1 destination: 0.0.0.0 mask: 0.0.0.0 gateway: 10.254.254.1 fib: 0 interface: mvneta0.4090 flags: <UP,GATEWAY,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0This is the same for 10.6.1.1.
The general config page for OVPN1 says:
This interface type does not support manual address configuration on this page.
Confirmed there is no IP for the two interfaces the console:
ovpnc1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500 description: OVPN1 options=80000<LINKSTATE> inet6 fe80::f2ad:4eff:fe18:9e32%ovpnc1 prefixlen 64 scopeid 0xd groups: tun openvpn nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> Opened by PID 69932 ovpnc3: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500 description: OVPN2 options=80000<LINKSTATE> inet6 fe80::f2ad:4eff:fe18:9e32%ovpnc3 prefixlen 64 scopeid 0xe groups: tun openvpn nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> Opened by PID 77818Do I set this up as a virtual IP?
-
@digininja99
After these changes you should reboot the box.If the error persists, check if you have IPv4 gateway for the VPNs in System > Routing.
How did you set the routes?
-
@viragomann I was told by the interface to restart things, which I did, I haven't rebooted.
I didn't setup any routing, what was there was setup automatically.
In System > Routing, there is no option to set the IP for the gateway, it just says Dynamic on the gateway IP.
There was nothing in static routes, so I tried adding one, but that hasn't changed anything, even with this config, the console still says the default gateway for the server IPs.
-
@digininja99 said in routing bounces between vpn tunnels:
I didn't setup any routing, what was there was setup automatically.
Yes, that is okay. But you should disable IPv6 in the interface settings if you don't need it.
There was nothing in static routes
No, there must not be added static routes for OpenVPN endpoints.
You have to enter the remote networks in the client settings in CIDR notation.
-
@viragomann I've removed the static routes and restarted things.
I have this setup in the OpenVPN config for both interfaces.
The bit I was missing was the IPv4 Tunnel Network IP, I just put that in and everything seems to be working!
I'm now going to back all this up and then grab a copy of this session as notes for if I ever need to add a third VPN.
Thanks very much for the help debugging this, it was more complex than I thought, but in the end it all makes sense I think. I'll re-read it all in the morning, it will probably have sunk in by then.
