OpenVPN tunnel established, one side's traffic gets lost
-
Hello!
I set up a site-to-site VPN using OpenVPN and I've so far been unable to get pings betwen hosts on opposite sides and for the life of me I cannot figure out why.
I've created a VPN server on my home pfSense as a Peer to Peer (SSL/TLS), UDP, tun. I've generated a CA and certificates on the main local machine. I've added the CA and remote certificate to the remote client, I've configured the local server with local certificate and the shared CA, copied the TLS key over to the remote machine. I've used unused CIDR for the tunnel network, added local LAN CIDR to the local local and remote remote networks; remote LAN CIDR to the local remote network and remote local networks. Then I've created the remote client with the matching details and the FQDN of the local server.
Both sides have wildcard firewall rules on LAN, added OpenVPN connections as interfaces and wildcard rules on those in the firewall section.
The tunnel connects and there's a few KiB of traffic in the status page. Pinging local and remote TEPs from local LAN works. Pinging local TEP from remote LAN works, pinging remote TEP from local LAN doesn't work. So it appears that there is a problem with firewall or routing or something on the remote end, but I cannot figure out what. I've added provisional static routes on each side for the other side's LAN to get it working as the local/remote networks option in the OpenVPN configuration didn't seem to do anything, if I remove these routes then neither side is able to ping the other side's pfSense.
Edit: "Post content was flagged as spam by Akismet.com"
-
@issuehaver said in OpenVPN tunnel established, one side's traffic gets lost:
Pinging local and remote TEPs from local LAN works.
pinging remote TEP from local LAN doesn't work.Apart from that I'm wondering what you call TEP here, these statements seem contrary.
Consider that the computers firewalls will block access from remote networks when using default settings.
-
@viragomann TEP is tunnel endpoint, sorry, the IP address assigned from tunnel network CIDR.
I have tested this from pfSense's Diagnostics -> Ping (going to the other pfSense), otherwise I did verify that the host is indeed pingable locally.
-
@viragomann said in OpenVPN tunnel established, one side's traffic gets lost:
Pinging local and remote TEPs from local LAN works.
pinging remote TEP from local LAN doesn't work.This is a typo, last one is supposed to be local TEP from remote LAN. I would correct it now, but editing after an hour has been forbidden. Apologies.
I just realized it works from inside pfSense in all directions, but doesn't work from outside pfSense. So when I ping local pfSense TEP from remote LAN, it doesn't work. Route tracing from remote LAN shows only response from remote LAN pfSense interface, next one is supposed to be remote TEP but nothing after the first line. So the tunnel is working but I cannot figure out what is blocking it when firewall is wide-open for all these interfaces.
-
@issuehaver
Are both endpoint the default gateways in their respective LAN?Use the pfSense Diagnostics > Ping tool, from the source drop-down select OpenVPN and try a ping to a local computer. Does this work as well?
-
@viragomann said in OpenVPN tunnel established, one side's traffic gets lost:
Are both endpoint the default gateways in their respective LAN?
No, the WAN interface is the default on both pfSenses.
I should mention that the remote side I am currently testing over a connection with CGN (added as WAN in pfSense directly), but since it is a client and the tunnel is established, I guess this is irrelevant? Since the connection has an address from private IP range I suppose pfSense could be trying to route the packets out of WAN interface, but it shouldn't be trying to do that, right? Also the static routes I have in place...
@viragomann said in OpenVPN tunnel established, one side's traffic gets lost:
Use the pfSense Diagnostics > Ping tool, from the source drop-down select OpenVPN and try a ping to a local computer. Does this work as well?
There's two, don't know the difference, I assume one is the interface I added, but the second one is also named the same but with "OpenVPN client:" prepended. I never got past pinging tunnel endpoints so I didn't test much further.
On local side I tried pinging with the default source address and both the VPN ones, none of them work.
On remote side I tried the same, it works with all three.I must be doing something stupid, but ignorance is bliss...
Edit: "Post content was flagged as spam by Akismet.com"
-
@issuehaver said in OpenVPN tunnel established, one side's traffic gets lost:
Are both endpoint the default gateways in their respective LAN?
No, the WAN interface is the default on both pfSenses.
The point is if both VPN endpoints, i.e. both pfSense boxes, are the default gateways for the LAN devices or if there is another router in use.
I should mention that the remote side I am currently testing over a connection with CGN (added as WAN in pfSense directly), but since it is a client and the tunnel is established, I guess this is irrelevant?
Yes, if it's on the client side it doesn't matter. And your connection is already established as you said.
-
@viragomann said in OpenVPN tunnel established, one side's traffic gets lost:
The point is if both VPN endpoints, i.e. both pfSense boxes, are the default gateways for the LAN devices or if there is another router in use.
Oh, no, these pfSense routers are the only routers for these LAN devices, on both sides, 1 per side. On the relevant part of the network at least, there's another one downstream, unrelated. The pfSense machines themselves are the only gateways, but the VPN endpoint is not the default gateway inside the router, the WAN connection is (just to clarify).
-
@issuehaver said in OpenVPN tunnel established, one side's traffic gets lost:
There's two, don't know the difference, I assume one is the interface I added, but the second one is also named the same but with "OpenVPN client:" prepended. I never got past pinging tunnel endpoints so I didn't test much further.
"OpenVPN" is an interface group which is implicitly created by pfSense when you set up an OpenVPN instance.
Therefore it's not wise to call an self assigned interface as well "OpenVPN".On local side I tried pinging with the default source address and both the VPN ones, none of them work.
At least it should work with the default source.
So possibly the destination device itself is blocking the access if it is coming from outside its network.
-
@viragomann said in OpenVPN tunnel established, one side's traffic gets lost:
"OpenVPN" is an interface group which is implicitly created by pfSense when you set up an OpenVPN instance.
Ah interface group, that's why there's a shared one.
There's a separate interface for the VPN as well though, so the interface I added, the VPN that's added by pfSense and the interface group OpenVPN, 3 of them. But nevermind that part.The problem is the same when I ping the pfSense interface on LAN the same way, same happens:
@issuehaver said in OpenVPN tunnel established, one side's traffic gets lost:
On local side I tried pinging with the default source address and both the VPN ones, none of them work.
On remote side I tried the same, it works with all three. -
This post is deleted! -
Ok, so, from pfSense I can ping both endpoints from both sides, but from a machine on the network I can ping all except from remote LAN to local tunnel endpoint.
Same when I try pinging pfSense's LAN address on local side from remote pfSense - all dropped. This looks like a problem with pfSense. -
I've checked the traffic on "OpenVPN Server: Site-to-site VPN" interface and the equivalent client interface; when I try to ping the local TEP from the remote side (from within pfSense), I can see the ICMP packets on the client side at the tunnel endpoint interface, and at the equivalent server TEP interface. But when I do it from remote LAN, I can see the packet at the LAN interface, and the remote TEP interface with 1 subtracted from TTL, but it doesn't appear at the server TEP interface. What would this indicate?
-
@viragomann said in OpenVPN tunnel established, one side's traffic gets lost:
Use the pfSense Diagnostics > Ping tool, from the source drop-down select OpenVPN and try a ping to a local computer. Does this work as well?
This simple test could shed some light. Do it on both sides and provide what you get.
-
@issuehaver said in OpenVPN tunnel established, one side's traffic gets lost:
On local side I tried pinging with the default source address and both the VPN ones, none of them work.
On remote side I tried the same, it works with all three.I did this already!
Unless you wanted me to ping machines local to the router on both sides, in which case it doesn't work on either side (pfSense ping to a machine on its own LAN). On local side I get packets dropped, on the remote side I get the following:PING 192.168.130.101 (192.168.130.101) from 192.168.240.2: 56 data bytes 92 bytes from 192.168.240.1: Redirect Host(New addr: 192.168.240.2) Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 0054 b167 0 0000 3f 01 d688 192.168.240.2 192.168.130.101 92 bytes from 192.168.240.1: Redirect Host(New addr: 192.168.240.2) Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 0054 7741 0 0000 3f 01 10af 192.168.240.2 192.168.130.101 92 bytes from 192.168.240.1: Redirect Host(New addr: 192.168.240.2) Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 0054 29e5 0 0000 3f 01 5e0b 192.168.240.2 192.168.130.101 --- 192.168.130.101 ping statistics --- 3 packets transmitted, 0 packets received, 100.0% packet lossThis looks like it tried to send the packet over VPN instead of locally and got the packet handed back to it by the other TEP and then it got lost. It looks as if routing isn't working anywhere.
The machine I am pinging on both sides definitely responds to out-of-subnet (if there are any) pings because I can ping it from other LANs, 192.168.91.0/24 for example.Also I am occasionally getting this error in OpenVPN log on the client side:
Jul 17 16:26:46 openvpn 60908 ERROR: FreeBSD route add command failed: external program exited with error status: 1 Jul 17 16:26:46 openvpn 60908 ERROR: FreeBSD route add command failed: external program exited with error status: 1 -
@issuehaver said in OpenVPN tunnel established, one side's traffic gets lost:
Also I am occasionally getting this error in OpenVPN log on the client side:
Jul 17 16:26:46 openvpn 60908 ERROR: FreeBSD route add command failed: external program exited with error status: 1
Jul 17 16:26:46 openvpn 60908 ERROR: FreeBSD route add command failed: external program exited with error status: 1Occasionally?
Is this all of the adding route issue you can find in the log?To troubleshoot this, need to know all subnets on the router and also the OpenVPN configuration. The pfSense routing tables would be helpful.
-
@viragomann Let's say it's all the time, I was troubleshooting and removed the "remote networks" in the remote side config and I think that's when it went away, but the server "local networks" was set all the time, there's an issue with something here but I don't see what. The error repeats each time the tunnel is connected.
This is all I see on the remote side, except transient error when disconnecting Internet. Local side has no errors that I can see except transients.@viragomann said in OpenVPN tunnel established, one side's traffic gets lost:
To troubleshoot this, need to know all subnets on the router and also the OpenVPN configuration. The pfSense routing tables would be helpful.
Tunnel is using 192.168.240.0/28, with assigned TEPs 192.168.240.1 for local and 192.168.240.2 for remote.
Remote network LAN is 192.168.130.0/24, WAN is an Android hotspot with 192.168.219.0/24 network.
Local network LAN is 192.168.1.0/24, WAN is direct Internet address.I've described what I did in the configuration in the first post, tell me if you need anything else.
As mentioned, I have provisional static routes in place for the opposite LANs, with local side having
192.168.130.0/24 with gateway 192.168.240.2
and remote side having
192.168.1.0/24 with 192.168.240.1.
If I remove this, the 192.168.1.0/24 route disappears from routing table.Local side routing table (the relevant part, there's more downstream subnets that are not relevant here):
Destination Gateway Flags Use Mtu Netif Expire default [ISP gateway] UGS 10266 1492 pppoe0 1.0.0.1 [ISP gateway] UGHS 32874 1492 pppoe0 1.1.1.1 [ISP gateway] UGHS 33231 1492 pppoe0 [Internet address] link#15 UHS 6 16384 lo0 127.0.0.1 link#6 UH 798349 16384 lo0 [ISP gateway] link#15 UH 60857 1492 pppoe0 192.168.1.0/24 link#1 U 146973429 1500 em0 192.168.1.250 link#1 UHS 0 16384 lo0 192.168.2.0/24 link#2 U 60651 1500 vmx0 192.168.130.0/24 192.168.240.2 UGS 0 1500 ovpns3 192.168.240.0/28 192.168.240.2 UGS 0 1500 ovpns3 192.168.240.1 link#18 UHS 0 16384 lo0 192.168.240.2 link#18 UH 65134 1500 ovpns3Remote side routing table:
Destination Gateway Flags Use Mtu Netif Expire default 192.168.219.216 UGS 3898 1400 ue0 127.0.0.1 link#4 UH 12126 16384 lo0 192.168.1.0/24 192.168.240.1 UGS 0 1500 ovpnc1 192.168.130.0/24 link#1 U 18522729 1500 vmx0 192.168.130.250 link#1 UHS 0 16384 lo0 192.168.219.0/24 link#7 U 1483 1400 ue0 192.168.219.134 link#7 UHS 13 16384 lo0 192.168.240.0/28 192.168.240.1 UGS 0 1500 ovpnc1 192.168.240.1 link#8 UH 1467 1500 ovpnc1 192.168.240.2 link#8 UHS 0 16384 lo0 -
@issuehaver
So I assume, the add routes issue was coming from having static routes in place, while OpenVPN tries to add a route for same route network.You shouldn't set static routes for networks across the VPN. This should be done by OpenVPN only. Use the "Remote Networks" box in the OpenVPN settings on both sites for setting the routes properly.
After removing the static routes and setting it in the OpenVPN you should see the equal routes when the connection is established.Further it's recommended to use a /30 tunnel subnet for a site-to-site VPN.
-
I removed the manual static route on both ends. Now the remote side has only one error:
"ERROR: FreeBSD route add command failed: external program exited with error status: 1"Okay, now the routing tables haven't changed compared to with static routes. I don't know why this wasn't working properly before, I had it without static routes and they were missing in the table.
Nothing changed regarding connectivity though; still can't get data through.
@viragomann said in OpenVPN tunnel established, one side's traffic gets lost:
Further it's recommended to use a /30 tunnel subnet for a site-to-site VPN.
I read in the instructions that /30 tunnels are used when they're the only planned remote site, to use larger than /30 if you plant to have more sites. So that's why I changed it a larger subnet. Also says it behaves differently with /30 subnet, with some things unavailable, like pushing routes and settings to clients, but most importantly the only one client part.
-
@issuehaver said in OpenVPN tunnel established, one side's traffic gets lost:
I removed the manual static route on both ends. Now the remote side has only one error:
"ERROR: FreeBSD route add command failed: external program exited with error status: 1"Have you entered a network on the local side at "Local Network"?
For a site-to-site I recommend to keep this empty and put the servers LANs into the "Remote Networks" box on the client.@issuehaver said in OpenVPN tunnel established, one side's traffic gets lost:
On local side I tried pinging with the default source address and both the VPN ones, none of them work.
On remote side I tried the same, it works with all three.I did this already!
Unless you wanted me to ping machines local to the router on both sides, in which case it doesn't work on either side (pfSense ping to a machine on its own LAN). On local side I get packets dropped, on the remote side I get the following:This seems quite strange to me at all.
To be clear, does the ping at least work when with the default source?