P2P VPN server can't reach client, but client can reach server

lifeboy

I recreated the tunnel completely, including a new client. This time I used the wizard and then changed the tunnel type to Peer-to-Peer SSL/TLS. The client connects to the server and I can ping the server LAN clients from the client firewall.

However, the same problem still exists on the client side. I can ping the tunnel network address (which is now 10.0.20.2) from the server firewall (which is 10.0.20.1).

If I do a traceroute, this is the result.

: traceroute -s 10.0.20.1 -P icmp 10.0.20.2
traceroute to 10.0.20.2 (10.0.20.2) from 10.0.20.1, 64 hops max, 48 byte packets
 1  10.0.20.2 (10.0.20.2)  17.378 ms  17.115 ms  17.082 ms

However, if I traceroute the LAN on the client side:

: traceroute -s 10.0.20.1 -P icmp 192.168.111.254
traceroute to 192.168.111.254 (192.168.111.254) from 10.0.20.1, 64 hops max, 48 byte packets
 1  *
 2  *^C

It doesn't seem to go via 10.0.20.2 ?

The routes have been added by OpenVPN.

10.0.20.0/24       link#19            U        ovpns6
10.0.20.1          link#11            UHS         lo0
192.168.111.0/24   10.0.20.2          UGS      ovpns6

Stuck at exactly the same spot that I was before.

lifeboy

I changed the tunnel network to /30. So it's 10.0.20.0/30. The server gets .1 and the client .2

I can ping .2 from the server, but not the client LAN. This is despite the being a route for 192.168.111.0/24 via 10.0.20.2 and having a n OpenVPN rule on the client to allow all traffic.

Either I'm a total idiot or something is just wrong internally with OpenVPN and pfSense.

Here the docs even say that with a /30 no CSO is needed...
https://docs.netgate.com/pfsense/en/latest/troubleshooting/openvpn-iroute.html

rlabaza

@lifeboy I am in the exact same boat as you. Have been for a few days. I have double, triple, quadruple checked everything.

From my 'Site B' (client side) any device on that LAN can essentially reach everything on my server side ('Site A').
From my 'Site A' (OpenVPN server side) pfsense box, I can ping anything on the Site B LAN, but from other devices on the Site A LAN, I get nothing.

My 'SiteA' pfSense box is at 23.09.1. My 'Site B' is an SG-1000 at 2.4.5-RELEASE-p1. I reconfigured OpenVPN from shared key to SSL/TLS following the netgate recipe at https://docs.netgate.com/.../recipes/openvpn-s2s-tls.html. The VPN is up and I can get from Site B to Site A, but I can not get from Site A to Site B. The recipe has this in the client firewall config: "If the other sites needs to initiate contact, then this traffic requires a firewall rule on the OpenVPN tab on the client firewall to allow traffic from other VPN sites to reach the Client-side LAN.". I did that. But I still can not ping from Site A LAN machines to anything at Site B.

Fast-forward to the trouble shooting doc (https://docs.netgate.com/.../troubleshooting/openvpn.html): "Test from different vantage points" -- On Site A pfsense GUI, I can ping anything at Site B using the LAN source or the OpenVPN (tun interface) as source.

Next on to "Trace the traffic with packet captures" -- I ran packet capture and I see pings to a Site B machine from a Site A LAN machine when I capture on the LAN interface, but nothing on the VPN interface!

However, if I ping from the Site A pfSense box, I see the traffic on packet captures for LAN and tun interfaces (as expected since ping works).

So I can only conclude that packets from my Site A LAN devices are not getting internally routed out the VPN interface when they hit the pfsense box. But I don't know why!

The "Check the system routing table" section in troubleshooting says check the routing table--it looks fine to me, and does not offer what an incorrect or missing route looks like. Site A LAN is 192.168.19.0/24, tunnel is 10.30.0.0/24 Site B LAN is 192.168.139.0/24 What am I missing? I think I set Local Network and Remote Network correctly in the OpenVPN server to those routes get created...

Following this thread with great interest...
rl.

rlabaza

@rlabaza

UPDATE: I got my configuration working. Found the answer in this post. I had one of these 'policy rules' that @reberhar pointed out, on my LAN firewall sending traffic out the WAN. I pinged the remote side, this time packet capturing on the WAN interface (wish I would have thought of that a few days ago), and sure enough the echo requests were going out the WAN, not to the VPN. Deleted that rule and its all good.

lifeboy

@rlabaza I carefully studied that post and the fix and it's clear as mud to me :-)

I have one rule for the LAN and that is to allow all traffic to anywhere. How can that route the traffic to the wrong destination to bypass the routing table?

lifeboy

I eventually found this:

https://docs.netgate.com/pfsense/en/latest/multiwan/policy-route.html

So from the LAN one should point the OpenVPN traffic not via the default gateway if I get the jist of what @reberhar has to say in your referenced post. However, under the advanced section on my very simple setup, there is no option to select any other gateway. I have only 2 LAN rules, the anti-lockout one and the "all traffic" rule.

rlabaza

@lifeboy Does your "LAN subnets" alias on the "all traffic" rule include the tunnel network? Try setting the source to 'any' and see what happens. The packet captures are what really cleared things up for me--watching where the ping requests were going on the different interfaces LAN, TUN, WAN. They have to be going somewhere. I read that post 3 times and it never clicked. The fourth time it finally did...that prompted me to packet capture the WAN ans sure enough that's were they were. My setup is simple as well, IDK where I got that policy rule--probably trying different things at 1 AM over several days. Once I removed it, the system took care of handling the routing of those pings not to the WAN, but to the TUN.

lifeboy

@jimp, since my bug report was closed, I can't comment there anymore, but this seems to be a documentation bug at the minimum. If there was a significant change in the way rules work in 2.7.0, then there should be updated documentation on how to set up an OpenVPN site-to-site connection. As it stands now, following the instruction does not result in a working site-to-site connection.

lifeboy

@rlabaza I watched pings come in on as below (from the server ovpn ip to the client ovpn ip. They seem to be arriving fine.

16:24:52.961958 IP 10.0.20.1 > 10.0.20.2: ICMP echo request, id 19195, seq 0, length 64
16:24:53.052314 IP 10.0.20.1 > 10.0.20.2: ICMP echo request, id 33787, seq 0, length 64
16:24:53.972680 IP 10.0.20.1 > 10.0.20.2: ICMP echo request, id 19195, seq 1, length 64

I assume that the issue is that the response is not routed correctly back to the server as per the routing tables, but how does one do that with a policy rule?

lifeboy

@rlabaza On the client I have added a rule to direct the OpenVPN ip addresses before the general "allow all rule", although it doesn't make sense to me. The "allow all rule" should take care of the OpenVPN addresses as well and send them to the default gateway?

What I don't get is that I have another point-to-point service to a different pfSense box, which is also site-to-site. It runs pfSense 2.6 and traffic between the server and client flows without any issues.

What obscure change was made to 2.7 that breaks the comms to the client without this mystery rule and why is not clearly stated in the documentation somewhere? Or am I just not able to find it?

lifeboy

My LAN rules:

Detail of the "OpenVPN_ips" rule:

I can only select default or WAN. Default is what the next rule "Allow to any" uses as well.

?

lifeboy

So now I have disabled all LAN rules, except the one that sends all LAN subnet traffic to the default gateway.

However, I still can't reach the client side LAN addresses from the server's side.

jimp

Please read and follow this exactly and do not skip any part of it no matter what:

https://docs.netgate.com/pfsense/en/latest/recipes/openvpn-s2s-tls.html

I just followed it word for word yet again for one client/one server and it resulted in a working LAN-to-LAN VPN as it has the last several times I tried it. Nothing changed here in the last several years.

You must have a Client-Specific Override even for one client. The override sets up the internal routing in OpenVPN that tells OpenVPN which client should receive traffic for a given subnet. It does not matter how many clients are on the VPN when setup this way, this is still a required step.

rlabaza

@lifeboy When I pinged the client side from the server side pfSense box, everything worked fine. But when I pinged from a server-side machine, it would not get to the VPN. Have you tried setting the source in your rule to "*" and not "LAN subnets" alias?

lifeboy

@jimp I will redo it again and follow the instructions step by step. After all, it's not a production environment yet unless I can get this thing to actually work.

lifeboy

@jimp We use ECDSA certificates instead of RSA. That would not break the routing of traffic from the server LAN to the client LAN, would it?

I'm redoing the tunnel and it connects fine, which to me means the certificates are ok.

lifeboy

Please see my reply and the resolution at https://forum.netgate.com/post/1151441.

Thanks to all for the massive effort!

rlabaza

@lifeboy Glad you're working now. What I learned on my journey to solve this problem is that there are many different causes that manifest in the same failure signature. The story of my (professional career) life. We were always the lightning rod.