P2P VPN server can't reach client, but client can reach server

lifeboy

I'm comparing the config between this new TLS connection (let's call it link2) and an existing P2P shared key link (that runs to another pfSense client). Call this link1.

The IPv4 Tunnel Network for link1 is 10.0.1.0/24 and when the tunnel starts, it assigns 10.0.1.1 to the server and 10.0.1.2 to the client. I can ping both these from the server and the client.

For link2 however (IPv4 Tunnel Network 10.0.111.0/24) , both the server and the client get 10.0.111.2 !! Surely that is a problem?

Look at the routing table on the server:

default			197.214.119.129	UGS	18	1500	vtnet1.650	
10.0.1.1		link#11		UHS	69	16384	lo0	
10.0.1.2		link#16		UH	68	1500	ovpns1	
10.0.111.0/24		link#17		U	40	1500	ovpns5	
10.0.111.1		link#11		UHS	71	16384	lo0	
10.210.10.0/24		192.168.142.102	UGS	19	1500	vtnet5	
10.210.20.0/24		192.168.142.102	UGS	19	1500	vtnet5	
10.210.30.0/24		192.168.142.102	UGS	19	1500	vtnet5	
10.210.40.0/24		192.168.142.102	UGS	19	1500	vtnet5	
127.0.0.1		link#11		UH	2	16384	lo0	
168.253.202.96/27	link#8		U	14	1500	vtnet7	
168.253.202.97		link#11		UHS	15	16384	lo0	
168.253.202.98		link#11		UHS	15	16384	lo0	
192.168.111.0/24	10.0.111.2	UGS	72	1500	ovpns5

(updated with more routes shown)
10.0.111.1 is on link#11, but ovpns5 is on link#19. Why is 10.0.111.1 on the wrong link? Or am I reading this incorrectly?

lifeboy

@lifeboy It seems that the GUI just displays the wrong address. It should have 10.0.111.1, instead of 10.0.111.2

lifeboy

I also see, since I enabled logging on the OVPN outgoing trafffic that the ping is leaving the server.

Jan 24 11:45:13	LAN	 Default allow LAN to any rule (1577106904)	  192.168.131.150	  10.0.111.2	ICMP

lifeboy

In the client I now see the traffic is actually blocked!

Jan 24 09:50:01 	ovpnc1 	Default deny rule IPv4 (1000000103) 	192.168.131.150		10.0.111.2		ICMP
Jan 24 09:50:01 	ovpnc1 	Default deny rule IPv4 (1000000103) 	10.0.111.1		10.0.111.2		ICMP
Jan 24 09:50:00 	ovpnc1 	Default deny rule IPv4 (1000000103) 	192.168.131.150		10.0.111.2		ICMP
Jan 24 09:50:00 	ovpnc1 	Default deny rule IPv4 (1000000103) 	10.0.111.1		10.0.111.2		ICMP
Jan 24 09:49:59 	ovpnc1 	Default deny rule IPv4 (1000000103) 	192.168.131.150		10.0.111.2		ICMP
Jan 24 09:49:59 	WAN 	Allow all ipv4 (1705832241) 	203.128.94.74:64008		156.38.137.242:445		TCP:S
Jan 24 09:49:59 	ovpnc1 	Default deny rule IPv4 (1000000103) 	10.0.111.1		10.0.111.2		ICMP
Jan 24 09:49:58 	ovpnc1 	Default deny rule IPv4 (1000000103) 	192.168.131.150		10.0.111.2		ICMP
Jan 24 09:49:58 	ovpnc1 	Default deny rule IPv4 (1000000103) 	10.0.111.1		10.0.111.2		ICMP

Surely these rules cancel the default deny rule?

viragomann

@lifeboy said in P2P VPN server can't reach client, but client can reach server:

The client has an allow-all traffic rule at this stage (just for the testing phase).

You need to allow incoming traffic on the OpenVPN interface.
Nothing will enter on you WAN. It's just the client connection going out there. So there is no rule needed on WAN for the site2site VPN on the client.

I'm comparing the config between this new TLS connection (let's call it link2) and an existing P2P shared key link (that runs to another pfSense client). Call this link1.

Shared key OpenVPN behaves different. A s2s doesn't require a CSO, even if the tunnel is wider. However, the server does only allow a single client to connect.

Look at the routing table on the server:
(updated with more routes shown)
10.0.111.1 is on link#11, but ovpns5 is on link#19. Why is 10.0.111.1 on the wrong link? Or am I reading this incorrectly?

10.0.111.1 is the loopback address, it's the servers virtual IP.
The links are just used by the OS internally to handle the connections.

It seems that the GUI just displays the wrong address. It should have 10.0.111.1, instead of 10.0.111.2

This is the server state. It lists up each connected client with its public and its virtual IP. 10.0.111.2 is the clients virtual IP.

lifeboy

@viragomann, the rules I have are below.

on the client:

and

On the server they are similar.

Of course there are many WAN rules, but the ones that are relevant are:

This allows the ovpn client to connect.

What else is needed to get the traffic flowing then?

lifeboy

After restarting the client for good measure, I can now ping 10.0.111.1 on the server from the client and 10.0.111.2 on the client from the server. (both from the pfSense command line)

However, I cannot now ping either side's other addresses.

I cannot ping 192.168.111.254 (the client firewall LAN address from the server.
I cannot ping 192.168.131.254 (the server firewall LAN address from the client.

One step forward, one step back!

viragomann

@lifeboy said in P2P VPN server can't reach client, but client can reach server:

However, I cannot now ping either side's other addresses.

I cannot ping 192.168.111.254 (the client firewall LAN address from the server.

This presumes a working CSO.

I cannot ping 192.168.131.254 (the server firewall LAN address from the client.

From the clients firewall or from a devices behind?
I'd expect that it works from the firewall itself. bit not from behind.

One step forward, one step back!

No, the CSO has never worked.

AND you should learn the lesson about the firewall rule basics.
The is no need at all to add a pass rule to the outbound interface. Rules have to be on the incoming interface only, both pass and block rules.
So I think, all your rules on the clients WAN are superfluous.

lifeboy

Here is my server config:

dev ovpns5
verb 4
dev-type tun
dev-node /dev/tun5
writepid /var/run/openvpn_server5.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 197.xxx.yyy.130
tls-server
server 10.0.111.0 255.255.255.0
client-config-dir /var/etc/openvpn/server5/csc
ifconfig 10.0.111.1 10.0.111.2
tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'Production_server' 1"
lport 1197
management /var/etc/openvpn/server5/sock unix
max-clients 3
push "route 192.168.131.0 255.255.255.0"
push "route 192.168.161.0 255.255.255.0"
duplicate-cn
remote-cert-tls client
route 192.168.111.0 255.255.255.0
capath /var/etc/openvpn/server5/ca
cert /var/etc/openvpn/server5/cert 
key /var/etc/openvpn/server5/key 
dh /etc/dh-parameters.2048
tls-auth /var/etc/openvpn/server5/tls-auth 0
data-ciphers AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305:AES-256-CBC
data-ciphers-fallback AES-256-CBC
allow-compression no
topology subnet
explicit-exit-notify 1

And my client config:

dev ovpnc1
verb 4
dev-type tun
dev-node /dev/tun1
writepid /var/run/openvpn_client1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 156.xxx.yyy.242
tls-client
lport 1194
management /var/etc/openvpn/client1/sock unix
remote fw.fast.xxx.yyy 1197 udp4
pull
remote-cert-tls server
capath /var/etc/openvpn/client1/ca
cert /var/etc/openvpn/client1/cert
key /var/etc/openvpn/client1/key
tls-auth /var/etc/openvpn/client1/tls-auth 1
data-ciphers AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305:AES-256-CBC
data-ciphers-fallback AES-256-CBC
allow-compression no
passtos
resolv-retry infinite
explicit-exit-notify 3

Is there anything in there that could be causing this behaviour I'm having?

lifeboy

I recreated the tunnel completely, including a new client. This time I used the wizard and then changed the tunnel type to Peer-to-Peer SSL/TLS. The client connects to the server and I can ping the server LAN clients from the client firewall.

However, the same problem still exists on the client side. I can ping the tunnel network address (which is now 10.0.20.2) from the server firewall (which is 10.0.20.1).

If I do a traceroute, this is the result.

: traceroute -s 10.0.20.1 -P icmp 10.0.20.2
traceroute to 10.0.20.2 (10.0.20.2) from 10.0.20.1, 64 hops max, 48 byte packets
 1  10.0.20.2 (10.0.20.2)  17.378 ms  17.115 ms  17.082 ms

However, if I traceroute the LAN on the client side:

: traceroute -s 10.0.20.1 -P icmp 192.168.111.254
traceroute to 192.168.111.254 (192.168.111.254) from 10.0.20.1, 64 hops max, 48 byte packets
 1  *
 2  *^C

It doesn't seem to go via 10.0.20.2 ?

The routes have been added by OpenVPN.

10.0.20.0/24       link#19            U        ovpns6
10.0.20.1          link#11            UHS         lo0
192.168.111.0/24   10.0.20.2          UGS      ovpns6

Stuck at exactly the same spot that I was before.

lifeboy

I changed the tunnel network to /30. So it's 10.0.20.0/30. The server gets .1 and the client .2

I can ping .2 from the server, but not the client LAN. This is despite the being a route for 192.168.111.0/24 via 10.0.20.2 and having a n OpenVPN rule on the client to allow all traffic.

Either I'm a total idiot or something is just wrong internally with OpenVPN and pfSense.

Here the docs even say that with a /30 no CSO is needed...
https://docs.netgate.com/pfsense/en/latest/troubleshooting/openvpn-iroute.html

rlabaza

@lifeboy I am in the exact same boat as you. Have been for a few days. I have double, triple, quadruple checked everything.

From my 'Site B' (client side) any device on that LAN can essentially reach everything on my server side ('Site A').
From my 'Site A' (OpenVPN server side) pfsense box, I can ping anything on the Site B LAN, but from other devices on the Site A LAN, I get nothing.

My 'SiteA' pfSense box is at 23.09.1. My 'Site B' is an SG-1000 at 2.4.5-RELEASE-p1. I reconfigured OpenVPN from shared key to SSL/TLS following the netgate recipe at https://docs.netgate.com/.../recipes/openvpn-s2s-tls.html. The VPN is up and I can get from Site B to Site A, but I can not get from Site A to Site B. The recipe has this in the client firewall config: "If the other sites needs to initiate contact, then this traffic requires a firewall rule on the OpenVPN tab on the client firewall to allow traffic from other VPN sites to reach the Client-side LAN.". I did that. But I still can not ping from Site A LAN machines to anything at Site B.

Fast-forward to the trouble shooting doc (https://docs.netgate.com/.../troubleshooting/openvpn.html): "Test from different vantage points" -- On Site A pfsense GUI, I can ping anything at Site B using the LAN source or the OpenVPN (tun interface) as source.

Next on to "Trace the traffic with packet captures" -- I ran packet capture and I see pings to a Site B machine from a Site A LAN machine when I capture on the LAN interface, but nothing on the VPN interface!

However, if I ping from the Site A pfSense box, I see the traffic on packet captures for LAN and tun interfaces (as expected since ping works).

So I can only conclude that packets from my Site A LAN devices are not getting internally routed out the VPN interface when they hit the pfsense box. But I don't know why!

The "Check the system routing table" section in troubleshooting says check the routing table--it looks fine to me, and does not offer what an incorrect or missing route looks like. Site A LAN is 192.168.19.0/24, tunnel is 10.30.0.0/24 Site B LAN is 192.168.139.0/24 What am I missing? I think I set Local Network and Remote Network correctly in the OpenVPN server to those routes get created...

Following this thread with great interest...
rl.

rlabaza

@rlabaza

UPDATE: I got my configuration working. Found the answer in this post. I had one of these 'policy rules' that @reberhar pointed out, on my LAN firewall sending traffic out the WAN. I pinged the remote side, this time packet capturing on the WAN interface (wish I would have thought of that a few days ago), and sure enough the echo requests were going out the WAN, not to the VPN. Deleted that rule and its all good.

lifeboy

@rlabaza I carefully studied that post and the fix and it's clear as mud to me :-)

I have one rule for the LAN and that is to allow all traffic to anywhere. How can that route the traffic to the wrong destination to bypass the routing table?

lifeboy

I eventually found this:

https://docs.netgate.com/pfsense/en/latest/multiwan/policy-route.html

So from the LAN one should point the OpenVPN traffic not via the default gateway if I get the jist of what @reberhar has to say in your referenced post. However, under the advanced section on my very simple setup, there is no option to select any other gateway. I have only 2 LAN rules, the anti-lockout one and the "all traffic" rule.

rlabaza

@lifeboy Does your "LAN subnets" alias on the "all traffic" rule include the tunnel network? Try setting the source to 'any' and see what happens. The packet captures are what really cleared things up for me--watching where the ping requests were going on the different interfaces LAN, TUN, WAN. They have to be going somewhere. I read that post 3 times and it never clicked. The fourth time it finally did...that prompted me to packet capture the WAN ans sure enough that's were they were. My setup is simple as well, IDK where I got that policy rule--probably trying different things at 1 AM over several days. Once I removed it, the system took care of handling the routing of those pings not to the WAN, but to the TUN.

lifeboy

@jimp, since my bug report was closed, I can't comment there anymore, but this seems to be a documentation bug at the minimum. If there was a significant change in the way rules work in 2.7.0, then there should be updated documentation on how to set up an OpenVPN site-to-site connection. As it stands now, following the instruction does not result in a working site-to-site connection.

lifeboy

@rlabaza I watched pings come in on as below (from the server ovpn ip to the client ovpn ip. They seem to be arriving fine.

16:24:52.961958 IP 10.0.20.1 > 10.0.20.2: ICMP echo request, id 19195, seq 0, length 64
16:24:53.052314 IP 10.0.20.1 > 10.0.20.2: ICMP echo request, id 33787, seq 0, length 64
16:24:53.972680 IP 10.0.20.1 > 10.0.20.2: ICMP echo request, id 19195, seq 1, length 64

I assume that the issue is that the response is not routed correctly back to the server as per the routing tables, but how does one do that with a policy rule?

lifeboy

@rlabaza On the client I have added a rule to direct the OpenVPN ip addresses before the general "allow all rule", although it doesn't make sense to me. The "allow all rule" should take care of the OpenVPN addresses as well and send them to the default gateway?

What I don't get is that I have another point-to-point service to a different pfSense box, which is also site-to-site. It runs pfSense 2.6 and traffic between the server and client flows without any issues.

What obscure change was made to 2.7 that breaks the comms to the client without this mystery rule and why is not clearly stated in the documentation somewhere? Or am I just not able to find it?

lifeboy

My LAN rules:

Detail of the "OpenVPN_ips" rule:

I can only select default or WAN. Default is what the next rule "Allow to any" uses as well.

?