P2P VPN server can't reach client, but client can reach server

lifeboy

Here's what the server logs report:

Jan 23 20:33:44 fw-1A openvpn[99185]: 156.38.137.242:1194 [JHB_backup] Peer Connection Initiated with [AF_INET]156.38.137.242:1194
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 MULTI_sva: pool returned IPv4=10.0.111.2, IPv6=(Not enabled)
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 MULTI: Learn: 10.0.111.2 -> JHB_backup/156.38.137.242:1194
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 MULTI: primary virtual IP for JHB_backup/156.38.137.242:1194: 10.0.111.2
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Data Channel MTU parms [ mss_fix:1400 max_frag:0 tun_mtu:1500 tun_max_mtu:1600 headroom:136 payload:1768 tailroom:562 ET:0 ]
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Outgoing dynamic tls-crypt: Cipher 'AES-256-CTR' initialized with 256 bit key
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Outgoing dynamic tls-crypt: Using 256 bit message hash 'SHA256' for HMAC authentication
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Incoming dynamic tls-crypt: Cipher 'AES-256-CTR' initialized with 256 bit key
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Incoming dynamic tls-crypt: Using 256 bit message hash 'SHA256' for HMAC authentication
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 SENT CONTROL [JHB_backup]: 'PUSH_REPLY,route 192.168.131.0 255.255.255.0,route 192.168.161.0 255.255.255.0,route-gateway 10.0.111.1,topology subnet,ping 10,ping-restart 
60,ifconfig 10.0.111.2 255.255.255.0,peer-id 0,cipher AES-256-GCM,protocol-flags cc-exit tls-ekm dyn-tls-crypt,tun-mtu 1500' (status=1)
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 MULTI: bad source address from client [::], packet dropped
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 MULTI: bad source address from client [::], packet dropped
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 MULTI: bad source address from client [::], packet dropped
Jan 23 20:33:46 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Data Channel: cipher 'AES-256-GCM', peer-id: 0
Jan 23 20:33:46 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Timers: ping 10, ping-restart 120
Jan 23 20:33:46 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Protocol options: explicit-exit-notify 1, protocol-flags cc-exit tls-ekm dyn-tls-crypt
Jan 23 20:34:09 fw-1A openvpn[71328]: MANAGEMENT: Client connected from /var/etc/openvpn/server2/sock
Jan 23 20:34:09 fw-1A openvpn[71328]: MANAGEMENT: CMD 'status 2'
Jan 23 20:34:10 fw-1A openvpn[71328]: MANAGEMENT: CMD 'quit'
Jan 23 20:34:10 fw-1A openvpn[71328]: MANAGEMENT: Client disconnected
Jan 23 20:34:10 fw-1A openvpn[95882]: MANAGEMENT: Client connected from /var/etc/openvpn/server4/sock
Jan 23 20:34:11 fw-1A openvpn[95882]: MANAGEMENT: CMD 'status 2'
Jan 23 20:34:11 fw-1A openvpn[95882]: MANAGEMENT: CMD 'quit'
Jan 23 20:34:11 fw-1A openvpn[95882]: MANAGEMENT: Client disconnected
Jan 23 20:34:20 fw-1A openvpn[71328]: MANAGEMENT: Client connected from /var/etc/openvpn/server2/sock
Jan 23 20:34:20 fw-1A openvpn[71328]: MANAGEMENT: CMD 'status 2'
Jan 23 20:34:20 fw-1A openvpn[71328]: MANAGEMENT: Client disconnected

lifeboy

I think I'll just delete the config (server and client) and redo it all. Something got messed up and I don't know where or what.

viragomann

@lifeboy said in P2P VPN server can't reach client, but client can reach server:

I do see this though:

[2.7.0-RELEASE][roland@fw-1A.fast.za.net]/home/roland: ping 10.0.111.2
PING 10.0.111.2 (10.0.111.2): 56 data bytes

At the server or at client?
I'd expect, that you can access it from both. However, for accessing from the server side you need a proper rule to allow this.

lifeboy

@viragomann That's on the server side.

Which is the problem I started out with. I cannot reach anything on the client side from the server, but from the client I can see the server and the LAN machines perfectly.

I wonder of Xneelo, the hosting provider, are doing something that blocks the traffic? Is it even possible to establish the tunnel and then block traffic over it?

viragomann

@lifeboy

Which is the problem I started out with. I cannot reach anything on the client side from the server, but from the client I can see the server and the LAN machines perfectly.

My assumption was that this concerns networks behind the client node. For this the CSO comes into play.
But you should be at least able to access the clients virtual IP, even independently from a working CSO, presumed the firewall rule at the client allows it.

I wonder of Xneelo, the hosting provider, are doing something that blocks the traffic? Is it even possible to establish the tunnel and then block traffic over it?

No, if the traffic flows well in the one direction it should also go into the other.

lifeboy

@viragomann said in P2P VPN server can't reach client, but client can reach server:

@lifeboy

Which is the problem I started out with. I cannot reach anything on the client side from the server, but from the client I can see the server and the LAN machines perfectly.

My assumption was that this concerns networks behind the client node. For this the CSO comes into play.
But you should be at least able to access the clients virtual IP, even independently from a working CSO, presumed the firewall rule at the client allows it.

The client has an allow-all traffic rule at this stage (just for the testing phase).

I wonder of Xneelo, the hosting provider, are doing something that blocks the traffic? Is it even possible to establish the tunnel and then block traffic over it?

No, if the traffic flows well in the one direction it should also go into the other.

This is quite perplexing. What else could be blocking traffic to flow from the server to the client that is not blocking traffic from the client to the server??

lifeboy

I'm comparing the config between this new TLS connection (let's call it link2) and an existing P2P shared key link (that runs to another pfSense client). Call this link1.

The IPv4 Tunnel Network for link1 is 10.0.1.0/24 and when the tunnel starts, it assigns 10.0.1.1 to the server and 10.0.1.2 to the client. I can ping both these from the server and the client.

For link2 however (IPv4 Tunnel Network 10.0.111.0/24) , both the server and the client get 10.0.111.2 !! Surely that is a problem?

Look at the routing table on the server:

default			197.214.119.129	UGS	18	1500	vtnet1.650	
10.0.1.1		link#11		UHS	69	16384	lo0	
10.0.1.2		link#16		UH	68	1500	ovpns1	
10.0.111.0/24		link#17		U	40	1500	ovpns5	
10.0.111.1		link#11		UHS	71	16384	lo0	
10.210.10.0/24		192.168.142.102	UGS	19	1500	vtnet5	
10.210.20.0/24		192.168.142.102	UGS	19	1500	vtnet5	
10.210.30.0/24		192.168.142.102	UGS	19	1500	vtnet5	
10.210.40.0/24		192.168.142.102	UGS	19	1500	vtnet5	
127.0.0.1		link#11		UH	2	16384	lo0	
168.253.202.96/27	link#8		U	14	1500	vtnet7	
168.253.202.97		link#11		UHS	15	16384	lo0	
168.253.202.98		link#11		UHS	15	16384	lo0	
192.168.111.0/24	10.0.111.2	UGS	72	1500	ovpns5

(updated with more routes shown)
10.0.111.1 is on link#11, but ovpns5 is on link#19. Why is 10.0.111.1 on the wrong link? Or am I reading this incorrectly?

lifeboy

@lifeboy It seems that the GUI just displays the wrong address. It should have 10.0.111.1, instead of 10.0.111.2

lifeboy

I also see, since I enabled logging on the OVPN outgoing trafffic that the ping is leaving the server.

Jan 24 11:45:13	LAN	 Default allow LAN to any rule (1577106904)	  192.168.131.150	  10.0.111.2	ICMP

lifeboy

In the client I now see the traffic is actually blocked!

Jan 24 09:50:01 	ovpnc1 	Default deny rule IPv4 (1000000103) 	192.168.131.150		10.0.111.2		ICMP
Jan 24 09:50:01 	ovpnc1 	Default deny rule IPv4 (1000000103) 	10.0.111.1		10.0.111.2		ICMP
Jan 24 09:50:00 	ovpnc1 	Default deny rule IPv4 (1000000103) 	192.168.131.150		10.0.111.2		ICMP
Jan 24 09:50:00 	ovpnc1 	Default deny rule IPv4 (1000000103) 	10.0.111.1		10.0.111.2		ICMP
Jan 24 09:49:59 	ovpnc1 	Default deny rule IPv4 (1000000103) 	192.168.131.150		10.0.111.2		ICMP
Jan 24 09:49:59 	WAN 	Allow all ipv4 (1705832241) 	203.128.94.74:64008		156.38.137.242:445		TCP:S
Jan 24 09:49:59 	ovpnc1 	Default deny rule IPv4 (1000000103) 	10.0.111.1		10.0.111.2		ICMP
Jan 24 09:49:58 	ovpnc1 	Default deny rule IPv4 (1000000103) 	192.168.131.150		10.0.111.2		ICMP
Jan 24 09:49:58 	ovpnc1 	Default deny rule IPv4 (1000000103) 	10.0.111.1		10.0.111.2		ICMP

Surely these rules cancel the default deny rule?

viragomann

@lifeboy said in P2P VPN server can't reach client, but client can reach server:

The client has an allow-all traffic rule at this stage (just for the testing phase).

You need to allow incoming traffic on the OpenVPN interface.
Nothing will enter on you WAN. It's just the client connection going out there. So there is no rule needed on WAN for the site2site VPN on the client.

I'm comparing the config between this new TLS connection (let's call it link2) and an existing P2P shared key link (that runs to another pfSense client). Call this link1.

Shared key OpenVPN behaves different. A s2s doesn't require a CSO, even if the tunnel is wider. However, the server does only allow a single client to connect.

Look at the routing table on the server:
(updated with more routes shown)
10.0.111.1 is on link#11, but ovpns5 is on link#19. Why is 10.0.111.1 on the wrong link? Or am I reading this incorrectly?

10.0.111.1 is the loopback address, it's the servers virtual IP.
The links are just used by the OS internally to handle the connections.

It seems that the GUI just displays the wrong address. It should have 10.0.111.1, instead of 10.0.111.2

This is the server state. It lists up each connected client with its public and its virtual IP. 10.0.111.2 is the clients virtual IP.

lifeboy

@viragomann, the rules I have are below.

on the client:

and

On the server they are similar.

Of course there are many WAN rules, but the ones that are relevant are:

This allows the ovpn client to connect.

What else is needed to get the traffic flowing then?

lifeboy

After restarting the client for good measure, I can now ping 10.0.111.1 on the server from the client and 10.0.111.2 on the client from the server. (both from the pfSense command line)

However, I cannot now ping either side's other addresses.

I cannot ping 192.168.111.254 (the client firewall LAN address from the server.
I cannot ping 192.168.131.254 (the server firewall LAN address from the client.

One step forward, one step back!

viragomann

@lifeboy said in P2P VPN server can't reach client, but client can reach server:

However, I cannot now ping either side's other addresses.

I cannot ping 192.168.111.254 (the client firewall LAN address from the server.

This presumes a working CSO.

I cannot ping 192.168.131.254 (the server firewall LAN address from the client.

From the clients firewall or from a devices behind?
I'd expect that it works from the firewall itself. bit not from behind.

One step forward, one step back!

No, the CSO has never worked.

AND you should learn the lesson about the firewall rule basics.
The is no need at all to add a pass rule to the outbound interface. Rules have to be on the incoming interface only, both pass and block rules.
So I think, all your rules on the clients WAN are superfluous.

lifeboy

Here is my server config:

dev ovpns5
verb 4
dev-type tun
dev-node /dev/tun5
writepid /var/run/openvpn_server5.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 197.xxx.yyy.130
tls-server
server 10.0.111.0 255.255.255.0
client-config-dir /var/etc/openvpn/server5/csc
ifconfig 10.0.111.1 10.0.111.2
tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'Production_server' 1"
lport 1197
management /var/etc/openvpn/server5/sock unix
max-clients 3
push "route 192.168.131.0 255.255.255.0"
push "route 192.168.161.0 255.255.255.0"
duplicate-cn
remote-cert-tls client
route 192.168.111.0 255.255.255.0
capath /var/etc/openvpn/server5/ca
cert /var/etc/openvpn/server5/cert 
key /var/etc/openvpn/server5/key 
dh /etc/dh-parameters.2048
tls-auth /var/etc/openvpn/server5/tls-auth 0
data-ciphers AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305:AES-256-CBC
data-ciphers-fallback AES-256-CBC
allow-compression no
topology subnet
explicit-exit-notify 1

And my client config:

dev ovpnc1
verb 4
dev-type tun
dev-node /dev/tun1
writepid /var/run/openvpn_client1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 156.xxx.yyy.242
tls-client
lport 1194
management /var/etc/openvpn/client1/sock unix
remote fw.fast.xxx.yyy 1197 udp4
pull
remote-cert-tls server
capath /var/etc/openvpn/client1/ca
cert /var/etc/openvpn/client1/cert
key /var/etc/openvpn/client1/key
tls-auth /var/etc/openvpn/client1/tls-auth 1
data-ciphers AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305:AES-256-CBC
data-ciphers-fallback AES-256-CBC
allow-compression no
passtos
resolv-retry infinite
explicit-exit-notify 3

Is there anything in there that could be causing this behaviour I'm having?

lifeboy

I recreated the tunnel completely, including a new client. This time I used the wizard and then changed the tunnel type to Peer-to-Peer SSL/TLS. The client connects to the server and I can ping the server LAN clients from the client firewall.

However, the same problem still exists on the client side. I can ping the tunnel network address (which is now 10.0.20.2) from the server firewall (which is 10.0.20.1).

If I do a traceroute, this is the result.

: traceroute -s 10.0.20.1 -P icmp 10.0.20.2
traceroute to 10.0.20.2 (10.0.20.2) from 10.0.20.1, 64 hops max, 48 byte packets
 1  10.0.20.2 (10.0.20.2)  17.378 ms  17.115 ms  17.082 ms

However, if I traceroute the LAN on the client side:

: traceroute -s 10.0.20.1 -P icmp 192.168.111.254
traceroute to 192.168.111.254 (192.168.111.254) from 10.0.20.1, 64 hops max, 48 byte packets
 1  *
 2  *^C

It doesn't seem to go via 10.0.20.2 ?

The routes have been added by OpenVPN.

10.0.20.0/24       link#19            U        ovpns6
10.0.20.1          link#11            UHS         lo0
192.168.111.0/24   10.0.20.2          UGS      ovpns6

Stuck at exactly the same spot that I was before.

lifeboy

I changed the tunnel network to /30. So it's 10.0.20.0/30. The server gets .1 and the client .2

I can ping .2 from the server, but not the client LAN. This is despite the being a route for 192.168.111.0/24 via 10.0.20.2 and having a n OpenVPN rule on the client to allow all traffic.

Either I'm a total idiot or something is just wrong internally with OpenVPN and pfSense.

Here the docs even say that with a /30 no CSO is needed...
https://docs.netgate.com/pfsense/en/latest/troubleshooting/openvpn-iroute.html

rlabaza

@lifeboy I am in the exact same boat as you. Have been for a few days. I have double, triple, quadruple checked everything.

From my 'Site B' (client side) any device on that LAN can essentially reach everything on my server side ('Site A').
From my 'Site A' (OpenVPN server side) pfsense box, I can ping anything on the Site B LAN, but from other devices on the Site A LAN, I get nothing.

My 'SiteA' pfSense box is at 23.09.1. My 'Site B' is an SG-1000 at 2.4.5-RELEASE-p1. I reconfigured OpenVPN from shared key to SSL/TLS following the netgate recipe at https://docs.netgate.com/.../recipes/openvpn-s2s-tls.html. The VPN is up and I can get from Site B to Site A, but I can not get from Site A to Site B. The recipe has this in the client firewall config: "If the other sites needs to initiate contact, then this traffic requires a firewall rule on the OpenVPN tab on the client firewall to allow traffic from other VPN sites to reach the Client-side LAN.". I did that. But I still can not ping from Site A LAN machines to anything at Site B.

Fast-forward to the trouble shooting doc (https://docs.netgate.com/.../troubleshooting/openvpn.html): "Test from different vantage points" -- On Site A pfsense GUI, I can ping anything at Site B using the LAN source or the OpenVPN (tun interface) as source.

Next on to "Trace the traffic with packet captures" -- I ran packet capture and I see pings to a Site B machine from a Site A LAN machine when I capture on the LAN interface, but nothing on the VPN interface!

However, if I ping from the Site A pfSense box, I see the traffic on packet captures for LAN and tun interfaces (as expected since ping works).

So I can only conclude that packets from my Site A LAN devices are not getting internally routed out the VPN interface when they hit the pfsense box. But I don't know why!

The "Check the system routing table" section in troubleshooting says check the routing table--it looks fine to me, and does not offer what an incorrect or missing route looks like. Site A LAN is 192.168.19.0/24, tunnel is 10.30.0.0/24 Site B LAN is 192.168.139.0/24 What am I missing? I think I set Local Network and Remote Network correctly in the OpenVPN server to those routes get created...

Following this thread with great interest...
rl.

rlabaza

@rlabaza

UPDATE: I got my configuration working. Found the answer in this post. I had one of these 'policy rules' that @reberhar pointed out, on my LAN firewall sending traffic out the WAN. I pinged the remote side, this time packet capturing on the WAN interface (wish I would have thought of that a few days ago), and sure enough the echo requests were going out the WAN, not to the VPN. Deleted that rule and its all good.

lifeboy

@rlabaza I carefully studied that post and the fix and it's clear as mud to me :-)

I have one rule for the LAN and that is to allow all traffic to anywhere. How can that route the traffic to the wrong destination to bypass the routing table?