P2P VPN server can't reach client, but client can reach server

lifeboy

@lifeboy At the client however, I see this is the log.

Jan 23 19:31:12 	openvpn 	9101 	Initialization Sequence Completed
Jan 23 19:31:12 	openvpn 	9101 	ERROR: FreeBSD route add command failed: external program exited with error status: 1
Jan 23 19:31:12 	openvpn 	9101 	/sbin/route add -net 192.168.111.0 10.0.111.1 255.255.255.0
Jan 23 19:31:12 	openvpn 	9101 	/sbin/route add -net 192.168.161.0 10.0.111.1 255.255.255.0
Jan 23 19:31:12 	openvpn 	9101 	ERROR: FreeBSD route add command failed: external program exited with error status: 1
Jan 23 19:31:12 	openvpn 	9101 	/sbin/route add -net 192.168.131.0 10.0.111.1 255.255.255.0
Jan 23 19:31:12 	openvpn 	9101 	/sbin/route add -net 192.168.131.0 10.0.111.1 255.255.255.0

The route add fails.

viragomann

@lifeboy
That's bad though.

Did you check "Don't add routes" in the client settings?

Are these routes even added successfully?
This could possibly happen if you have added the remote networks in the client settings, and also the server is pushing them. In this case you can remote the subnets from the "Local networks" box at the server.

lifeboy

@viragomann No, I don't have networks specced in the client. I did have before, but since removed them to make sure there's no clash.

I do see this though:

[2.7.0-RELEASE][roland@fw-1A.fast.za.net]/home/roland: ping 10.0.111.2
PING 10.0.111.2 (10.0.111.2): 56 data bytes
^C
--- 10.0.111.2 ping statistics ---
7 packets transmitted, 0 packets received, 100.0% packet loss
[2.7.0-RELEASE][roland@fw-1A.fast.za.net]/home/roland: ping 10.0.111.1
PING 10.0.111.1 (10.0.111.1): 56 data bytes
64 bytes from 10.0.111.1: icmp_seq=0 ttl=64 time=0.871 ms
64 bytes from 10.0.111.1: icmp_seq=1 ttl=64 time=0.088 ms
64 bytes from 10.0.111.1: icmp_seq=2 ttl=64 time=0.076 ms
^C
--- 10.0.111.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.076/0.345/0.871/0.372 ms
[2.7.0-RELEASE][roland@fw-1A.fast.za.net]/home/roland:

Which doesn't make sense. It's almost as if the connection is not up, but it is.

lifeboy

I have removed the network specs in the CSO now the route add error is gone. Still can't reach the client from the server though or even ping 10.0.111.2.

Jan 23 20:33:44 	openvpn 	79155 	Protocol options: explicit-exit-notify 3, protocol-flags cc-exit tls-ekm dyn-tls-crypt
Jan 23 20:33:44 	openvpn 	79155 	Timers: ping 10, ping-restart 60
Jan 23 20:33:44 	openvpn 	79155 	Data Channel: cipher 'AES-256-GCM', peer-id: 0
Jan 23 20:33:44 	openvpn 	79155 	Initialization Sequence Completed
Jan 23 20:33:44 	openvpn 	79155 	Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Jan 23 20:33:44 	openvpn 	79155 	Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Jan 23 20:33:44 	openvpn 	79155 	Incoming dynamic tls-crypt: Using 256 bit message hash 'SHA256' for HMAC authentication
Jan 23 20:33:44 	openvpn 	79155 	Incoming dynamic tls-crypt: Cipher 'AES-256-CTR' initialized with 256 bit key
Jan 23 20:33:44 	openvpn 	79155 	Outgoing dynamic tls-crypt: Using 256 bit message hash 'SHA256' for HMAC authentication
Jan 23 20:33:44 	openvpn 	79155 	Outgoing dynamic tls-crypt: Cipher 'AES-256-CTR' initialized with 256 bit key
Jan 23 20:33:44 	openvpn 	79155 	Data Channel MTU parms [ mss_fix:1400 max_frag:0 tun_mtu:1500 tun_max_mtu:1600 headroom:136 payload:1768 tailroom:562 ET:0 ]
Jan 23 20:33:44 	openvpn 	79155 	/sbin/route add -net 192.168.161.0 10.0.111.1 255.255.255.0
Jan 23 20:33:44 	openvpn 	79155 	/sbin/route add -net 192.168.131.0 10.0.111.1 255.255.255.0
Jan 23 20:33:44 	openvpn 	79155 	/usr/local/sbin/ovpn-linkup ovpnc1 1500 0 10.0.111.2 255.255.255.0 init
Jan 23 20:33:44 	openvpn 	79155 	/sbin/ifconfig ovpnc1 10.0.111.2/24 mtu 1500 up
Jan 23 20:33:44 	openvpn 	79155 	do_ifconfig, ipv4=1, ipv6=0
Jan 23 20:33:44 	openvpn 	79155 	TUN/TAP device /dev/tun1 opened
Jan 23 20:33:44 	openvpn 	79155 	TUN/TAP device ovpnc1 exists previously, keep at program end
Jan 23 20:33:44 	openvpn 	79155 	ROUTE_GATEWAY 156.38.137.241/255.255.255.248 IFACE=vtnet1 HWADDR=bc:24:11:cb:4f:1e
Jan 23 20:33:44 	openvpn 	79155 	OPTIONS IMPORT: tun-mtu set to 1500
Jan 23 20:33:44 	openvpn 	79155 	OPTIONS IMPORT: route-related options modified
Jan 23 20:33:44 	openvpn 	79155 	OPTIONS IMPORT: route options modified
Jan 23 20:33:44 	openvpn 	79155 	OPTIONS IMPORT: --ifconfig/up options modified
Jan 23 20:33:44 	openvpn 	79155 	PUSH: Received control message: 'PUSH_REPLY,route 192.168.131.0 255.255.255.0,route 192.168.161.0 255.255.255.0,route-gateway 10.0.111.1,topology subnet,ping 10,ping-restart 60,ifconfig 10.0.111.2 255.255.255.0,peer-id 0,cipher AES-256-GCM,protocol-flags cc-exit tls-ekm dyn-tls-crypt,tun-mtu 1500'

lifeboy

Here's what the server logs report:

Jan 23 20:33:44 fw-1A openvpn[99185]: 156.38.137.242:1194 [JHB_backup] Peer Connection Initiated with [AF_INET]156.38.137.242:1194
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 MULTI_sva: pool returned IPv4=10.0.111.2, IPv6=(Not enabled)
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 MULTI: Learn: 10.0.111.2 -> JHB_backup/156.38.137.242:1194
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 MULTI: primary virtual IP for JHB_backup/156.38.137.242:1194: 10.0.111.2
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Data Channel MTU parms [ mss_fix:1400 max_frag:0 tun_mtu:1500 tun_max_mtu:1600 headroom:136 payload:1768 tailroom:562 ET:0 ]
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Outgoing dynamic tls-crypt: Cipher 'AES-256-CTR' initialized with 256 bit key
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Outgoing dynamic tls-crypt: Using 256 bit message hash 'SHA256' for HMAC authentication
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Incoming dynamic tls-crypt: Cipher 'AES-256-CTR' initialized with 256 bit key
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Incoming dynamic tls-crypt: Using 256 bit message hash 'SHA256' for HMAC authentication
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 SENT CONTROL [JHB_backup]: 'PUSH_REPLY,route 192.168.131.0 255.255.255.0,route 192.168.161.0 255.255.255.0,route-gateway 10.0.111.1,topology subnet,ping 10,ping-restart 
60,ifconfig 10.0.111.2 255.255.255.0,peer-id 0,cipher AES-256-GCM,protocol-flags cc-exit tls-ekm dyn-tls-crypt,tun-mtu 1500' (status=1)
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 MULTI: bad source address from client [::], packet dropped
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 MULTI: bad source address from client [::], packet dropped
Jan 23 20:33:44 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 MULTI: bad source address from client [::], packet dropped
Jan 23 20:33:46 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Data Channel: cipher 'AES-256-GCM', peer-id: 0
Jan 23 20:33:46 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Timers: ping 10, ping-restart 120
Jan 23 20:33:46 fw-1A openvpn[99185]: JHB_backup/156.38.137.242:1194 Protocol options: explicit-exit-notify 1, protocol-flags cc-exit tls-ekm dyn-tls-crypt
Jan 23 20:34:09 fw-1A openvpn[71328]: MANAGEMENT: Client connected from /var/etc/openvpn/server2/sock
Jan 23 20:34:09 fw-1A openvpn[71328]: MANAGEMENT: CMD 'status 2'
Jan 23 20:34:10 fw-1A openvpn[71328]: MANAGEMENT: CMD 'quit'
Jan 23 20:34:10 fw-1A openvpn[71328]: MANAGEMENT: Client disconnected
Jan 23 20:34:10 fw-1A openvpn[95882]: MANAGEMENT: Client connected from /var/etc/openvpn/server4/sock
Jan 23 20:34:11 fw-1A openvpn[95882]: MANAGEMENT: CMD 'status 2'
Jan 23 20:34:11 fw-1A openvpn[95882]: MANAGEMENT: CMD 'quit'
Jan 23 20:34:11 fw-1A openvpn[95882]: MANAGEMENT: Client disconnected
Jan 23 20:34:20 fw-1A openvpn[71328]: MANAGEMENT: Client connected from /var/etc/openvpn/server2/sock
Jan 23 20:34:20 fw-1A openvpn[71328]: MANAGEMENT: CMD 'status 2'
Jan 23 20:34:20 fw-1A openvpn[71328]: MANAGEMENT: Client disconnected

lifeboy

I think I'll just delete the config (server and client) and redo it all. Something got messed up and I don't know where or what.

viragomann

@lifeboy said in P2P VPN server can't reach client, but client can reach server:

I do see this though:

[2.7.0-RELEASE][roland@fw-1A.fast.za.net]/home/roland: ping 10.0.111.2
PING 10.0.111.2 (10.0.111.2): 56 data bytes

At the server or at client?
I'd expect, that you can access it from both. However, for accessing from the server side you need a proper rule to allow this.

lifeboy

@viragomann That's on the server side.

Which is the problem I started out with. I cannot reach anything on the client side from the server, but from the client I can see the server and the LAN machines perfectly.

I wonder of Xneelo, the hosting provider, are doing something that blocks the traffic? Is it even possible to establish the tunnel and then block traffic over it?

viragomann

@lifeboy

Which is the problem I started out with. I cannot reach anything on the client side from the server, but from the client I can see the server and the LAN machines perfectly.

My assumption was that this concerns networks behind the client node. For this the CSO comes into play.
But you should be at least able to access the clients virtual IP, even independently from a working CSO, presumed the firewall rule at the client allows it.

I wonder of Xneelo, the hosting provider, are doing something that blocks the traffic? Is it even possible to establish the tunnel and then block traffic over it?

No, if the traffic flows well in the one direction it should also go into the other.

lifeboy

@viragomann said in P2P VPN server can't reach client, but client can reach server:

@lifeboy

Which is the problem I started out with. I cannot reach anything on the client side from the server, but from the client I can see the server and the LAN machines perfectly.

My assumption was that this concerns networks behind the client node. For this the CSO comes into play.
But you should be at least able to access the clients virtual IP, even independently from a working CSO, presumed the firewall rule at the client allows it.

The client has an allow-all traffic rule at this stage (just for the testing phase).

I wonder of Xneelo, the hosting provider, are doing something that blocks the traffic? Is it even possible to establish the tunnel and then block traffic over it?

No, if the traffic flows well in the one direction it should also go into the other.

This is quite perplexing. What else could be blocking traffic to flow from the server to the client that is not blocking traffic from the client to the server??

lifeboy

I'm comparing the config between this new TLS connection (let's call it link2) and an existing P2P shared key link (that runs to another pfSense client). Call this link1.

The IPv4 Tunnel Network for link1 is 10.0.1.0/24 and when the tunnel starts, it assigns 10.0.1.1 to the server and 10.0.1.2 to the client. I can ping both these from the server and the client.

For link2 however (IPv4 Tunnel Network 10.0.111.0/24) , both the server and the client get 10.0.111.2 !! Surely that is a problem?

Look at the routing table on the server:

default			197.214.119.129	UGS	18	1500	vtnet1.650	
10.0.1.1		link#11		UHS	69	16384	lo0	
10.0.1.2		link#16		UH	68	1500	ovpns1	
10.0.111.0/24		link#17		U	40	1500	ovpns5	
10.0.111.1		link#11		UHS	71	16384	lo0	
10.210.10.0/24		192.168.142.102	UGS	19	1500	vtnet5	
10.210.20.0/24		192.168.142.102	UGS	19	1500	vtnet5	
10.210.30.0/24		192.168.142.102	UGS	19	1500	vtnet5	
10.210.40.0/24		192.168.142.102	UGS	19	1500	vtnet5	
127.0.0.1		link#11		UH	2	16384	lo0	
168.253.202.96/27	link#8		U	14	1500	vtnet7	
168.253.202.97		link#11		UHS	15	16384	lo0	
168.253.202.98		link#11		UHS	15	16384	lo0	
192.168.111.0/24	10.0.111.2	UGS	72	1500	ovpns5

(updated with more routes shown)
10.0.111.1 is on link#11, but ovpns5 is on link#19. Why is 10.0.111.1 on the wrong link? Or am I reading this incorrectly?

lifeboy

@lifeboy It seems that the GUI just displays the wrong address. It should have 10.0.111.1, instead of 10.0.111.2

lifeboy

I also see, since I enabled logging on the OVPN outgoing trafffic that the ping is leaving the server.

Jan 24 11:45:13	LAN	 Default allow LAN to any rule (1577106904)	  192.168.131.150	  10.0.111.2	ICMP

lifeboy

In the client I now see the traffic is actually blocked!

Jan 24 09:50:01 	ovpnc1 	Default deny rule IPv4 (1000000103) 	192.168.131.150		10.0.111.2		ICMP
Jan 24 09:50:01 	ovpnc1 	Default deny rule IPv4 (1000000103) 	10.0.111.1		10.0.111.2		ICMP
Jan 24 09:50:00 	ovpnc1 	Default deny rule IPv4 (1000000103) 	192.168.131.150		10.0.111.2		ICMP
Jan 24 09:50:00 	ovpnc1 	Default deny rule IPv4 (1000000103) 	10.0.111.1		10.0.111.2		ICMP
Jan 24 09:49:59 	ovpnc1 	Default deny rule IPv4 (1000000103) 	192.168.131.150		10.0.111.2		ICMP
Jan 24 09:49:59 	WAN 	Allow all ipv4 (1705832241) 	203.128.94.74:64008		156.38.137.242:445		TCP:S
Jan 24 09:49:59 	ovpnc1 	Default deny rule IPv4 (1000000103) 	10.0.111.1		10.0.111.2		ICMP
Jan 24 09:49:58 	ovpnc1 	Default deny rule IPv4 (1000000103) 	192.168.131.150		10.0.111.2		ICMP
Jan 24 09:49:58 	ovpnc1 	Default deny rule IPv4 (1000000103) 	10.0.111.1		10.0.111.2		ICMP

Surely these rules cancel the default deny rule?

viragomann

@lifeboy said in P2P VPN server can't reach client, but client can reach server:

The client has an allow-all traffic rule at this stage (just for the testing phase).

You need to allow incoming traffic on the OpenVPN interface.
Nothing will enter on you WAN. It's just the client connection going out there. So there is no rule needed on WAN for the site2site VPN on the client.

I'm comparing the config between this new TLS connection (let's call it link2) and an existing P2P shared key link (that runs to another pfSense client). Call this link1.

Shared key OpenVPN behaves different. A s2s doesn't require a CSO, even if the tunnel is wider. However, the server does only allow a single client to connect.

Look at the routing table on the server:
(updated with more routes shown)
10.0.111.1 is on link#11, but ovpns5 is on link#19. Why is 10.0.111.1 on the wrong link? Or am I reading this incorrectly?

10.0.111.1 is the loopback address, it's the servers virtual IP.
The links are just used by the OS internally to handle the connections.

It seems that the GUI just displays the wrong address. It should have 10.0.111.1, instead of 10.0.111.2

This is the server state. It lists up each connected client with its public and its virtual IP. 10.0.111.2 is the clients virtual IP.

lifeboy

@viragomann, the rules I have are below.

on the client:

and

On the server they are similar.

Of course there are many WAN rules, but the ones that are relevant are:

This allows the ovpn client to connect.

What else is needed to get the traffic flowing then?

lifeboy

After restarting the client for good measure, I can now ping 10.0.111.1 on the server from the client and 10.0.111.2 on the client from the server. (both from the pfSense command line)

However, I cannot now ping either side's other addresses.

I cannot ping 192.168.111.254 (the client firewall LAN address from the server.
I cannot ping 192.168.131.254 (the server firewall LAN address from the client.

One step forward, one step back!

viragomann

@lifeboy said in P2P VPN server can't reach client, but client can reach server:

However, I cannot now ping either side's other addresses.

I cannot ping 192.168.111.254 (the client firewall LAN address from the server.

This presumes a working CSO.

I cannot ping 192.168.131.254 (the server firewall LAN address from the client.

From the clients firewall or from a devices behind?
I'd expect that it works from the firewall itself. bit not from behind.

One step forward, one step back!

No, the CSO has never worked.

AND you should learn the lesson about the firewall rule basics.
The is no need at all to add a pass rule to the outbound interface. Rules have to be on the incoming interface only, both pass and block rules.
So I think, all your rules on the clients WAN are superfluous.

lifeboy

Here is my server config:

dev ovpns5
verb 4
dev-type tun
dev-node /dev/tun5
writepid /var/run/openvpn_server5.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 197.xxx.yyy.130
tls-server
server 10.0.111.0 255.255.255.0
client-config-dir /var/etc/openvpn/server5/csc
ifconfig 10.0.111.1 10.0.111.2
tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'Production_server' 1"
lport 1197
management /var/etc/openvpn/server5/sock unix
max-clients 3
push "route 192.168.131.0 255.255.255.0"
push "route 192.168.161.0 255.255.255.0"
duplicate-cn
remote-cert-tls client
route 192.168.111.0 255.255.255.0
capath /var/etc/openvpn/server5/ca
cert /var/etc/openvpn/server5/cert 
key /var/etc/openvpn/server5/key 
dh /etc/dh-parameters.2048
tls-auth /var/etc/openvpn/server5/tls-auth 0
data-ciphers AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305:AES-256-CBC
data-ciphers-fallback AES-256-CBC
allow-compression no
topology subnet
explicit-exit-notify 1

And my client config:

dev ovpnc1
verb 4
dev-type tun
dev-node /dev/tun1
writepid /var/run/openvpn_client1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 156.xxx.yyy.242
tls-client
lport 1194
management /var/etc/openvpn/client1/sock unix
remote fw.fast.xxx.yyy 1197 udp4
pull
remote-cert-tls server
capath /var/etc/openvpn/client1/ca
cert /var/etc/openvpn/client1/cert
key /var/etc/openvpn/client1/key
tls-auth /var/etc/openvpn/client1/tls-auth 1
data-ciphers AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305:AES-256-CBC
data-ciphers-fallback AES-256-CBC
allow-compression no
passtos
resolv-retry infinite
explicit-exit-notify 3

Is there anything in there that could be causing this behaviour I'm having?

lifeboy

I recreated the tunnel completely, including a new client. This time I used the wizard and then changed the tunnel type to Peer-to-Peer SSL/TLS. The client connects to the server and I can ping the server LAN clients from the client firewall.

However, the same problem still exists on the client side. I can ping the tunnel network address (which is now 10.0.20.2) from the server firewall (which is 10.0.20.1).

If I do a traceroute, this is the result.

: traceroute -s 10.0.20.1 -P icmp 10.0.20.2
traceroute to 10.0.20.2 (10.0.20.2) from 10.0.20.1, 64 hops max, 48 byte packets
 1  10.0.20.2 (10.0.20.2)  17.378 ms  17.115 ms  17.082 ms

However, if I traceroute the LAN on the client side:

: traceroute -s 10.0.20.1 -P icmp 192.168.111.254
traceroute to 192.168.111.254 (192.168.111.254) from 10.0.20.1, 64 hops max, 48 byte packets
 1  *
 2  *^C

It doesn't seem to go via 10.0.20.2 ?

The routes have been added by OpenVPN.

10.0.20.0/24       link#19            U        ovpns6
10.0.20.1          link#11            UHS         lo0
192.168.111.0/24   10.0.20.2          UGS      ovpns6

Stuck at exactly the same spot that I was before.