OpenVPN connected but can reach any LAN host

damq · May 31, 2021, 7:35 AM

Since I've upgraded to pfSense 2.5.x a few weeks ago, I've some deep trouble with OpenVPN server module.
I'm now upgraded to 2.5.1 version.

Any idea of my mistake on OpenVPN settings ?

Global Context

We are using pfSense as entry point in a global network for application hosting : it's used for

firewalling
proxy (HAProxy module)
VPN admin access (OpenVPN)
VPN for access from hardware devices (kind of setup box for an owned digital signage network)

I've got about 80 Raspberry Pi (3 & 4) connected with OpenVPN client on my network. RPI are based on Raspbian (Debian 9)

Random issue on routing from client to LAN

After a few settings, quite the whole features are running well, but I'm really stuck with a random bug on OpenVPN Server module.

It allows us to monitor & access to a digital signage network and pushing some new contents / features.

On a first watch, we had considered that it worked : all clients were marked as connected and everything looked fine, as previously in 2.4.x version.
BUT after a deeper analysis, all hosts are indeed connected but some of them (randomly) can not reach any LAN host :

they can connect to a internet host (with IP) —> ping or wget work
but they can not reach either the VPN server IP or any other host from LAN (including gateway)

The consequence is that OpenVPN restart regularly & these devices are totally unstable
What is really strange: nearly all devices falls in this position, but not every time.

I've got 1 other OpenVPN server in this pfSense instance. It seems to work perfectly (but it's not used in the same way : fewer clients, not connected permanently)

Logs details

I can not ping 10.2.0.1.
Here is routing table

Destination     Passerelle      Genmask         Indic Metric Ref    Use Iface
default         192.168.95.1    0.0.0.0         UG    0      0        0 eth0
10.2.0.0        0.0.0.0         255.255.0.0     U     0      0        0 tun0
192.168.95.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.95.0    0.0.0.0         255.255.255.0   U     0      0        0 wlan0
192.168.100.0   10.2.0.1        255.255.255.0   UG    0      0        0 tun0

OpenVPN.log

Inactivity timeout (--ping-restart), restarting
SIGUSR1[soft,ping-restart] received, process restarting
Restart pause, 10 second(s)
TCP/UDP: Preserving recently used remote address: [AF_INET]X.X.X.X:1194
Socket Buffers: R=[180224->180224] S=[180224->180224]
UDP link local (bound): [AF_INET][undef]:1194
UDP link remote: [AF_INET]X.X.X.X:1194
TLS: Initial packet from [AF_INET]X.X.X.X:1194, sid=020b5e04 8a42cf9d 
VERIFY OK: depth=1, CN=vpnaccess, C=FR, ST=HS, L=Paris, O=Company
++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
OK
VERIFY X509NAME OK: CN=myhost.com, C=FR, ST=HS, L=PARIS, O=Company
VERIFY OK: depth=0, CN=myhost.com, C=FR, ST=HS, L=Paris, O=Company 
Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 2048 bit RSA
[myhost.com] Peer Connection Initiated with [AF_INET]X.X.X.X:1194
SENT CONTROL [myhost.com]: 'PUSH_REQUEST' (status=1)
PUSH: Received control message: 'PUSH_REPLY,route 192.168.100.0 255.255.255.0,route 10.1.1.0 25
ta│5.255.255.0,dhcp-option DOMAIN Company.intra,dhcp-option DNS 192.168.100.1,comp-lzo yes,route-gateway 10.2.0.1,topology
20│ subnet,ping 10,ping-restart 60,ifconfig 10.2.0.25 255.255.0.0,peer-id 24,cipher AES-128-CBC'
OPTIONS IMPORT: timers and/or timeouts modified
OPTIONS IMPORT: compression parms modified
OPTIONS IMPORT: --ifconfig/up options modified
OPTIONS IMPORT: route options modified
OPTIONS IMPORT: route-related options modified
OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options 
OPTIONS IMPORT: peer-id set
OPTIONS IMPORT: adjusting link_mtu to 1625
OPTIONS IMPORT: data channel crypto options modified
Outgoing Data Channel: Cipher 'AES-128-CBC' initialized with 128 bit key
Outgoing Data Channel: Using 256 bit message hash 'SHA256' for HMAC authentication
Incoming Data Channel: Cipher 'AES-128-CBC' initialized with 128 bit key
Incoming Data Channel: Using 256 bit message hash 'SHA256' for HMAC authentication  
Preserving previous TUN/TAP instance: tun0
Initialization Sequence Completed
[myhost.com] Inactivity timeout (--ping-restart), restarting
SIGUSR1[soft,ping-restart] received, process restarting
Restart pause, 10 second(s)
TCP/UDP: Preserving recently used remote address: [AF_INET]X.X.X.X:1194
Socket Buffers: R=[180224->180224] S=[180224->180224]
UDP link local (bound): [AF_INET][undef]:1194
UDP link remote: [AF_INET]X.X.X.X:1194
TLS: Initial packet from [AF_INET]X.X.X.X:1194, sid=64de0e5f 79d87795
VERIFY OK: depth=1, CN=vpnaccess, C=FR, ST=HS, L=Paris, O=Company 
VERIFY KU OK
Validating certificate extended key usage
++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
VERIFY EKU OK
VERIFY X509NAME OK: CN=myhost.com, C=FR, ST=HS, L=Paris, O=Company 
VERIFY OK: depth=0, CN=myhost.com, C=FR, ST=HS, L=Paris, O=Company
Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 2048 bit RSA
[myhost.com] Peer Connection Initiated with [AF_INET]X.X.X.X:1194
SENT CONTROL [myhost.com]: 'PUSH_REQUEST' (status=1)
PUSH: Received control message: 'PUSH_REPLY,route 192.168.100.0 255.255.255.0,route 10.1.1.0 255.255.255.0,dhcp-option DOMAIN company.intra,dhcp-option DNS 192.168.100.1,comp-lzo yes,route-gateway 10.2.0.1,topology subnet,ping 10,ping-restart 60,ifconfig 10.2.0.25 255.255.0.0,peer-id 24,cipher AES-128-CBC'
OPTIONS IMPORT: timers and/or timeouts modified
OPTIONS IMPORT: compression parms modified
OPTIONS IMPORT: --ifconfig/up options modified
OPTIONS IMPORT: route options modified
OPTIONS IMPORT: route-related options modified
OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
OPTIONS IMPORT: peer-id set
OPTIONS IMPORT: adjusting link_mtu to 1625
OPTIONS IMPORT: data channel crypto options modified
Outgoing Data Channel: Cipher 'AES-128-CBC' initialized with 128 bit key
Outgoing Data Channel: Using 256 bit message hash 'SHA256' for HMAC authentication
Incoming Data Channel: Cipher 'AES-128-CBC' initialized with 128 bit key
Incoming Data Channel: Using 256 bit message hash 'SHA256' for HMAC authentication
Preserving previous TUN/TAP instance: tun0
Initialization Sequence Completed

In /var/run/openvpn/client.status, I've got 0 byte on statistics

OpenVPN STATISTICS
Updated,Thu May 20 06:05:45 2021
TUN/TAP read bytes,5160
TUN/TAP write bytes,0
TCP/UDP read bytes,4150
TCP/UDP write bytes,11261
Auth read bytes,0
pre-compress bytes,0
post-compress bytes,0
pre-decompress bytes,0
post-decompress bytes,0
END

Additional Tests

If I restart client, there is no change.
If I kill connection from server → that restart deeply connection, and it's fixed the issue
A workaround is to stop OpenVPN daemon on client device. Waiting almost 30s, and then restart OpenVPN client. I do not understand why a simple restart has not the same result. (I use systemctl restart openvpn command)

Is there any way to force OpenVPN (on server or client side) to refresh IP (DHCP lease) on reconnection ?

Client side setup

I’ve checked the client side configuration.
I found no difference in configurations.
I've checked

VPN client configuration
routing table
OpenVPN version (2.4.7)

In order to check any client-side bug, I've created a new OpenVPN singleton server, based on Debian & hosted on a server located behind the same pfSense instance (with NAT rule) --> it runs perfectly.

Server Configuration

Here is global OpenVPN server settings

dev ovpns1
verb 0
dev-type tun
dev-node /dev/tun1
writepid /var/run/openvpn_server1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
client-connect /usr/local/sbin/openvpn.attributes.sh
client-disconnect /usr/local/sbin/openvpn.attributes.sh
local X.X.X.X
tls-server
server 10.2.0.0 255.255.0.0
client-config-dir /var/etc/openvpn/server1/csc
username-as-common-name
plugin /usr/local/lib/openvpn/plugins/openvpn-plugin-auth-script.so /usr/local/sbin/ovpn_auth_verify_async user TG9jYWwgRGF0YWJhc2U= false server1 1194
tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'proxy.mydomain.com' 1"
lport 1194
management /var/etc/openvpn/server1/sock unix
max-clients 300
push "route 192.168.100.0 255.255.255.0"
push "route 10.1.1.0 255.255.255.0"
push "dhcp-option DOMAIN mydomain.intra"
push "dhcp-option DNS 8.8.8.8"
client-to-client
duplicate-cn
capath /var/etc/openvpn/server1/ca
cert /var/etc/openvpn/server1/cert 
key /var/etc/openvpn/server1/key 
dh /etc/dh-parameters.2048
tls-crypt /var/etc/openvpn/server1/tls-crypt 
ncp-disable
cipher AES-128-CBC
allow-compression asym
comp-lzo yes
push "comp-lzo yes"
persist-remote-ip
float
topology subnet

cdunbar · Jun 9, 2021, 8:25 PM

Hello,

Thank you for documenting this. I believe I am experiencing the same problem on one of my pfSense systems. Sometimes OpenVPN works fine, but then a small number of users connect but cannot reach the LAN. It happened a few minutes ago and restarting the OpenVPN service seemed to clear the jam. That's what prompted me to go looking here. Looks like a known issue - has Netgate confirmed anything?

Regards,
Chris

damq · Jun 10, 2021, 7:27 AM

@cdunbar No news from netgate.

In my case, I can not manage the situation with a manual restart of OpenVPN service.
As a workaround, I've created an adhoc OpenVPN virtual machine (based on Debian) on my private network with NAT & routing settings on the pfSense firewall.

It's too bad and more complicated, but I had no choice.