OpenVPN client fatal error

pfsenser_ca

We have an OpenVPN server + several clients that form a site-to-site VPN that have been working well for over a year. Each site is using an SG-3100 running pfSense 2.4.4-RELEASE-p3.

Recently one of the clients will disconnect intermittently (a few days or weeks between incidents), with the following error on the client side:

Jan  8 08:12:44 firewall openvpn[49806]: Assertion failed at crypto.c:258 (cipher_ctx_final(ctx->cipher, BEND(&work), &outlen))
Jan  8 08:12:44 firewall openvpn[49806]: Exiting due to fatal error

On the server side, the only log entry around that same time is the following:

Jan  8 08:14:45 firewall openvpn[16421]: xxxxxxxx/x.x.x.x:1194 [xxxxxxxx] Inactivity timeout (--ping-restart), restarting

The users at the client site will reboot the device to restore access.

On the server side, we are also seeing the following types of errors, even when the client tunnel is working:

Jan  8 09:11:20 firewall openvpn[16421]: xxxxxxxx/x.x.x.x:1194 Authenticate/Decrypt packet error: bad packet ID (may be a replay): [ #633636 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings

Below is the server config:

dev ovpns1
verb 1
dev-type tun
dev-node /dev/tun1
writepid /var/run/openvpn_server1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
cipher AES-128-CBC
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local x.x.x.x
engine cryptodev
tls-server
server 172.16.0.0 255.255.255.0
client-config-dir /var/etc/openvpn-csc/server1
ifconfig 172.16.0.1 172.16.0.2
tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'xxxxxxxx-openvpn' 1"
lport 1194
management /var/etc/openvpn/server1.sock unix
push "route 10.1.0.0 255.255.255.0"
route 10.1.1.0 255.255.255.0
route 10.1.2.0 255.255.255.0
route 10.1.3.0 255.255.255.0
route 10.1.5.0 255.255.255.0
route 10.1.6.0 255.255.255.0
route 10.1.7.0 255.255.255.0
route 192.168.2.0 255.255.255.0
ca /var/etc/openvpn/server1.ca 
cert /var/etc/openvpn/server1.cert 
key /var/etc/openvpn/server1.key 
dh /etc/dh-parameters.2048
crl-verify /var/etc/openvpn/server1.crl-verify 
tls-auth /var/etc/openvpn/server1.tls-auth 0
ncp-disable
topology subnet

Below is the client config:

dev ovpnc1
verb 1
dev-type tun
dev-node /dev/tun1
writepid /var/run/openvpn_client1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
cipher AES-128-CBC
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local x.x.x.x
engine cryptodev
tls-client
client
lport 1194
management /var/etc/openvpn/client1.sock unix
remote x.x.x.x 1194
ifconfig 172.16.0.2 172.16.0.1
route 10.1.0.0 255.255.255.0
route 10.1.1.0 255.255.255.0
route 10.1.2.0 255.255.255.0
route 10.1.5.0 255.255.255.0
route 10.1.6.0 255.255.255.0
route 10.1.7.0 255.255.255.0
route 192.168.2.0 255.255.255.0
route 172.16.9.0 255.255.255.0
ca /var/etc/openvpn/client1.ca 
cert /var/etc/openvpn/client1.cert 
key /var/etc/openvpn/client1.key 
crl-verify /var/etc/openvpn/client1.crl-verify 
tls-auth /var/etc/openvpn/client1.tls-auth 1
ncp-disable
resolv-retry infinite
topology subnet

I haven't been able to locate any related reports of such an issue. Any ideas would be welcomed. Thanks.

DaddyGo

@pfsenser_ca said in OpenVPN client fatal error:

Any ideas would be welcomed.

I would do this for the first time:
https://docs.netgate.com/pfsense/en/latest/releases/2-4-5-p1.html

pfsenser_ca

We updated both endpoints to 2.4.5_1 this weekend. Will see how the VPN performs this week. Thanks.

DaddyGo

@pfsenser_ca said in OpenVPN client fatal error:

Will see how the VPN performs this week. Thanks.

I would even change the "cipher" to AES-256-GCM, if not too much work because of the endpoints....

pfsenser_ca

We experienced the same issue again today, only 2 days after updating the systems. The error message on the client side is slightly different now:

Jan 12 14:27:33 firewall openvpn[47886]: cipher_ctx_update: EVP_CipherUpdate() failed
Jan 12 14:27:33 firewall openvpn[47886]: Exiting due to fatal error

There is a single previous report of this issue noted here:

https://forum.netgate.com/topic/133889/openvpn-sometimes-stops-working-with-cipher_ctx_update-evp_cipherupdate-failed/4

Those users also reported the issue on an SG-3100 but no clear resolution.

Our OpenVPN client is set to use "BSD cryptodev engine" currently, so we might try changing that to "None" to see if it helps.

DaddyGo

@pfsenser_ca said in OpenVPN client fatal error:

Our OpenVPN client is set to use "BSD cryptodev engine" currently

This is definitely not a problem, at least I never met that (we have 100 - 120 OpenVPN clients- with BSD crypto)

Try even these steps:

NCP disabling (if it is checked)
cipher change AES-256-GCM (GCM is faster and safer anyway, in principle)