OpenVPN sometimes stops working with cipher_ctx_update: EVP_CipherUpdate() failed

SipriusPT

Hello,

I have a OpenVPN client from site to site who have been working without any issue in last 2 months, since it was installed, and today from time to time, it stops working, and cannot turn it on again manually, till a restart is made.

There is no issues with lack of hardware performance/resourses that could be triggering this, the behaviour is the same like it was until I start having this issue.

Anyone knows what can be?

This is the error that it gives:

Aug 17 15:36:41	openvpn	20992	MANAGEMENT: Client connected from /var/etc/openvpn/client1.sock
Aug 17 15:36:41	openvpn	20992	MANAGEMENT: CMD 'state 1'
Aug 17 15:36:41	openvpn	20992	MANAGEMENT: CMD 'status 2'
Aug 17 15:36:41	openvpn	20992	MANAGEMENT: Client disconnected
Aug 17 15:37:21	openvpn	20992	MANAGEMENT: Client connected from /var/etc/openvpn/client1.sock
Aug 17 15:37:21	openvpn	20992	MANAGEMENT: CMD 'state 1'
Aug 17 15:37:21	openvpn	20992	MANAGEMENT: CMD 'status 2'
Aug 17 15:37:21	openvpn	20992	MANAGEMENT: Client disconnected
Aug 17 15:37:24	openvpn	20992	cipher_ctx_update: EVP_CipherUpdate() failed
Aug 17 15:37:24	openvpn	20992	Exiting due to fatal error
Aug 17 15:37:24	openvpn	20992	/sbin/route delete -net 10.0.0.0 10.0.9.1 255.255.255.0
Aug 17 15:37:25	openvpn	20992	Closing TUN/TAP interface
Aug 17 15:37:25	openvpn	20992	/usr/local/sbin/ovpn-linkdown ovpnc1 1500 1575 10.0.9.2 10.0.9.1 init
Aug 17 15:38:00	openvpn	11923	disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
Aug 17 15:38:00	openvpn	11923	OpenVPN 2.4.4 armv6-portbld-freebsd11.1 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Mar 16 2018
Aug 17 15:38:00	openvpn	11923	library versions: OpenSSL 1.0.2m-freebsd 2 Nov 2017, LZO 2.10
Aug 17 15:38:00	openvpn	12227	MANAGEMENT: unix domain socket listening on /var/etc/openvpn/client1.sock
Aug 17 15:38:00	openvpn	12227	NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Aug 17 15:38:00	openvpn	12227	Initializing OpenSSL support for engine 'cryptodev'
Aug 17 15:38:00	openvpn	12227	Outgoing Static Key Encryption: Cipher 'AES-128-CBC' initialized with 128 bit key
Aug 17 15:38:00	openvpn	12227	Outgoing Static Key Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Aug 17 15:38:00	openvpn	12227	Incoming Static Key Encryption: Cipher 'AES-128-CBC' initialized with 128 bit key
Aug 17 15:38:00	openvpn	12227	Incoming Static Key Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Aug 17 15:38:00	openvpn	12227	ROUTE_GATEWAY 192.168.1.254/255.255.255.0 IFACE=mvneta2 HWADDR=00:08:a2:0d:8c:2e
Aug 17 15:38:00	openvpn	12227	TUN/TAP device ovpnc1 exists previously, keep at program end
Aug 17 15:38:00	openvpn	12227	TUN/TAP device /dev/tun1 opened
Aug 17 15:38:00	openvpn	12227	do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Aug 17 15:38:00	openvpn	12227	/sbin/ifconfig ovpnc1 10.0.9.2 10.0.9.1 mtu 1500 netmask 255.255.255.255 up
Aug 17 15:38:00	openvpn	12227	/usr/local/sbin/ovpn-linkup ovpnc1 1500 1575 10.0.9.2 10.0.9.1 init
Aug 17 15:38:00	openvpn	12227	/sbin/route add -net 10.0.0.0 10.0.9.1 255.255.255.0
Aug 17 15:38:00	openvpn	12227	TCP/UDP: Preserving recently used remote address: [AF_INET]OPENVPN_SERVER_EXTERNAL_IP:51195
Aug 17 15:38:00	openvpn	12227	Socket Buffers: R=[65228->65228] S=[65228->65228]
Aug 17 15:38:00	openvpn	12227	Attempting to establish TCP connection with [AF_INET]x:51195 [nonblock]
Aug 17 15:38:01	openvpn	12227	TCP connection established with [AF_INET]OPENVPN_SERVER_EXTERNAL_IP:51195
Aug 17 15:38:01	openvpn	12227	TCPv4_CLIENT link local (bound): [AF_INET]192.168.1.147:0
Aug 17 15:38:01	openvpn	12227	TCPv4_CLIENT link remote: [AF_INET]OPENVPN_SERVER_EXTERNAL_IP:51195
Aug 17 15:38:01	openvpn	12227	cipher_ctx_update: EVP_CipherUpdate() failed
Aug 17 15:38:01	openvpn	12227	Exiting due to fatal error

useru0284t35

Sorry to resurrect an old post, but I'm seeing the same thing. Same log data as above. Seem to just happen after a period of time. Reboot resolves it.

SipriusPT

I have "solved" this issue in the same way, by just rebooting this system, and since the day that I have started this thread, I didnt had this problem again.

luckman212

Happened to one of my units today—client was a Netgate SG-3100, running 2.4.4-p2
Nothing I could do from the commandline fixed it, just had to reboot the box.
Uptime was 89 days previous to that.

It's a real head scratcher.

useru0284t35

@luckman212 Unfortunately disabling hardware crypto on the SG-3100 was the "solution" for me. Everything has been reliably stable since.

luckman212

@useru0284t35 Huh. Ok, well good to know I guess. Pros and Cons.

SipriusPT

This problem that I had one time, was with that unit too, the SG-3100.

luckman212

@useru0284t35 Actually - sad to say that I checked the settings on my affected unit and Crypto hardware was already set to "None"

So I guess that isn't actually the cause, and maybe you were just lucky...

pfsenser_ca

Sorry to poke an old thread, but we're also seeing this on just one of our SG-3100 units, after flawless operation for many months:

https://forum.netgate.com/topic/159722/openvpn-client-fatal-error

adamw

Still and issue on SG-3100 running 2.4.5-p1:

cipher_ctx_update: EVP_CipherUpdate() failed
Exiting due to fatal error

Nobody can connect, service crashes and continue crashing after restarting.
The only solution seems to be a full firewall reboot.

This sounds like it's a bug in OpenSSL: http://cve.circl.lu/cve/CVE-2021-23840

On our SG-3100 we have 1.0.2u, which is affected. Changelog says it was fixed in 1.1.1j in Feb 2021: https://www.openssl.org/news/changelog.html

Is the fix likely to ever find its way to 2.4.x or was 2.4.5-p1 the final release?

PTZ-M

A similar problem on 2.4 downgraded from 2.5. Appear randomly.
All three clients cannot connect to the server until manually restart the daemon.
There is nothing unusual in the logs.

adamw

@ptz-m For me restarting the service from web GUI didn't work. It was crashing within seconds. It only came back after a full firewall reboot.