OpenVPN on 2.7.1 crashes on some circumstances

Volui · Nov 22, 2023, 5:08 AM

Hi there! I recently update my installation from 2.7 to 2.7.1 and I have discovered some strange issue after it:
I have 3 tunnels running on my server, two in UDP mode on non-default openvpn ports, and one in TCP mode on port 443 (just as a backup option). All tunnels works just fine for months without any issues. On TCP 443 tunnel i can see (obvious) many connection attempts\portscans but it has never affected the performance of the service itself. Now, after upgrade, it's work just fine, mine clients as before may connect to TCP server vithout any problem and connection still stable. But somtimes, someone from unknown to me IP tries to connect to that (TCP) server and it crashes. Server may work couple of days before that someone unknown tries to connect and it causes the service to crash. All servers configured in same type: Mode: Remote Access ( SSL/TLS + User Auth ) Data Ciphers: AES-256-GCM Digest: SHA256 D-H Params: 2048 bits. There some logs from server, all time when service crash it has same record on log files:

On openvpn.log:
Nov 18 09:02:38 pf openvpn[11964]: 62.233.50.179:65059 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Nov 18 09:02:38 pf openvpn[11964]: 62.233.50.179:65059 TLS Error: TLS handshake failed

Nov 20 16:45:37 pf openvpn[12689]: 147.235.216.91:11353 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Nov 20 16:45:37 pf openvpn[12689]: 147.235.216.91:11353 TLS Error: TLS handshake failed

On system.log:
Nov 18 09:02:38 pf kernel: pid 11964 (openvpn), jid 0, uid 0: exited on signal 11 (core dumped)
Nov 18 09:02:38 pf kernel: ovpns1: link state changed to DOWN

Nov 20 16:45:37 pf kernel: pid 12689 (openvpn), jid 0, uid 0: exited on signal 11 (core dumped)
Nov 20 16:45:37 pf kernel: ovpns1: link state changed to DOWN

In same time legitimate clients can connect and operate vithout any problems.

On UDP tunnels i don't have that problem just because it sit on hi-range ports and no one dont try to connect to it excluding legitimate clients

My config is: i5-2540M CPU with 4GB RAM and 60GB SSD, two GigE adapters (Intel and Marvell). I'am already have run a Memtest 6.20 (3 passes without error) and check if CPU will overheating (they not).

This is not so much a request for help with solving a problem as an attempt to draw attention to its very existence. Although I do not rule out that this is a problem solely with my installation.

P.S. I restarted that tunnel with log verbosity level 6 and I'll just wait until it crashes again.

ogghi · Nov 22, 2023, 5:08 AM

@Volui-0
Hi there, no solution from my end, but just wanting to say: Affected, too!

We had a difficult upgrade from 2.6.0 to 2.7.0 where the SSD was not booting anymore. Installed fresh 2.7.1 over it (could not even find 2.6.0 image to download) and imported settings backup.
All was up and working again.

Only issue is the VPN server (UDP port 1443 here) crashing randomly. I'll now monitor the system.log.
Any other log file to look at?

OpenVPN 2.6.7 amd64-portbld-freebsd14.0 [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [MH/RECVDA] [AEAD] [DCO]
library versions: OpenSSL 3.0.12 24 Oct 2023, LZO 2.10
Is what is currently installed on here

ogghi · Nov 22, 2023, 5:32 AM

@Volui-0
As found on a Reddit post:

There was a patch to openvpn that you can install using the CLI.
openvpn: 2.6.7 -> 2.6.7_1 [pfSense]

pkg upgrade

I did this and restarted the VPN server in question, hoping it's enough. Let's see!

Volui · Nov 22, 2023, 8:48 AM

@ogghi
I will waiting until it crashes again with more verbose log and try to catch the bug in it with extended info about it. Then, i will post that logs there and try your solution, thanks!

Volui · Nov 23, 2023, 10:18 AM

Ok, the server crashed again. The more detailed log didn't show much, but here it is nonetheless:

openvpn.log:
Nov 23 04:33:51 pf openvpn[56470]: MULTI: multi_create_instance called
Nov 23 04:33:51 pf openvpn[56470]: Re-using SSL/TLS context
Nov 23 04:33:51 pf openvpn[56470]: Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 23 04:33:51 pf openvpn[56470]: Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 23 04:33:51 pf openvpn[56470]: Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 23 04:33:51 pf openvpn[56470]: Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 23 04:33:51 pf openvpn[56470]: Control Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1250 tun_max_mtu:0 headroom:126 payload:1600 tailroom:126 ET:0 ]
Nov 23 04:33:51 pf openvpn[56470]: Data Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1500 tun_max_mtu:1600 headroom:136 payload:1768 tailroom:562 ET:0 ]
Nov 23 04:33:51 pf openvpn[56470]: TCP connection established with [AF_INET]45.79.168.172:34222
Nov 23 04:33:51 pf openvpn[56470]: TCPv4_SERVER link local: (not bound)
Nov 23 04:33:51 pf openvpn[56470]: TCPv4_SERVER link remote: [AF_INET]45.79.168.172:34222
Nov 23 04:33:53 pf openvpn[56470]: MULTI: multi_create_instance called
Nov 23 04:33:53 pf openvpn[56470]: Re-using SSL/TLS context
Nov 23 04:33:53 pf openvpn[56470]: Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 23 04:33:53 pf openvpn[56470]: Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 23 04:33:53 pf openvpn[56470]: Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 23 04:33:53 pf openvpn[56470]: Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 23 04:33:53 pf openvpn[56470]: Control Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1250 tun_max_mtu:0 headroom:126 payload:1600 tailroom:126 ET:0 ]
Nov 23 04:33:53 pf openvpn[56470]: Data Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1500 tun_max_mtu:1600 headroom:136 payload:1768 tailroom:562 ET:0 ]
Nov 23 04:33:53 pf openvpn[56470]: TCP connection established with [AF_INET]45.79.168.172:34218
Nov 23 04:33:53 pf openvpn[56470]: TCPv4_SERVER link local: (not bound)
Nov 23 04:33:53 pf openvpn[56470]: TCPv4_SERVER link remote: [AF_INET]45.79.168.172:34218
Nov 23 04:33:57 pf openvpn[56470]: 45.79.168.172:34215 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Nov 23 04:33:57 pf openvpn[56470]: 45.79.168.172:34215 TLS Error: TLS handshake failed

system.log:
Nov 23 04:33:57 kernel pid 56470 (openvpn), jid 0, uid 0: exited on signal 11 (core dumped)
Nov 23 04:33:57 kernel ovpns1: link state changed to DOWN

In openvpn.log there several connection attempts in a row from the same IP address, then the server crashed. Now all that remains is to try the solution suggested by ogghi above (pkg upgrade). I hope this works.

ogghi · Nov 23, 2023, 11:49 AM

@Volui-0
I think it will work.
It hasn't crashed here anymore!

I am wondering if there is any official statement from Netgate or so?

Volui · Nov 23, 2023, 12:10 PM

@ogghi
Yes, I also updated the OpenVPN package via pkg upgrade and now all that remains is to monitor the stability of the server. You are right, it looks like netgate has updated the openvpn package in its repository since the release of 2.7.1, but has not said anything about it anywhere. In any case, pkg upgrade is the only thing we can do about this problem for now.

ogghi · Nov 27, 2023, 6:17 AM

It seems stable here in regards to disconnects, but people using remote desktop sometimes get some timeouts as it seems. Keeping an eye on logs today.

Happy Monday ppl!

jimp · Nov 29, 2023, 4:46 PM

OpenVPN released OpenVPN 2.6.8 which addresses a segfault that some users see with 2.6.7:

https://github.com/OpenVPN/openvpn/issues/449

https://openvpn.net/community-downloads/

We're still discussing what the best course of action is to address it since it doesn't seem to be widespread.

slu · Nov 30, 2023, 8:19 AM

@jimp
we are also affected by random openvpn crashes, this setup running before over 8 years without any issue.

There are many different openvpn clients and version on this server, maybe this combination trigger the issue more...

jimp · Nov 29, 2023, 7:20 PM

We'll be bringing in OpenVPN 2.6.8 to the next patch release (Plus 23.09.1, CE 2.7.2) which should be out here in the next week or so if all goes according to plan.

https://redmine.pfsense.org/issues/15049

michmoor · Nov 29, 2023, 8:43 PM

@jimp
Hey Jim. Any release notes on 23.09.1 ?

jimp · Nov 30, 2023, 4:23 PM

@michmoor said in OpenVPN on 2.7.1 crashes on some circumstances:

@jimp
Hey Jim. Any release notes on 23.09.1 ?

Nothing public yet, but it's primarily security/stability things. The ZFS corruption bug, OpenVPN needed a version bump, so did strongSwan, various fixes in PHP code. Not a long list, but significant enough to warrant a patch release.

michmoor · Nov 30, 2023, 5:37 PM

@jimp
sounds good to me. thanks!

jimp · Nov 30, 2023, 6:18 PM

FYI- If you are on Plus 23.09 or CE 2.7.1 we picked back the update to the current repositories so you can also get OpenVPN 2.6.8_1 right now from the console or SSH shell prompt:

# pkg update
# pkg upgrade -y openvpn

And then either restart each instance manually or reboot.

tedquade · Nov 30, 2023, 6:34 PM

@jimp Done.

Thanks
Ted

OhYeah 0 · Dec 7, 2023, 1:37 PM

@jimp said in OpenVPN on 2.7.1 crashes on some circumstances:

FYI- If you are on Plus 23.09 or CE 2.7.1 we picked back the update to the current repositories so you can also get OpenVPN 2.6.8_1 right now from the console or SSH shell prompt:
# pkg update
# pkg upgrade -y openvpn
And then either restart each instance manually or reboot.

We had one virtual instance of pfsense that had the OpenVPN remote access server crash every 2-3 days. After pkg update the server has been running without problems for 5+ days.

slu · Dec 7, 2023, 1:37 PM

We can confirm this, no issue since 2.6.8_1, thank you @jimp

Simply reinstall of the package openvpn-client-export does also the OpenVPN upgrade.

thuyetti · Jan 2, 2024, 9:34 PM

@OhYeah-0 said in OpenVPN on 2.7.1 crashes on some circumstances:

@jimp said in OpenVPN on 2.7.1 crashes on some circumstances:
FYI- If you are on Plus 23.09 or CE 2.7.1 we picked back the update to the current repositories so you can also get OpenVPN 2.6.8_1 right now from the console or SSH shell prompt:
# pkg update
# pkg upgrade -y openvpn
And then either restart each instance manually or reboot.
We had one virtual instance of pfsense that had the OpenVPN remote access server crash every 2-3 days. After pkg update the server has been running without problems for 5+ days.

Thank you a lot @jimp , it works for me since more than 20 days now. We have 3 OpenVPN servers.

Volui · Jan 3, 2024, 11:02 AM

It's seems to be solved for now (running almost month without crashes). To solve problem do from console from root:

pkg update
pkg upgrade -y openvpn

Or just upgrade to latest release! For me, openVPN run as rock solid after i do upgraded the package and then update system to the latest stable release 2.7.2! Thanks All Guys!