OpenVPN on 2.7.1 crashes on some circumstances

ogghi

@Volui-0
Hi there, no solution from my end, but just wanting to say: Affected, too!

We had a difficult upgrade from 2.6.0 to 2.7.0 where the SSD was not booting anymore. Installed fresh 2.7.1 over it (could not even find 2.6.0 image to download) and imported settings backup.
All was up and working again.

Only issue is the VPN server (UDP port 1443 here) crashing randomly. I'll now monitor the system.log.
Any other log file to look at?

OpenVPN 2.6.7 amd64-portbld-freebsd14.0 [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [MH/RECVDA] [AEAD] [DCO]
library versions: OpenSSL 3.0.12 24 Oct 2023, LZO 2.10
Is what is currently installed on here

ogghi

@Volui-0
As found on a Reddit post:

There was a patch to openvpn that you can install using the CLI.
openvpn: 2.6.7 -> 2.6.7_1 [pfSense]

pkg upgrade

I did this and restarted the VPN server in question, hoping it's enough. Let's see!

Volui

@ogghi
I will waiting until it crashes again with more verbose log and try to catch the bug in it with extended info about it. Then, i will post that logs there and try your solution, thanks!

Volui

Ok, the server crashed again. The more detailed log didn't show much, but here it is nonetheless:

openvpn.log:
Nov 23 04:33:51 pf openvpn[56470]: MULTI: multi_create_instance called
Nov 23 04:33:51 pf openvpn[56470]: Re-using SSL/TLS context
Nov 23 04:33:51 pf openvpn[56470]: Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 23 04:33:51 pf openvpn[56470]: Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 23 04:33:51 pf openvpn[56470]: Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 23 04:33:51 pf openvpn[56470]: Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 23 04:33:51 pf openvpn[56470]: Control Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1250 tun_max_mtu:0 headroom:126 payload:1600 tailroom:126 ET:0 ]
Nov 23 04:33:51 pf openvpn[56470]: Data Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1500 tun_max_mtu:1600 headroom:136 payload:1768 tailroom:562 ET:0 ]
Nov 23 04:33:51 pf openvpn[56470]: TCP connection established with [AF_INET]45.79.168.172:34222
Nov 23 04:33:51 pf openvpn[56470]: TCPv4_SERVER link local: (not bound)
Nov 23 04:33:51 pf openvpn[56470]: TCPv4_SERVER link remote: [AF_INET]45.79.168.172:34222
Nov 23 04:33:53 pf openvpn[56470]: MULTI: multi_create_instance called
Nov 23 04:33:53 pf openvpn[56470]: Re-using SSL/TLS context
Nov 23 04:33:53 pf openvpn[56470]: Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 23 04:33:53 pf openvpn[56470]: Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 23 04:33:53 pf openvpn[56470]: Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Nov 23 04:33:53 pf openvpn[56470]: Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Nov 23 04:33:53 pf openvpn[56470]: Control Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1250 tun_max_mtu:0 headroom:126 payload:1600 tailroom:126 ET:0 ]
Nov 23 04:33:53 pf openvpn[56470]: Data Channel MTU parms [ mss_fix:0 max_frag:0 tun_mtu:1500 tun_max_mtu:1600 headroom:136 payload:1768 tailroom:562 ET:0 ]
Nov 23 04:33:53 pf openvpn[56470]: TCP connection established with [AF_INET]45.79.168.172:34218
Nov 23 04:33:53 pf openvpn[56470]: TCPv4_SERVER link local: (not bound)
Nov 23 04:33:53 pf openvpn[56470]: TCPv4_SERVER link remote: [AF_INET]45.79.168.172:34218
Nov 23 04:33:57 pf openvpn[56470]: 45.79.168.172:34215 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Nov 23 04:33:57 pf openvpn[56470]: 45.79.168.172:34215 TLS Error: TLS handshake failed

system.log:
Nov 23 04:33:57 kernel pid 56470 (openvpn), jid 0, uid 0: exited on signal 11 (core dumped)
Nov 23 04:33:57 kernel ovpns1: link state changed to DOWN

In openvpn.log there several connection attempts in a row from the same IP address, then the server crashed. Now all that remains is to try the solution suggested by ogghi above (pkg upgrade). I hope this works.

ogghi

@Volui-0
I think it will work.
It hasn't crashed here anymore!

I am wondering if there is any official statement from Netgate or so?

Volui

@ogghi
Yes, I also updated the OpenVPN package via pkg upgrade and now all that remains is to monitor the stability of the server. You are right, it looks like netgate has updated the openvpn package in its repository since the release of 2.7.1, but has not said anything about it anywhere. In any case, pkg upgrade is the only thing we can do about this problem for now.

ogghi

It seems stable here in regards to disconnects, but people using remote desktop sometimes get some timeouts as it seems. Keeping an eye on logs today.

Happy Monday ppl!

jimp

OpenVPN released OpenVPN 2.6.8 which addresses a segfault that some users see with 2.6.7:

https://github.com/OpenVPN/openvpn/issues/449

https://openvpn.net/community-downloads/

We're still discussing what the best course of action is to address it since it doesn't seem to be widespread.

slu

@jimp
we are also affected by random openvpn crashes, this setup running before over 8 years without any issue.

There are many different openvpn clients and version on this server, maybe this combination trigger the issue more...

jimp

We'll be bringing in OpenVPN 2.6.8 to the next patch release (Plus 23.09.1, CE 2.7.2) which should be out here in the next week or so if all goes according to plan.

https://redmine.pfsense.org/issues/15049

michmoor

@jimp
Hey Jim. Any release notes on 23.09.1 ?

jimp

@michmoor said in OpenVPN on 2.7.1 crashes on some circumstances:

@jimp
Hey Jim. Any release notes on 23.09.1 ?

Nothing public yet, but it's primarily security/stability things. The ZFS corruption bug, OpenVPN needed a version bump, so did strongSwan, various fixes in PHP code. Not a long list, but significant enough to warrant a patch release.

michmoor

@jimp
sounds good to me. thanks!

jimp

FYI- If you are on Plus 23.09 or CE 2.7.1 we picked back the update to the current repositories so you can also get OpenVPN 2.6.8_1 right now from the console or SSH shell prompt:

# pkg update
# pkg upgrade -y openvpn

And then either restart each instance manually or reboot.

tedquade

@jimp Done.

Thanks
Ted

OhYeah 0

@jimp said in OpenVPN on 2.7.1 crashes on some circumstances:

FYI- If you are on Plus 23.09 or CE 2.7.1 we picked back the update to the current repositories so you can also get OpenVPN 2.6.8_1 right now from the console or SSH shell prompt:
# pkg update
# pkg upgrade -y openvpn
And then either restart each instance manually or reboot.

We had one virtual instance of pfsense that had the OpenVPN remote access server crash every 2-3 days. After pkg update the server has been running without problems for 5+ days.

slu

We can confirm this, no issue since 2.6.8_1, thank you @jimp

Simply reinstall of the package openvpn-client-export does also the OpenVPN upgrade.

thuyetti

@OhYeah-0 said in OpenVPN on 2.7.1 crashes on some circumstances:

@jimp said in OpenVPN on 2.7.1 crashes on some circumstances:
FYI- If you are on Plus 23.09 or CE 2.7.1 we picked back the update to the current repositories so you can also get OpenVPN 2.6.8_1 right now from the console or SSH shell prompt:
# pkg update
# pkg upgrade -y openvpn
And then either restart each instance manually or reboot.
We had one virtual instance of pfsense that had the OpenVPN remote access server crash every 2-3 days. After pkg update the server has been running without problems for 5+ days.

Thank you a lot @jimp , it works for me since more than 20 days now. We have 3 OpenVPN servers.

Volui

It's seems to be solved for now (running almost month without crashes). To solve problem do from console from root:

pkg update
pkg upgrade -y openvpn

Or just upgrade to latest release! For me, openVPN run as rock solid after i do upgraded the package and then update system to the latest stable release 2.7.2! Thanks All Guys!

ncohafmuta

I just ran into this crash, same errors as OP.
This is the first time i've ever seen the openvpn server crash. Usually my prod system is a i3-4130 (for years now), but i've been testing on an n100 system for about a week now and this crash just happened. Same pfsense version and config from the i3 system.
I'm running pfsense 2.7.1 w/openvpn 2.6.8_1 so the new version doesn't fix it

# pkg info
openvpn-2.6.8_1 Secure IP/Ethernet tunnel daemon

# openvpn --version
OpenVPN 2.6.8 amd64-portbld-freebsd14.0 [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [MH/RECVDA] [AEAD] [DCO]
library versions: OpenSSL 3.0.12 24 Oct 2023, LZO 2.10
DCO version: FreeBSD 14.0-CURRENT amd64 1400094 #1 RELENG_2_7_1-n255918-774957be06d: Wed Nov 15 17:41:06 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_1-main/obj/amd64/GScwGwyy/var/jenkins/workspace/pfSense-CE-snapshots-2_7_1-main/sources/F