OpenVPN 2.5.0 Certificate Verification Fails

fr3ddie

@gribnut
thank you for your information.
But there is still something that I can't understand: how a length issue on the certificate subject can be fixed by just enclosing the subject in apexes?

What I believe is that, probably, there is more than one issue here in fcgicli: an issue linked to the subject length and maybe another issue linked to "special" characters (spaces? slashes?) included in the certificate subject that is fixed by simply enclosing the subject between apexes.

divsys

@nycspud Do you know what error messages OpenVPN was logging while it was failing?
I'd be curious what happens if you set Certificate Depth back to Level one, does OpenVPN start failing again or is it "magically" OK?

As noted below, there seems to be an internal interaction were not catching that trips the Certficate checks and causes OpenVPN to fail. My experiences so far have been odd in that I've seen hard fails right after an upgrade but once I managed to massage the certificate depth or the ovpn_auth_verify "fix", I could no longer replicate the issue by backing out the fix.

nycspud

@divsys Yes, setting Certificate Depth = 1 or higher causes it to fail.
With Certificate Depth = 1 or higher
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 WARNING: Failed running command (--tls-verify script): external program exited with error status: 1
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 VERIFY SCRIPT ERROR: depth=1, C=US, ST=XX, L=XX, O=XX, emailAddress=XX@XX.XX, CN=ovpn
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 OpenSSL: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 TLS_ERROR: BIO read tls_read_plaintext error
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 TLS Error: TLS object -> incoming plaintext read error
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 TLS Error: TLS handshake failed
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 SIGUSR1[soft,tls-error] received, client-instance restarting

On the client side when Certificate Depth = 1 or higher
2021-02-23 07:11:48 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
2021-02-23 07:11:48 TLS Error: TLS handshake failed
2021-02-23 07:11:48 SIGUSR1[soft,tls-error] received, process restarting
2021-02-23 07:11:48 MANAGEMENT: >STATE:1614093108,RECONNECTING,tls-error,,,,,
2021-02-23 07:11:48 Restart pause, 5 second(s)

I only have a self signed root CA, and no intermediate CA.

jimp

If it is from fcgicli, you might try the original change for #9460 (before it was fixed properly last time) by using the System Patches package and then create an entry for ce76f299853dccb036de229f08a30013593c98fd to apply the change. It will use php-cgi instead of fcgicli.

gribnut

@fr3ddie
I agree on skepticism that surrounding one of the key values with quotes in key/value pairs submitted to fcgicli actually fixes the problem. It's possible that while it returns OK using workaround I listed it could cause a different behavior that may or may not work for accurately detecting cert depth. Since saving openvpn config changes overwrites ovpn_auth_verify anyway, I just went with workaround to disable cert depth check until (or if) issue is resolved with fcgicli and lengthy args to -d from bug reported in redmine.

DeaDSouL

@gribnut I tried your fix and it worked for me. (I'm on pfSense 2.5.0-RELEASE)
Thank you

wtw

I tried the patch on pfSense 2.5.0-RELEASE also. No change. PIA VPN connection will not connect.
Feb 24 13:45:36 openvpn 31850 Restart pause, 5 second(s)
Feb 24 13:45:36 openvpn 31850 SIGUSR1[soft,tls-error] received, process restarting
Feb 24 13:45:36 openvpn 31850 TCP/UDP: Closing socket
Feb 24 13:45:36 openvpn 31850 TLS Error: TLS handshake failed
Feb 24 13:45:36 openvpn 31850 TLS Error: TLS object -> incoming plaintext read error
Feb 24 13:45:36 openvpn 31850 TLS_ERROR: BIO read tls_read_plaintext error
Feb 24 13:45:36 openvpn 31850 OpenSSL: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
Feb 24 13:45:36 openvpn 31850 VERIFY ERROR: depth=1, error=self signed certificate in certificate chain: C=US, ST=CA, L=LosAngeles, O=Private Internet Access, OU=Private Internet Access, CN=Private Internet Access, name=Private Internet Access, emailAddress=secure@privateinternetaccess.com, serial=11996199170155461251

PIA suggested rolling back OpenVPN from 2.5 to 2.4.
Any suggestions?
Thanks.

jimp

@wtw said in OpenVPN 2.5.0 Certificate Verification Fails:

I tried the patch on pfSense 2.5.0-RELEASE also. No change. PIA VPN connection will not connect.

The problem in this thread is for OpenVPN servers on pfSense, not clients. Your problem is unlikely to be related. Start a new thread for your issue if you haven't already.

gribnut

As @jimp noted, the failure of cert depth check with long cert subject name is due fcgicli inability to parse long key/value strings. I have verified that replacing fcgicli with php-cgi as noted in patch for #4521 ( and #9460 ) resolves. When using php-cgi, cert depth check works as expected with my self generated cert, intermediate and root CA.

wtw

@gribnut said in OpenVPN 2.5.0 Certificate Verification Fails:

As @jimp noted, the failure of cert depth check with long cert subject name is due fcgicli inability to parse long key/value strings. I have verified that replacing fcgicli with php-cgi as noted in patch for #4521 ( and #9460 ) resolves. When using php-cgi, cert depth check works as expected with my self generated cert, intermediate and root CA.

I tried the suggestion you provded "The fifth arg ($5)" above. No change.
I tried the patch #9460; ce76f299853dccb036de229f08a30013593c98fd as suggested. No change.
I started a new topic: pfSense 2.5.0-RELEASE OpenVPN Cert bug
What worked was creating a new self-signed cert using the same data. The difference in the backup XML is provided there.
Is patch #4521 different than #9460?
If the patch in only for an OpenVPN server, then this would be expected.

chrullrich

@jimp said in OpenVPN 2.5.0 Certificate Verification Fails:

If it is from fcgicli, you might try the original change for #9460 (before it was fixed properly last time) by using the System Patches package and then create an entry for ce76f299853dccb036de229f08a30013593c98fd to apply the change. It will use php-cgi instead of fcgicli.

I had the same issue immediately after upgrading from 2.4.5p1 to 2.5.0, and this fix did not work for me either. However, this was because it fixes the wrong thing.

The commit above modifies /usr/local/sbin/ovpn_auth_verify_async, while OpenVPN actually calls /usr/local/sbin/ovpn_auth_verify. I changed this script in the same way the commit does:

change /usr/local/sbin/fcgicli to /usr/local/bin/php-cgi
remove the -d in front of the query string

This worked; OpenVPN now again allows incoming connections.

Here is the patch (apply with strip 0 in /usr/local/sbin) for reference:

--- ovpn_auth_verify.orig       2021-03-07 18:20:59.312509000 +0000
+++ ovpn_auth_verify    2021-03-07 18:21:42.060270000 +0000
@@ -24,14 +24,14 @@
        for check_depth in $(/usr/bin/seq ${3} -1 0)
        do
                eval serial="\$tls_serial_${check_depth}"
-               RESULT=$(/usr/local/sbin/fcgicli -f /etc/inc/openvpn.tls-verify.php -d "servercn=$2&depth=$3&certdepth=$4&certsubject=$5&serial=$serial&config=$config")
+               RESULT=$(/usr/local/bin/php-cgi -f /etc/inc/openvpn.tls-verify.php "servercn=$2&depth=$3&certdepth=$4&certsubject=$5&serial=$serial&config=$config")
        done
 else
        # Single quoting $password breaks getting the value from the variable.
        # Base64 and urlEncode usernames and passwords
        password=$(echo -n "${password}" | openssl enc -base64 | sed -e 's_=_%3D_g;s_+_%2B_g;s_/_%2F_g')
        username=$(echo -n "${username}" | openssl enc -base64 | sed -e 's_=_%3D_g;s_+_%2B_g;s_/_%2F_g')
-       RESULT=$(/usr/local/sbin/fcgicli -f /etc/inc/openvpn.auth-user.php -d "username=$username&password=$password&cn=$common_name&strictcn=$3&authcfg=$2&modeid=$4&nas_port=$5")
+       RESULT=$(/usr/local/bin/php-cgi -f /etc/inc/openvpn.auth-user.php "username=$username&password=$password&cn=$common_name&strictcn=$3&authcfg=$2&modeid=$4&nas_port=$5")
 fi

 if [ "${RESULT}" = "OK" ]; then

--
Christian

shpokas

I can only add that I was bitten by this issue, too.
What is interesting is that certificate verification failed for some users, but not for all.
CA CN contains spaces and user certificates contain spaces. And yet some users can use OpenVPN and some cannot. Disabling Certificate Depth verification fixed that.
I hope this gets classified as a bug and will be fixed in the future.

gribnut

@shpokas
The initial assumption of having space in the cert Subject was a red herring. Problem is actually due to length of string passed to fcgicli for key value pairs. If the cert Subject was a long value, there was a good chance the command fcgicli would fail. As noted above, it is a known issue. See #4521 for more info and patch. The patch replaces fcgicli with php-cgi in script called by OpenVPN for cert depth check. php-cgi does not have issue with the longer string argument that includes cert Subject name. You can apply the patch as a temp fix until it is applied to future version of pfsense.

lan2021

I can confirm this.

It is always the same thing: I googled and found do nothing
appropriate.

Than i dived into the things and figured out the length problem myself. Definitely, length is the problem. Reproducable on the commandline.

When googling again with fcgicli in the search, i found this thread.

Manually applied the patch and everthing works fine.

vinceent

good day to all same problem i have after update pfsense to 2.5.1
this messages i recived in server

Jul 5 13:20:08 openvpn 90254 ip:43573 TLS Error: TLS handshake failed
Jul 5 13:20:08 openvpn 90254 ip:43573 TLS Error: TLS object -> incoming plaintext read error
Jul 5 13:20:08 openvpn 90254 ip:43573 TLS_ERROR: BIO read tls_read_plaintext error
Jul 5 13:20:08 openvpn 90254 ip:43573 OpenSSL: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
Jul 5 13:20:08 openvpn 90254 ip:43573 VERIFY ERROR: depth=0, error=unsupported certificate purpose: C=RO, ST=HD, L=MT, O=ITL, emailAddress=mail, CN=pfsmtsrv, OU=IT, serial=1
Jul 5 13:19:28 openvpn 90254 Initialization Sequence Completed
Jul 5 13:19:28 openvpn 90254 UDPv4 link remote: [AF_UNSPEC]
Jul 5 13:19:28 openvpn 90254 UDPv4 link local (bound): [AF_INET]ip:44442
Jul 5 13:19:28 openvpn 90254 /usr/local/sbin/ovpn-linkup ovpns4 1500 1622 ip 255.255.255.0 init
Jul 5 13:19:28 openvpn 90254 /sbin/ifconfig ovpns4 10.1.2.1 10.1.2.2 mtu 1500 netmask 255.255.255.0 up
Jul 5 13:19:28 openvpn 90254 TUN/TAP device /dev/tun4 opened
Jul 5 13:19:28 openvpn 90254 TUN/TAP device ovpns4 exists previously, keep at program end
Jul 5 13:19:28 openvpn 90254 WARNING: experimental option --capath /var/etc/openvpn/server4/ca
Jul 5 13:19:28 openvpn 90254 Initializing OpenSSL support for engine 'devcrypto'
Jul 5 13:19:28 openvpn 90254 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jul 5 13:19:28 openvpn 90243 library versions: OpenSSL 1.1.1k-freebsd 25 Mar 2021, LZO 2.10
Jul 5 13:19:28 openvpn 90243 OpenVPN 2.5.1 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Apr 5 2021
Jul 5 13:19:28 openvpn 90243 WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless "allow-compression yes" is also set.

This messages is in client

Jul 5 13:24:23 openvpn 34328 UDPv4 link remote: [AF_INET]ip:44442
Jul 5 13:24:23 openvpn 34328 UDPv4 link local (bound): [AF_INET]ip:0
Jul 5 13:24:23 openvpn 34328 TCP/UDP: Preserving recently used remote address: [AF_INET]ip:44442
Jul 5 13:24:23 openvpn 34328 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jul 5 13:24:23 openvpn 34328 WARNING: No server certificate verification method has been enabled
Jul 5 13:24:18 openvpn 34328 SIGUSR1[soft,ping-restart] received, process restarting
Jul 5 13:24:18 openvpn 34328 [pfsmtsrv] Inactivity timeout (--ping-restart), restarting
Jul 5 13:22:13 openvpn 34328 UDPv4 link remote: [AF_INET]ip:44442
Jul 5 13:22:13 openvpn 34328 UDPv4 link local (bound): [AF_INET]ip:0
Jul 5 13:22:13 openvpn 34328 TCP/UDP: Preserving recently used remote address: [AF_INET]89.121.228.158:44442
Jul 5 13:22:13 openvpn 34328 WARNING: experimental option --capath /var/etc/openvpn/client2/ca
Jul 5 13:22:13 openvpn 34328 Initializing OpenSSL support for engine 'devcrypto'
Jul 5 13:22:13 openvpn 34328 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jul 5 13:22:13 openvpn 34328 WARNING: No server certificate verification method has been enabled.
Jul 5 13:22:13 openvpn 34328 WARNING: using --pull/--client and --ifconfig together is probably not what you want
Jul 5 13:22:13 openvpn 34039 library versions: OpenSSL 1.1.1k-freebsd 25 Mar 2021, LZO 2.10
Jul 5 13:22:13 openvpn 34039 OpenVPN 2.5.1 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Apr 5 2021
Jul 5 13:22:13 openvpn 34039 WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless "allow-compression yes" is also set.

Summer

Hi,

I've just upgraded to 23.05.1 and facing same error:

Fatal TLS error (check_tls_errors_co), restarting
OpenSSL: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed

I've tried the Check Depth trick, but did not work, any idea?

Thanks, BR

jimp

@Summer Start your own thread, this one is over two years old and highly unlikely to be the same issue.