OpenVPN 2.5.0 Certificate Verification Fails

joshh

I'm having issues with my OpenVPN site configurations after upgrading to 2.5.0 (reproduced on two pfsense boxes). I previously have not had any issues while upgrading with previous pfsense versions. It seems that certificate verification may be broken or working differently in 2.5.0 - I can connect successfully only after disabling certificate verification.

For my setup, I have a self-signed root CA, intermediate CA (signed by the root CA), and server/user certificates (signed by the intermediate CA). The root CA, intermediate CA, and server/user certificates are all imported into pfsense. My certificate depth verification is set to Two (Client+Intermediate+Server). The peer certificate authority is set to the intermediate CA. I get the following logs when attempting to connect from a client (as well as a pfsense to pfsense site-to-site setup):

Feb 21 13:16:20	openvpn	99514	<user ip>:51909 VERIFY WARNING: depth=0, unable to get certificate CRL: <user cert>
Feb 21 13:16:20	openvpn	99514	<user ip>:51909 VERIFY WARNING: depth=1, unable to get certificate CRL: <intermediate cert>
Feb 21 13:16:20	openvpn	99514	<user ip>:51909 VERIFY WARNING: depth=2, unable to get certificate CRL: <root cert>
Feb 21 13:16:20	openvpn	99514	<user ip>:51909 VERIFY SCRIPT OK: depth=2, <root cert>
Feb 21 13:16:20	openvpn	99514	<user ip>:51909 VERIFY OK: depth=2, <root cert>
Feb 21 13:16:20	openvpn	99514	<user ip>:51909 WARNING: Failed running command (--tls-verify script): external program exited with error status: 1
Feb 21 13:16:20	openvpn	99514	<user ip>:51909 VERIFY SCRIPT ERROR: depth=1, <intermediate cert>
Feb 21 13:16:21	openvpn	99514	<user ip>:51909 SSL alert (write): fatal: unknown CA
Feb 21 13:16:21	openvpn	99514	<user ip>:51909 OpenSSL: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
Feb 21 13:16:21	openvpn	99514	<user ip>:51909 TLS_ERROR: BIO read tls_read_plaintext error
Feb 21 13:16:21	openvpn	99514	<user ip>:51909 TLS Error: TLS object -> incoming plaintext read error
Feb 21 13:16:21	openvpn	99514	<user ip>:51909 TLS Error: TLS handshake failed

It seems like there is a verification of the intermediate CA at depth=1 which fails with unknown CA. I'm not sure what is normally done in previous versions of pfsense.

gribnut

@joshh
I have observed same failure to connect after updating to pfsense 21.02 and OpenVPN 2.5.0. I also use a similar setup where I have a self generated root CA, intermediate CA with server/client certs signed by the intermediate CA.

I don't have a copy of the previous openvpn server config generated by pfsense, but it appears that pfsense enabled an additional feature tls-verify that wasn't used in previous version of pfsense. After upgrading pfsense, the "Certificate Depth" option was set to "Two (Client + Intermediate + Server)".

This results in following config line in the generated config file /var/etc/openvpn/server1/config.ovpn:

tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'myhostname.mydomain.com' 2"

One would have thought this would work since both client and server are signed by the same intermediate cert. But as with your failure, the script returns a non zero value causing openvpn to fail the connection. The script "/usr/local/sbin/ovpn_auth_verify" appears to be maintained by pfsense.

I've not yet spent anytime troubleshooting the script "/usr/local/sbin/ovpn_auth_verify" to determine why it's failing with my self generated root CA and intermediate cert. As a workaround, disabling "Certificate Depth" by setting to "Do Not Check" allows clients to connect until able to resolve why the tls-verify script is failing.

gribnut

I think I may have found cause of failure. The tls-verify cmd calls "/usr/local/sbin/ovpn_auth_verify" which in turn calls following "/usr/local/sbin/fcgicli"

The fifth arg ($5) to ovpn_auth_verify looks to be the subject of the root CA (passed by openvpn). I found that if the CN of the this subject contains a space (the last element of subject), the call to fcgicli fails with error message "Something wrong happened while reading request". I suspect that fcgicli fails the parsing of key value pairs if the last element of the subject contains a space. Adding some escapes/quotes to the command in "/usr/local/sbin/ovpn_auth_verify" seems to fix. If I set cert depth to 2 (or more) with fix applied, the RESULT is "OK" and passes. If I set to depth of 1, it fails as expected with "FAILED".

> diff ovpn_auth_verify ovpn_auth_verify.fixed 
27c27
<               RESULT=$(/usr/local/sbin/fcgicli -f /etc/inc/openvpn.tls-verify.php -d "servercn=$2&depth=$3&certdepth=$4&certsubject=$5&serial=$serial&config=$config")
---
>               RESULT=$(/usr/local/sbin/fcgicli -f /etc/inc/openvpn.tls-verify.php -d "servercn=$2&depth=$3&certdepth=$4&certsubject=\\"$5\\"&serial=$serial&config=$config")

gribnut

Searching bug reports, seems that problem may be more of string length rather than whitespace in CN (my testing also confirms). The length of my subject name 137 chars which looks to be a little larger than length that causes problem per bug report.

See bug report #10560 and #4521.

gbitglenn

@gribnut 's fix didn't work for me, however ...

For some reason, OpenVPN confuses things by having TWO clients: "OpenVPN Connect v3", (The one you download from openvpn.net), "OpenVPN GUI v11", which is what PFSense uses when you download the full client installer from "Client Export".

Since some clients have more than one VPN profile they need to connect to, and not everyone is using PFSense with their fancy bundled deploy, I use the OpenVPN Connect client. It's got a nicer UI that's easier to use for the end-users plus has a nice traffic graph. It's also the one readily downloadable from openvpn.net.

PFSense seems to have broken the OpenVPN Connect client. If you use the crappy client PFSense bundles with Client Export it works for me.

fr3ddie

@gribnut

Same problem here, same fix as you suggested.
CA CN is "Home CA" (note the space) and client is an old OpenVPN v2.3.4 (with the associated network-manager-openvpn plugin v0.9.10) on a Debian Jessie.
pfSense v2.4.5p1 has no issue, upgraded to pfSense v2.5.0 it is impossible to connect with the "client certificate verification failed" error already reported in the starting post.
Applied your fix (\\"$5\\" in the verification script) and everything works again as expected for the old Debian Jessie.

Android phone with "OpenVPN for Android" v0.7.21 has no issue whatsoever, before or after the fix, thus the issue seems also related to the OpenVPN client used.
For sure the /usr/local/sbin/ovpn_auth_verify script distributed with pfSense v2.5.0 is buggy: v2.4.5p1 had no problem at all.

divsys

@gribnut This fix seems to have solved a similar problem I have been having with TLS/SSL OpenVPN connections.
Converted a stable 2.4.5p1 to 2.4.5p1 pair to 2.5.0 - 2.5.0 and lost the existing S2S OpenVPN link. I rebuilt the CA/Certs for Server and Client under 2.5.0 and got the link back.
Later I realized the Server had a secondary RoadWarrior setup that was also now failing previously stable clients. Logs pointed to failure to find CA.
Tried your fix and Voila it all came back to life.

Looks like a definite bug here, what's interesting is my CA common name has no spaces as far as I can tell.
It's "811pow-ovpn-rdwar-ca" although I'm not sure if there's a leading space buried in there. I've used an OpenSSL command to dump the subject, but it's not clear if the output adds a space to the entries as they are printed.

gribnut

@divsys
I was a bit off when I though was due to space as removing space appeared to fix problem. Looks like the root problem is length of string (all key value pairs) for -d arg to fcgicli. If combination of server hostname (as used by client), cert subject and serial number is too long, fcgicli bombs and returns "Something wrong happened while reading request". Guessing the length of various values in your environment also exceed what works for fcgicli.
Looks like bug #4521 was reported some time ago and no indication of when will be fixed. I added comment to confirm it is still a problem in hopes it gets fixed in near future. Fortunately, I don't have a dependency upon limiting cert depth.

gribnut

@fr3ddie
I forgot to mention that anytime you save openvpn config, it will write over any changes made to /usr/local/sbin/ovpn_auth_verify. I've not looked to see which file updates /usr/local/sbin/ovpn_auth_verify, so I just disabled cert depth check altogether until long arg to fcgicli is resolved.

Just guessing, but suspect that reason one of your clients might work whereas as others don't is due to length of cert subject for client. The tls_verify will iterate through entire chain (depending up configured depth to search). The length of string for subject (the entire subject not just CN) could end up causing arg passed to fcgiclie to exceed value that works.

divsys

@gribnut Thanks for your help in tracking this down. Makes me much more hopeful about converting up to 2.5.0.

JKnott

@gribnut

So, does disabling that depth check allow the VPN to work? Currently, I can get it to work on the same LAN, but not from outside the firewall.

gribnut

@jknott
Disabling depth check is only a workaround if the tls-verify script is failing. Won't fix other problems. Recommend checking your logs to see cause of failure.

divsys

@gribnut Hmmmmm, I think there's a bit more to this one than meets the eye.
On ahunch, I backed out the code change to ovpn_auth_verify back to factory.

Roadwarrior links still good

Restarted the RoadWarror server process

Roadwarrior links still good

Rebooted the remote pfSense box

Roadwarrior links still good

So now I 'm at the point that I can't make it fail anymore after backing all my changes out <sigh>

One thing to note in all this - none of my OpenVPN Servers do more than a depth check of 1.

nycspud

@gribnut I just upgraded to 2.5 today and noticed right away that I couldn't connect to OpenVPN.
Once I set Certificate Depth to Do not check it worked again.

fr3ddie

@gribnut
thank you for your information.
But there is still something that I can't understand: how a length issue on the certificate subject can be fixed by just enclosing the subject in apexes?

What I believe is that, probably, there is more than one issue here in fcgicli: an issue linked to the subject length and maybe another issue linked to "special" characters (spaces? slashes?) included in the certificate subject that is fixed by simply enclosing the subject between apexes.

divsys

@nycspud Do you know what error messages OpenVPN was logging while it was failing?
I'd be curious what happens if you set Certificate Depth back to Level one, does OpenVPN start failing again or is it "magically" OK?

As noted below, there seems to be an internal interaction were not catching that trips the Certficate checks and causes OpenVPN to fail. My experiences so far have been odd in that I've seen hard fails right after an upgrade but once I managed to massage the certificate depth or the ovpn_auth_verify "fix", I could no longer replicate the issue by backing out the fix.

nycspud

@divsys Yes, setting Certificate Depth = 1 or higher causes it to fail.
With Certificate Depth = 1 or higher
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 WARNING: Failed running command (--tls-verify script): external program exited with error status: 1
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 VERIFY SCRIPT ERROR: depth=1, C=US, ST=XX, L=XX, O=XX, emailAddress=XX@XX.XX, CN=ovpn
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 OpenSSL: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 TLS_ERROR: BIO read tls_read_plaintext error
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 TLS Error: TLS object -> incoming plaintext read error
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 TLS Error: TLS handshake failed
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 SIGUSR1[soft,tls-error] received, client-instance restarting

On the client side when Certificate Depth = 1 or higher
2021-02-23 07:11:48 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
2021-02-23 07:11:48 TLS Error: TLS handshake failed
2021-02-23 07:11:48 SIGUSR1[soft,tls-error] received, process restarting
2021-02-23 07:11:48 MANAGEMENT: >STATE:1614093108,RECONNECTING,tls-error,,,,,
2021-02-23 07:11:48 Restart pause, 5 second(s)

I only have a self signed root CA, and no intermediate CA.

jimp

If it is from fcgicli, you might try the original change for #9460 (before it was fixed properly last time) by using the System Patches package and then create an entry for ce76f299853dccb036de229f08a30013593c98fd to apply the change. It will use php-cgi instead of fcgicli.

gribnut

@fr3ddie
I agree on skepticism that surrounding one of the key values with quotes in key/value pairs submitted to fcgicli actually fixes the problem. It's possible that while it returns OK using workaround I listed it could cause a different behavior that may or may not work for accurately detecting cert depth. Since saving openvpn config changes overwrites ovpn_auth_verify anyway, I just went with workaround to disable cert depth check until (or if) issue is resolved with fcgicli and lengthy args to -d from bug reported in redmine.

DeaDSouL

@gribnut I tried your fix and it worked for me. (I'm on pfSense 2.5.0-RELEASE)
Thank you