OpenVPN 2.5.0 Certificate Verification Fails

gribnut

@fr3ddie
I forgot to mention that anytime you save openvpn config, it will write over any changes made to /usr/local/sbin/ovpn_auth_verify. I've not looked to see which file updates /usr/local/sbin/ovpn_auth_verify, so I just disabled cert depth check altogether until long arg to fcgicli is resolved.

Just guessing, but suspect that reason one of your clients might work whereas as others don't is due to length of cert subject for client. The tls_verify will iterate through entire chain (depending up configured depth to search). The length of string for subject (the entire subject not just CN) could end up causing arg passed to fcgiclie to exceed value that works.

divsys

@gribnut Thanks for your help in tracking this down. Makes me much more hopeful about converting up to 2.5.0.

JKnott

@gribnut

So, does disabling that depth check allow the VPN to work? Currently, I can get it to work on the same LAN, but not from outside the firewall.

gribnut

@jknott
Disabling depth check is only a workaround if the tls-verify script is failing. Won't fix other problems. Recommend checking your logs to see cause of failure.

divsys

@gribnut Hmmmmm, I think there's a bit more to this one than meets the eye.
On ahunch, I backed out the code change to ovpn_auth_verify back to factory.

Roadwarrior links still good

Restarted the RoadWarror server process

Roadwarrior links still good

Rebooted the remote pfSense box

Roadwarrior links still good

So now I 'm at the point that I can't make it fail anymore after backing all my changes out <sigh>

One thing to note in all this - none of my OpenVPN Servers do more than a depth check of 1.

nycspud

@gribnut I just upgraded to 2.5 today and noticed right away that I couldn't connect to OpenVPN.
Once I set Certificate Depth to Do not check it worked again.

fr3ddie

@gribnut
thank you for your information.
But there is still something that I can't understand: how a length issue on the certificate subject can be fixed by just enclosing the subject in apexes?

What I believe is that, probably, there is more than one issue here in fcgicli: an issue linked to the subject length and maybe another issue linked to "special" characters (spaces? slashes?) included in the certificate subject that is fixed by simply enclosing the subject between apexes.

divsys

@nycspud Do you know what error messages OpenVPN was logging while it was failing?
I'd be curious what happens if you set Certificate Depth back to Level one, does OpenVPN start failing again or is it "magically" OK?

As noted below, there seems to be an internal interaction were not catching that trips the Certficate checks and causes OpenVPN to fail. My experiences so far have been odd in that I've seen hard fails right after an upgrade but once I managed to massage the certificate depth or the ovpn_auth_verify "fix", I could no longer replicate the issue by backing out the fix.

nycspud

@divsys Yes, setting Certificate Depth = 1 or higher causes it to fail.
With Certificate Depth = 1 or higher
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 WARNING: Failed running command (--tls-verify script): external program exited with error status: 1
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 VERIFY SCRIPT ERROR: depth=1, C=US, ST=XX, L=XX, O=XX, emailAddress=XX@XX.XX, CN=ovpn
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 OpenSSL: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 TLS_ERROR: BIO read tls_read_plaintext error
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 TLS Error: TLS object -> incoming plaintext read error
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 TLS Error: TLS handshake failed
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 SIGUSR1[soft,tls-error] received, client-instance restarting

On the client side when Certificate Depth = 1 or higher
2021-02-23 07:11:48 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
2021-02-23 07:11:48 TLS Error: TLS handshake failed
2021-02-23 07:11:48 SIGUSR1[soft,tls-error] received, process restarting
2021-02-23 07:11:48 MANAGEMENT: >STATE:1614093108,RECONNECTING,tls-error,,,,,
2021-02-23 07:11:48 Restart pause, 5 second(s)

I only have a self signed root CA, and no intermediate CA.

jimp

If it is from fcgicli, you might try the original change for #9460 (before it was fixed properly last time) by using the System Patches package and then create an entry for ce76f299853dccb036de229f08a30013593c98fd to apply the change. It will use php-cgi instead of fcgicli.

gribnut

@fr3ddie
I agree on skepticism that surrounding one of the key values with quotes in key/value pairs submitted to fcgicli actually fixes the problem. It's possible that while it returns OK using workaround I listed it could cause a different behavior that may or may not work for accurately detecting cert depth. Since saving openvpn config changes overwrites ovpn_auth_verify anyway, I just went with workaround to disable cert depth check until (or if) issue is resolved with fcgicli and lengthy args to -d from bug reported in redmine.

DeaDSouL

@gribnut I tried your fix and it worked for me. (I'm on pfSense 2.5.0-RELEASE)
Thank you

wtw

I tried the patch on pfSense 2.5.0-RELEASE also. No change. PIA VPN connection will not connect.
Feb 24 13:45:36 openvpn 31850 Restart pause, 5 second(s)
Feb 24 13:45:36 openvpn 31850 SIGUSR1[soft,tls-error] received, process restarting
Feb 24 13:45:36 openvpn 31850 TCP/UDP: Closing socket
Feb 24 13:45:36 openvpn 31850 TLS Error: TLS handshake failed
Feb 24 13:45:36 openvpn 31850 TLS Error: TLS object -> incoming plaintext read error
Feb 24 13:45:36 openvpn 31850 TLS_ERROR: BIO read tls_read_plaintext error
Feb 24 13:45:36 openvpn 31850 OpenSSL: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
Feb 24 13:45:36 openvpn 31850 VERIFY ERROR: depth=1, error=self signed certificate in certificate chain: C=US, ST=CA, L=LosAngeles, O=Private Internet Access, OU=Private Internet Access, CN=Private Internet Access, name=Private Internet Access, emailAddress=secure@privateinternetaccess.com, serial=11996199170155461251

PIA suggested rolling back OpenVPN from 2.5 to 2.4.
Any suggestions?
Thanks.

jimp

@wtw said in OpenVPN 2.5.0 Certificate Verification Fails:

I tried the patch on pfSense 2.5.0-RELEASE also. No change. PIA VPN connection will not connect.

The problem in this thread is for OpenVPN servers on pfSense, not clients. Your problem is unlikely to be related. Start a new thread for your issue if you haven't already.

gribnut

As @jimp noted, the failure of cert depth check with long cert subject name is due fcgicli inability to parse long key/value strings. I have verified that replacing fcgicli with php-cgi as noted in patch for #4521 ( and #9460 ) resolves. When using php-cgi, cert depth check works as expected with my self generated cert, intermediate and root CA.

wtw

@gribnut said in OpenVPN 2.5.0 Certificate Verification Fails:

As @jimp noted, the failure of cert depth check with long cert subject name is due fcgicli inability to parse long key/value strings. I have verified that replacing fcgicli with php-cgi as noted in patch for #4521 ( and #9460 ) resolves. When using php-cgi, cert depth check works as expected with my self generated cert, intermediate and root CA.

I tried the suggestion you provded "The fifth arg ($5)" above. No change.
I tried the patch #9460; ce76f299853dccb036de229f08a30013593c98fd as suggested. No change.
I started a new topic: pfSense 2.5.0-RELEASE OpenVPN Cert bug
What worked was creating a new self-signed cert using the same data. The difference in the backup XML is provided there.
Is patch #4521 different than #9460?
If the patch in only for an OpenVPN server, then this would be expected.

chrullrich

@jimp said in OpenVPN 2.5.0 Certificate Verification Fails:

If it is from fcgicli, you might try the original change for #9460 (before it was fixed properly last time) by using the System Patches package and then create an entry for ce76f299853dccb036de229f08a30013593c98fd to apply the change. It will use php-cgi instead of fcgicli.

I had the same issue immediately after upgrading from 2.4.5p1 to 2.5.0, and this fix did not work for me either. However, this was because it fixes the wrong thing.

The commit above modifies /usr/local/sbin/ovpn_auth_verify_async, while OpenVPN actually calls /usr/local/sbin/ovpn_auth_verify. I changed this script in the same way the commit does:

change /usr/local/sbin/fcgicli to /usr/local/bin/php-cgi
remove the -d in front of the query string

This worked; OpenVPN now again allows incoming connections.

Here is the patch (apply with strip 0 in /usr/local/sbin) for reference:

--- ovpn_auth_verify.orig       2021-03-07 18:20:59.312509000 +0000
+++ ovpn_auth_verify    2021-03-07 18:21:42.060270000 +0000
@@ -24,14 +24,14 @@
        for check_depth in $(/usr/bin/seq ${3} -1 0)
        do
                eval serial="\$tls_serial_${check_depth}"
-               RESULT=$(/usr/local/sbin/fcgicli -f /etc/inc/openvpn.tls-verify.php -d "servercn=$2&depth=$3&certdepth=$4&certsubject=$5&serial=$serial&config=$config")
+               RESULT=$(/usr/local/bin/php-cgi -f /etc/inc/openvpn.tls-verify.php "servercn=$2&depth=$3&certdepth=$4&certsubject=$5&serial=$serial&config=$config")
        done
 else
        # Single quoting $password breaks getting the value from the variable.
        # Base64 and urlEncode usernames and passwords
        password=$(echo -n "${password}" | openssl enc -base64 | sed -e 's_=_%3D_g;s_+_%2B_g;s_/_%2F_g')
        username=$(echo -n "${username}" | openssl enc -base64 | sed -e 's_=_%3D_g;s_+_%2B_g;s_/_%2F_g')
-       RESULT=$(/usr/local/sbin/fcgicli -f /etc/inc/openvpn.auth-user.php -d "username=$username&password=$password&cn=$common_name&strictcn=$3&authcfg=$2&modeid=$4&nas_port=$5")
+       RESULT=$(/usr/local/bin/php-cgi -f /etc/inc/openvpn.auth-user.php "username=$username&password=$password&cn=$common_name&strictcn=$3&authcfg=$2&modeid=$4&nas_port=$5")
 fi

 if [ "${RESULT}" = "OK" ]; then

--
Christian

shpokas

I can only add that I was bitten by this issue, too.
What is interesting is that certificate verification failed for some users, but not for all.
CA CN contains spaces and user certificates contain spaces. And yet some users can use OpenVPN and some cannot. Disabling Certificate Depth verification fixed that.
I hope this gets classified as a bug and will be fixed in the future.

gribnut

@shpokas
The initial assumption of having space in the cert Subject was a red herring. Problem is actually due to length of string passed to fcgicli for key value pairs. If the cert Subject was a long value, there was a good chance the command fcgicli would fail. As noted above, it is a known issue. See #4521 for more info and patch. The patch replaces fcgicli with php-cgi in script called by OpenVPN for cert depth check. php-cgi does not have issue with the longer string argument that includes cert Subject name. You can apply the patch as a temp fix until it is applied to future version of pfsense.

lan2021

I can confirm this.

It is always the same thing: I googled and found do nothing
appropriate.

Than i dived into the things and figured out the length problem myself. Definitely, length is the problem. Reproducable on the commandline.

When googling again with fcgicli in the search, i found this thread.

Manually applied the patch and everthing works fine.