OpenVPN 2.5.0 Certificate Verification Fails
-
@gribnut This fix seems to have solved a similar problem I have been having with TLS/SSL OpenVPN connections.
Converted a stable 2.4.5p1 to 2.4.5p1 pair to 2.5.0 - 2.5.0 and lost the existing S2S OpenVPN link. I rebuilt the CA/Certs for Server and Client under 2.5.0 and got the link back.
Later I realized the Server had a secondary RoadWarrior setup that was also now failing previously stable clients. Logs pointed to failure to find CA.
Tried your fix and Voila it all came back to life.Looks like a definite bug here, what's interesting is my CA common name has no spaces as far as I can tell.
It's "811pow-ovpn-rdwar-ca" although I'm not sure if there's a leading space buried in there. I've used an OpenSSL command to dump the subject, but it's not clear if the output adds a space to the entries as they are printed. -
@divsys
I was a bit off when I though was due to space as removing space appeared to fix problem. Looks like the root problem is length of string (all key value pairs) for -d arg to fcgicli. If combination of server hostname (as used by client), cert subject and serial number is too long, fcgicli bombs and returns "Something wrong happened while reading request". Guessing the length of various values in your environment also exceed what works for fcgicli.
Looks like bug #4521 was reported some time ago and no indication of when will be fixed. I added comment to confirm it is still a problem in hopes it gets fixed in near future. Fortunately, I don't have a dependency upon limiting cert depth. -
@fr3ddie
I forgot to mention that anytime you save openvpn config, it will write over any changes made to /usr/local/sbin/ovpn_auth_verify. I've not looked to see which file updates /usr/local/sbin/ovpn_auth_verify, so I just disabled cert depth check altogether until long arg to fcgicli is resolved.Just guessing, but suspect that reason one of your clients might work whereas as others don't is due to length of cert subject for client. The tls_verify will iterate through entire chain (depending up configured depth to search). The length of string for subject (the entire subject not just CN) could end up causing arg passed to fcgiclie to exceed value that works.
-
@gribnut Thanks for your help in tracking this down. Makes me much more hopeful about converting up to 2.5.0.
-
So, does disabling that depth check allow the VPN to work? Currently, I can get it to work on the same LAN, but not from outside the firewall.
-
@jknott
Disabling depth check is only a workaround if the tls-verify script is failing. Won't fix other problems. Recommend checking your logs to see cause of failure. -
@gribnut Hmmmmm, I think there's a bit more to this one than meets the eye.
On ahunch, I backed out the code change to ovpn_auth_verify back to factory.Roadwarrior links still good
Restarted the RoadWarror server process
Roadwarrior links still good
Rebooted the remote pfSense box
Roadwarrior links still good
So now I 'm at the point that I can't make it fail anymore after backing all my changes out <sigh>
One thing to note in all this - none of my OpenVPN Servers do more than a depth check of 1.
-
@gribnut I just upgraded to 2.5 today and noticed right away that I couldn't connect to OpenVPN.
Once I set Certificate Depth to Do not check it worked again. -
@gribnut
thank you for your information.
But there is still something that I can't understand: how a length issue on the certificate subject can be fixed by just enclosing the subject in apexes?What I believe is that, probably, there is more than one issue here in fcgicli: an issue linked to the subject length and maybe another issue linked to "special" characters (spaces? slashes?) included in the certificate subject that is fixed by simply enclosing the subject between apexes.
-
@nycspud Do you know what error messages OpenVPN was logging while it was failing?
I'd be curious what happens if you set Certificate Depth back to Level one, does OpenVPN start failing again or is it "magically" OK?As noted below, there seems to be an internal interaction were not catching that trips the Certficate checks and causes OpenVPN to fail. My experiences so far have been odd in that I've seen hard fails right after an upgrade but once I managed to massage the certificate depth or the ovpn_auth_verify "fix", I could no longer replicate the issue by backing out the fix.
-
@divsys Yes, setting Certificate Depth = 1 or higher causes it to fail.
With Certificate Depth = 1 or higher
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 WARNING: Failed running command (--tls-verify script): external program exited with error status: 1
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 VERIFY SCRIPT ERROR: depth=1, C=US, ST=XX, L=XX, O=XX, emailAddress=XX@XX.XX, CN=ovpn
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 OpenSSL: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 TLS_ERROR: BIO read tls_read_plaintext error
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 TLS Error: TLS object -> incoming plaintext read error
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 TLS Error: TLS handshake failed
Feb 23 07:10:48 openvpn 21518 xx.xx.xx.xx:53817 SIGUSR1[soft,tls-error] received, client-instance restartingOn the client side when Certificate Depth = 1 or higher
2021-02-23 07:11:48 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
2021-02-23 07:11:48 TLS Error: TLS handshake failed
2021-02-23 07:11:48 SIGUSR1[soft,tls-error] received, process restarting
2021-02-23 07:11:48 MANAGEMENT: >STATE:1614093108,RECONNECTING,tls-error,,,,,
2021-02-23 07:11:48 Restart pause, 5 second(s)I only have a self signed root CA, and no intermediate CA.
-
If it is from fcgicli, you might try the original change for #9460 (before it was fixed properly last time) by using the System Patches package and then create an entry for
ce76f299853dccb036de229f08a30013593c98fdto apply the change. It will use php-cgi instead of fcgicli. -
@fr3ddie
I agree on skepticism that surrounding one of the key values with quotes in key/value pairs submitted to fcgicli actually fixes the problem. It's possible that while it returns OK using workaround I listed it could cause a different behavior that may or may not work for accurately detecting cert depth. Since saving openvpn config changes overwrites ovpn_auth_verify anyway, I just went with workaround to disable cert depth check until (or if) issue is resolved with fcgicli and lengthy args to -d from bug reported in redmine. -
@gribnut I tried your fix and it worked for me. (I'm on pfSense 2.5.0-RELEASE)
Thank you -
I tried the patch on pfSense 2.5.0-RELEASE also. No change. PIA VPN connection will not connect.
Feb 24 13:45:36 openvpn 31850 Restart pause, 5 second(s)
Feb 24 13:45:36 openvpn 31850 SIGUSR1[soft,tls-error] received, process restarting
Feb 24 13:45:36 openvpn 31850 TCP/UDP: Closing socket
Feb 24 13:45:36 openvpn 31850 TLS Error: TLS handshake failed
Feb 24 13:45:36 openvpn 31850 TLS Error: TLS object -> incoming plaintext read error
Feb 24 13:45:36 openvpn 31850 TLS_ERROR: BIO read tls_read_plaintext error
Feb 24 13:45:36 openvpn 31850 OpenSSL: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
Feb 24 13:45:36 openvpn 31850 VERIFY ERROR: depth=1, error=self signed certificate in certificate chain: C=US, ST=CA, L=LosAngeles, O=Private Internet Access, OU=Private Internet Access, CN=Private Internet Access, name=Private Internet Access, emailAddress=secure@privateinternetaccess.com, serial=11996199170155461251PIA suggested rolling back OpenVPN from 2.5 to 2.4.
Any suggestions?
Thanks. -
@wtw said in OpenVPN 2.5.0 Certificate Verification Fails:
I tried the patch on pfSense 2.5.0-RELEASE also. No change. PIA VPN connection will not connect.
The problem in this thread is for OpenVPN servers on pfSense, not clients. Your problem is unlikely to be related. Start a new thread for your issue if you haven't already.
-
As @jimp noted, the failure of cert depth check with long cert subject name is due fcgicli inability to parse long key/value strings. I have verified that replacing fcgicli with php-cgi as noted in patch for #4521 ( and #9460 ) resolves. When using php-cgi, cert depth check works as expected with my self generated cert, intermediate and root CA.
-
@gribnut said in OpenVPN 2.5.0 Certificate Verification Fails:
As @jimp noted, the failure of cert depth check with long cert subject name is due fcgicli inability to parse long key/value strings. I have verified that replacing fcgicli with php-cgi as noted in patch for #4521 ( and #9460 ) resolves. When using php-cgi, cert depth check works as expected with my self generated cert, intermediate and root CA.
I tried the suggestion you provded "The fifth arg ($5)" above. No change.
I tried the patch #9460; ce76f299853dccb036de229f08a30013593c98fd as suggested. No change.
I started a new topic: pfSense 2.5.0-RELEASE OpenVPN Cert bug
What worked was creating a new self-signed cert using the same data. The difference in the backup XML is provided there.
Is patch #4521 different than #9460?
If the patch in only for an OpenVPN server, then this would be expected. -
@jimp said in OpenVPN 2.5.0 Certificate Verification Fails:
If it is from fcgicli, you might try the original change for #9460 (before it was fixed properly last time) by using the System Patches package and then create an entry for
ce76f299853dccb036de229f08a30013593c98fdto apply the change. It will use php-cgi instead of fcgicli.I had the same issue immediately after upgrading from 2.4.5p1 to 2.5.0, and this fix did not work for me either. However, this was because it fixes the wrong thing.
The commit above modifies
/usr/local/sbin/ovpn_auth_verify_async, while OpenVPN actually calls/usr/local/sbin/ovpn_auth_verify. I changed this script in the same way the commit does:- change
/usr/local/sbin/fcgiclito/usr/local/bin/php-cgi - remove the
-din front of the query string
This worked; OpenVPN now again allows incoming connections.
Here is the patch (apply with strip 0 in /usr/local/sbin) for reference:
--- ovpn_auth_verify.orig 2021-03-07 18:20:59.312509000 +0000 +++ ovpn_auth_verify 2021-03-07 18:21:42.060270000 +0000 @@ -24,14 +24,14 @@ for check_depth in $(/usr/bin/seq ${3} -1 0) do eval serial="\$tls_serial_${check_depth}" - RESULT=$(/usr/local/sbin/fcgicli -f /etc/inc/openvpn.tls-verify.php -d "servercn=$2&depth=$3&certdepth=$4&certsubject=$5&serial=$serial&config=$config") + RESULT=$(/usr/local/bin/php-cgi -f /etc/inc/openvpn.tls-verify.php "servercn=$2&depth=$3&certdepth=$4&certsubject=$5&serial=$serial&config=$config") done else # Single quoting $password breaks getting the value from the variable. # Base64 and urlEncode usernames and passwords password=$(echo -n "${password}" | openssl enc -base64 | sed -e 's_=_%3D_g;s_+_%2B_g;s_/_%2F_g') username=$(echo -n "${username}" | openssl enc -base64 | sed -e 's_=_%3D_g;s_+_%2B_g;s_/_%2F_g') - RESULT=$(/usr/local/sbin/fcgicli -f /etc/inc/openvpn.auth-user.php -d "username=$username&password=$password&cn=$common_name&strictcn=$3&authcfg=$2&modeid=$4&nas_port=$5") + RESULT=$(/usr/local/bin/php-cgi -f /etc/inc/openvpn.auth-user.php "username=$username&password=$password&cn=$common_name&strictcn=$3&authcfg=$2&modeid=$4&nas_port=$5") fi if [ "${RESULT}" = "OK" ]; then--
Christian - change
-
I can only add that I was bitten by this issue, too.
What is interesting is that certificate verification failed for some users, but not for all.
CA CN contains spaces and user certificates contain spaces. And yet some users can use OpenVPN and some cannot. Disabling Certificate Depth verification fixed that.
I hope this gets classified as a bug and will be fixed in the future.
