OpenVPN server certificate verify failed on pfSense 2.6.0

blasterspike

Hi All!
Yesterday I upgraded pfSense Community Edition from 2.5.2 to 2.6.0 and the OpenVPN Server has stopped establishing connections.
I have only 1 user and the authentication is "Remote Access (SSL/TLS + User Auth)".

This is the error I get on the server:

Apr 22 13:55:24 	openvpn 	40410 	192.168.0.59:53529 SIGUSR1[soft,tls-error] received, client-instance restarting
Apr 22 13:55:24 	openvpn 	40410 	192.168.0.59:53529 TLS Error: TLS handshake failed
Apr 22 13:55:24 	openvpn 	40410 	192.168.0.59:53529 TLS Error: TLS object -> incoming plaintext read error
Apr 22 13:55:24 	openvpn 	40410 	192.168.0.59:53529 TLS_ERROR: BIO read tls_read_plaintext error
Apr 22 13:55:24 	openvpn 	40410 	192.168.0.59:53529 OpenSSL: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
Apr 22 13:55:24 	openvpn 	40410 	192.168.0.59:53529 VERIFY SCRIPT ERROR: depth=1, CN=pfSense-CA, C=GB, ST=UK, L=London, O=Test Ltd.
Apr 22 13:55:24 	openvpn 	40410 	192.168.0.59:53529 WARNING: Failed running command (--tls-verify script): external program exited with error status: 1
Apr 22 13:55:23 	openvpn 	40410 	192.168.0.59:53529 VERIFY WARNING: depth=1, unable to get certificate CRL: CN=pfSense-CA, C=GB, ST=UK, L=London, O=Test Ltd.
Apr 22 13:55:23 	openvpn 	40410 	192.168.0.59:53529 VERIFY WARNING: depth=0, unable to get certificate CRL: CN=spike, C=GB, ST=UK, L=London, O=Test Ltd.
Apr 22 13:55:23 	openvpn 	40410 	192.168.0.59:53529 TLS: Initial packet from [AF_INET]192.168.0.59:53529, sid=65d042ea 93aca844

This is what I get on the client, Tunnelblick on macOS

2022-04-22 14:55:23.728911 TLS: Initial packet from [AF_INET]84.9.xxx.xxx:1194, sid=dd8e10ce 12f07843
2022-04-22 14:55:23.799390 VERIFY OK: depth=1, CN=pfSense-CA, C=GB, ST=UK, L=London, O=Test Ltd.
2022-04-22 14:55:23.800444 VERIFY KU OK
2022-04-22 14:55:23.800484 Validating certificate extended key usage
2022-04-22 14:55:23.800499 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
2022-04-22 14:55:23.800509 VERIFY EKU OK
2022-04-22 14:55:23.800518 VERIFY X509NAME OK: CN=OpenVPNServer, C=GB, ST=UK, L=London, O=Test Ltd.
2022-04-22 14:55:23.800526 VERIFY OK: depth=0, CN=OpenVPNServer, C=GB, ST=UK, L=London, O=Test Ltd.

So I see that the Verify is OK the client side but something fails on the server.
This is with Verbosity = 3. I increased it to 11 but I don't get anything meaningful (at least for me).

I tried with different clients:
Tunnelblick 3.8.7a (build 5770) on macOS
OpenVPN Connect 3.3.5 (410) on macOS
OpenVPN 3.2.3 (3760) on iOS
but the error is always the same.

In 2019 I configured OpenVPN server following this guide
https://www.sparklabs.com/support/kb/article/setting-up-an-openvpn-server-with-pfsense-and-viscosity/
and never had a problem.
The CA and user certificate are managed in pfSense.

Worried that something in my configuration wasn't compatible anymore with the most recent version of OpenVPN, I decided to wipe out the OpenVPN Server configuration, the CA and the user certificate, so that I was starting almost from scratch.
I reconfigured the server following this guide
https://www.comparitech.com/blog/vpn-privacy/openvpn-server-pfsense/
but still the same problem.

I wiped out everything again and I used the Wizard but again same issue.
Unfortunately I don't understand what is wrong.

This is the configuration of the server

OpenVPN server configuration

This is what the "Client Export" is generating

dev tun
persist-tun
persist-key
data-ciphers AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305:AES-256-CBC
data-ciphers-fallback AES-256-CBC
auth SHA512
tls-client
client
resolv-retry infinite
remote 84.9.xxx.xxx 1194 udp4
nobind
verify-x509-name "OpenVPNServer" name
auth-user-pass
remote-cert-tls server
explicit-exit-notify

<ca>
-----BEGIN CERTIFICATE-----
***
-----END CERTIFICATE-----
</ca>
<cert>
-----BEGIN CERTIFICATE-----
***
-----END CERTIFICATE-----
</cert>
<key>
-----BEGIN PRIVATE KEY-----
***
-----END PRIVATE KEY-----
</key>
key-direction 1
<tls-auth>
#
# 2048 bit OpenVPN static key
#
-----BEGIN OpenVPN Static key V1-----
***
-----END OpenVPN Static key V1-----
</tls-auth>

Any help is appreciated because I don't know what else I need to check.

Thanks!

blasterspike

Actually by checking carefully the OpenVPN server log at Verbosity = 11 I have spotted the following

Apr 22 13:57:55 	openvpn 	17319 	192.168.0.59:61823 TLS Error: TLS handshake failed
Apr 22 13:57:55 	openvpn 	17319 	192.168.0.59:61823 TLS Error: TLS object -> incoming plaintext read error
Apr 22 13:57:55 	openvpn 	17319 	192.168.0.59:61823 TLS_ERROR: BIO read tls_read_plaintext error
Apr 22 13:57:55 	openvpn 	17319 	192.168.0.59:61823 OpenSSL: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
Apr 22 13:57:55 	openvpn 	17319 	192.168.0.59:61823 SSL alert (write): fatal: unknown CA
Apr 22 13:57:55 	openvpn 	17319 	192.168.0.59:61823 VERIFY SCRIPT ERROR: depth=1, CN=pfSense-CA, C=GB, ST=UK, L=London, O=Test Ltd.
Apr 22 13:57:55 	openvpn 	17319 	192.168.0.59:61823 WARNING: Failed running command (--tls-verify script): external program exited with error status: 1
Apr 22 13:57:55 	openvpn 	17319 	192.168.0.59:61823 TLS: executing verify command: /usr/local/sbin/ovpn_auth_verify tls OpenVPNServer 1 1 CN=pfSense-CA, C=GB, ST=UK, L=London, O=Test Ltd.
Apr 22 13:57:55 	openvpn 	17319 	192.168.0.59:61823 VERIFY WARNING: depth=1, unable to get certificate CRL: CN=pfSense-CA, C=GB, ST=UK, L=London, O=Test Ltd.
Apr 22 13:57:55 	openvpn 	17319 	192.168.0.59:61823 VERIFY WARNING: depth=0, unable to get certificate CRL: CN=spike, C=GB, ST=UK, L=London, O=Test Ltd.
Apr 22 13:57:55 	openvpn 	17319 	192.168.0.59:61823 SSL state (accept): TLSv1.3 early data
Apr 22 13:57:55 	openvpn 	17319 	192.168.0.59:61823 Incoming Ciphertext -> TLS

If I run manually

usr/local/sbin/ovpn_auth_verify tls OpenVPNServer 1 1 CN=pfSense-CA, C=GB, ST=UK, L=London, O=Test Ltd.

on the shell, the exit code is 1.
If I run the same with sh -x I get

sh -x /usr/local/sbin/ovpn_auth_verify tls OpenVPN+server 1 1 CN=pfSense-ca, ST=UK, L=London, O=Test Ltd.
+ [ tls '=' tls ]
+ /usr/bin/seq 1 -1 0
+ eval 'serial=$tls_serial_1'
+ serial=''
+ [ -n '' ]
+ eval 'serial=$tls_serial_0'
+ serial=''
+ [ -n '' ]
+ [ '' '=' OK ]
+ exit 1

I'm trying to understand how that

eval serial="\$tls_serial_${check_depth}"

is evaluated.

I have followed this other thread and changed the code to \"$5\" but I still get an exit 1.
I have recreated the certificates to remove the optional
Country Code
State or Province
City
Organization
Organizational Unit
so that they are only CN=pfSense-CA and CN=spike but still the exit code is 1.

If I set "Certificate Depth = Do Not Check" as they specify in that thread, Tunnelblick returns with

Authentication failed

The username and password were not accepted by the remote VPN server.

and definitely they are both correct.
Even with OpenVPN Connect same error.

blasterspike

Still following the thread I mentioned above, I saw that the eval previously was right before RESULT=.
I have tried to comment the if statement block and move eval, so this way

                # eval serial="\$tls_serial_${check_depth}"
                # if [ -n "$serial" ]; then
                        eval serial="\$tls_serial_${check_depth}"
                        RESULT=$(/usr/local/bin/php-cgi -q /etc/inc/openvpn.tls-verify.php "servercn=$2&depth=$3&certdepth=$4&certsubject=$5&serial=$serial&co
nfig=$config")
                        if [ "${RESULT}" = "FAILED" ]; then
                                exit 1
                        fi
                # fi

and I don't get anymore the error on the certificate!
I don't know if I need to open an issue about this.

However, now I get the error about the user authentication

SENT CONTROL [spike]: 'AUTH_FAILED' (status=1)

like I was getting when I set "Certificate Depth = Do Not Check".
I looks like I'm not the only one having this issue.