pfSense 2.5.0-RELEASE OpenVPN Cert bug

wtw

When upgrading from pfSense 2.4.5 to 2.5.0 the OpenVPN client to PIA stopped working.

Feb 24 13:45:36 openvpn 31850 Restart pause, 5 second(s)
Feb 24 13:45:36 openvpn 31850 SIGUSR1[soft,tls-error] received, process restarting
Feb 24 13:45:36 openvpn 31850 TCP/UDP: Closing socket
Feb 24 13:45:36 openvpn 31850 TLS Error: TLS handshake failed
Feb 24 13:45:36 openvpn 31850 TLS Error: TLS object -> incoming plaintext read error
Feb 24 13:45:36 openvpn 31850 TLS_ERROR: BIO read tls_read_plaintext error
Feb 24 13:45:36 openvpn 31850 OpenSSL: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
Feb 24 13:45:36 openvpn 31850 VERIFY ERROR: depth=1, error=self signed certificate in certificate chain: C=US, ST=CA, L=LosAngeles, O=Private Internet Access, OU=Private Internet Access, CN=Private Internet Access, name=Private Internet Access, emailAddress=secure@privateinternetaccess.com, serial=11996199170155461251

The CA certificate is an import with Certificate data only.
Creating a "new" CA using the same data and referencing that "new" CA from OpenVPN fixed the problem.
There appears to be a bug handling some certificates in pfSense 2.5.0. This should be easily repeatable. It was very exasperating.

jimp

I haven't seen anything like that happen when I've been working in certs between 2.4.x and 2.5.x.

What is the difference in the configuration between the old CA entry and new CA entry?

You should be able to use the configuration history tab under Diagnostics > Backup/Restore to see what changed between the config before you removed the entry and created it again.

wtw

Unfortunately, the config history does not go back in time far enough. Too many changes.
I compared the XML backup from before the upgrade with the current one (post upgrade). I did not delete the old CA (haven't figured out how to do that yet).
The same CA matches exactly; as expected.
The new CA body (cert) matches the old exactly also; as expected.
However, I did find a difference; the cause of which is unknown to me (except the serial number difference).

Old CA:
<refid>5f11c78c9b37f</refid>
<descr><![CDATA[PIA.2048]]></descr>
<crt>...</crt>
<serial>0</serial>
<caref>5f1305a403377</caref>

New CA:
<refid>6036d1a985420</refid>
<descr><![CDATA[PIA.2048.1]]></descr>
<trust>disabled</trust>
<randomserial>disabled</randomserial>
<crt>...</crt>
<serial>1</serial>

The old CA is listed as issued by CA: PIA.4096. The new CA is listed as self-signed.
pfSense 2.4.5 had no issue with the old CA for OpenVPN.
Maybe the old CA should not have worked like it does not now. I do not know.
CA PIA.4096 (self-signed) has the exact same info: (different cert data)
name=Private Internet Access, emailAddress=secure@privateinternetaccess.com, ST=CA, OU=Private Internet Access, O=Private Internet Access, L=LosAngeles, CN=Private Internet Access, C=US
And the same valid dates.
So, it seems like there is either something wrong, or there was something wrong. The behaviors are inconsistent.
The fact that <trust> is disabled obviously solved my problem.
The questions is: Why did it work before and not now?
So, the old CA had a chain that referenced back to a self-signed CA; the new one does not.
Unfortunately, I do not recall exactly how I created the old CAs (PIA.4096 & PIA.2048) except that the cert data was imported as text.

wtw

@wtw
I found another issue: In the config history, I looked at a difference for: (2 instances)
(system): stunnel configuration changed
The only difference is:

```
  <time>1614213136</time>
```

```
  <time>1614213078</time>
```

This pollutes the history with timestamp differences, which are not really meaningful and renders the history useless because these changes are occurring often, in my case. The STunnel is used for E-mail, so when E-mail updates, it appears that the tunnel is being re-established (or something) and a config change is recorded. I can only see about a couple dozen entries and do not see a more history available option.

jimp

@wtw said in pfSense 2.5.0-RELEASE OpenVPN Cert bug:

So, it seems like there is either something wrong, or there was something wrong. The behaviors are inconsistent.
The fact that <trust> is disabled obviously solved my problem.

Not necessarily, when that's set on CAs in 2.5.0 they are added to the global trust store so anything on the system can trust them. That's a new feature for 2.5.0.

The questions is: Why did it work before and not now?

I don't think we have enough information to say for sure yet.

So, the old CA had a chain that referenced back to a self-signed CA; the new one does not.

Did the old CA entry have both the root CA an an intermediate in a single CA list entry? That was a common practice in the past but it hasn't been necessary for a while now. I could maybe see that kind of entry breaking on a current system.

wtw

@jimp said in pfSense 2.5.0-RELEASE OpenVPN Cert bug:

So, the old CA had a chain that referenced back to a self-signed CA; the new one does not.

Did the old CA entry have both the root CA an an intermediate in a single CA list entry? That was a common practice in the past but it hasn't been necessary for a while now. I could maybe see that kind of entry breaking on a current system.

I do not know. How would one determine that?
Remember, the cert data is identical for these two PIA.2048 CAs.
I don't know what is in the PIA.4096 cert data.
I did try editing all of them (without saving) to see how they were entered. Nothing special. Copy cert data into the top box (import existing CA) , provide a name and done.
I do not know what created the reference, yet it is there.
None of these CAs show any intermediate info in the edit window.

jimp

If it had both you'd see two PEM blocks in the cert data:

BEGIN
...
END
BEGIN
...
END

Normally a root CA would be self-signed and intermediates will show as being signed by another CA.

wtw

That is simple. No, they are all single certs. Only one begin and end. So the question of why it worked before and no longer does remains.

It is weird. I changed the cert for the VPN to the PIA.4096 CA and it errors the same, with the same CA serial number as the PIA.2048 (original) CA. I checked the PIA.4096 serial number using openssl and it is definitely different. Neither CA worked. I had to switch back to the "new" one (PIA.2048.1).
So, there must be coding error somewhere.

I have seen many complaints about this kind of behavior online (verify error). There were diferent strategies some had success with. There is one where someone stated that they had upgraded OpenVPN to 2.5 while using pfSense 2.4.5 and it worked fine. When they upgraded to pfSense 2.5.0 they had a CA error, like this. They recreated the CA and it worked. Unfortunately, I cannot find the link now.

wtw

@wtw
Had some time to further investigate this issue.
The <trust> tag with disabled just indicates that this CA was not added to the local CA trust store.
This should not have changed anything that would allow the new CA to work.
Which only leaves the <caref> tag (CA Reference), which creates a certificate chain.
This was not set for the new certificate, so, no chain, no chain error.
I do not know how I created this CA reference.

The error, assuming the error message is correct, states that a self signed certificate in the certificate chain is not allowed. The fact that this worked for pfSense 2.4.5 seems to indicate that this was acceptable for the previous version, but not in version 2.5.0. The error appears to indicate it came from OpenVPN.
Checked the pfSense source and OpenVPN source codes. The error does not originate in pfSense.
It does originate in the OpenVPN code. It appears to be ssl_verify_openssl.c in verify_callback() at line 93, where preverify_ok controls.
This gets called in ssl_openssl.c by tls_ctx_set_options() at line 347; which sets SSL_VERIFY_PEER (Require peer certificate verification) and calls OpenSSL library's SSL_CTX_set_verify()
The certificate chain is checked starting with the deepest nesting level (the root CA certificate) and worked upward to the peer's certificate. At each level signatures and issuer attributes are checked. Whenever a verification error is found, the error number is stored in x509_ctx and verify_callback is called with preverify_ok=0.

So, it appears to be an issue with how the CAs are combined and passed through OpenVPN to OpenSSL and possibly the CA definition, which for some reason worked in pfSense 2.4.5.

Any suggestions on how to view the call that invokes OpenVPN to check the command and argument syntax between the 2 CAs?

Thanks.

wtw

@wtw
Any suggestions on how to view the calls?
It appears the place to check are the calls in ovpn_auth_verify during the TLS verify (line 27).