No Clients Can Connect To OpenVPN Due to CRL Expiry

jimp

What hardware are you using? If it's a Netgate 3100 or 1000, those are 32-bit ARM, and it could be the 2038 UNIX time rollover. If it's amd64 then it might be the 2050 issue noted on that blog post you linked.

pfSense regenerates internal CRLs whenever they are saved so they really don't need an expiration that long anyhow, so we probably should reduce that drastically.

That said, OpenVPN and/or OpenSSL itself might have a bug there since it should be accounting for that kind of rollover internally and they are not.

mmulqueen

@jimp It's a KVM guest and the arch is amd64, so unlikely to be a 2038 rollover bug. My money is definitely on a 2050 rollover bug. The RFC for X.509 specifies a different time format for dates from 2050 onwards https://datatracker.ietf.org/doc/html/rfc2459#section-4.1.2.5 - I expect it's a date formatting bug when the CRL is (re)generated rather than an integer overflow.

I agree that there's no need for a 9999 day life on a CRL.

mmulqueen

I haven't looked too deeply, but I think the bug is probably here, in a library that pfSense uses: https://github.com/ukrbublik/openssl_x509_crl/blob/master/src/X509_CRL.php

The library appears to support both formats described in 4.1.2.5 of the RFC, but when generating the CRL, it seems to hard code the use of the UTCTime (2 digit years) instead of GeneralizedTime (4 digit years). There's no logic in there to validate whether UTCTime can be validly used for a given date.

Woody 1

@jimp

I ran into exactly this problem today. No longer able to make an OpenVPN connection to my pfSense, and the error in the log read:

Aug 17 14:23:43 pfSense openvpn[75777]: XX.XX.XX.XX:47729 VERIFY ERROR: depth=0, error=CRL has expired: C=NL, ST=NH, L=MyTown, O=MyCompany, emailAddress=me@somewhere.nl, CN=Pete, serial=10
Aug 17 14:23:43 pfSense openvpn[75777]: XX.XX.XX.XX:47729 OpenSSL: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed

I created a new CRL, with the default 9999 days expiry, pointed the OpenVPN server to it but still no joy.

I then created another new CRL, but this time with a 999 days expiry. That got me back in business.

Tried a bit more and it turns out that the default, 9999 days, does no longer work. Any other value up to and including 9998 days does the trick.

I think @mmulqueen above hit the nail on the head when they found it has something to do with today being 9999 days away from jan 1st, 2050. I'll try 9998 days tomorrow and see if that gives me the same problem then :-)

I did upgrade from CE 2.5.x to 2.6.0 last week, but IIRC I tested OpenVPN as usual after an update. This morning I made a config change in OpenVPN as I was playing with IPv6 after which OpenVPN stopped working.

BTW, I'm using Intel Core i5-7200 based hardware

jimp

I created a Redmine entry for this (https://redmine.pfsense.org/issues/13424) and I'll be working on a fix shortly. When I have one, I'll also create an entry in the System Patches package for it.

e_404

Well, add me to the line. Exactly the same issue occurred today after I updated CRL (in ver. 2.4.5-p1). Sudden loss of VPN connections of all clients and OpenVPN stating CRL expired during initialization on re-connection. Realized whats going on after I saw 'next update=1st of Jan 1950' in CRL properties. Had to create a new list with shorter validity, after this things got back to normal. Dates roooollin` (over).... :)

jimp

I merged the fix in yesterday evening.

You can install the System Patches package and then create an entry for a3c1589086ea67d25a28ec14ab95d7fd9ab25fa2 to apply the fix.

It will be added as a "Recommended Patch" in the System Patches package soon, but in the meantime it is safe to add a manual entry to obtain the fix now.

jeffreyn

Thank you @jimp for the speedy patch on this. I am in the middle of a a rollout to end users and got hit this morning when I made a configuration change. I applied the patch and re-saved the openvpn configuration and I'm back up now. Thank you again.

Woody 1

@jimp

And thanks from me too! I applied the patch as per your instructions (did not even know about the 'patches' package) and OpenVPN is working fine again. pfSense is a brilliantly supported firewall

Stux

Just another

Started hearing from WFHers that the VPN was down.

Figured out the CRL was reporting 1950 as next update, and found this post.

The system patch package is worth knowing about :)

OpenVPN restored after installing patch, and reloading the service. Great.

Thanks

vbredjp

Got this problem today was pulling hairs why my open vpn server not working found this topic
reduced crl time to 200 days and fixed thank you.

why

@vbredjp @jimp @mmulqueen

Would also like to add my gratitude for quick identification and patch solution for this issue.

It bit me yesterday and I not see what was wrong with the path I had trodden many times before in setting up a link. Discovered this solution and the patch fixed it. Thank you.

dougs

Got bit by this bug when our firewall rebooted due to a power blackout after being up for 187 days. Was so glad to come across this System Patches and be able to apply the needed patch and get back up quickly! pfSense is a great firewall product!

flat4

just ran into this since i had not use my vpn, thanks to everyone it got me fixed up.

jeffreyn

@jimp I applied the patch when it was released. I'm reading the release notes for 23.01 and see Issue #13424 has been addressed in the new version. Do I need to do anything like remove the patch before or after I upgrade? Or does everything take care of itself?

jimp

@jeffreyn said in No Clients Can Connect To OpenVPN Due to CRL Expiry:

@jimp I applied the patch when it was released. I'm reading the release notes for 23.01 and see Issue #13424 has been addressed in the new version. Do I need to do anything like remove the patch before or after I upgrade? Or does everything take care of itself?

You do not need to do anything with the patch after upgrading. You can delete the entry from the system patches package.