No Clients Can Connect To OpenVPN Due to CRL Expiry
-
I've found a bug in pfSense to do with CRLs and OpenVPN that prevents any clients from being able to connect. A temporary fix is to reduce the CRL validity to a shorter period of time.
Description and work around:
No clients were able to connect to connect to OpenVPN server.
Messages like this seen in log: REDACTED:50095 VERIFY ERROR: depth=0, error=CRL has expired: CN=REDACTED, C=GB, ST=REDACTED, L=REDACTED, O=REDACTED, serial=24
I attempted to create a new CRL and switch over to it, but the problem persisted. This was using the default expiry of 9999 days.
When I inspected the CRL file, I was able to see that the Next Update field was a date in the past (year 1950). 9999 days from today is 1st January 2050, so this looks like a roll over bug. The general issue is explained here https://www.redhat.com/sysadmin/certificate-y2k20-bug
I then created another CRL, this time with an expiry of 730 days and switched it over in the web configurator. Clients were able to connect again and the Next Update field was a date in the future (year 2024) as expected.
I have not investigated this further to see where the bad date is generated.
OpenVPN 2.5.4 amd64-portbld-freebsd12.3 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Jan 12 2022
pfSense base system 2.6.0I'll post on the bug tracker later today to give a chance for discussion first.
-
What hardware are you using? If it's a Netgate 3100 or 1000, those are 32-bit ARM, and it could be the 2038 UNIX time rollover. If it's amd64 then it might be the 2050 issue noted on that blog post you linked.
pfSense regenerates internal CRLs whenever they are saved so they really don't need an expiration that long anyhow, so we probably should reduce that drastically.
That said, OpenVPN and/or OpenSSL itself might have a bug there since it should be accounting for that kind of rollover internally and they are not.
-
@jimp It's a KVM guest and the arch is amd64, so unlikely to be a 2038 rollover bug. My money is definitely on a 2050 rollover bug. The RFC for X.509 specifies a different time format for dates from 2050 onwards https://datatracker.ietf.org/doc/html/rfc2459#section-4.1.2.5 - I expect it's a date formatting bug when the CRL is (re)generated rather than an integer overflow.
I agree that there's no need for a 9999 day life on a CRL.
-
I haven't looked too deeply, but I think the bug is probably here, in a library that pfSense uses: https://github.com/ukrbublik/openssl_x509_crl/blob/master/src/X509_CRL.php
The library appears to support both formats described in 4.1.2.5 of the RFC, but when generating the CRL, it seems to hard code the use of the UTCTime (2 digit years) instead of GeneralizedTime (4 digit years). There's no logic in there to validate whether UTCTime can be validly used for a given date.
-
I ran into exactly this problem today. No longer able to make an OpenVPN connection to my pfSense, and the error in the log read:
Aug 17 14:23:43 pfSense openvpn[75777]: XX.XX.XX.XX:47729 VERIFY ERROR: depth=0, error=CRL has expired: C=NL, ST=NH, L=MyTown, O=MyCompany, emailAddress=me@somewhere.nl, CN=Pete, serial=10
Aug 17 14:23:43 pfSense openvpn[75777]: XX.XX.XX.XX:47729 OpenSSL: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failedI created a new CRL, with the default 9999 days expiry, pointed the OpenVPN server to it but still no joy.
I then created another new CRL, but this time with a 999 days expiry. That got me back in business.
Tried a bit more and it turns out that the default, 9999 days, does no longer work. Any other value up to and including 9998 days does the trick.
I think @mmulqueen above hit the nail on the head when they found it has something to do with today being 9999 days away from jan 1st, 2050. I'll try 9998 days tomorrow and see if that gives me the same problem then :-)
I did upgrade from CE 2.5.x to 2.6.0 last week, but IIRC I tested OpenVPN as usual after an update. This morning I made a config change in OpenVPN as I was playing with IPv6 after which OpenVPN stopped working.
BTW, I'm using Intel Core i5-7200 based hardware
-
Referenced by M mmulqueen
-
Referenced by M mmulqueen
-
I created a Redmine entry for this (https://redmine.pfsense.org/issues/13424) and I'll be working on a fix shortly. When I have one, I'll also create an entry in the System Patches package for it.
-
Well, add me to the line. Exactly the same issue occurred today after I updated CRL (in ver. 2.4.5-p1). Sudden loss of VPN connections of all clients and OpenVPN stating CRL expired during initialization on re-connection. Realized whats going on after I saw 'next update=1st of Jan 1950' in CRL properties. Had to create a new list with shorter validity, after this things got back to normal. Dates roooollin` (over).... :)
-
I merged the fix in yesterday evening.
You can install the System Patches package and then create an entry for
a3c1589086ea67d25a28ec14ab95d7fd9ab25fa2to apply the fix.It will be added as a "Recommended Patch" in the System Patches package soon, but in the meantime it is safe to add a manual entry to obtain the fix now.
Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!
Need help fast? Netgate Global Support!
Do not Chat/PM for help!
-
Thank you @jimp for the speedy patch on this. I am in the middle of a a rollout to end users and got hit this morning when I made a configuration change. I applied the patch and re-saved the openvpn configuration and I'm back up now. Thank you again.
-
And thanks from me too! I applied the patch as per your instructions (did not even know about the 'patches' package) and OpenVPN is working fine again. pfSense is a brilliantly supported firewall
-
Just another
Started hearing from WFHers that the VPN was down.
Figured out the CRL was reporting 1950 as next update, and found this post.
The system patch package is worth knowing about :)
OpenVPN restored after installing patch, and reloading the service. Great.
Thanks
-
Got this problem today was pulling hairs why my open vpn server not working found this topic
reduced crl time to 200 days and fixed thank you. -
Would also like to add my gratitude for quick identification and patch solution for this issue.
It bit me yesterday and I not see what was wrong with the path I had trodden many times before in setting up a link. Discovered this solution and the patch fixed it. Thank you.
-
Got bit by this bug when our firewall rebooted due to a power blackout after being up for 187 days. Was so glad to come across this System Patches and be able to apply the needed patch and get back up quickly! pfSense is a great firewall product!
-
just ran into this since i had not use my vpn, thanks to everyone it got me fixed up.
-
@jimp I applied the patch when it was released. I'm reading the release notes for 23.01 and see Issue #13424 has been addressed in the new version. Do I need to do anything like remove the patch before or after I upgrade? Or does everything take care of itself?
-
@jeffreyn said in No Clients Can Connect To OpenVPN Due to CRL Expiry:
@jimp I applied the patch when it was released. I'm reading the release notes for 23.01 and see Issue #13424 has been addressed in the new version. Do I need to do anything like remove the patch before or after I upgrade? Or does everything take care of itself?
You do not need to do anything with the patch after upgrading. You can delete the entry from the system patches package.
