Expired Let's Encrypt CA when using it as a client

securvark

So my certs look good in the gui, ive been reading the posts here and my cert on pfsense is renewed through lets encrypt using the acme script.

However, when I try to git push from the cli, I get the error that the certificate of our gitlab is expired. This one is also using a lets encrypt certificate.

When I do the following from my local laptop:

# openssl s_client -connect  gitlab.our.domain.org:443 -servername gitlab.our.domain.org

I see proper cert for lets encrypt ca and chain.

When I do the same from pfsense its invalid.

# openssl version -d
/etc/ssl

The files there are not updated, how do I update those?
pkg update doesn't seem to do that.

VioletDragon

@securvark Are you using a older Mac ? if so the old Cert Chain needs to be removed from Keychain Access. If not test it from CLI on the Firewall itself.

securvark

@violetdragon

No no, its from pfsense cli itself.

Edit:
in other words, I think the /etc/ssl files are outdated on pfsense, that is what openssl command is using (and what git uses to push).

Gertjan

@securvark said in HEADS UP: DST Root CA X3 Expiration (September 2021):

in other words, I think the /etc/ssl files are outdated ...

Yep, the "DST Root" it's there :
Symlinked from :/usr/local/share/certs/ca-root-nns.cert :

            44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Sep 30 21:12:19 2000 GMT
            Not After : Sep 30 14:01:15 2021 GMT
        Subject: O = Digital Signature Trust Co., CN = DST Root CA X3
        Subject Public Key Info:

But, as it's expired, it isn't used any more.
And you shouldn't use it neither ;)

If your "gitlab.our.domain.org" makes use of "DST Root CA X3" then openssl on pfSense will yell.
If it uses a non expired one, all will be well.

My turn to test :

openssl s_client -connect test-domaine.fr:443 -servername test-domaine.fr
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = test-domaine.fr
verify return:1
---
Certificate chain
 0 s:CN = test-domaine.fr
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
......
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---
read:errno=0

My Letencrypt certs (test-domaine.fr) use the newer "ISRG Root X1" so all is ok.

Nice detail : The cert test-domaine.fr was created 23 August, so "DST Root" was still valid back then.
But, a newer one is also included in the cert chain "just to be sure" so my certs stays valid even after 30/09

securvark

@gertjan

Thanks!

Maybe I misunderstand you, or I haven't explained myself very well.

Either way, our gitlab cert is new and uses the new lets encrypt ca.

Our pfsense on the other hand does not have this new lets encrypt ca, so it cannot verify the gitlab certificate, hence openssl and git push will throw an error.

I need to update /etc/ssl on pfsense, so that it has the new lets encrypt ca to validate certs that use that new chain.

johnpoz

@securvark said in HEADS UP: DST Root CA X3 Expiration (September 2021):

Our pfsense on the other hand does not have this new lets encrypt ca

And what version of pfsense are you running? As @Gertjan shows the current CA is there. The old CA still being there is not a problem.

securvark

@johnpoz

Its 2.4.5-RELEASE-p1.

When i run that command it gives this:

[2.4.5-RELEASE][root@hostname.org]/etc/ssl: openssl s_client -connect  gitlab.our.domain.org:443 -servername gitlab.our.domain.org
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
---
Certificate chain
 0 s:/CN=*.our.domain.org
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
.....
    Start Time: 1633695560
    Timeout   : 300 (sec)
    Verify return code: 10 (certificate has expired)

jimp

This has nothing to do with the original thread where it was posted, so I forked it and moved it.

You need to upgrade or you need to manually replace the root certificate list on your firewall because it's out of date. But the easiest way to do that is to upgrade to a supported version.

securvark

@jimp Alright, will do (upgrade is planned) but for the time being I will update them manually from another already updated pfsense.

Thanks @jimp!

johnpoz

@jimp said in Expired Let's Encrypt CA when using it as a client:

But the easiest way to do that is to upgrade to a supported version.

Just playing devils advocate here.. And I am the first one to advocate being current.. But when you say supported version.. From here - 2.4.5p1 is still supported ;)

https://docs.netgate.com/pfsense/en/latest/releases/versions.html

Such a case I think is more of a one-off sort of thing, pfsense as a client to something using acme prob not a lot of use cases?

Is there a easy way for such users to update the certs on pfsense without having to manually do it? Some sort of pkg upgrade or something that doesn't actually update the version of pfsense, but would update that cert.pem file to current?

I am all for updating to current version, but in this time of covid and company locations being closed, etc. It might not always be possible or prudent to update at this time. I have a couple of boxes that are behind for sure..

jimp

@johnpoz said in Expired Let's Encrypt CA when using it as a client:

Such a case I think is more of a one-off sort of thing, pfsense as a client to something using acme prob not a lot of use cases?

Hard to say. There are a few different ways something on the firewall may reach out to a system which could be using an ACME chain, like URL table aliases, pfBlockerNG lists, etc.

Is there a easy way for such users to update the certs on pfsense without having to manually do it? Some sort of pkg upgrade or something that doesn't actually update the version of pfsense, but would update that cert.pem file to current?

Not unless we were to update the ca_root_nss package which we wouldn't do for older versions.

I mentioned it in another thread recently but you can copy the file in that package, /usr/local/share/certs/ca-root-nss.crt, from a new system to the old one. Or hand edit that file and replace the expired CA with the new one.

I am all for updating to current version, but in this time of covid and company locations being closed, etc. It might not always be possible or prudent to update at this time. I have a couple of boxes that are behind for sure..

I get that to some extent, but at some point the benefits outweigh the risks.

johnpoz

@jimp said in Expired Let's Encrypt CA when using it as a client:

I get that to some extent, but at some point the benefits outweigh the risks.

very true..

firewall may reach out to a system which could be using an ACME chain, like URL table aliases, pfBlockerNG

Quite true! These are very good examples of where users might run into an issue that I didn't think off the top of my head..

I wonder if some sort of blog post, or sticky thread with specific instructions on how to update might be in order? With maybe link to download current file, or instructions on how to edit the specific CA in the pem might very helpful.

Not everyone might have access or ability to fire up a current version of pfsense where they could obtain the current version of the file.

jimp

I don't see us doing a blog post or docs entry for it since it's really a non-issue for most users, and the only official answer we'll give is "upgrade".

If someone wants to keep using older versions they have to take on the responsibility of handling the problems that come with that choice.

johnpoz

@jimp all true, all true!

jimmychoosshoes

For me, I wasnt using ACME but my /etc/ssl/cert.pem or /usr/local/share/certs/ca-root-nss.crt was out of date due to me using 2.4.5

If anyone else needs a pointer on updating 2.4.5 this is what I did: I updated the ca-root-nss.crt manually from nss package but also needed to remove the old DST Root CA X3 from ca-root-nss.crt after this openssl retrieved the correct cert CA from the external website.

bingo600

I couldn't update bogons on my 2.4.5-p1 , due to fetch using the expired certificate.

I spend quite some time to solve it here , due to my FreeBSD inexperience.
https://forum.netgate.com/topic/167276/solved-can-t-update-bogons-on-a-2-4-5-p1-cert-expired

Success in the end.

/Bingo