Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    FreeRadius - Acme and the reappearing DST Root CA X3 ca

    Scheduled Pinned Locked Moved
    pfSense Packages
    3
    8
    1.4k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • V
      VictorRobellini
      last edited by VictorRobellini

      pfSense: 2.6.0-RELEASE (amd64)
      Acme: 0.7.1_1
      FreeRadius3: 0.15.7_33

      Last year (before 2.6 upgrade - format, fresh install and restore) when the DST Root CA expired I made sure to upgrade all of my certs. Almost everything worked great, HTTPS pfSense web interface and haproxy SSL offloading were all showing the updated cert and chain. The one issue I ran into was my WPA2-Enterprise was working for everything except android. According to the freeradius debug logs my cert was invalid since it was expired, no other supplicant had any issues accepting it. This drove me nuts and since it's for my house and kids (VLAN association), I switched to using EAP-PWD as a quick fix. Today I decided to dig in a little more and figure out what's going on and discovered something bizarre. For some reason, Freeradius is building an invalid fullchain cert.

      Here's how I can consistently reproduce:
      Services -> FreeRadius -> EAP -> SSL CA Certificate = Acmecert CN=R3
      SSL Server Cert: pfsense.mydomain.com

      openssl crl2pkcs7 -nocrl -certfile /usr/local/etc/raddb/certs/server_cert.pem | openssl pkcs7 -print_certs -noout
subject=CN = pfsense.mydomain.com
issuer=C = US, O = Let's Encrypt, CN = R3

subject=C = US, O = Let's Encrypt, CN = R3
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3

      I even switched to a test domain that I made today, long after the expired root CA was deleted and replaced and I deleted all my freeradius config.

      • I backed up my pfSense config, edited out anything to do with freeradius in the XML and then reimported so I could start the config from scratch.

      SSL Server Cert: test.mydomain.com

      openssl crl2pkcs7 -nocrl -certfile /usr/local/etc/raddb/certs/server_cert.pem | openssl pkcs7 -print_certs -noout
subject=CN = test.mydomain.com
issuer=C = US, O = Let's Encrypt, CN = R3

subject=C = US, O = Let's Encrypt, CN = R3
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3

      How the hell did DST Root CA X3 get in there?! If I change the SSL Server Cert in FreeRadius to any other cert, somehow the DST Root CA X3 shows up in the chain regardless! It doesn't exist in my cert manager or Acme config.

      If I check my certs at /cf/conf/acme/ (.fullchain .all.pem .crt) there is no reference to the expired CA, everything shows the correct chain.

      openssl crl2pkcs7 -nocrl -certfile /cf/conf/acme/pfsense.mydomain.com.fullchain | openssl pkcs7 -print_certs -noout
subject=CN = pfsense.mydomain.com
issuer=C = US, O = Let's Encrypt, CN = R3

subject=C = US, O = Let's Encrypt, CN = R3
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1

      And my test domain

      openssl crl2pkcs7 -nocrl -certfile /cf/conf/acme/test.mydomain.com.fullchain | openssl pkcs7 -print_certs -noout
subject=CN = test.mydomain.com
issuer=C = US, O = Let's Encrypt, CN = R3

subject=C = US, O = Let's Encrypt, CN = R3
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1

      If I manually replace the freeradius generated fullchain with the Acme fullchain and restart the freeradius service, all of the issues go away.

      cp /cf/conf/acme/pfsense.mydomain.com.fullchain /usr/local/etc/raddb/certs/server_cert.pem

      Does anyone know where this old CA cert is coming from? I would prefer to not have to rely on a manual file copy to keep things working.

      It's possibly the same issue that @darkfire had here: https://forum.netgate.com/topic/168813/freeradius-let-s-encrypt-dst-root-ca-x3

      Thanks

      Sergei_ShablovskyS GertjanG 2 Replies Last reply Reply Quote 0
      • Sergei_ShablovskyS
        Sergei_Shablovsky @VictorRobellini
        last edited by

        @victorrobellini How you resolve this? Re-installing all? Or next pfSense update fix this?

        —
        CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
        Help Ukraine to resist, save civilians people’s lives !
        (Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.)

        V 1 Reply Last reply Reply Quote 0
        • V
          VictorRobellini @Sergei_Shablovsky
          last edited by VictorRobellini

          @sergei_shablovsky Unfortunately, a fresh install an upgrade did not resolve this. I just refer back to my post here and copy over the cert. I've also migrated most of my client to wpa2-personal since I got tired of this issue.

          I also posted on reddit looking for help.

          1 Reply Last reply Reply Quote 0
          • GertjanG
            Gertjan @VictorRobellini
            last edited by

            @victorrobellini said in FreeRadius - Acme and the reappearing DST Root CA X3 ca:

            How the hell did DST Root CA X3 get in there?!

            pfSense 2.6.0, so you can't update acme.sh (I guess ?)

            dbc52669-e0b2-49bc-a3d4-b1e069f0724f-image.png

            But, maybe a solution :
            Edit : /usr/local/share/certs/ca-root-nss.crt (not with the GUI, this file sis heavy, use SSH or console access)
            and locate "DST Root CA X3" and nuke it.

            For info : on mine (pfSense 23.01) it doesn't exist.

            I'm pretty sure 'Letsenscrypt' doesn't send you over expired certificates , root or intermediate, when you you renew your local certificate.

            My theory is : both certificates are in the file, the expired and the valid one. They have the same name (I guess) and the first one that matches is used. The winner was of course the expired one.
            =>This is me thinking out loud ...

            No "help me" PM's please. Use the forum, the community will thank you.
            Edit : and where are the logs ??

            V 1 Reply Last reply Reply Quote 1
            • V
              VictorRobellini @Gertjan
              last edited by VictorRobellini

              @gertjan thanks for the hint!

              Interesting

              grep -e "DST Root CA X3" -e "ISRG Root X1" /usr/local/share/certs/ca-root-nss.crt

Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
Subject: O = Digital Signature Trust Co., CN = DST Root CA X3
Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1
              • I removed the X3 block from /usr/local/share/certs/ca-root-nss.crt
              • Set FreeRadius to use a self signed CA and cert and confirmed that the self signed CA & cert were in use at /usr/local/etc/raddb/certs/server_cert.pem
              • Changed the FreeRadius CA back to X1 and used the correct signed cert
              • I checked /usr/local/etc/raddb/certs/server_cert.pem and the damn X3 is back in place!

              I'm currently waiting to see what comes back from

              grep -R -e "DST Root CA X3"  /*
              GertjanG 1 Reply Last reply Reply Quote 0
              • GertjanG
                Gertjan @VictorRobellini
                last edited by Gertjan

                @victorrobellini said in FreeRadius - Acme and the reappearing DST Root CA X3 ca:

                Changed the FreeRadius CA back to X1 and used the correct signed cert
                I checked /usr/local/etc/raddb/certs/server_cert.pem and the damn X3 is back in place!

                I did not find any suspected certificate in the acme.sh folders (/usr/local/pkg/acme) and/or the "System Certificate Manager" pfSense certificate storage.

                When I renbew my wildcard pfSense certicate, I see, after several seconds, a 'green' block as a reply.
                The first certificate is misted.
                But in the 'log' file, there are more certificates :

                [Thu May  4 09:10:40 CEST 2023] code='200'
[Thu May  4 09:10:40 CEST 2023] original='-----BEGIN CERTIFICATE-----
MIIGRzCCBS+gAwIBAgISA9KBRQOkD9R8NaKPG2ZD1OyKMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
......
23rw6WjecejxE+Pa3CzmxRPurGlbXLVhh0A0aZbOVrQ7tT472pl2t7g4qeS5v8aw
wjA6/EpTfZ+H0MkmVuvSmyQy4kd/TKIH1ySPybFOXuB6QjgKpKYBpLhlAUs7x9tQ
gfqNXBc2FCRp6tCqCulo8fpuRfQhkmfhzrgK
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
.......
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
....
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----'
[Thu May  4 09:10:40 CEST 2023] response='-----BEGIN CERTIFICATE-----
MIIGRzCCBS+gAwIBAgISA9KBRQOkD9R8NaKPG2ZD1OyKMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
.....
23rw6WjecejxE+Pa3CzmxRPurGlbXLVhh0A0aZbOVrQ7tT472pl2t7g4qeS5v8aw
wjA6/EpTfZ+H0MkmVuvSmyQy4kd/TKIH1ySPybFOXuB6QjgKpKYBpLhlAUs7x9tQ
gfqNXBc2FCRp6tCqCulo8fpuRfQhkmfhzrgK
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
.....
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
....
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----'
[Thu May  4 09:10:40 CEST 2023] Found cert chain
[Thu May  4 09:10:40 CEST 2023] _end_n='36'

                They are listed twice, so 3 certs came back.
                Yours, the intermediate and the CA.

                Check if one of these (the last) is the wrong = "DST Root CA X3".

                Btw : always check the acme.sh log file :
                /tmp/acme/[domain]/acme_issuecert.log
                It's one of the best logs file I ever found 😊

                No "help me" PM's please. Use the forum, the community will thank you.
                Edit : and where are the logs ??

                V 1 Reply Last reply Reply Quote 1
                • V
                  VictorRobellini @Gertjan
                  last edited by VictorRobellini

                  @gertjan
                  I dug through the logs but there's no reference of the X3 CA.

                  I even checked all the certs, fullchains, CA and everything else I could find in the acme dir and found no trace of the X3 CA

                  ]: bash
]$ for i in /cf/conf/acme/*.{fullchain,pem,crt,ca}; do openssl crl2pkcs7 -nocrl -certfile "$i" | openssl pkcs7 -print_certs -noout; done

                  Returns nothing.

                  Must go deeper!

                  #!/usr/bin/env bash

find / -type f \( -iname \*.fullchain -o -iname \*.pem -o -iname \*.crt -iname \*.ca \) -print0 | while read -d $'\0' file

do
        echo "$file"
        openssl crl2pkcs7 -nocrl -certfile "$file" | openssl pkcs7 -print_certs -noout | grep "DST Root CA X3" | awk '{print "\t",$0,"\n"}'
done

                  I found the following files still contain the X3 Ca:

                  • /var/etc/haproxy/shared-frontend.pem
                  • /usr/local/openssl/cert.pem
                  • /usr/local/etc/ssl/cert.pem
                  • /usr/local/etc/raddb/certs/server_cert.pem
                  • /usr/local/etc/raddb/certs/ca_cert.pem
                  • /usr/share/certs/trusted/DST_Root_CA_X3.pem (makes sense!)

                  I avoid screwing around the with the FS since this is my router and I treat it as one, but damn. I'll clean these up and if it appears back in my freeradius certs, I may just go insane.

                  V 1 Reply Last reply Reply Quote 0
                  • V
                    VictorRobellini @VictorRobellini
                    last edited by VictorRobellini

                    Well crap! I edited every cert I found with a X3 reference and it keeps coming back. It now only exists in the backup directory where I saved the unedited certs.

                    The strange thing is that not only is it still appearing in the FreeRadius chain, but also in the haproxy shared-frontend.pem.

                    What am I missing?

                    Maybe it's in the config.xml?

                    I made a copy of my config from /conf/config.xml at /root/backup

                    xmllint --xpath "//*/crt" /root/backup/config.xml > cert.base64
split -p "\n" ./cert.base64

                    Here is where I lost the will to script stuff out. All the certs are base64 encoded and there's no base64 tool installed and I was struggling with using openssl so I went back to basics.

                    cat xaa | perl -MMIME::Base64=decode_base64 -e 'print decode_base64 join"",<>' > xaa.b64
cat xab | perl -MMIME::Base64=decode_base64 -e 'print decode_base64 join"",<>' > xab.b64
cat xac ...

                    Now to see if any contain the dreaded and undying X3 cert

                    openssl crl2pkcs7 -nocrl -certfile /root/backup/xaa.b64 | openssl pkcs7 -print_certs -noout | grep -i x3
openssl crl2pkcs7 -nocrl -certfile /root/backup/xab.b64 | openssl pkcs7 -print_certs -noout | grep -i x3
openssl crl2pkcs7 -nocrl -certfile /root/backup/xac.b64...

                    2 of these files actually contain the damn X3 cert.

                    xaj.b64 contains it, so I just searched the config.xml for the contents of the xaj file and found it here:

                    <ca>
	<refid>61563c252dbc8</refid>
	<descr><![CDATA[Acmecert: O=Internet Security Research Group, CN=ISRG Root X1, C=US]]></descr>
	<crt>LS0tLS1CRUdJ...S0tLS0=</crt>
	<serial>0</serial>
</ca>

                    There was also another match I traced back to:

                    	<ca>
		<refid>61578b1bd6592</refid>
		<descr><![CDATA[Acmecert: O=(STAGING) Internet Security Research Group, CN=(STAGING) Pretend Pear X1, C=US]]></descr>
		<crt>LS0tL...tLQ==</crt>
		<serial>0</serial>
	</ca>

                    I grabbed the serial number of the certs with X3 using this tool and then manually went through my cert manager looking for anything that matches. I found them in my cert manager and deleted them and nothing has changed. FreeRadius and HAProxy arestill somehow finding the X3 cert. I even extracted all the certs again from the updated config.xml, now with fewer certs and yet it's still appearing.

                    1 Reply Last reply Reply Quote 0
                    • First post
                      Last post