Certificate chain causing application authentication failure

rhbauman

A few days ago Amazon Alexa started failing to initiate my Home Assistant (HA) operations although browser access to HA worked fine. My ACME package was up to date and the certificate showed valid in the browser. But testing the website with wget returned

ERROR: The certificate of ‘...’ is not trusted.
ERROR: The certificate of ‘...’ doesn't have a known issuer.

An SSL Labs report showed a broken certificate chain as a result of a revoked certificate. Digging around with Google led me to the following thread which explains the issue in detail and the current workaround.

https://community.letsencrypt.org/t/pfsense-invalid-chains-after-y-generation-cert/247667/27

I don't know if this an issue with the ACME package, certbot or acme.sh but I thought I'd start a topic here first.

johnpoz

@rhbauman and what version of pfsense and the acme client are you using?

blknight88

@rhbauman sounds like you have the same issue here where ACME is not putting all the certs in the fullchain file:

Pfsense invalid chains after Y generation cert

johnpoz

@blknight88 yeah that looks like he already knows that, since he linked to the same thread in his OP.

Maybe I just skimmed it too quick, but I did not see any mention of specific version of the acme client being used on a specific pfsense version.

blknight88

@johnpoz oh yeah I missed that, well I have the same issue since I am the one who started that thread on the Let's Encrypt forums.

I am running the latest version of 1.2 in my pfsense currently.

johnpoz

@blknight88 I will try and update my certs when I get back - walking out the door here in a bit. Don't expire til aug. But looked currently chain is E7 and X1

edit: Uggh, now I am going to find myself down a rabbit hole. I ran test against one of the services I have exposed using acme cert on ssl labs and saw this new

"This server does not support PQC (Post-Quantum Cryptography) key exchange. "

While it didn't report any issues with the chain "Chain issues None" I can see the rabbit hole I am going to be going down ;) hehehe

There was recently a thread about PQ for ssh, and my fav ssh client securecrt doesn't support it yet, but got ssh on pfsense and my linux boxes fixed up for that ;)

rhbauman

@johnpoz pfSense 2.8.1 ACME 1.2

johnpoz

@rhbauman ok just renewed my cert, and while it is a bit odd that now have 2 paths.. Not seeing any issues with the cert chain, and other than the the last cert in the chain all are sent by server.

And the new YE stuff is in the path and show that in pfsense cert store.

IwjnnavSnM

@johnpoz Can you take a look at the '/cf/conf/acme/yourdomain.tld.fullchain' file on the pfsense? For my understanding the file should contain 3 (for Path #1) or 4 (for Path #2) certificates to be 'full chain' (leaving out the 'ISRG Root X1' cert as it has to be on the clients trust store).

johnpoz

@IwjnnavSnM said in Certificate chain causing application authentication failure:

/cf/conf/acme/yourdomain.tld.fullchain

no only 2, my cert and cert of ye2

Seems like its pulling others from the cert store, which has both the ye2 and root ye certs. And then the 2nd path that x2 cert is also in the cert store.

IwjnnavSnM

@johnpoz said in Certificate chain causing application authentication failure:

no only 2, my cert and cert of ye2
Seems like its pulling others from the cert store,

These missing certs are causing verification errors when the .fullchain cert-file is copied to another machine where it is necessary to port-forward instead of using any type of reverse-proxy.

Is there a reason why the certs are being truncated in the process of moving them from '/tmp/acme/yourdomain.tld/yourdomain.tld/fullchain.cer' to '/cf/conf/acme/yourdomain.tld.fullchain'?

johnpoz

@IwjnnavSnM have no idea - but in tmp/acme the fullchain.cer has 4 in it.

Gertjan

@johnpoz said in Certificate chain causing application authentication failure:

"This server does not support PQC (Post-Quantum Cryptography) key exchange. "

Saw that one last week (2 weeks ago) showing up.
It's the new kid on the block.
I mostly use apache2 web servers and my Debian source told me : Get the latest Debian version, 13 (Trixie) and switch to 'testing' = 'release candidates' =aka : go live on the bleeding edge, and you can have a web server (apache2 latest dev) that supports "the latest 4 bit LLM + AI processor TLS security".
Wtf : I just upgraded to Bookworm (12) and unlearn all the syslog stuff, and start to get used to systemd ... (that will take a decade or so, I postponed this choice for way to long already).
I didn’t check if nginx, the pfSense web server, if they - the authors - had anything to say about it, but you've tested that : the pfSense nginx version isn't yet 'Quantum aware'.
postfix : the same thing.
Courier/dovecot : they aren't even full TLS 1.3 yet.
And so on.

I don't care for now, my servers are - imho - safe.
Still using classic RSA4K certs, nothing fancy.

Btw : joke aside: when I read "Post-Quantum Cryptography" do we need to upgrade our pfSense hardware so it includes some kind of 'nvidia 500 Watt' AI security ship or other M1 ?

johnpoz

@Gertjan the brief look into this, not available on the openssl on pfsense from what I can tell with.

[26.03.1-RELEASE][admin@sg4860.home.arpa]/root: openssl list -key-exchange-algorithms
  ECDH @ default
  { 1.3.101.110, X25519 } @ default
  { 1.3.101.111, X448 } @ default
  { 1.2.840.113549.1.3.1, DH, dhKeyAgreement } @ default
  TLS1-PRF @ default
  HKDF @ default
  { 1.3.6.1.4.1.11591.4.11, id-scrypt, SCRYPT } @ default
[26.03.1-RELEASE][admin@sg4860.home.arpa]/root:

Seems like you can add them as sort of module, but nothing going to start playing around with on my pfsense install. And I use ssl offloading in ha-proxy. So would be required for openssl to support it. Could prob play with a local instance of something. I don't really think its ready for prime time as of yet. Unlike the ssh stuff where support is actually builtin to the latest versions.

rhbauman

@johnpoz as @IwjnnavSnM pointed out, the problem is that the .fullchain file in /cf/conf does not contain the Root YR certificate but it is in fullchain.cer in /tmp/acme. Root YR is not in the trusted certificate authorities (yet). Here's the discussion of why certificate chains are failing without the Root YR certificate: https://community.letsencrypt.org/t/today-re-issued-certificates-from-issuer-yr2-have-a-revoked-cert-in-chain/247612/16. It would seem that for certs obtained from Let's Encrypt, the fullchain.cer file should be copied over without modification to the [domain].fullchain file. When the Root YR certificate finally shows up in trusted certificates, Let's Encrypt can remove it from the .fullchain file.

Is this the right place to submit bug reports for the ACME package?

johnpoz

@rhbauman said in Certificate chain causing application authentication failure:

Root YR is not in the trusted certificate authorities (yet).

it is on mine

See my post from above. I didn't put it there by hand that is for sure.

bugs/issues would be submitted via redmine for pfsense

https://redmine.pfsense.org/

But that showed up right after I renewed my cert, first thing I did was check it with ssl labs after it renewed.. Now I do run a sh script in my acme.. One sec let me look to exactly what is in there.. brb

ah - that just restarts haproxy

"/usr/local/etc/rc.d/haproxy.sh restart"

And have this checked in general.

Which I would think then the fullchain should be actually full.. So yeah something seems off.

rhbauman

@johnpoz the certs in my fullchain.cer from /tmp/acme:

        Issuer: C=US, O=Let's Encrypt, CN=YR2
        Subject: CN=[domain]
        Issuer: C=US, O=ISRG, CN=Root YR
        Subject: C=US, O=Let's Encrypt, CN=YR2
        Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
        Subject: C=US, O=ISRG, CN=Root YR

and [domain].fullchain:

        Issuer: C=US, O=Let's Encrypt, CN=YR2
        Subject: CN=[domain]
        Issuer: C=US, O=ISRG, CN=Root YR
        Subject: C=US, O=Let's Encrypt, CN=YR2

I'm running Debian 12 (bookworm) up-to-date and ca-certificates doesn't include the Root YR cert:

root@snowy:~# dpkg -L ca-certificates | grep ISRG
/usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
/usr/share/ca-certificates/mozilla/ISRG_Root_X2.crt

I guess it's taking awhile to percolate the YR cert to Linux distros.

GPz1100

Subscribing to this as I just got hit by this new YR bug as well.

For the time being I copied the /tmp/acme/yourdomain.tld/yourdomain.tld/fullchain.cer to the server and to the /conf/acme.... path.

MalagaFirewall8

For anyone filing this, I’d include both paths in the report: the full /tmp/acme/.../fullchain.cer contents and the shortened /cf/conf/acme/<domain>.fullchain contents after save/deploy. That makes it clearer whether the package is intentionally trimming trust anchors or accidentally dropping the current extra YR certificate needed by some clients. As a temporary workaround, copying the /tmp/acme fullchain into the deployed path (or doing that in the ACME action list before restarting HAProxy/nginx) seems safer than editing the certificate in the GUI, since the next renewal may overwrite it again.

Laxarus

any redmine bug report for this yet?