ACME v1.1_1 25.11.1 Release Cloudflare letsencrypt issue
-
On my router logs show ACME uses cloudflare-dns.com to verify the TXT record for DNS challenge has been added successfully.
That's the pfSense ACME package's validation process, not LE's.
If you wish to block
cloudflare-dns.com/DoH/DoT/DoQ on your LAN via pfBlockerNG, then you might consider the "DNS Sleep" option underServices > ACME > Certificates > Edit > Validation. -
Misconfiguration.
-
@tinfoilmatt Ran some tests. Put cloudflare-dns.com back in SafeSearch block list. Created new cert definition in ACME, no DNS timeout. Tried creation, failed. Put in 120s timeout and retried, cert created. Its now well past ethanol time in UK, will look at logs and do more tests tomorrow.
-
The description for that package setting, "
DNS Sleep", is the key:Disables automatic DNS polling for DNS validation methods and configures a specific amount of time, in seconds, ACME waits before attempting verification after adding TXT records.
The default behavior is to automatically poll public DNS servers for records until ACME finds them, rather than waiting a set amount of time.
[Emphasis added.]
Also, from the docs:
For DNS-based methods of validating FQDN SANs, Let’s Encrypt checks for a TXT record in the form of
_acme-challenge.<domain name>which must contain the authorization value.[Emphasis added.]
In other words—when you specify a wait time, package polling (i.e., from the pfSense host) is skipped, and the script simply waits for the Let's Encrypt service to complete the DNS challenge 'blind'. If the challenge is successful, then the certificate is created/renewed without the pfSense host ever having to 'poll' for the TXT record itself.
-
@stephenw10 Like I mentioned, I haven’t made any configuration changes as such. My DNS resolver is set to ALL on outbound interfaces. DNS resolution behaviour is set to Use Local DNS (127.0.0.1), Ignore remote. This has always been my config including all pfblockerng settings where I have enabled to block DoH/DoT/DoQ Blocking and no changes made there as well. There definitely has been some changes but either Netgate pfsense+, pfblockerng or acme somewhere.
-
There definitely has been some changes [with] either Netgate pfsense+, pfblockerng or acme somewhere.
In your of yet unsubstantiated opinion.
Put everything back the way you had it, configure the "
DNS Sleep" option, initiate a renewal, and report back. -
@ketsman @stephenw10 @johnpoz Done more tests and investigation. The short answer is I made a change in February. I don't think anything has changed in pfSense or ACME.
In February I had a reorganisation of certificates and Cloudflare API keys. Did a test on a staging domain with a new API key, certificate issued ok. Thought I’d try removing the DNS sleep time that I had set at 120s. Certificate issued ok, what I didn’t see was log entry “mydomain.tld is already verified, skipping dns-01” followed by “Skipping dns.”, so my test wasn't valid, but I removed DNS sleep from all my domain entries.
I’m now back with cloudflare-dns.com blocked in safe search and DNS sleep time set at 120s in all production certificates. I have tried 30s in a staging certificate which worked.
-
@AMG-A35 said in ACME v1.1_1 25.11.1 Release Cloudflare letsencrypt issue:
Thought I’d try removing the DNS sleep time that I had set at 120s
That might be ok-ish, but set it to 1x, not 0.
( "0" activates something else ...)acme.sh uses a script you've chosen that handles the insertion (and deletion) of a TXT record into your 'master' DNS domain name server.
As domain names have always two (or more) DNS domain name slave servers, the master will 'notify' the slaves that 'new info' is available.
The DNS domain name slaves will come back, whenever it suits them, ( !) and initial a zone transfer, the so called "XFER even". This is a pull operation. The DNS master server doesn't 'push' the info to the slaves.
There is no guarantee that this will happen within 'x seconds'. After all, the DNS master and slave have to deal with many thousands of domains names.
Example : when you ask for a wild card certificate for a domain name, two TXT will get created. You could also ask for several individual like host1.your-domain.tld, host1.your-domain.tld, host1.your-domain.tld etc, every individual host name will have its own TXT record.
DNS server admins are smart people : they have put in place a delay after the last notification came in, which means DNS slaves will sync after this delay.From what I've seen, using 'free', like dns.het.net, afraid.org, and the more obvious "your own registration", the sync can happens within several seconds, up until minutes ( ! ). I've seen 300 sec and more.
What happens when the DNS Sleep times out ? acme.sh signal’s Letsencrypt that it can proceed with the check = the proof you own the domain, as only you, the owner, can create these TXT records.
The good old days are over : Letsencrypt could check just the master domain name server, and call it a day. But that's not what happens these days : they are all checked, and all have to contain the same, correct info. So if just one DNS domain name slave is lagging behind - didn't sync up yet, the check fails.Btw : I use acme.sh myself.
I have my own DNS domain name master and slaves servers (ISC bind (named) of course) so I only have to deal with my own domain names (about 10). So I can see the acme.sh interactions in real time both at the acmes.sh side, and the server(s) side and things normally settled down within 10 seconds.
So, for me acm.sh's DNSsleep couldn't be set lower as "10" (seconds). -
I have tried 30s in a staging certificate which worked.
Selecting any 'sleep' time value will disable the local TXT query/'poll' completely.
-
Thanks.
With yourSelecting any 'sleep' time value will disable the local TXT query/'poll' completely.
I dived into acme.sh to understand what it was doing if DNS Sleep is set to zero.
If DNS Sleep is set to zero ... then there will be a 20 seconds delay.
After that, acme.sh does the checking itself.
See _check_dns_entries().
Here is the acme.sh DNS Check Wiki page.
The acme.sh will use DOH** and picks DOH CLOUDFLARE, DOH_GOOGLE, or DOH_ALI.
From here, it starts looping around up until the moment all DNS NS server reproduce the same correct result = a TXT record with the correct name and correct content.
This DNS test run can last for 1200 seconds or 10 minutes max.
As soon as a total match is found : good TXT value and good TXT content, then the process continues : the actual certificate renewal.Humm.
This is actually a smarter way of doing things
** but : If you've blocked DOH with for example pfBlockerng, or blocked Google and or Cloudflare DNS IPS, then you've reached the typical 'shoot in the foot' situation.
I do block all DOH with pfBLockerng, so that explains for me why :
never worked for me.The default behavior is to automatically poll public DNS servers for records until ACME finds them, rather than waiting a set amount of time.
The reality :
The default behavior is to automatically poll public DOH DNS servers for records until ACME finds them, rather than waiting a set amount of time.
Furthermore, the public DOH DNS servers, and ordinary DNS servers are known and listed. If pfBlocker blocks these, then DNS Sleep = 0 will probably fail.
Thanks, @tinfoilmatt , I now (better) understand what DNS Sleep '0' does.
edit
Further investigation : we can't set up our own DOH server to be used by acme.sh.
acme.sh has 4 build in (hardcoded) DOH servers, and you have to pick one and if you don't, Cloudflare is used by default.
The "DNS Sleep" setting is "0" by default, and for good reasons it is exposed in the pfSense GUI.
Read for example this acme.sh 'issue', and everything is now clear.
I could activate DOH on my own DNS domain name servers, but that means I have to patch acme.sh. My own DOH wouldn't be blacklisted (DNSBL) by pfBlockerng ^^I do understand why acmes.sh default to use DOH : the challenge code, created by Letsencrypt, and put in place at the DNS master domain name server by acme.sh, has to be protected 'at all costs'. After all, the one who possesses (intercepts) this challenge code could obtain a certificate for that (your) domain name. That would be a huge disaster.
