Unable To Renew Certs Since last ACME package Update
-
Hi guys - I'm no longer able to renew any of my certs via the ACME package in Pfsense 2.4.5 since the last ACME package update (I presume)
I'm using the dns-01 method with Cloudflare. Not sure if this is a Coudflare issue or the ACME package. This guy https://community.cloudflare.com/t/lets-encrypt-acme-pfsense-plugin-fail-to-create-dns-records-with-token/168973 seems to be having the same issue.
Here's my log upon attempting to renew one of my certs:-
[Tue Apr 28 20:04:46 BST 2020] _postContentType='application/jose+json' [Tue Apr 28 20:04:46 BST 2020] Http already initialized. [Tue Apr 28 20:04:46 BST 2020] _CURL='curl -L --silent --dump-header /tmp/acme/conference.somedomain.com//http.header -g ' [Tue Apr 28 20:04:47 BST 2020] _ret='0' [Tue Apr 28 20:04:47 BST 2020] responseHeaders='HTTP/2 200 server: nginx date: Tue, 28 Apr 2020 19:04:47 GMT content-type: application/json content-length: 184 boulder-requester: 84718288 cache-control: public, max-age=0, no-cache link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index" link: <https://acme-v02.api.letsencrypt.org/acme/authz-v3/4238164555>;rel="up" location: https://acme-v02.api.letsencrypt.org/acme/chall-v3/4238164555/Cpd4VA replay-nonce: 0101bbBsqPZfMLc8658XszzbLQuToNMJLhH1BIf9_75pyT8 x-frame-options: DENY strict-transport-security: max-age=604800 ' [Tue Apr 28 20:04:47 BST 2020] code='200' [Tue Apr 28 20:04:47 BST 2020] original='{ "type": "dns-01", "status": "pending", "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/4238164555/Cpd4VA", "token": "AblXJ3KyDj7ri2P_z7ErsetDd7mOBDwIIhVDCDT9MGY" }' [Tue Apr 28 20:04:47 BST 2020] response='{"type":"dns-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/4238164555/Cpd4VA","token":"AblXJ3KyDj7ri2P_z7ErsetDd7mOBDwIIhVDCDT9MGY"}' [Tue Apr 28 20:04:47 BST 2020] pid [Tue Apr 28 20:04:47 BST 2020] No need to restore nginx, skip. [Tue Apr 28 20:04:47 BST 2020] _clearupdns [Tue Apr 28 20:04:47 BST 2020] dns_entries [Tue Apr 28 20:04:47 BST 2020] skip dns.
Many thanks
-
I'm also seeing this after updating to ACME 0.6.7. It was not an issue with 2.4.5 + 0.6.6 from what I can tell.
I'm seeing the following error when renewing:
[Tue Apr 28 16:42:57 EDT 2020] Single domain='*.DOMAIN.com' [Tue Apr 28 16:42:57 EDT 2020] Getting domain auth token for each domain [Tue Apr 28 16:42:58 EDT 2020] Getting webroot for domain='*.DOMAIN.com' [Tue Apr 28 16:42:58 EDT 2020] Adding txt value: <txt value> for domain: _acme-challenge.<DOMAIN>.com [Tue Apr 28 16:42:59 EDT 2020] Error [Tue Apr 28 16:42:59 EDT 2020] Error add txt for domain:_acme-challenge.<DOMAIN>.com [Tue Apr 28 16:42:59 EDT 2020] Please check log file for more details: /tmp/acme/DOMAIN.com/acme_issuecert.log
Happy to provide the extended log too if needed.
-
Looks like an issue with the latest package update. I may try to do a cert renewal manually using acme.sh on one of my linux VM's to confirm everything is working on the Cloudflare side.
-
I have the same issue.
-
FYI - I created a support ticket. Request ID [#INC-50562]. Let me know if you need any logs, or other information that would help.
Thanks in advance.
-
Ok we see that and are investigating.
Yes, please add any logs you have on the ticket.
Steve
-
@stephenw10
Done - uploaded the error, log detail as well as info on my DNS and API Token setup from CloudFlare.Thanks again for your help.
-
@stephenw10 Thanks for looking into this. Much appreciated.
-
I think I'm having the same issue. In addition to the error text shown earlier, I have what I think is a related error further up the in the acme_issuecert.log
[Wed Apr 29 03:16:05 EDT 2020] response='{"success":false,"errors":[{"code":0,"message":"Actor 'com.cloudflare.api.token.<api_token_that_I_created>' requires permission 'com.cloudflare.api.account.zone.list' to list zones"}],"messages":[],"result":null}' "token": "<another_token_here>"
I gave one API token a Zone.Zone and Zone.DNS edit rights, which I then pasted in pfsense during the ACME configuration using Cloudflare as my DNS.
-
Yes, we see the issue and are working on an update to fix it.
Steve
-
I think thy are working to have a fix soon. In the meantime, here's a workaround:
Use Manual DNS
- Create a TXT record in your Cloudflare DNS
- Change the dropdown to Manual DNS
-
Save
-
Click Issue
-
Copy the _acme-challenge key (you will see this displayed after hitting issue)
-
Go in to your DNS and update the key in the TXT record
-
Wait for 2-3 Minutes (this is to give time for the DNS record to propagate)
-
Go back in to the Acme and click Renew
-
Thank you! That worked perfectly. Pasted part of the log below. I do have a question - would this also work for wildcard domains? That's at least what I'm trying to do with my setup. I just have a single entry in my Domain SAN list for the wildcard domain.
Successful Result:
[Wed Apr 29 12:38:58 EDT 2020] Single domain='*.subdomain.domain.com' [Wed Apr 29 12:38:58 EDT 2020] Getting domain auth token for each domain [Wed Apr 29 12:38:58 EDT 2020] Verifying: *.subdomain.domain.com [Wed Apr 29 12:39:02 EDT 2020] Success [Wed Apr 29 12:39:02 EDT 2020] Verify finished, start to sign. [Wed Apr 29 12:39:02 EDT 2020] Lets finalize the order, Le_OrderFinalize: https://acme-staging-v02.api.letsencrypt.org/acme/finalize/13385284/... [Wed Apr 29 12:39:03 EDT 2020] Download cert, Le_LinkCert: https://acme-staging-v02.api.letsencrypt.org/acme/cert/fa7e... [Wed Apr 29 12:39:03 EDT 2020] Cert success.
-
Yes. You need to create two SANs: one for the example.com and one for *.example.com
It probable doesn't mater, but I would list the example.com domain first.
For example:
Also, make sure you your domain listed in your Cloudflare DNS record too. Eg, have example.com.
-
@costanzo
Updated my configs. Working still with both SANs being list, and I also see the resulting certs in the filesystem for both my wildcard and standard domains.
I didn't test it extensively, but the order of the SAN list doesn't seem to matter. I did leave the wildcard listing last as you recommended. -
Switched over to a production setup, and there seems to be an issue. When I first go to issue, there's two TXT values that are being asked to enter into Cloudflare. One for the wildcard cert, one for the regular cert. But they're both looking for the same key value (_acme-challenge.subdomain.domain.com).
Right now I'm able to get the wildcard cert to return, but not the normal cert. Maybe manual-DNS doesn't work for wildcard certs in Production? -
Update the ACME package and try again, there was a change to the CloudFlare script in the ACME.sh repo which is in the new version. Worth a try.
-
@jimp Thanks for the quick turn around! Confirmed fixed for me.
-
I'm seeing another issue, not sure if related to this. I'm using DNS-Hurricane Electric method.
I think the referrer should be my external IP, but also when I query my FQDN of my pfsense box, it returns 192.168.2.1, so is it a DNS issue?2020/04/29 13:27:18 [error] 44990#100120: *3720 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.2.25, server: , request: "POST /acme/acme_certificates.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.2.1", referrer: "https://192.168.2.1/acme/acme_certificates.php"
-
@bartkowski Start a new thread for that. This one is only for CloudFlare. And make sure you have updated to the new ACME package (just put up an hour or two ago)
-
@jimp
Updated to the latest package, but wildcard certs aren't coming down for my domain. Normal cert is coming in, properly signed with the Prod Let's Encrypt CA. Maybe because the second domain in the SAN list is the wildcard listing?EDIT: Yes, if I switch the SAN order to the wildcard first, it comes down. With this workaround using DNS-Manual, should I switch back to the DNS-Cloudflare option? Maybe the new ACME package fixes this?
-
If that's also on CloudFlare it may be a different issue.
The one the update had a fix for is https://github.com/acmesh-official/acme.sh/issues/2888
-
@raiderj said in Unable To Renew Certs Since last ACME package Update:
One for the wildcard cert, one for the regular cert. But they're both looking for the same key valu
Odd. Can you post a screen capture of the message show when issuing the cert. Be sure to blur out any sensitive data. I am not seeing this issue, but may be different for you.
Also, keep in mind generating too many certs under production will cap you out and you will need to wait a week. Best to use the staging option.
-
@costanzo said in Unable To Renew Certs Since last ACME package Update:
@raiderj said in Unable To Renew Certs Since last ACME package Update:
One for the wildcard cert, one for the regular cert. But they're both looking for the same key valu
Odd. Can you post a screen capture of the message show when issuing the cert. Be sure to blur out any sensitive data. I am not seeing this issue, but may be different for you.
Also, keep in mind generating too many certs under production will cap you out and you will need to wait a week. Best to use the staging option.
I'll do that if I see it again. I can't replicate the issue at the moment. The only issue I have now, with both staging and production is that the second domain in the SAN list won't get an updated cert. Haven't tried switching to the DNS-Cloudflare option though.
-
@jimp Thanks for the quick turn around! I can confirmed this also fixed it for me. I tested the cert gen using the staging.
-
@raiderj said in Unable To Renew Certs Since last ACME package Update:
@costanzo said in Unable To Renew Certs Since last ACME package Update:
@raiderj said in Unable To Renew Certs Since last ACME package Update:
One for the wildcard cert, one for the regular cert. But they're both looking for the same key valu
Odd. Can you post a screen capture of the message show when issuing the cert. Be sure to blur out any sensitive data. I am not seeing this issue, but may be different for you.
Also, keep in mind generating too many certs under production will cap you out and you will need to wait a week. Best to use the staging option.
I'll do that if I see it again. I can't replicate the issue at the moment. The only issue I have now, with both staging and production is that the second domain in the SAN list won't get an updated cert. Haven't tried switching to the DNS-Cloudflare option though.
Ok, I believe it's working now with the original "DNS-Cloudflare" option. I think I can also remove the new API token I created as well. So, if I'm understanding this properly, all I need for cert generation is the global API key from Cloudflare.
Wildcard plus normal cert generation in the same request seems to still not work. I still only get the first domain in the SAN list.
-
I'll hitch onto this, but my error seems a bit different:
- running ACME with DNS challenge
- using Cloudflare
- DNS challenge seems to be working and certificate to be issued
- but automatic renewal fails (as per pfsense email notification)
Manual renewal runs into the following issues: XMLRPC errors
pfsense error message (my domain masked with FQDN):
[...] [Sun Jun 7 10:48:57 CEST 2020] Your cert is in /tmp/acme/FQDN//*.FQDN/*.FQDN.cer [Sun Jun 7 10:48:57 CEST 2020] Your cert key is in /tmp/acme/FQDN//*.FQDN/*.FQDN.key [Sun Jun 7 10:48:57 CEST 2020] The intermediate CA cert is in /tmp/acme/FQDN//*.FQDN/ca.cer [Sun Jun 7 10:48:57 CEST 2020] And the full chain certs is there: /tmp/acme/FQDN//*.FQDN/fullchain.cer [Sun Jun 7 10:48:57 CEST 2020] Run reload cmd: /tmp/acme/FQDN/reloadcmd.sh IMPORT CERT FQDN, /tmp/acme/FQDN/*.FQDN/*.FQDN.key, /tmp/acme/FQDN/*.FQDN/*.FQDN.cer update cert! Fatal error: Uncaught XML_RPC2_InvalidUriException: Client URI 'https://admin:@:444/xmlrpc.php' is not valid in /usr/local/share/pear/XML/RPC2/Client.php:167 Stack trace: #0 /usr/local/share/pear/XML/RPC2/Backend/Php/Client.php(80): XML_RPC2_Client->__construct('https://admin:@...', Array) #1 /usr/local/share/pear/XML/RPC2/Client.php(238): XML_RPC2_Backend_Php_Client->__construct('https://admin:@...', Array) #2 /etc/inc/xmlrpc_client.inc(92): XML_RPC2_Client::create('https://admin:@...', Array) #3 /etc/inc/xmlrpc_client.inc(148): pfsense_xmlrpc_client->xmlrpc_internal('exec_php', 'require_once('s...', 240) #4 /usr/local/pkg/acme/acme.inc(107): pfsense_xmlrpc_client->xmlrpc_exec_php('require_once('s...') #5 /usr/local/pkg/acme/acme_command.sh(76): acme_xmlrpc_restart_service('haproxy', 'array (\n)') #6 {main} thrown in /usr/local/share/pear/XML/RPC2/Client.php on line 167 PHP ERROR: Type: 1, File: /usr/local/share/pear/XML/RPC2/Client.php, Line: 167, Message: Uncaught XML_RPC2_InvalidUriException: Client URI 'https://admin:@:444/xmlrpc.php' is not valid in /usr/local/share/pear/XML/RPC2/Client.php:167 Stack trace: #0 /usr/local/share/pear/XML/RPC2/Backend/Php/Client.php(80): XML_RPC2_Client->__construct('https://admin:@...', Array) #1 /usr/local/share/pear/XML/RPC2/Client.php(238): XML_RPC2_Backend_Php_Client->__construct('https://admin:@...', Array) #2 /etc/inc/xmlrpc_client.inc(92): XML_RPC2_Client::create('https://admin:@...', Array) #3 /etc/inc/xmlrpc_client.inc(148): pfsense_xmlrpc_client->xmlrpc_internal('exec_php', 'require_once('s...', 240) #4 /usr/local/pkg/acme/acme.inc(107): pfsense_xmlrpc_client->xmlrpc_exec_php('require_once('s...') #5 /usr/local/pkg/acme/acme_command.sh(76): acme_xmlrpc_restart_service('haproxy', 'array (\n)') #6 {main} thrown[Sun Jun 7 10:49:07 CEST 2020] Reload error for :
Couple of things to note:
- Yes, I currently employ HAproxy, and my pfsense port has been changed to :444
- NO, I do not use CARP nor sync nor HAProxy Sync (all are off / disabled)
- The same error occurs when I disable HAProxy.
- it seems that a valid certificate is created and - partly - updated into pfsense, because I can see an updated certificate in the pfsense certificate manager.
- pfsense still uses the old (soon-to-expire) certificate, not the renewed one shown in the certificate manager (??)
-
@asche
What 'actions' have you configured on the acme certificate configuration? Perhaps a "Restart Remote Service (XMLRPC)" action? -
@PiBa said in Unable To Renew Certs Since last ACME package Update:
@asche
What 'actions' have you configured on the acme certificate configuration? Perhaps a "Restart Remote Service (XMLRPC)" action?You nailed it -- "Enabled haproxy Restart Remote Service (XMLRPC)"
Should I delete this?
-
@asche said in Unable To Renew Certs Since last ACME package Update:
NO, I do not use CARP nor sync nor HAProxy Sync (all are off / disabled)
If you dont sync to anywhere, then you probably don't have anything to restart 'remotely', so yes delete that action. You might want to restart the local haproxy service and webgui though. Use the example "shell command" options for that.