Unable To Renew Certs Since last ACME package Update

stephenw10

Yes, we see the issue and are working on an update to fix it.

Steve

costanzo

@raiderj

I think thy are working to have a fix soon. In the meantime, here's a workaround:

Use Manual DNS

Create a TXT record in your Cloudflare DNS

Change the dropdown to Manual DNS

Save
Click Issue
Copy the _acme-challenge key (you will see this displayed after hitting issue)
Go in to your DNS and update the key in the TXT record
Wait for 2-3 Minutes (this is to give time for the DNS record to propagate)
Go back in to the Acme and click Renew

raiderj

@costanzo

Thank you! That worked perfectly. Pasted part of the log below. I do have a question - would this also work for wildcard domains? That's at least what I'm trying to do with my setup. I just have a single entry in my Domain SAN list for the wildcard domain.

Successful Result:

[Wed Apr 29 12:38:58 EDT 2020] Single domain='*.subdomain.domain.com'
[Wed Apr 29 12:38:58 EDT 2020] Getting domain auth token for each domain
[Wed Apr 29 12:38:58 EDT 2020] Verifying: *.subdomain.domain.com
[Wed Apr 29 12:39:02 EDT 2020] Success
[Wed Apr 29 12:39:02 EDT 2020] Verify finished, start to sign.
[Wed Apr 29 12:39:02 EDT 2020] Lets finalize the order, Le_OrderFinalize: https://acme-staging-v02.api.letsencrypt.org/acme/finalize/13385284/...
[Wed Apr 29 12:39:03 EDT 2020] Download cert, Le_LinkCert: https://acme-staging-v02.api.letsencrypt.org/acme/cert/fa7e...
[Wed Apr 29 12:39:03 EDT 2020] Cert success.

costanzo

@raiderj

Yes. You need to create two SANs: one for the example.com and one for *.example.com

It probable doesn't mater, but I would list the example.com domain first.

For example:

Also, make sure you your domain listed in your Cloudflare DNS record too. Eg, have example.com.

raiderj

@costanzo
Updated my configs. Working still with both SANs being list, and I also see the resulting certs in the filesystem for both my wildcard and standard domains.
I didn't test it extensively, but the order of the SAN list doesn't seem to matter. I did leave the wildcard listing last as you recommended.

raiderj

Switched over to a production setup, and there seems to be an issue. When I first go to issue, there's two TXT values that are being asked to enter into Cloudflare. One for the wildcard cert, one for the regular cert. But they're both looking for the same key value (_acme-challenge.subdomain.domain.com).
Right now I'm able to get the wildcard cert to return, but not the normal cert. Maybe manual-DNS doesn't work for wildcard certs in Production?

jimp

Update the ACME package and try again, there was a change to the CloudFlare script in the ACME.sh repo which is in the new version. Worth a try.

cakewalk

@jimp Thanks for the quick turn around! Confirmed fixed for me.

bartkowski

I'm seeing another issue, not sure if related to this. I'm using DNS-Hurricane Electric method.
I think the referrer should be my external IP, but also when I query my FQDN of my pfsense box, it returns 192.168.2.1, so is it a DNS issue?

2020/04/29 13:27:18 [error] 44990#100120: *3720 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.2.25, server: , request: "POST /acme/acme_certificates.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.2.1", referrer: "https://192.168.2.1/acme/acme_certificates.php"

jimp

@bartkowski Start a new thread for that. This one is only for CloudFlare. And make sure you have updated to the new ACME package (just put up an hour or two ago)

raiderj

@jimp
Updated to the latest package, but wildcard certs aren't coming down for my domain. Normal cert is coming in, properly signed with the Prod Let's Encrypt CA. Maybe because the second domain in the SAN list is the wildcard listing?

EDIT: Yes, if I switch the SAN order to the wildcard first, it comes down. With this workaround using DNS-Manual, should I switch back to the DNS-Cloudflare option? Maybe the new ACME package fixes this?

jimp

If that's also on CloudFlare it may be a different issue.

The one the update had a fix for is https://github.com/acmesh-official/acme.sh/issues/2888

costanzo

@raiderj said in Unable To Renew Certs Since last ACME package Update:

One for the wildcard cert, one for the regular cert. But they're both looking for the same key valu

Odd. Can you post a screen capture of the message show when issuing the cert. Be sure to blur out any sensitive data. I am not seeing this issue, but may be different for you.

Also, keep in mind generating too many certs under production will cap you out and you will need to wait a week. Best to use the staging option.

raiderj

@costanzo said in Unable To Renew Certs Since last ACME package Update:

@raiderj said in Unable To Renew Certs Since last ACME package Update:

One for the wildcard cert, one for the regular cert. But they're both looking for the same key valu

Odd. Can you post a screen capture of the message show when issuing the cert. Be sure to blur out any sensitive data. I am not seeing this issue, but may be different for you.

Also, keep in mind generating too many certs under production will cap you out and you will need to wait a week. Best to use the staging option.

I'll do that if I see it again. I can't replicate the issue at the moment. The only issue I have now, with both staging and production is that the second domain in the SAN list won't get an updated cert. Haven't tried switching to the DNS-Cloudflare option though.

costanzo

@jimp Thanks for the quick turn around! I can confirmed this also fixed it for me. I tested the cert gen using the staging.

raiderj

@raiderj said in Unable To Renew Certs Since last ACME package Update:

@costanzo said in Unable To Renew Certs Since last ACME package Update:

@raiderj said in Unable To Renew Certs Since last ACME package Update:

One for the wildcard cert, one for the regular cert. But they're both looking for the same key valu

Odd. Can you post a screen capture of the message show when issuing the cert. Be sure to blur out any sensitive data. I am not seeing this issue, but may be different for you.

Also, keep in mind generating too many certs under production will cap you out and you will need to wait a week. Best to use the staging option.

I'll do that if I see it again. I can't replicate the issue at the moment. The only issue I have now, with both staging and production is that the second domain in the SAN list won't get an updated cert. Haven't tried switching to the DNS-Cloudflare option though.

Ok, I believe it's working now with the original "DNS-Cloudflare" option. I think I can also remove the new API token I created as well. So, if I'm understanding this properly, all I need for cert generation is the global API key from Cloudflare.

Wildcard plus normal cert generation in the same request seems to still not work. I still only get the first domain in the SAN list.

asche

I'll hitch onto this, but my error seems a bit different:

running ACME with DNS challenge
using Cloudflare
DNS challenge seems to be working and certificate to be issued
but automatic renewal fails (as per pfsense email notification)

Manual renewal runs into the following issues: XMLRPC errors

pfsense error message (my domain masked with FQDN):

[...]
[Sun Jun  7 10:48:57 CEST 2020] Your cert is in  /tmp/acme/FQDN//*.FQDN/*.FQDN.cer 
[Sun Jun  7 10:48:57 CEST 2020] Your cert key is in  /tmp/acme/FQDN//*.FQDN/*.FQDN.key 
[Sun Jun  7 10:48:57 CEST 2020] The intermediate CA cert is in  /tmp/acme/FQDN//*.FQDN/ca.cer 
[Sun Jun  7 10:48:57 CEST 2020] And the full chain certs is there:  /tmp/acme/FQDN//*.FQDN/fullchain.cer 
[Sun Jun  7 10:48:57 CEST 2020] Run reload cmd: /tmp/acme/FQDN/reloadcmd.sh

IMPORT CERT FQDN, /tmp/acme/FQDN/*.FQDN/*.FQDN.key, /tmp/acme/FQDN/*.FQDN/*.FQDN.cer
update cert!
Fatal error: Uncaught XML_RPC2_InvalidUriException: Client URI 'https://admin:@:444/xmlrpc.php' is not valid in /usr/local/share/pear/XML/RPC2/Client.php:167
Stack trace:
#0 /usr/local/share/pear/XML/RPC2/Backend/Php/Client.php(80): XML_RPC2_Client->__construct('https://admin:@...', Array)
#1 /usr/local/share/pear/XML/RPC2/Client.php(238): XML_RPC2_Backend_Php_Client->__construct('https://admin:@...', Array)
#2 /etc/inc/xmlrpc_client.inc(92): XML_RPC2_Client::create('https://admin:@...', Array)
#3 /etc/inc/xmlrpc_client.inc(148): pfsense_xmlrpc_client->xmlrpc_internal('exec_php', 'require_once('s...', 240)
#4 /usr/local/pkg/acme/acme.inc(107): pfsense_xmlrpc_client->xmlrpc_exec_php('require_once('s...')
#5 /usr/local/pkg/acme/acme_command.sh(76): acme_xmlrpc_restart_service('haproxy', 'array (\n)')
#6 {main}
  thrown in /usr/local/share/pear/XML/RPC2/Client.php on line 167
PHP ERROR: Type: 1, File: /usr/local/share/pear/XML/RPC2/Client.php, Line: 167, Message: Uncaught XML_RPC2_InvalidUriException: Client URI 'https://admin:@:444/xmlrpc.php' is not valid in /usr/local/share/pear/XML/RPC2/Client.php:167
Stack trace:
#0 /usr/local/share/pear/XML/RPC2/Backend/Php/Client.php(80): XML_RPC2_Client->__construct('https://admin:@...', Array)
#1 /usr/local/share/pear/XML/RPC2/Client.php(238): XML_RPC2_Backend_Php_Client->__construct('https://admin:@...', Array)
#2 /etc/inc/xmlrpc_client.inc(92): XML_RPC2_Client::create('https://admin:@...', Array)
#3 /etc/inc/xmlrpc_client.inc(148): pfsense_xmlrpc_client->xmlrpc_internal('exec_php', 'require_once('s...', 240)
#4 /usr/local/pkg/acme/acme.inc(107): pfsense_xmlrpc_client->xmlrpc_exec_php('require_once('s...')
#5 /usr/local/pkg/acme/acme_command.sh(76): acme_xmlrpc_restart_service('haproxy', 'array (\n)')
#6 {main}
  thrown[Sun Jun  7 10:49:07 CEST 2020] Reload error for :

Couple of things to note:

Yes, I currently employ HAproxy, and my pfsense port has been changed to :444
NO, I do not use CARP nor sync nor HAProxy Sync (all are off / disabled)
The same error occurs when I disable HAProxy.
it seems that a valid certificate is created and - partly - updated into pfsense, because I can see an updated certificate in the pfsense certificate manager.
pfsense still uses the old (soon-to-expire) certificate, not the renewed one shown in the certificate manager (??)

PiBa

@asche
What 'actions' have you configured on the acme certificate configuration? Perhaps a "Restart Remote Service (XMLRPC)" action?

asche

@PiBa

@PiBa said in Unable To Renew Certs Since last ACME package Update:

@asche
What 'actions' have you configured on the acme certificate configuration? Perhaps a "Restart Remote Service (XMLRPC)" action?

You nailed it -- "Enabled haproxy Restart Remote Service (XMLRPC)"

Should I delete this?

PiBa

@asche said in Unable To Renew Certs Since last ACME package Update:

NO, I do not use CARP nor sync nor HAProxy Sync (all are off / disabled)

If you dont sync to anywhere, then you probably don't have anything to restart 'remotely', so yes delete that action. You might want to restart the local haproxy service and webgui though. Use the example "shell command" options for that.