Unable To Renew Certs Since last ACME package Update


  • @stephenw10
    Done - uploaded the error, log detail as well as info on my DNS and API Token setup from CloudFlare.

    Thanks again for your help.


  • @stephenw10 Thanks for looking into this. Much appreciated.


  • I think I'm having the same issue. In addition to the error text shown earlier, I have what I think is a related error further up the in the acme_issuecert.log

    [Wed Apr 29 03:16:05 EDT 2020] response='{"success":false,"errors":[{"code":0,"message":"Actor 'com.cloudflare.api.token.<api_token_that_I_created>' requires permission 'com.cloudflare.api.account.zone.list' to list zones"}],"messages":[],"result":null}'
      "token": "<another_token_here>"
    

    I gave one API token a Zone.Zone and Zone.DNS edit rights, which I then pasted in pfsense during the ACME configuration using Cloudflare as my DNS.

  • Netgate Administrator

    Yes, we see the issue and are working on an update to fix it.

    Steve


  • @raiderj

    I think thy are working to have a fix soon. In the meantime, here's a workaround:

    Use Manual DNS

    • Create a TXT record in your Cloudflare DNS

    aff0212b-4962-4540-978d-7f1502d9c614-image.png

    • Change the dropdown to Manual DNS

    462b98ee-714d-4da9-bf6a-6e1967e0632d-image.png

    • Save

    • Click Issue

    • Copy the _acme-challenge key (you will see this displayed after hitting issue)

    • Go in to your DNS and update the key in the TXT record

    • Wait for 2-3 Minutes (this is to give time for the DNS record to propagate)

    • Go back in to the Acme and click Renew


  • @costanzo

    Thank you! That worked perfectly. Pasted part of the log below. I do have a question - would this also work for wildcard domains? That's at least what I'm trying to do with my setup. I just have a single entry in my Domain SAN list for the wildcard domain.

    Successful Result:

    [Wed Apr 29 12:38:58 EDT 2020] Single domain='*.subdomain.domain.com'
    [Wed Apr 29 12:38:58 EDT 2020] Getting domain auth token for each domain
    [Wed Apr 29 12:38:58 EDT 2020] Verifying: *.subdomain.domain.com
    [Wed Apr 29 12:39:02 EDT 2020] Success
    [Wed Apr 29 12:39:02 EDT 2020] Verify finished, start to sign.
    [Wed Apr 29 12:39:02 EDT 2020] Lets finalize the order, Le_OrderFinalize: https://acme-staging-v02.api.letsencrypt.org/acme/finalize/13385284/...
    [Wed Apr 29 12:39:03 EDT 2020] Download cert, Le_LinkCert: https://acme-staging-v02.api.letsencrypt.org/acme/cert/fa7e...
    [Wed Apr 29 12:39:03 EDT 2020] Cert success.
    

  • @raiderj

    Yes. You need to create two SANs: one for the example.com and one for *.example.com

    It probable doesn't mater, but I would list the example.com domain first.

    For example:

    c530659e-597a-44c4-b431-b1318eda9201-image.png

    Also, make sure you your domain listed in your Cloudflare DNS record too. Eg, have example.com.


  • @costanzo
    Updated my configs. Working still with both SANs being list, and I also see the resulting certs in the filesystem for both my wildcard and standard domains.
    I didn't test it extensively, but the order of the SAN list doesn't seem to matter. I did leave the wildcard listing last as you recommended.


  • Switched over to a production setup, and there seems to be an issue. When I first go to issue, there's two TXT values that are being asked to enter into Cloudflare. One for the wildcard cert, one for the regular cert. But they're both looking for the same key value (_acme-challenge.subdomain.domain.com).
    Right now I'm able to get the wildcard cert to return, but not the normal cert. Maybe manual-DNS doesn't work for wildcard certs in Production?

  • Rebel Alliance Developer Netgate

    Update the ACME package and try again, there was a change to the CloudFlare script in the ACME.sh repo which is in the new version. Worth a try.


  • @jimp Thanks for the quick turn around! Confirmed fixed for me.


  • I'm seeing another issue, not sure if related to this. I'm using DNS-Hurricane Electric method.
    I think the referrer should be my external IP, but also when I query my FQDN of my pfsense box, it returns 192.168.2.1, so is it a DNS issue?

    2020/04/29 13:27:18 [error] 44990#100120: *3720 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.2.25, server: , request: "POST /acme/acme_certificates.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.2.1", referrer: "https://192.168.2.1/acme/acme_certificates.php"

  • Rebel Alliance Developer Netgate

    @bartkowski Start a new thread for that. This one is only for CloudFlare. And make sure you have updated to the new ACME package (just put up an hour or two ago)


  • @jimp
    Updated to the latest package, but wildcard certs aren't coming down for my domain. Normal cert is coming in, properly signed with the Prod Let's Encrypt CA. Maybe because the second domain in the SAN list is the wildcard listing?

    EDIT: Yes, if I switch the SAN order to the wildcard first, it comes down. With this workaround using DNS-Manual, should I switch back to the DNS-Cloudflare option? Maybe the new ACME package fixes this?

  • Rebel Alliance Developer Netgate

    If that's also on CloudFlare it may be a different issue.

    The one the update had a fix for is https://github.com/acmesh-official/acme.sh/issues/2888


  • @raiderj said in Unable To Renew Certs Since last ACME package Update:

    One for the wildcard cert, one for the regular cert. But they're both looking for the same key valu

    Odd. Can you post a screen capture of the message show when issuing the cert. Be sure to blur out any sensitive data. I am not seeing this issue, but may be different for you.

    Also, keep in mind generating too many certs under production will cap you out and you will need to wait a week. Best to use the staging option.


  • @costanzo said in Unable To Renew Certs Since last ACME package Update:

    @raiderj said in Unable To Renew Certs Since last ACME package Update:

    One for the wildcard cert, one for the regular cert. But they're both looking for the same key valu

    Odd. Can you post a screen capture of the message show when issuing the cert. Be sure to blur out any sensitive data. I am not seeing this issue, but may be different for you.

    Also, keep in mind generating too many certs under production will cap you out and you will need to wait a week. Best to use the staging option.

    I'll do that if I see it again. I can't replicate the issue at the moment. The only issue I have now, with both staging and production is that the second domain in the SAN list won't get an updated cert. Haven't tried switching to the DNS-Cloudflare option though.


  • @jimp Thanks for the quick turn around! I can confirmed this also fixed it for me. I tested the cert gen using the staging.


  • @raiderj said in Unable To Renew Certs Since last ACME package Update:

    @costanzo said in Unable To Renew Certs Since last ACME package Update:

    @raiderj said in Unable To Renew Certs Since last ACME package Update:

    One for the wildcard cert, one for the regular cert. But they're both looking for the same key valu

    Odd. Can you post a screen capture of the message show when issuing the cert. Be sure to blur out any sensitive data. I am not seeing this issue, but may be different for you.

    Also, keep in mind generating too many certs under production will cap you out and you will need to wait a week. Best to use the staging option.

    I'll do that if I see it again. I can't replicate the issue at the moment. The only issue I have now, with both staging and production is that the second domain in the SAN list won't get an updated cert. Haven't tried switching to the DNS-Cloudflare option though.

    Ok, I believe it's working now with the original "DNS-Cloudflare" option. I think I can also remove the new API token I created as well. So, if I'm understanding this properly, all I need for cert generation is the global API key from Cloudflare.

    Wildcard plus normal cert generation in the same request seems to still not work. I still only get the first domain in the SAN list.


  • I'll hitch onto this, but my error seems a bit different:

    • running ACME with DNS challenge
    • using Cloudflare
    • DNS challenge seems to be working and certificate to be issued
    • but automatic renewal fails (as per pfsense email notification)

    Manual renewal runs into the following issues: XMLRPC errors

    pfsense error message (my domain masked with FQDN):

    [...]
    [Sun Jun  7 10:48:57 CEST 2020] Your cert is in  /tmp/acme/FQDN//*.FQDN/*.FQDN.cer 
    [Sun Jun  7 10:48:57 CEST 2020] Your cert key is in  /tmp/acme/FQDN//*.FQDN/*.FQDN.key 
    [Sun Jun  7 10:48:57 CEST 2020] The intermediate CA cert is in  /tmp/acme/FQDN//*.FQDN/ca.cer 
    [Sun Jun  7 10:48:57 CEST 2020] And the full chain certs is there:  /tmp/acme/FQDN//*.FQDN/fullchain.cer 
    [Sun Jun  7 10:48:57 CEST 2020] Run reload cmd: /tmp/acme/FQDN/reloadcmd.sh
    
    IMPORT CERT FQDN, /tmp/acme/FQDN/*.FQDN/*.FQDN.key, /tmp/acme/FQDN/*.FQDN/*.FQDN.cer
    update cert!
    Fatal error: Uncaught XML_RPC2_InvalidUriException: Client URI 'https://admin:@:444/xmlrpc.php' is not valid in /usr/local/share/pear/XML/RPC2/Client.php:167
    Stack trace:
    #0 /usr/local/share/pear/XML/RPC2/Backend/Php/Client.php(80): XML_RPC2_Client->__construct('https://admin:@...', Array)
    #1 /usr/local/share/pear/XML/RPC2/Client.php(238): XML_RPC2_Backend_Php_Client->__construct('https://admin:@...', Array)
    #2 /etc/inc/xmlrpc_client.inc(92): XML_RPC2_Client::create('https://admin:@...', Array)
    #3 /etc/inc/xmlrpc_client.inc(148): pfsense_xmlrpc_client->xmlrpc_internal('exec_php', 'require_once('s...', 240)
    #4 /usr/local/pkg/acme/acme.inc(107): pfsense_xmlrpc_client->xmlrpc_exec_php('require_once('s...')
    #5 /usr/local/pkg/acme/acme_command.sh(76): acme_xmlrpc_restart_service('haproxy', 'array (\n)')
    #6 {main}
      thrown in /usr/local/share/pear/XML/RPC2/Client.php on line 167
    PHP ERROR: Type: 1, File: /usr/local/share/pear/XML/RPC2/Client.php, Line: 167, Message: Uncaught XML_RPC2_InvalidUriException: Client URI 'https://admin:@:444/xmlrpc.php' is not valid in /usr/local/share/pear/XML/RPC2/Client.php:167
    Stack trace:
    #0 /usr/local/share/pear/XML/RPC2/Backend/Php/Client.php(80): XML_RPC2_Client->__construct('https://admin:@...', Array)
    #1 /usr/local/share/pear/XML/RPC2/Client.php(238): XML_RPC2_Backend_Php_Client->__construct('https://admin:@...', Array)
    #2 /etc/inc/xmlrpc_client.inc(92): XML_RPC2_Client::create('https://admin:@...', Array)
    #3 /etc/inc/xmlrpc_client.inc(148): pfsense_xmlrpc_client->xmlrpc_internal('exec_php', 'require_once('s...', 240)
    #4 /usr/local/pkg/acme/acme.inc(107): pfsense_xmlrpc_client->xmlrpc_exec_php('require_once('s...')
    #5 /usr/local/pkg/acme/acme_command.sh(76): acme_xmlrpc_restart_service('haproxy', 'array (\n)')
    #6 {main}
      thrown[Sun Jun  7 10:49:07 CEST 2020] Reload error for :
    

    Couple of things to note:

    1. Yes, I currently employ HAproxy, and my pfsense port has been changed to :444
    2. NO, I do not use CARP nor sync nor HAProxy Sync (all are off / disabled)
    3. The same error occurs when I disable HAProxy.
    4. it seems that a valid certificate is created and - partly - updated into pfsense, because I can see an updated certificate in the pfsense certificate manager.
    5. pfsense still uses the old (soon-to-expire) certificate, not the renewed one shown in the certificate manager (??)

  • @asche
    What 'actions' have you configured on the acme certificate configuration? Perhaps a "Restart Remote Service (XMLRPC)" action?


  • @PiBa

    @PiBa said in Unable To Renew Certs Since last ACME package Update:

    @asche
    What 'actions' have you configured on the acme certificate configuration? Perhaps a "Restart Remote Service (XMLRPC)" action?

    You nailed it -- "Enabled haproxy Restart Remote Service (XMLRPC)"

    Should I delete this?


  • @asche said in Unable To Renew Certs Since last ACME package Update:

    NO, I do not use CARP nor sync nor HAProxy Sync (all are off / disabled)

    If you dont sync to anywhere, then you probably don't have anything to restart 'remotely', so yes delete that action. You might want to restart the local haproxy service and webgui though. Use the example "shell command" options for that.