ACME issue with Godaddy DNS
-
After upgrading my firewall and the acme client(0.6.8) I am unable to renew my cert through the Godaddy DNS option. I can get a cert through the staging V2 but not through the production V2 server. I am seeing:
_CURL='curl -L --silent --dump-header /tmp/acme/mydomain.com-wildcard//http.header -g ' Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 60
I have all of the other logs. Not sure what I should put on here since there is a lot of personal info in them.
Any help is appreciated. -
cURL error 60 implies that cURL failed to validate the server certificate... Which seems unlikely
CURLE_PEER_FAILED_VERIFICATION (60)
The remote server's SSL certificate or SSH md5 fingerprint was deemed not OK. This error code has been unified with CURLE_SSL_CACERT since 7.62.0. Its previous value was 51.
Do you have IPv6 on that system? Maybe try nudging the firewall to prefer IPv4 (System > Advanced, Networking tab)
-
I checked the Prefer to use IPv4 even if IPv6 is available and rebooted. I am getting the same results still. I have a static ipv4 public IP on my firewall. I am using haproxy to redirect traffic to a couple of different sites as well as the firewall gui.
Is the acme dns process trying to send something back to the firewall that may be getting caught there? -
No, requests from cURL would be outbound HTTPS requests like any other browser would make.
-
That's what I thought. Are there any other steps I can take to troubleshoot this?
-
Check your local date/clock and make sure it's OK, maybe look back in the log on the filesystem and see if it shows the exact curl command it used and see what happens when you run it by hand.
-
The date and time are confirmed correct and are using ntp to a GPS clock with Stratum 1 on my local network. I am not sure where to look for the exact curl command being ran. I am looking in the logs under /tmp/acme/mydomain.com-wildcard/acme-issuecert.log and I see a log line that reads similar to this:
_CURL='curl -L --silent --dump-header /tmp/acme/mydomain.com-wildcard//http.header -g '
I tried running the curl command and it says no URL specified.
I see this same curl command in the staging which goes through without issue.
-
There should be a URL in the log somewhere around that message, I thought. I don't have one handy to check, though.
-
I see this in the logs(I have redacted some personal info):
[Wed May 6 09:26:38 CDT 2020] _sub_domain='_acme-challenge' [Wed May 6 09:26:38 CDT 2020] _domain='mydomain.com' [Wed May 6 09:26:38 CDT 2020] Getting existing records [Wed May 6 09:26:38 CDT 2020] domains/mydomain.com/records/TXT/_acme-challenge [Wed May 6 09:26:38 CDT 2020] GET [Wed May 6 09:26:38 CDT 2020] url='https://api.godaddy.com/v1/domains/mydomain.com/records/TXT/_acme-challenge' [Wed May 6 09:26:38 CDT 2020] timeout= [Wed May 6 09:26:38 CDT 2020] Http already initialized. [Wed May 6 09:26:38 CDT 2020] _CURL='curl -L --silent --dump-header /tmp/acme/mydomain.com-wildcard//http.header -g ' [Wed May 6 09:26:38 CDT 2020] ret='0' [Wed May 6 09:26:38 CDT 2020] response='[{"data":"","name":"_acme-challenge","ttl":600,"type":"TXT"},{"data":"DqWXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX","name":"_acme-challenge","ttl":600,"type":"TXT"},{"data":"umVXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX","name":"_acme-challenge","ttl":600,"type":"TXT"},{"data":"WUUXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX","name":"_acme-challenge","ttl":600,"type":"TXT"}]' [Wed May 6 09:26:38 CDT 2020] The record is existing, skip [Wed May 6 09:26:38 CDT 2020] The txt record is added: Success. [Wed May 6 09:26:38 CDT 2020] mydomain.com,_acme-challenge.mydomain.com,,dns_gd,DqWXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,/usr/local/pkg/acme/dnsapi/dns_gd.sh [Wed May 6 09:26:38 CDT 2020] Let's check each dns records now. Sleep 20 seconds first. [Wed May 6 09:26:39 CDT 2020] Let's wait 10 seconds and check again. [Wed May 6 09:26:41 CDT 2020] _is_idn_d='_acme-challenge.mydomain.com' [Wed May 6 09:26:41 CDT 2020] _idn_temp [Wed May 6 09:26:41 CDT 2020] _is_idn_d='_acme-challenge.mydomain.com' [Wed May 6 09:26:41 CDT 2020] _idn_temp [Wed May 6 09:26:41 CDT 2020] d='mydomain.com' [Wed May 6 09:26:41 CDT 2020] txtdomain='_acme-challenge.mydomain.com' [Wed May 6 09:26:41 CDT 2020] aliasDomain='_acme-challenge.mydomain.com' [Wed May 6 09:26:41 CDT 2020] txt='WUUXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' [Wed May 6 09:26:41 CDT 2020] d_api='/usr/local/pkg/acme/dnsapi/dns_gd.sh' [Wed May 6 09:26:41 CDT 2020] Checking mydomain.com for _acme-challenge.mydomain.com [Wed May 6 09:26:41 CDT 2020] _c_txtdomain='_acme-challenge.mydomain.com' [Wed May 6 09:26:41 CDT 2020] _c_aliasdomain='_acme-challenge.mydomain.com' [Wed May 6 09:26:41 CDT 2020] _c_txt='WUUXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' [Wed May 6 09:26:41 CDT 2020] Detect dns server first. [Wed May 6 09:26:41 CDT 2020] GET [Wed May 6 09:26:41 CDT 2020] url='https://cloudflare-dns.com' [Wed May 6 09:26:41 CDT 2020] timeout= [Wed May 6 09:26:41 CDT 2020] Http already initialized. [Wed May 6 09:26:41 CDT 2020] _CURL='curl -L --silent --dump-header /tmp/acme/mydomain.com-wildcard//http.header -g ' [Wed May 6 09:26:41 CDT 2020] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 60 [Wed May 6 09:26:41 CDT 2020] ret='60' [Wed May 6 09:26:41 CDT 2020] Use google doh server [Wed May 6 09:26:41 CDT 2020] _ns_ep='https://dns.google/resolve' [Wed May 6 09:26:41 CDT 2020] _ns_domain='_acme-challenge.mydomain.com' [Wed May 6 09:26:41 CDT 2020] _ns_type='TXT' [Wed May 6 09:26:41 CDT 2020] GET [Wed May 6 09:26:41 CDT 2020] url='https://dns.google/resolve?name=_acme-challenge.mydomain.com&type=TXT' [Wed May 6 09:26:41 CDT 2020] timeout= [Wed May 6 09:26:41 CDT 2020] Http already initialized. [Wed May 6 09:26:41 CDT 2020] _CURL='curl -L --silent --dump-header /tmp/acme/mydomain.com-wildcard//http.header -g ' [Wed May 6 09:26:41 CDT 2020] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 60 [Wed May 6 09:26:41 CDT 2020] ret='60' [Wed May 6 09:26:41 CDT 2020] response [Wed May 6 09:26:41 CDT 2020] Not valid yet, let's wait 10 seconds and check next one. [Wed May 6 09:26:41 CDT 2020] _p_txtdomain='_acme-challenge.mydomain.com' [Wed May 6 09:26:41 CDT 2020] Cloudflare purge TXT record for domain _acme-challenge.mydomain.com [Wed May 6 09:26:41 CDT 2020] POST [Wed May 6 09:26:41 CDT 2020] _post_url='https://cloudflare-dns.com/api/v1/purge?domain=_acme-challenge.mydomain.com&type=TXT' [Wed May 6 09:26:41 CDT 2020] body [Wed May 6 09:26:41 CDT 2020] _postContentType [Wed May 6 09:26:41 CDT 2020] Http already initialized. [Wed May 6 09:26:41 CDT 2020] _CURL='curl -L --silent --dump-header /tmp/acme/mydomain.com-wildcard//http.header -g ' [Wed May 6 09:26:41 CDT 2020] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 60 [Wed May 6 09:26:41 CDT 2020] _ret='60' [Wed May 6 09:26:41 CDT 2020] response [Wed May 6 09:26:46 CDT 2020] _is_idn_d='_acme-challenge.mydomain.com' [Wed May 6 09:26:46 CDT 2020] _idn_temp [Wed May 6 09:26:46 CDT 2020] _is_idn_d='_acme-challenge.mydomain.com' [Wed May 6 09:26:46 CDT 2020] _idn_temp [Wed May 6 09:26:46 CDT 2020] d='mydomain.com' [Wed May 6 09:26:46 CDT 2020] txtdomain='_acme-challenge.mydomain.com' [Wed May 6 09:26:46 CDT 2020] aliasDomain='_acme-challenge.mydomain.com' [Wed May 6 09:26:46 CDT 2020] txt='umVXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' [Wed May 6 09:26:46 CDT 2020] d_api='/usr/local/pkg/acme/dnsapi/dns_gd.sh' [Wed May 6 09:26:46 CDT 2020] Checking mydomain.com for _acme-challenge.mydomain.com [Wed May 6 09:26:46 CDT 2020] _c_txtdomain='_acme-challenge.mydomain.com' [Wed May 6 09:26:46 CDT 2020] _c_aliasdomain='_acme-challenge.mydomain.com' [Wed May 6 09:26:46 CDT 2020] _c_txt='umVXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' [Wed May 6 09:26:46 CDT 2020] Detect dns server first. [Wed May 6 09:26:46 CDT 2020] GET [Wed May 6 09:26:46 CDT 2020] url='https://cloudflare-dns.com' [Wed May 6 09:26:46 CDT 2020] timeout= [Wed May 6 09:26:46 CDT 2020] Http already initialized.
-
So it's failing to validate
dns.google
andcloudflare-dns.com
... Are you doing anything that might interfere with those DNS queries locally? Maybe something like DNSBL or similar messing with your DNS responses? Or a DNS redirect?Though that should be the same for ACME staging or prod, that part doesn't have anything to do with Let's Encrypt directly.
Except maybe if you had a recent authz still valid for staging it may be skipping that part entirely, so it's not a fair test.
-
I had my dns pointing to opendns servers. I moved them to google and the issue went away. I wouldn't have dreamed that opendns was causing this. Thank you so much for your help!
-
Hi there. Just wanted to say that this thread was helpful to me as I had the same issue with NextDNS when working with the ACME package and GoDaddy. Thank you! Also, I realize this thread is old but for others coming across this in a google search of the curl error I thought I would provide a bit more detail and an additional workaround for those who cannot or do not want to change their DNS provider.
DNS filtering providers like OpenDNS/NextDNS/etc. appear to resolve requests to public DNS provider hosts, like dns.google and cloudflare-dns.com (which ACME uses), to '0.0.0.0' possibly to protect against content filter circumvention. Because ACME calls the DoH (port 443) versions of these sites for validation the resolution to 0.0.0.0 actually sends the request to the pfSense web GUI (assuming you are running on the default webConfigurator HTTPS port of 443). This causes the certificate validation failure by curl.
This issue can be demonstrated quite easily by checking dns.google:443 or clouldflare-dns.com:443 with openssl like so:
openssl s_client -connect cloudflare-dns.com:443
If this is your issue, the openssl command output will show a certificate chain containing the webConfigurator self-signed certs from pfSense and not the proper ones curl expects for Google or CloudFlare.
If you have set the pfSense system-wide DNS servers to use OpenDNS/NextDNS/etc. and don't wish to change these in each individual DHCP range assignment, you can simply add manual '/etc/hosts' entries for dns.google and cloudflare-dns.com on your pfSense box. This will allow DNS validation to succeed for ACME but leave the rest of your DNS setup unaffected. I'm sure there are other workarounds as well with their own pluses/deltas but the hosts file entry is a quick and dirty fix. Cheers!
-
The last paragraph about the '/etc/hosts' workaround in pfSense was incorrect; I forgot that '/etc/hosts' gets wiped periodically by pfSense. The real workaround is below:
If you have set the pfSense system-wide DNS servers to use OpenDNS/NextDNS/etc. and don't wish to change these in each individual DHCP range assignment, you can simply add 'Allowlist' entries for dns.google and cloudflare-dns.com in the web console for your DNS provider ('Allowlist' may be called something else but that is what NextDNS calls it). This will allow DNS validation to succeed for ACME. If you are concerned about clients circumventing your DNS provider due to whitelisting the Google and Cloudflare DNS names, you can always redirect all DNS traffic on your LAN to make sure it goes through your DNS provider:
https://docs.netgate.com/pfsense/en/latest/recipes/dns-redirect.html