ACME & HAProxy secondary step failed after backup restore

Foxi352

After trying myself for some days, time is running out now before needed renewals ....

Situation: pfSense was running in bhyve on FreeNAS. I did full config backup, installed Proxmox and clean new pfSense, restored backup. Everything went smooth and everything is working. Except ACME renewal.

All certificates are setup to use standalone HTTP server on port 5080. HAProxy redirects to ACME on '/.well-known/acme-challenge/'. Nothing has changed on this config for the past two or more years.

When i try to renew or issue i get:

registry.mydomain.family
Renewing certificate 
account: mydomain.family-production 
server: letsencrypt-staging-2 

/usr/local/pkg/acme/acme.sh  --issue  -d 'registry.mydomain.family' --standalone --listen-v4 --httpport '5080' --home '/tmp/acme/registry.mydomain.family/' --accountconf '/tmp/acme/registry.mydomain.family/accountconf.conf' --force --reloadCmd '/tmp/acme/registry.mydomain.family/reloadcmd.sh' --log-level 3 --log '/tmp/acme/registry.mydomain.family/acme_issuecert.log'
Array
(
    [path] => /etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin/
    [PATH] => /etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin/
    [port] => 5080
    [ipv6] => 
)
[Tue Feb 25 08:34:42 CET 2020] Standalone mode.
[Tue Feb 25 08:34:42 CET 2020] Single domain='registry.mydomain.family'
[Tue Feb 25 08:34:42 CET 2020] Getting domain auth token for each domain
[Tue Feb 25 08:34:47 CET 2020] Getting webroot for domain='registry.mydomain.family'
[Tue Feb 25 08:34:47 CET 2020] Verifying: registry.mydomain.family
[Tue Feb 25 08:34:47 CET 2020] Standalone mode server
[Tue Feb 25 08:34:52 CET 2020] Pending
[... another 30 times the same line ...]
[Tue Feb 25 08:36:17 CET 2020] Pending
[Tue Feb 25 08:36:17 CET 2020] registry.mydomain.family:Timeout
[Tue Feb 25 08:36:17 CET 2020] Please check log file for more details: /tmp/acme/registry.mydomain.family/acme_issuecert.log

During the renewal process i can curl from the outside:

curl http://registry.mydomain.family/.well-known/acme-challenge/

b7_N9T7DNNx6xixC1P02Ytbx8oXqGBKzcUx9QrDxxU8.LPnxRXIPxxzlxGr-6_Os9vaSrxbGuLRhxuxxfBxxyqI%

So for me everything looks OK. But the secondary validation step, whatever that is, is failing.
I hope someone can point me to the right direction as to where to look for error.

I will attach the complete log as file, but i think the relevant lines are the following:

[Tue Feb 25 08:40:11 CET 2020] responseHeaders='HTTP/2 200
server: nginx
date: Tue, 25 Feb 2020 07:40:11 GMT
content-type: application/json
content-length: 823
boulder-requester: 12557860
cache-control: public, max-age=0, no-cache
link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
link: <https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/40503614>;rel="up"
location: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/40503614/nfDUPg
replay-nonce: 00023KQIUQjZNzC_aCZLVbaZOC6wqjPqHPZt0jApDgxANKM
x-frame-options: DENY
strict-transport-security: max-age=604800
'
[Tue Feb 25 08:40:11 CET 2020] code='200'
[Tue Feb 25 08:40:11 CET 2020] original='{
  "type": "http-01",
  "status": "invalid",
  "error": {
    "type": "urn:ietf:params:acme:error:connection",
    "detail": "During secondary validation: Fetching http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8: Timeout after connect (your server may be slow or overloaded)",
    "status": 400
  },
  "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/40503614/nfDUPg",
  "token": "b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8",
  "validationRecord": [
    {
      "url": "http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8",
      "hostname": "registry.mydomain.family",
      "port": "80",
      "addressesResolved": [
        "158.64.172.18"
      ],
      "addressUsed": "158.64.172.18"
    }
  ]
}'
[Tue Feb 25 08:40:11 CET 2020] response='{"type":"http-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:connection","detail":"During secondary validation: Fetching http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8: Timeout after connect (your server may be slow or overloaded)","status": 400},"url":"https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/40503614/nfDUPg","token":"b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8","validationRecord":[{"url":"http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8","hostname":"registry.mydomain.family","port":"80","addressesResolved":["158.64.172.18"],"addressUsed":"158.64.172.18"}]}'
[Tue Feb 25 08:40:11 CET 2020] original='{"type":"http-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:connection","detail":"During secondary validation: Fetching http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8: Timeout after connect (your server may be slow or overloaded)","status": 400},"url":"https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/40503614/nfDUPg","token":"b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8","validationRecord":[{"url":"http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8","hostname":"registry.mydomain.family","port":"80","addressesResolved":["158.64.172.18"],"addressUsed":"158.64.172.18"}]}'
[Tue Feb 25 08:40:11 CET 2020] response='{"type":"http-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:connection","detail":"During secondary validation: Fetching http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8: Timeout after connect (your server may be slow or overloaded)","status": 400},"url":"https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/40503614/nfDUPg","token":"b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8","validationRecord":[{"url":"http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8","hostname":"registry.mydomain.family","port":"80","addressesResolved":["158.64.172.18"],"addressUsed":"158.64.172.18"}]}'
[Tue Feb 25 08:40:11 CET 2020] error='"error":{"type":"urn:ietf:params:acme:error:connection","detail":"During secondary validation: Fetching http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8: Timeout after connect (your server may be slow or overloaded)","status": 400'
[Tue Feb 25 08:40:11 CET 2020] errordetail='During secondary validation: Fetching http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8: Timeout after connect (your server may be slow or overloaded)'
[Tue Feb 25 08:40:11 CET 2020] registry.mydomain.family:Verify error:During secondary validation: Fetching http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8: Timeout after connect (your server may be slow or overloaded)

acme_issuecert.log.txt [0_1582619230669_acme_issuecert.log](Uploading 100%)

PiBa

@Foxi352
Are you blocking any requests from outside? Like from different country's or something?

Perhaps run a tcpdump / packetcapture on the wan interface (filtered on port 80) while the certificate is being requested? It should show if requests are being made, from where, and if indeed they are 'hanging'..?

Also perhaps try 'the-other' haproxy package.? Which one and what version of it are you using? 'haproxy' or 'haproxy-devel' ?

Foxi352

@PiBa I use the 'simple' haproxy package. I am not blocking anything special. pfSense and all packages are on newest versions. I also disabled snort, pfblocker and everything else during debugging. As i said, it worked before moving from bhyve to proxmox. Or probably what counts more is before backup up and restoring on a clean install.

But nevermind, i got it working now by using the haproxy chroot'ed webroot instead of standalone http server. After hours or fiddeling i have up on the other solution and switched all my certs to webroot. This is even faster as the script does not have to spin up an extra http server ...

But thank you for taking the time to reply ;-)