ACME & HAProxy secondary step failed after backup restore



  • After trying myself for some days, time is running out now before needed renewals ....

    Situation: pfSense was running in bhyve on FreeNAS. I did full config backup, installed Proxmox and clean new pfSense, restored backup. Everything went smooth and everything is working. Except ACME renewal.

    All certificates are setup to use standalone HTTP server on port 5080. HAProxy redirects to ACME on '/.well-known/acme-challenge/'. Nothing has changed on this config for the past two or more years.

    When i try to renew or issue i get:

    registry.mydomain.family
    Renewing certificate 
    account: mydomain.family-production 
    server: letsencrypt-staging-2 
    
    /usr/local/pkg/acme/acme.sh  --issue  -d 'registry.mydomain.family' --standalone --listen-v4 --httpport '5080' --home '/tmp/acme/registry.mydomain.family/' --accountconf '/tmp/acme/registry.mydomain.family/accountconf.conf' --force --reloadCmd '/tmp/acme/registry.mydomain.family/reloadcmd.sh' --log-level 3 --log '/tmp/acme/registry.mydomain.family/acme_issuecert.log'
    Array
    (
        [path] => /etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin/
        [PATH] => /etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin/
        [port] => 5080
        [ipv6] => 
    )
    [Tue Feb 25 08:34:42 CET 2020] Standalone mode.
    [Tue Feb 25 08:34:42 CET 2020] Single domain='registry.mydomain.family'
    [Tue Feb 25 08:34:42 CET 2020] Getting domain auth token for each domain
    [Tue Feb 25 08:34:47 CET 2020] Getting webroot for domain='registry.mydomain.family'
    [Tue Feb 25 08:34:47 CET 2020] Verifying: registry.mydomain.family
    [Tue Feb 25 08:34:47 CET 2020] Standalone mode server
    [Tue Feb 25 08:34:52 CET 2020] Pending
    [... another 30 times the same line ...]
    [Tue Feb 25 08:36:17 CET 2020] Pending
    [Tue Feb 25 08:36:17 CET 2020] registry.mydomain.family:Timeout
    [Tue Feb 25 08:36:17 CET 2020] Please check log file for more details: /tmp/acme/registry.mydomain.family/acme_issuecert.log
    

    During the renewal process i can curl from the outside:

    curl http://registry.mydomain.family/.well-known/acme-challenge/
    
    b7_N9T7DNNx6xixC1P02Ytbx8oXqGBKzcUx9QrDxxU8.LPnxRXIPxxzlxGr-6_Os9vaSrxbGuLRhxuxxfBxxyqI%
    

    So for me everything looks OK. But the secondary validation step, whatever that is, is failing.
    I hope someone can point me to the right direction as to where to look for error.

    I will attach the complete log as file, but i think the relevant lines are the following:

    [Tue Feb 25 08:40:11 CET 2020] responseHeaders='HTTP/2 200
    server: nginx
    date: Tue, 25 Feb 2020 07:40:11 GMT
    content-type: application/json
    content-length: 823
    boulder-requester: 12557860
    cache-control: public, max-age=0, no-cache
    link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
    link: <https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/40503614>;rel="up"
    location: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/40503614/nfDUPg
    replay-nonce: 00023KQIUQjZNzC_aCZLVbaZOC6wqjPqHPZt0jApDgxANKM
    x-frame-options: DENY
    strict-transport-security: max-age=604800
    '
    [Tue Feb 25 08:40:11 CET 2020] code='200'
    [Tue Feb 25 08:40:11 CET 2020] original='{
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "During secondary validation: Fetching http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8: Timeout after connect (your server may be slow or overloaded)",
        "status": 400
      },
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/40503614/nfDUPg",
      "token": "b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8",
      "validationRecord": [
        {
          "url": "http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8",
          "hostname": "registry.mydomain.family",
          "port": "80",
          "addressesResolved": [
            "158.64.172.18"
          ],
          "addressUsed": "158.64.172.18"
        }
      ]
    }'
    [Tue Feb 25 08:40:11 CET 2020] response='{"type":"http-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:connection","detail":"During secondary validation: Fetching http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8: Timeout after connect (your server may be slow or overloaded)","status": 400},"url":"https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/40503614/nfDUPg","token":"b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8","validationRecord":[{"url":"http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8","hostname":"registry.mydomain.family","port":"80","addressesResolved":["158.64.172.18"],"addressUsed":"158.64.172.18"}]}'
    [Tue Feb 25 08:40:11 CET 2020] original='{"type":"http-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:connection","detail":"During secondary validation: Fetching http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8: Timeout after connect (your server may be slow or overloaded)","status": 400},"url":"https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/40503614/nfDUPg","token":"b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8","validationRecord":[{"url":"http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8","hostname":"registry.mydomain.family","port":"80","addressesResolved":["158.64.172.18"],"addressUsed":"158.64.172.18"}]}'
    [Tue Feb 25 08:40:11 CET 2020] response='{"type":"http-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:connection","detail":"During secondary validation: Fetching http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8: Timeout after connect (your server may be slow or overloaded)","status": 400},"url":"https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/40503614/nfDUPg","token":"b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8","validationRecord":[{"url":"http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8","hostname":"registry.mydomain.family","port":"80","addressesResolved":["158.64.172.18"],"addressUsed":"158.64.172.18"}]}'
    [Tue Feb 25 08:40:11 CET 2020] error='"error":{"type":"urn:ietf:params:acme:error:connection","detail":"During secondary validation: Fetching http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8: Timeout after connect (your server may be slow or overloaded)","status": 400'
    [Tue Feb 25 08:40:11 CET 2020] errordetail='During secondary validation: Fetching http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8: Timeout after connect (your server may be slow or overloaded)'
    [Tue Feb 25 08:40:11 CET 2020] registry.mydomain.family:Verify error:During secondary validation: Fetching http://registry.mydomain.family/.well-known/acme-challenge/b7_N9T7DNNw6MiMC1P02YtbM8oXqGBKzcUE9QrDwwU8: Timeout after connect (your server may be slow or overloaded)
    

    acme_issuecert.log.txt [0_1582619230669_acme_issuecert.log](Uploading 100%)



  • @Foxi352
    Are you blocking any requests from outside? Like from different country's or something?

    Perhaps run a tcpdump / packetcapture on the wan interface (filtered on port 80) while the certificate is being requested? It should show if requests are being made, from where, and if indeed they are 'hanging'..?

    Also perhaps try 'the-other' haproxy package.? Which one and what version of it are you using? 'haproxy' or 'haproxy-devel' ?



  • @PiBa I use the 'simple' haproxy package. I am not blocking anything special. pfSense and all packages are on newest versions. I also disabled snort, pfblocker and everything else during debugging. As i said, it worked before moving from bhyve to proxmox. Or probably what counts more is before backup up and restoring on a clean install.

    But nevermind, i got it working now by using the haproxy chroot'ed webroot instead of standalone http server. After hours or fiddeling i have up on the other solution and switched all my certs to webroot. This is even faster as the script does not have to spin up an extra http server ...

    But thank you for taking the time to reply ;-)


Log in to reply