ACME sftp webroot validation fails / path issue?

[pfSense_user 0]

Hi all!
[Disclaimer: My technical grasp of the ACME validation process might be sketchy. However, I have done my best to describe my question. Thanks for your patience!]

(1) Situation:

• I am hosting a handful of domains on a provider's webspace (no physical server, no dedicated VM)

• I have a common ftp account for uploading content to all domains, which is structured /accountroot/domainfolder_1, /accountroot/domainfolder_2 etc. [see screenshot of ftp further below]

(2) Goal:

• I would like to issue and renew letsencrypt certificates for each domain via pfSense and have set up sftp webroot according to the documentation [see screenshot of pfSense ACME page]
  

(3) Problem:

• Validation fails every time [see attached message from pfSense GUI ACME package & attached ACME log]

GUI_pfSense-Services-ACME-Certificates.txt
log_acme_issuecert.txt

• ACME successfully writes to the webserver [see screenshot of ftp client]
  

• When I change the path for ACME to write to into /exampledomain_1/.well-known/acme-challenge/ , it successfully writes there, however, validation also fails.

(4) My tentative hypothesis

• Because my 'sftp root' is different from the individual 'domain roots', ACME tries to read back from a different filesystem location than it writes to.

(5) Questions

• Is my hypothesis plausible from the point of view of a more knowledgeable person?

• Is there a solution to my problem?

Any help ist greatly appreciated! Thanks!

[pfSense_user 0]

In restrospect, I may not have been clear enough under (4) in my original post.

What I mean is that, maybe, no matter what folder I tell ACME to write to, the readout attempt will always be misplaced by one layer of filesystem hierarchy.

• If ACME package writes to /sampledomain_1/.well-known/acme-challenge/

• The providers server will point an http request to www.sampledomain_1.tld/sampledomain_1/.well-known/acme-challenge, which does not exist,

• whereas the file would be found at www.sampledomain_1.tld/.well-known/acme-challenge

[pfSense_user 0]

Rephrasing Problem:
Seems NOT related to filesystem hierarchy

Please somebody help?

• pfSense ACME succesfully writes to the folder ftproot/domainfolder/.well-knowm/acme-challenge/ and, as far as I can interpret the logs, ACME also seems to try to veryfy there.

• I am able to access the token file via sftp client.

• I am NOT able to access the token file via web browser (http://my_domain/.well-known/acme-challenge/sdaQTOOENK9boBB6qzZf9fnRLOVhrRqzMQvc7_OXiFg with Safari or Firefox on macOS), however I am not sure, how a browser is supposed to handle the token file, which does not bear a filetype extension. [In order to upload the token file to the forum, I have appended ".txt"]

• Uploaded logs redacted for privacy (password, server etc.)
  

GUI-message-pfSense-ACME.txt

acme_issuecert.txt

sdaQTOOENK9boBB6qzZf9fnRLOVhrRqzMQvc7_OXiFg.txt

Could it be that somehow my provider blocks the verification process, because they want to sell their own overpriced TLS-certificate bundles?

[ http://my_domain.tld shows the expected index.html as usual, so port 80 is open on the providers webserver. ]

Has anybody out there tried ACME on pfSense with webspace from the provider "ionos" (formerly 1and1 / 1und1)?

[Gertjan]

@pfsense_user-0 said in ACME sftp webroot validation fails / path issue?:

sdaQTOOENK9boBB6qzZf9fnRLOVhrRqzMQvc7_OXiFg

The .well-known folder should be places in the webroot (often called 'www' folder)
In there, there should be the 'acme-challenge' folder.
In that folder, the random file name (your example) sdaQTOOENK9boBB6qzZf9fnRLOVhrRqzMQvc7_OXiFg should be placed - I've put 'Hi !' in that file ;)

I own the domain name test-domaine.fr.
So I tested :
https://www.test-domaine.fr/.well-known/acme-challenge/sdaQTOOENK9boBB6qzZf9fnRLOVhrRqzMQvc7_OXiFg

The thing is : sftp should place the file in the web server's root folder.

[pfSense_user 0]

@gertjan
Thanks for your reply!!!

I do not have access to the "true" root folder of the server, because I merely have rented a webspace from a provider.

However, in my understanding, on the webspace administration webpage, I have associated each domain that I host with them, with a specific subfolder on my webspace, which becomes the root folder for the logical webserver. Or is this thinking flawed?

Screenshot 1 shows my domains (only one SSL certificate is included in my contract ["Vertrag"]). The status column shows the respective folder, where all files of the domain in question reside (e.g. index.html, /css/... and so on)

Screenshot 2 shows the details for the one domain, for which I have tried ACME verification. The row "destination" ["Ziel"] again denotes the webspace folder, where all the domain's data reside.

I have tried to duplicate your example, and have exchanged the original token file for a file containing "Hi, too!" ;-) which can be seen in the appropriate location of the directory tree.

However, I can not view this file from a browser, although it is in the same directory (domain blued out), where index.html etc. work as expected.

[pfSense_user 0]

@gertjan
Addendum:

I have confirmed, that my .htaccess file for this domain does not include any prohibitions.

I can list the content of another subfolder (which is on the same file system hierarchy level as the .well-known folder) without problems:

For testing purposes, I have put an .htaccess file into the .well-known folder, which only contains "Options +Indexes". I am then able to view it's content (and thus it should be in the logical webroot of my domain):

However, I cannot enter the /acme-challenge folder from my webbrowser, even when I put another .htaccess with "Options +Indexes" there. whenever I click on the /acme-challenge link, nothing happens, the same as when I try to open any file contained within that directory by specifyint it's name un the URL.

[Gertjan]

@pfsense_user-0 said in ACME sftp webroot validation fails / path issue?:

However, I can not view this file from a browser, although it is in the same directory (domain blued out), where index.html etc. work as expected

That might be an issue.
If the web server, your host is controlling it - doesn't not 'want' to show files with unknown extensions, then the "web browser request" that LE makes when checking the file and it content won't work neither.

You have a domain name.
Most registrars these days give you an API access. Isn't it way easier to use some more classic DNS acme solution ?

[pfSense_user 0]

@gertjan
It gets even more weird: Once I move my test file (still without extension) one level up into the /.well-known folder itself, i can view and access it...

And:

The access problem seems to be limited to the /acme-challenge subfolder ::scratching head::

Unfortunately, the provider does not give access to API or automated DNS record changes with their affordable hosting contracts. Of course, they want customers to upgrade or to buy their SSL-certificates, both of which, however, are not competitively priced. I shy away from switching providers, because many e-mail adresses in use by the whole family depend on my domains, thus moving house would be a pain.

Thus, I am stuck with ftp validation, getting it to work would really, really be great!

[edited typos]

[pfSense_user 0]

@gertjan THANKS for pointing me towards the API!

Although fiendishly hidden away on their website, my provider IONOS offers free of charge sign-up to "developer APIs".

Now that I have a valid API key, and luckily, pfSense-acme has a preset for IONOS, I have tried that.

Both staging and production went through without an error message from pfSense-acme package.

Concurrent with validation

I could see a temporary TXT record in my administrative hosting page, which disappeared after validation finalized, which seems normal cleanup behaviour.

I just presume that the broken symbol on the first screenshot will be present during validation, whereas the tick indicates successful validation

However, I still get an SSL error when trying to access my domain [sorry, screenshot is in German, but it is the generic Firefox SSL error page; a corresponding error also shows up in Safari and in google chrome ("ERR_SSL_PROTOCOL_ERROR")

I have already

• flushed my pfSense resolver cache,

• flushed the browser cache,

• tried viewing the domain from my mobile over LTE,

same result.

Could this be a question of DNS-propagation? I presumed that the tick implied finalized verification and propagation, but I might be wrong.

@gertjan Thanks for bearing with me!

[Gertjan]

@pfsense_user-0

Where is this web server ?
The certificate(s) is ready to be used, and you can see it here :
System > Certificate Manager > Certificates

Now yo have to export it, and bring them over to server, there where the web server (like apache2, nginx, etc) runs, so it can use that certificate.

The acme pfsense package was created so you could get a cert for pfSense, the web GUI. For any other device, you have to copy the certificate over to that device.
Or write a script that automates that process, and restarts the web server on that device so the the new cert is taken in account.
The effect will be immediate.

[pfSense_user 0]

@gertjan Ehemm... It is hard for me to admit that I was so naive to think that, with a IONOS preset and an API key, pfSense-acme would also magically, on validation, put the certificate onto the server.

Thanks for the clarification and all your help!

[pfSense_user 0]

@gertjan Unfortunately, after all this work, there seems to be no way for me to upload the letsencrypt certificate to my IONOS webspace.

The API only allows for management of certificates purchased from IONOS. And I do not have access to the SSL root, nor can I restart the server.

[Gertjan]

@pfsense_user-0 said in ACME sftp webroot validation fails / path issue?:

IONOS webspace.

If the webserver is controlled by the host, then you can't do things like 'add' a certificate.
For the certificate to be taken in account, the web server has to be restarted.
Or, 'your' web server is shared with hundreds if not thousand other users ....

But : there is still good news : every web hosting company offers signed certs (mostly from LE) as every web server is using https these days. The host company will take care of everything.
"http" isn't used any more, Google (and others) isn't even indexing them.

[pfSense_user 0]

@gertjan I know about http pointing to Nirvana these days ;-) That's why I am desperately looking for a solution...

Unfortunately, the good news ist not so good with IONOS, as they know about their customer's desperation for SSL certificates. Their pricing policy for adding certificates to existing domains is rather steep. All my websites are hobby projects, so they don't create any revenue. IONOS exclusively offers GeoTrust QuickSSL certificates and offers them in bundles at a hefty premium.

Maybe it is time to cancel some of my domains...

[Gertjan]

@pfsense_user-0

I'm not affiliated with these guys, but https://www.ovhcloud.com/de/web-hosting/ and you have it all - and even more.
Certs are LE so not your issue.

I was using them for years (web hosting), but then my web sites became to big - their mail (MX) handling is good for average guy, but not for a company. I went bare bone 'dedicated server' (not some fog based device). Solved many issued.

[pfSense_user 0]

@gertjan Thanks for the provider info, and also for all of your replies, which were extremely helpful to me! Kind regards.