Failure updating ACME certificate
-
I am getting an error attempting to renew a certificate via the Services/Acme/Certificates, clicking on the Issue/Renew button:
-
When you ask acme to get a certificate for you, some preparation is needed.
the acme .sh scripts contacts Letsencrypt with it's intention, and Letsencrypt will give a random file name, this one :and in this file is placed another random number.
Btw : these numbers have only a meaning at that very moment, for your domain. They are not valid - can not be re used etc. No need to hide them ^^The file name is use d by the acme.sh script to create file file in the web root directory, in the directory called ".well-known". The acme.sh uses the "dnsapi" you've selected to create this file, and put in the "secret" content.
When done, it signals "Letsencrypt : go ahead : check please".
Letencrypt will do a DNS lookup to find your
login-to-view
and when it got an IP (A or AAAA record) it will connect to your web server and ask for the file :
login-to-view
in the ".well-known" folder.
The content of this file should be the "code" that Letensrypy has given to the acme.sh script.The thing is : the file wasn't created. Letsencrypt got a 404 (file not found) error.
Here :
you will find more info.
What is the dnsapi method you've chosen ?
Was it working before ? -
@AudioDave Check to make sure the port is open.
-
@Gertjan I'll have to investigate further. I used the certificate manager to obtain the original certificates (CA and Server) and I am now receiving a notice that the server certificate is approaching expiration, and I am attempting to use the Issue/Renew the certificate.
This is the first time I'm attempting a renewal, so I'm not sure how to answer your inquiry as to "working before".
My first review of the log file did not reveal anything obvious to me.
Not sure how to answer your question regarding DNS API.
-
@Popolou Not sure what "the port" is referring to.
-
Upon revisiting this issue, I am looking at the System/Certificates/Certificates page and the server certificate I am attempting to renew is not showing the icon for reissue/renew for that individual server license.
-
What have you entered here (bottom of page where you've set up your cert ) :
Under Services > Cron [install the pfSense CRON package ] > Settings you'll find :
which means that my acme is run every day at 03h16 acme checks if it is time to renew :
If this auto renewal process fails, it time to look for the 'why' question.
It's here : /tmp/acme/[your-cert-name]/ and in this folder you'll find a file called "acme_issuecert.log"@AudioDave said in Failure updating ACME certificate:
Not sure how to answer your question regarding DNS API.
It's the method acme uses so letsencrypt can test that you are the 'owner' of your domain :
For example, I use the good old RFC2136 :
-
Here's the renewal date. I'm assuming the default 60 days.
Here's the cron entry:
I'm noticing an error sending email. I'll have to verify why this is occurring. Looks like an issue with authorization, so email may be misconfigured.
You can see the renewal command being issued. I'm wondering if HTTP port 80 is part of the issue. Does this need to be HTTPS port?
Checking the log file, it is seeing status 403 and 404:
[Thu Mar 7 03:16:19 CST 2024] error='"error":{"type":"urn:ietf:params:acme:error:unauthorized","detail":"174.62.133.164: Invalid response from http://burrisaudio.ddns.net/.well-known/acme-challenge/VHEP-Y8LRTIgddsxAiXXXXXXXXXXXXXG6Jl2f_m6Az05KQ: 404","status": 403'
[Thu Mar 7 03:16:19 CST 2024] errordetail='174.62.133.164: Invalid response from http://burrisaudio.ddns.net/.well-known/acme-challenge/VHEP-Y8LRTIgddsxAiXXXXXXXXXXXXXG6Jl2f_m6Az05KQ: 404'
[Thu Mar 7 03:16:19 CST 2024] burrisaudio.ddns.net:Verify error:174.62.133.164: Invalid response from http://burrisaudio.ddns.net/.well-known/acme-challenge/VHEP-Y8LRTIgddsxAiXXXXXXXXXXXXXG6Jl2f_m6Az05KQ: 404First things first, I need to fix the email failure. Then, assuming this is not the cause, I will need to determine why this is failing. I obscured the actual file path, though apparently it is not needed, per your earlier response.
-
The email issue appears to be resolved. It was a password issue.
-
My web root is /var/www/html
In that directory, I do not see $PATH/.well-known/*
I'm assuming that since a script is trying to create this, that I have a permissions issue. Do I need to make changes to .htaccess or file/directory permission changes so the script can create the file(s)?
Thanks!
-
@AudioDave said in Failure updating ACME certificate:
My web root is /var/www/html
Not sure if that path is even used.
When you use the 'standalone' mode, acme.sh will use 'socat'.
This is a freebsd binary that will listen on the port and address you've given.
It will behave like a very 'low bud' web server.This means that Letsencrypt will connect to your URL, and that this request should wind up on your WAN interface. If you have an upstream router : don't forget the port 80 TCP NAT rule to pfSense (!)
This means that you need to allow port 80 TCP (or whatever port you've chosen) on your WAN interface, otherwise Letencrypt will not be able to reach your 'socat mini web server'.
Just keep in mind that the pfSense GUI can also use port 80, TCP - this can create conflicts. -
@Gertjan Admitttedly, I manage to get myself confused. I somewhat recall that in order to originally issue the Letsencrypt certificate, that I had to disable the port forwarding and have forgotten all the details of that.
I have several web admin interfaces on the network that I have had to create a port forward for and I tend to get confused about what port is open on what server/device.
So, the issue I suppose I need to answer is, what is the best practice for setting up the certificate renewal AND getting the renewed certificate into the correct location on my web server.
Currently, my configuration is to forward port 80 and 443 incoming from the WAN to my Fedora web server. This server resides on its own subnet. I created a separate port forward for internal requests to the web server, where the DNS lookup returns the INTERNAL IP for requests coming from the LAN side.
If I understand correctly, the acme.sh script is creating a listener, also on port 80, but on the pfsense. On the router, port 80 is forwarded to the web server, so the script listener never receives the responses it's expecting.
I was thinking that the .well-known was attempting to be created on the web server, thus my confusion. In actuality, it seems the .well-known is actually a path on the pfsense itself, not the web server, so I think this gets at the WHY it's not working.
This now brings me to the question of best practices for setting this up so that it works without or minimal manual intervention. It seems that the acme.sh script needs to have its own listen port that sees the incoming request rather than forwarding to the web server. This raises a few issues:
- The acme script needs a dedicated listen port for "the socal mini-web-server".
- Subsequently, the chosen port must also be open to requests incoming on the WAN side for the request to succeed.
- Currently, the incoming request is being forwarded to the web server and NOT seen by the acme.sh script.
Assuming I'm on the right track, there are a few key questions:
- Do I need to change the listen port since port 80 is already being forwarded to the web server?
- Does this listen port need to be accessible 24/7 or can I use a cron job, or other method to open the port only when I know the renewal will need access?
- Are there any security issues concerning the acme.sh script or the socal listener?
- Once the renewal issue is resolved, how do I replace the renewed certificate on the web server in order to keep a working HTTPS certificate?
I still have something like 24 days before the certificate expires.
Additional info:
I have several web accessible devices/servers on various subnets, and several associated port forwards that would "normally" be listening on ports 80/445. Most of these are only accessible from an internal LAN request. Examples are:- HAOS on a Raspberry Pi, including a port for Node Red.
- Hubitat hub.
- Fedora server admin web interface.
- Fedora server terminal emulation interface.
- Fedora web server on ports 80/445.
These listeners reside on multiple subnets, with firewall rules attempting to maintain security across subnets. It took a little while to everything working as expected. The holdout to be resolved is getting this acme.sh and certificate renewal resolved.
I think I have a better understanding of what is happening, but I'm not entirely sure the best way to resolve this.
Thanks!
-
@AudioDave seems like you have it overly complex.. I run a couple of services behind pfsense that are open to the public internet.. But I just ha proxy handle all the ssl.. And just update the acme certs via dns..
ha proxy can direct to your different backends based on the fqdn.. If they are all in the same domain you could just use 1 cert (wildcard) and only need to update 1 cert, etc.
-
@johnpoz It may be overly complex, no opinion on that. As it is, everything is working EXCEPT the automatic renewal of the server certificate for my web server. Prior to changing out my router for a pfsense router, I had longer duration certificate(s), but since then there have been limits put on the duration of certificates to 90 days. Thus my need to get this working.
All the other port forwards occur only for access from the LAN/VLAN side for security.
The ONLY access from the WAN side is ports 80/445 for the web server, and standard ports for OpenVPN and IPsec for VPN access. The VPN ports can easily be disabled when not needed to reduce the footprint for external port scans.
I do not access most of these web admin interfaces from the WAN side.
I certainly can reconfigure if there's a compelling reason to do that. However, I do not want to open another can of worms unnecessarily. I'm only trying to resolve what I've done wrong for the certificate renewals to function.
-
@AudioDave said in Failure updating ACME certificate:
I had longer duration certificate(s), but since then there have been limits put on the duration of certificates to 90 days.
ACME has never been longer than 90 days... Yeah if you got your certs from some CA then you could of have longer..
Maybe you manually installed these certs that you had for longer than 90 days?
If you do not even access them via outside, why even have them available? Just offering up a suggestion to simplify your cert management.. If you are going to use ACME for your cert needs.. If you have multiple services that are accessed from the public, I would just put them behind haproxy, this allows you to handle all the ssl for all over your services in one spot. If you want your services to handle their own certs, then run the acme client on those boxes to handle their certs.
Either way I would use dns method of validating your certs.
If you have your ssl offloaded via haproxy - you can access your services via ha proxy while your internal to your network.
-
@johnpoz said in Failure updating ACME certificate:
Maybe you manually installed these certs that you had for longer than 90 days?
If you do not even access them via outside, why even have them available? Just offering up a suggestion to simplify your cert management. If you are going to use ACME for your cert needs.. If you have multiple services that are accessed from the public, I would just put them behind haproxy, this allows you to handle all the ssl for all over your services in one spot. If you want your services to handle their own certs, then run the acme client on those boxes to handle their certs.
Either way I would use dns method of validating your certs.
This is getting a little off track. If there is a compelling reason to redo everything, I'm certainly willing to reconsider. However, my original question is simply how to resolve the fact that the automatic renewal is failing.
I was previously using a self-signed certificate that I created using OpenSSL. Newer browser versions do not appear particularly happy with self-signed certificates from an untrusted CA.
Will I open some of these services to access from the WAN at some point? Maybe. Most of these I simply configure from the LAN side, but still have configured such that the certificates work with ssl/https. The DNS query on the LAN side returns an internal IP to keep the URL happy.I don't understand your comment about "using the DNS method". How does that differ from what I'm attempting and how would I get to that result?
I'm not sure where to go with your suggestions, partly because I do not understand the how or why to go that route. Can you point to example(s) or documentation?
Thanks! Not meaning to be argumentative. I simply want to address the issue of automatic certificate renewal. If I need to describe my complete setup to get to an answer, I can certainly attempt to do that, providing it's necessary to answer the question(s). I did not keep extensive notes about how I got to the current configuration. I simply resolved each issue I discovered as I progressively got things working after switching out a Zyxel router for a pfsense one.
p.s. I think what I did to get the original certificate was simply to disable the port forwarding on port 80, which allowed the script to get the responses from Letsencrypt. Perhaps I will try that for the renewal as well, if nothing else to confirm the issue.
-
@AudioDave Just trying to give you an easy way to manage certs for multiple devices behind pfsense... But hey you do you..
I simply want to address the issue of automatic certificate renewal.
Well if you want to use the web server approach then yeah you would have to open up pfsense wan if you want acme on pfsense to validate.. If you want something behind pfsense to use certbot and renew its certs then you would have to forward the port to the client.
Or just use dns method where ever you run the lets encrypt script to renew a cert... But leaving some port open to something behind pfsense is not going to renew the cert on the acme client on pfsense.. Because that would have the correct info that is created when it checks, etc. and the cert wouldn't be on your remote device even if it did work..
-
@AudioDave have you read https://docs.netgate.com/pfsense/en/latest/packages/acme/settings-validation.html This should help to illustrate the DNS method that is being encouraged here.
--Larry
-
@LarryFahnoe I have not but will check it out. Thanks!
-
@AudioDave said in Failure updating ACME certificate:
However, my original question is simply how to resolve the fact that the automatic renewal is failing.
The web root method needs an "open port TCP 80" ending up on the WAN interface of pfSense.
If you already have that port forwarded else - using a NAT rule - you have a conflict, as enabling and disabling NAT and/or firewall rules isn't going to happen.I'm not sure, but :
=> It must be port 80 or port 443 - and the help text says so : traffic must arrive at the WAN, so if need, take car of upstream NAT if you have an upstream ISP router.
I've read some of the https://www.google.com/search?client=firefox-b-d&q=ddns.net+letsencrypt and it really looks like a "yes, it is possible, but is is a pure pain to handle".
By far, I prefer the original solution 'DNS' and this needs a 'real' domain name that you rent (la couple of $ a year) with a capable registrar that supports a DNS method, and call it a day.
Not that the webroot / standalone method is bad : if you didn't have any ports forwarder into your network, it doesn't matter that ones in 60 days, a 'listener' (the socat tool) fires up to handle the incoming request from Letsencrypt, but : you still have an open hole in your WAN.Far more better (= easier) would be : as said : use a DNS method. Check if your registrar which one it supports, and use it : no more hassle, no more 'ports to open' (could be a a security issue).
Not that it is the best way, but I've pfSense acme asking for a wildcard certificate for my internal network domain name, and copy it over to some other devices (nas, printers etc) ones every 60 days. Not that this really needed, I could still access their GUI anyway.
