Failure updating ACME certificate
-
@Gertjan Admitttedly, I manage to get myself confused. I somewhat recall that in order to originally issue the Letsencrypt certificate, that I had to disable the port forwarding and have forgotten all the details of that.
I have several web admin interfaces on the network that I have had to create a port forward for and I tend to get confused about what port is open on what server/device.
So, the issue I suppose I need to answer is, what is the best practice for setting up the certificate renewal AND getting the renewed certificate into the correct location on my web server.
Currently, my configuration is to forward port 80 and 443 incoming from the WAN to my Fedora web server. This server resides on its own subnet. I created a separate port forward for internal requests to the web server, where the DNS lookup returns the INTERNAL IP for requests coming from the LAN side.
If I understand correctly, the acme.sh script is creating a listener, also on port 80, but on the pfsense. On the router, port 80 is forwarded to the web server, so the script listener never receives the responses it's expecting.
I was thinking that the .well-known was attempting to be created on the web server, thus my confusion. In actuality, it seems the .well-known is actually a path on the pfsense itself, not the web server, so I think this gets at the WHY it's not working.
This now brings me to the question of best practices for setting this up so that it works without or minimal manual intervention. It seems that the acme.sh script needs to have its own listen port that sees the incoming request rather than forwarding to the web server. This raises a few issues:
- The acme script needs a dedicated listen port for "the socal mini-web-server".
- Subsequently, the chosen port must also be open to requests incoming on the WAN side for the request to succeed.
- Currently, the incoming request is being forwarded to the web server and NOT seen by the acme.sh script.
Assuming I'm on the right track, there are a few key questions:
- Do I need to change the listen port since port 80 is already being forwarded to the web server?
- Does this listen port need to be accessible 24/7 or can I use a cron job, or other method to open the port only when I know the renewal will need access?
- Are there any security issues concerning the acme.sh script or the socal listener?
- Once the renewal issue is resolved, how do I replace the renewed certificate on the web server in order to keep a working HTTPS certificate?
I still have something like 24 days before the certificate expires.
Additional info:
I have several web accessible devices/servers on various subnets, and several associated port forwards that would "normally" be listening on ports 80/445. Most of these are only accessible from an internal LAN request. Examples are:- HAOS on a Raspberry Pi, including a port for Node Red.
- Hubitat hub.
- Fedora server admin web interface.
- Fedora server terminal emulation interface.
- Fedora web server on ports 80/445.
These listeners reside on multiple subnets, with firewall rules attempting to maintain security across subnets. It took a little while to everything working as expected. The holdout to be resolved is getting this acme.sh and certificate renewal resolved.
I think I have a better understanding of what is happening, but I'm not entirely sure the best way to resolve this.
Thanks!
-
@AudioDave seems like you have it overly complex.. I run a couple of services behind pfsense that are open to the public internet.. But I just ha proxy handle all the ssl.. And just update the acme certs via dns..
ha proxy can direct to your different backends based on the fqdn.. If they are all in the same domain you could just use 1 cert (wildcard) and only need to update 1 cert, etc.
-
@johnpoz It may be overly complex, no opinion on that. As it is, everything is working EXCEPT the automatic renewal of the server certificate for my web server. Prior to changing out my router for a pfsense router, I had longer duration certificate(s), but since then there have been limits put on the duration of certificates to 90 days. Thus my need to get this working.
All the other port forwards occur only for access from the LAN/VLAN side for security.
The ONLY access from the WAN side is ports 80/445 for the web server, and standard ports for OpenVPN and IPsec for VPN access. The VPN ports can easily be disabled when not needed to reduce the footprint for external port scans.
I do not access most of these web admin interfaces from the WAN side.
I certainly can reconfigure if there's a compelling reason to do that. However, I do not want to open another can of worms unnecessarily. I'm only trying to resolve what I've done wrong for the certificate renewals to function.
-
@AudioDave said in Failure updating ACME certificate:
I had longer duration certificate(s), but since then there have been limits put on the duration of certificates to 90 days.
ACME has never been longer than 90 days... Yeah if you got your certs from some CA then you could of have longer..
Maybe you manually installed these certs that you had for longer than 90 days?
If you do not even access them via outside, why even have them available? Just offering up a suggestion to simplify your cert management.. If you are going to use ACME for your cert needs.. If you have multiple services that are accessed from the public, I would just put them behind haproxy, this allows you to handle all the ssl for all over your services in one spot. If you want your services to handle their own certs, then run the acme client on those boxes to handle their certs.
Either way I would use dns method of validating your certs.
If you have your ssl offloaded via haproxy - you can access your services via ha proxy while your internal to your network.
-
@johnpoz said in Failure updating ACME certificate:
Maybe you manually installed these certs that you had for longer than 90 days?
If you do not even access them via outside, why even have them available? Just offering up a suggestion to simplify your cert management. If you are going to use ACME for your cert needs.. If you have multiple services that are accessed from the public, I would just put them behind haproxy, this allows you to handle all the ssl for all over your services in one spot. If you want your services to handle their own certs, then run the acme client on those boxes to handle their certs.
Either way I would use dns method of validating your certs.
This is getting a little off track. If there is a compelling reason to redo everything, I'm certainly willing to reconsider. However, my original question is simply how to resolve the fact that the automatic renewal is failing.
I was previously using a self-signed certificate that I created using OpenSSL. Newer browser versions do not appear particularly happy with self-signed certificates from an untrusted CA.
Will I open some of these services to access from the WAN at some point? Maybe. Most of these I simply configure from the LAN side, but still have configured such that the certificates work with ssl/https. The DNS query on the LAN side returns an internal IP to keep the URL happy.I don't understand your comment about "using the DNS method". How does that differ from what I'm attempting and how would I get to that result?
I'm not sure where to go with your suggestions, partly because I do not understand the how or why to go that route. Can you point to example(s) or documentation?
Thanks! Not meaning to be argumentative. I simply want to address the issue of automatic certificate renewal. If I need to describe my complete setup to get to an answer, I can certainly attempt to do that, providing it's necessary to answer the question(s). I did not keep extensive notes about how I got to the current configuration. I simply resolved each issue I discovered as I progressively got things working after switching out a Zyxel router for a pfsense one.
p.s. I think what I did to get the original certificate was simply to disable the port forwarding on port 80, which allowed the script to get the responses from Letsencrypt. Perhaps I will try that for the renewal as well, if nothing else to confirm the issue.
-
@AudioDave Just trying to give you an easy way to manage certs for multiple devices behind pfsense... But hey you do you..
I simply want to address the issue of automatic certificate renewal.
Well if you want to use the web server approach then yeah you would have to open up pfsense wan if you want acme on pfsense to validate.. If you want something behind pfsense to use certbot and renew its certs then you would have to forward the port to the client.
Or just use dns method where ever you run the lets encrypt script to renew a cert... But leaving some port open to something behind pfsense is not going to renew the cert on the acme client on pfsense.. Because that would have the correct info that is created when it checks, etc. and the cert wouldn't be on your remote device even if it did work..
-
@AudioDave have you read https://docs.netgate.com/pfsense/en/latest/packages/acme/settings-validation.html This should help to illustrate the DNS method that is being encouraged here.
--Larry
-
@LarryFahnoe I have not but will check it out. Thanks!
-
@AudioDave said in Failure updating ACME certificate:
However, my original question is simply how to resolve the fact that the automatic renewal is failing.
The web root method needs an "open port TCP 80" ending up on the WAN interface of pfSense.
If you already have that port forwarded else - using a NAT rule - you have a conflict, as enabling and disabling NAT and/or firewall rules isn't going to happen.I'm not sure, but :
=> It must be port 80 or port 443 - and the help text says so : traffic must arrive at the WAN, so if need, take car of upstream NAT if you have an upstream ISP router.
I've read some of the https://www.google.com/search?client=firefox-b-d&q=ddns.net+letsencrypt and it really looks like a "yes, it is possible, but is is a pure pain to handle".
By far, I prefer the original solution 'DNS' and this needs a 'real' domain name that you rent (la couple of $ a year) with a capable registrar that supports a DNS method, and call it a day.
Not that the webroot / standalone method is bad : if you didn't have any ports forwarder into your network, it doesn't matter that ones in 60 days, a 'listener' (the socat tool) fires up to handle the incoming request from Letsencrypt, but : you still have an open hole in your WAN.Far more better (= easier) would be : as said : use a DNS method. Check if your registrar which one it supports, and use it : no more hassle, no more 'ports to open' (could be a a security issue).
Not that it is the best way, but I've pfSense acme asking for a wildcard certificate for my internal network domain name, and copy it over to some other devices (nas, printers etc) ones every 60 days. Not that this really needed, I could still access their GUI anyway. -
@Gertjan Well, this is all just to keep encryption (HTTPS) on a small forum on my network and has already been a PITA for a small amount of benefit.
I'm not a network expert, so I end up investing time to understand and implement this and frankly, it's becoming less attractive to manage it.
I'll dig a little more, but it may ultimately not be worth the trouble. I may also revert to how I originally managed certificates, which removes the 90-day restriction. I only really moved to Letsencrypt because it appeared that managing certs had been automated and that the certificates were from a "valid" CA.
No free lunch, I suppose!
I'm using a free dynamic DNS provider and am not motivated to pay fees for a little-used server. I may just drop the idea entirely.
Even for the VPN access, I've managed years without this. It just seemed it would be nice to have a VPN for use on the road. Even there, I find myself monitoring logs to track port scanners banging on the open ports. Ultimately, it's becoming less attractive.
-
@AudioDave said in Failure updating ACME certificate:
I'm using a free dynamic DNS provider and am not motivated to pay fees for a little-used server. I may just drop the idea entirely.
If your current DNS provider is amongst those listed in the ACME configuration, this is a very easy route & gives you the low maintenance automated that you're looking for. Easy enough to set up a test cert to see if/how the dynamic DNS bits work with ACME and then decide if you want to use it to generate other certs.
--Larry
-
@AudioDave said in Failure updating ACME certificate:
However, my original question is simply how to resolve the fact that the automatic renewal is failing
I did point out your problem several days ago and what you needed to do.
