Automating Certificate imports with letencrypt script

tylerhoadey

Hey All,

I'm in the process of flipping my CA from StartSSL to Lets Encrypt SSL Certs. I have setup a Docker container to automate the 90 day renewal process and have scripted the whole process to my servers/services that require the Certificates, chain and key, (Haproxy, apache, webmin etc)

The problem I'm trying to over come is automating this into pfsense webui. I've tried replacing the /var/etc/cert.pem file and restarting the webui but by the looks of the /etc/rc.restart_webgui, this calls the system.inc file and replaces it with the default set.

So I'm curious where the existing certificates are stored? DB, cache files or if someone has a curl POST script handy in order to push it into the Certificate Manager and setting as WebUI default.

Currently I'm doing this all local on the lets encrypt container so would be ideal to include pfsense right inside the same script, however if I have to do this locally on pfsense, via shell, thats fine too. Just want a way to automate it. every Post I've come across relies on point and click methods (I'm a *nix guy).

Any help would be appreciated

Cheers,

Update

I've uploaded my code to github
https://github.com/tylerhoadley/letsencrypt-automation

Gertjan

You were close.
Everything is in the one and only config file : /conf/config.xml :)

tylerhoadey

Ok that makes sense. so everytime its reloads, its that function what writes out the systems config replacing the static confs.

i was looking at this file prior to posting and was trying for a scp or curl method. now however ill have to craft a script to replace the lines in cert manager portion, then reload the config. ill have some google'in to do.

if anyone has code snippit to do this replace, and reload, that would be great. i can post my end method once i get it setup and tested.

thanks

radioruck

@tylerhoadey:

i was looking at this file prior to posting and was trying for a scp or curl method. now however ill have to craft a script to replace the lines in cert manager portion, then reload the config. ill have some google'in to do.

Any luck with scripting an update to config.xml?

Gertjan

It's not only the scripting …

Much has been said about implementing LetEncrypt into pfSense, like https://redmine.pfsense.org/issues/5434#note-2
( Use Google to find them alll )

Install LetEncrypt on your own (web) server on the net, just to see what happens ..... it is somewhat scarry, this huge pile of dependencies ...
Remember : pfSense is just a "firewall".

radioruck

I've been looking at using https://github.com/Neilpang/acme.sh. Haven't done a dry run yet but in planning I think copying certs into config.xml is my missing puzzle piece.

tylerhoadey

Sorry, no update… haven't had the time to sit down and build the regex replace. figured I would get back to this when the 90 day cert is due for renewal (and I'll be able to test it all from start to finish)

30 days or so left, so will slot some time out to get it done.

radioruck

I did come across this which seems like a pretty straightforward way of editing the config: https://forum.pfsense.org/index.php?topic=56956.msg304026#msg304026

I need to dig in a little more to see how the certs are stores in config.xml and what I can key off of to make the update like this.

tylerhoadey

Ok, spend a bit of time this AM getting the pattern down, and reading advance sed substitution. It hasn't been fully tested but believe the guts of the task are done, with hopefully minor tweaks.

Just to ensure everyone reading this is aware, I will be running this from a dedicated Lets Encrypt server I have that renews and copies my certs to few different servers and services, apache, Haproxy etc, I use ssh keys to seamlessly copy/execute from LE server to pfsense.

the script below is using variables so adjust to your needs.


#!/bin/bash

# Letencrypt certificate filename (and path)
FILE=test.crt

# pfsense server name or IP
SERVER=pfsense.example.com

#remove newlines and Cert tags from certificate file

CERT=`grep -v 'CERT' $FILE | grep '^[^<]' | tr -d '\n'`

# copy the template to a sed script file
cp pattern.template pattern.sub

# replace the placeholder string in the pattern file with certificate information

sed -i 's/REPLACEHOLDER/$CERT/g' pattern.sub

# scp the pattern file to the pfsense system
scp pattern.sub $SERVER:/tmp/

# execute sed replace against the config.xml and reload the configuration

ssh $SERVER 'sed -f /tmp/pattern.sub /conf/config.xml'
ssh $SERVER 'rm /tmp/config.cache'

as you can read above, I'm using sed -f to minimize the special characters and to match multi line substitution. seem to be simpler to format and test. I've also had to use a template approach as I don't want to keep matching older certs to insert new certs. that could be a pain, and I like to keep it as simple as I can (or at least for now).

create the pattern.template file

*Note LetsEncrypt is the name of the certificate in pfsense certificate manager. aka the "Descriptive name" field. I recommend you keep this 100% unique within the configuration when you do the initial import. this approach assumes you have imported at least once in order to replace it


/.*LetsEncrypt.*/{
		N
		/<crt>.*<\/crt>/{
		s//<crt>REPLACEHOLDER<\/crt>/
		P
		D
		}
}</crt></crt>

Please don't use this directly against your "true" config.xml unless you have done further testing. I've uploaded this before I've done more than just quick testing against copied files.

I have not done true Production testing against my firewall yet… although thats next on my list.

Use at own risk.

*** I have an update (escape characters were driving me crazy on the $CERT sed replace... I have changed it to use awk and have also inserted backslashes in to the CERT= variable) will post update tomorrow.

tylerhoadey

Like I posted yesterday, I believe I'm really close to having this down. Had to overcome escape characters issues with the sed script and the certs character '/' being passed, but now I see a bigger problem that I didn't see coming. Hoping a PFSense expert can help me out here.

The issue I see is that in the config.xml, the cert and chain look like there hashed or in a different cert format.
example.

in the config.xml has my cert and chain starting with LS and ending in ==

when I replace the cert in the file, its in the original pem format.

What format does the config.xml present the cert and chain in?

If I can prepare the cert for this format and insert… Then I have a working model script that will do this automation.

*** base64 encoding
-- php snippit from system_certmanager.php --
$exp_data = base64_decode($a_cert[$id]['crt']);

–

tylerhoadey

Working model. tested and confirm certificate usage. * had to include the key, as Lets encrypt changes the default key on new generation/request.

it requires php5 (CLI) on lets encrypt server.

Remotescript.sh


#!/bin/bash

PHP=`which php`

# Letencrypt certificate and chain filename (and path)
# 
# CHANGE THESE TO YOUR CERT/CHAIN PATH 
# 
CRT=/etc/letsencrypt/live/DNS/fullchain.pem
KEY=/etc/letsencrypt/live/DNS/privkey.pem

# pfsense server name or IP (using ssk keys for passwordless login)
#
SERVER=192.168.2.1

# use php base64_encode function to convert certificate and chain
#
ENCRT=`$PHP -r '$cert = file_get_contents( $argv[1] , true);  echo base64_encode("$cert");' $CRT`
ENKEY=`$PHP -r '$key = file_get_contents( $argv[1] , true);  echo base64_encode("$key");' $KEY`

# replace the placeholder string in the pattern template with certificate encoded information.
# redirect it out to the sub file so it can be scp to the pfsense server
#
cat pattern.template | awk '$1=$1' FS="CRTPLACEHOLDER" OFS="$ENCRT"  | awk '$1=$1' FS="KEYPLACEHOLDER" OFS="$ENKEY" > pattern.sub

# scp the pattern file to the pfsense system
#
scp pattern.sub $SERVER:/tmp/

# execute sed replace against the config.xml and reload the configuration
#
ssh $SERVER 'cp /conf/config.xml /tmp/config.xml && sed -f /tmp/pattern.sub < /tmp/config.xml > /conf/config.xml && rm /tmp/config.cache && /etc/rc.restart_webgui'

the above code, reads in the cert and chain files, encodes them with php (which makes sense when dealing with the special characters in the certs). we then use a sed template to replace the Placeholder with the base64 encoding. once that pattern.sub is ready, we scp and run it with sed.

create the template file, Change LetsEncrypt to the description name you used to import. I would recommend keeping this unique or understand this script is looking for any'NAME'any with <crt…crt>in the next line.

pattern.template


/.*LetsEncrypt.*/{
		N
		/<crt>.*<\/crt>/{
		s//<crt>CRTPLACEHOLDER<\/crt>/
                }
                N
                /<prv>.*<\/prv>/{
                s//<prv>KEYPLACEHOLDER<\/prv>/
                P
                D
                }

}</prv></prv></crt></crt>

Of course I know there is some clean up and maybe a few things that could be done more efficiently, so please feel free to improve and optimize. this will get me by for now but would be open to removing php base64_encode function out and use base64 command line (more of a time issue right now to research)… pretty sure the sed -f command could be cleaner too with inplace rewriting on the file.</crt…crt>

dig1234

Just curios why you're using a dedicated Letsencrypt server? There is a an sh script which runs perfectly on pfsense with no dependencies. I used this myself with success. If someone could add the key insertion script into the cron job we'd be all set. Check it out at:
https://github.com/Neilpang/acme.sh

Regarding the resistance to supporting Lets Encrypt in pfsense I don't see how not using a letsencrypt cert makes anyone safer. If the skeptics are right that illegitimate LE certs are easy to obtain (which I'm not convinced of), that affects us all whether we use LE Certs or not. Fact is the LE CA is trusted (by default) whether they like it or not. Someone correct me if I'm missing something.

tylerhoadey

In my environment it makes sense to use a dedicated server (which I will convert to a docker container). My ssl termination all happens behind pfsense. Don't get me wrong pfsense I know has lots of features, and have use haproxy amoung other services on it prior. However I like using pfsense for true firewall and nat rules (with dedicate resources for that purpose). I use other scale-able systems in the background to ensure services availability. and in my case a dedicated process in order to renew the LE certs. this keeps my systems clean of other dependencies.

I am glad to hear you were able to get it running on pfsense. Like I said pfsense is feature rich and I know a lot of chatter out there on getting this component native or as an install package (and I am for it) Thats what makes pfsense fit for many use cases and setups.

hvisage

Firstly: Thank you @tylerhoadey for the sed/php scripting you shared 8)
Secondly: Thank you @dig1234 for pointing to NeilDang's acme.sh

https://gitlab.com/snippets/26220

Now just my modifications to make it run on the pfSense it self:

install acme.sh, I used:

curl https://get.acme.sh | sh

mkdir /root/le and put script.sh and pattern.template (below) in there.
Script.sh added some test for file existence, and pattern.template I somehow found the need to remove the tabs/spaces to make it work on pfSense/FreeBSD's sed.
chmod +x script.sh ;)
I disabled the webConfigurator's use of port 80, but allowed port 80 on the rules to use the standalone mechanism

5) ```
/root/.acme.sh/acme.sh --issue --standalone -d <domainname></domainname>


6) The important part here is to now to add this certificate in the Certificate manager with the name atleast containing "LetsEncrypt"
Use  /root/.acme.sh/<domainname>/fullchain.cer for the certifiacte
and the private key would be /root/.acme.sh/<domainname>/<domainname>.key. The string LetsEncrypt in what is used by the sed pattern below to find the right certificate to update.

7)```
/root/le/script.sh <domainname></domainname>

My next steps would be to run /root/.acme.sh/.acme.sh –renewAll weekly/monthly from cron, followed by /root/le/script.sh <domainname>script.sh

#!/bin/sh
#
# Thanks to tylerhoadey
#  https://forum.pfsense.org/index.php?action=profile;u=282215

PHP=`which php`

# Letencrypt certificate and chain filename (and path)
#
# CHANGE THESE TO YOUR CERT/CHAIN PATH
#
DNS=$1
CRT=/root/.acme.sh/${DNS}/fullchain.cer
KEY=/root/.acme.sh/${DNS}/${DNS}.key

# use php base64_encode function to convert certificate and chain
#
ENCRT=`$PHP -r '$cert = file_get_contents( $argv[1] , true);  echo base64_encode("$cert");' $CRT`
ENKEY=`$PHP -r '$key = file_get_contents( $argv[1] , true);  echo base64_encode("$key");' $KEY`

# replace the placeholder string in the pattern template with certificate encoded information.
# redirect it out to the sub file so it can be scp to the pfsense server
#
cat pattern.template | awk '$1=$1' FS="CRTPLACEHOLDER" OFS="$ENCRT"  | awk '$1=$1' FS="KEYPLACEHOLDER" OFS="$ENKEY" > pattern.sub

# scp the pattern file to the pfsense system
#
cp pattern.sub /tmp/

# execute sed replace against the config.xml and reload the configuration
#
cp /conf/config.xml /tmp/config.xml && \
 sed -f /tmp/pattern.sub < /tmp/config.xml > /root/le/config.xml-tmp && \
 cp /root/le/config.xml-tmp /conf/config.xml && \
 rm /tmp/config.cache && /etc/rc.restart_webgui

pattern.template


/.*LetsEncrypt.*/{
N
/<crt.* {<br="">s//<crt>CRTPLACEHOLDER<\/crt>/
}
N
/<prv.* {<br="">s//<prv>KEYPLACEHOLDER<\/prv>/
P
D
}
}</prv></prv.*></crt></crt.*> 
```</domainname></domainname></domainname></domainname>

robinvanleeuwen

I altered your script to work on the PFSense server itself. It makes use of dehydrated (which was formerly known as letsencrypt.sh .. but they updated the name of the repo not yet)

It uses dehydrated to generate new certificates…. The certificate must be manually inserted in the cert manager with a known prefix once .... In my case sslcertificate- <domain>.... and your script automagically replaces the cert and privkey in the config.xml

In this way i am able to use a cron job to generate new certfificates with dehydrated/ldehydrated. And use your tool to renew the certs and privkeys in the confix.xml

https://github.com/lukas2511/dehydrated is the dehydrated generated tool
And your script is the tool that updates the certs on teh PFSense Server itself.
It's available on https://github.com/robinvanleeuwen/update-ssl-certs-on-pfsense.git

The Certs have to be made manually once with a placeholder in the name for the first time to recognize it to be updatyed : for example sslcertificate- <mydomin>to correctly replace it with the pattern.template.... But it works....

I just editted your files and uploaded it to my own github repo.... If you want your name mentioned or some other kind of info about you, please let me know. I'll be happy to add it since your script was my first initial an best starting point!....

Tnx!</mydomin></domain>

dig1234

Just curious, did you see the new LE pFsense package? Would seem to replace any need for scripts. You can find it under Available Packages..

robinvanleeuwen

https://github.com/rigtersys/update-certificates-on-other-servers/blob/master/README.md

It's a simple script to generate new certs on pfsense and rsync them to places behind your PFSense / HAProxy setup. I Needed since i placed a catchall on /.well-known/acme-challenge/* on all my IP's in ha-proxy, so i could not generate certrs inside my network with LE-certbot. I now let dehydrated genererate the certs and ssh/rsync them to the server i ran the script from…. It's a bit hacky tool... but maybe someone can get some hints/tips from it... If you have any suggestions how to improve it, let me know! so it can better all suit our needs... :)

luisenrique

@hvisage said in Automating Certificate imports with letencrypt script:

enewAll weekly/monthly from cron, followed by

Hi @hvisage thats script still alive? it work on recent vertions?? i'm stating to automate the copy of my acme generated certificates on one pfsense/dns server to other servers web, mail on my network, i'm looking for some solutions scripting for that end...
ofther runn my script it give to me some error related permission over config.xml and config.cache so thats permissions cant be changed by a regular user.
has ben pass 2 years of this post but never is late to make the work fine ... thanks

yobyot

I realize this topic is old and about importing LE certs -- but I have the opposite problem. In 2.4.4, if in the Acme package for creating LE certs, in Settings, General Settings, I select the option to "Write ACME certificates to /conf/acme/ in various formats for use by other scripts or daemons which do not integrate with the certificate manager," the certs do NOT get added to /conf/config.xml.

My use case is that I WANT the certs to be in both /conf/config.xml and on the file system. I distribute the LE certs to other devices that need a cert on my LAN.

Does anyone have a suggestion on how I can get LE certs that are created by the Acme package into BOTH the Certificate Manager AND the file system?

Thanks.

Gertjan

See https://forum.netgate.com/topic/150603/certs-not-written-to-conf-config-xml-if-write-certificates-selected-in-general-settings/2

Ask yourself this question : if you re install pfSense 'from scratch' and you import your settings using the config.xml file, thinks like certs and CA"s are also imported.
And yes, when the package Acme received a cert from LE after a successful request, it load (replace an older cert with the same "ID") it into the System > Certificate Manager > Certificates list.