Automating Certificate imports with letencrypt script



  • Hey All,

    I'm in the process of flipping my CA from StartSSL to Lets Encrypt SSL Certs. I have setup a Docker container to automate the 90 day renewal process and have scripted the whole process to my servers/services that require the Certificates, chain and key, (Haproxy, apache, webmin etc)

    The problem I'm trying to over come is automating this into pfsense webui. I've tried replacing the /var/etc/cert.pem file and restarting the webui but by the looks of the /etc/rc.restart_webgui, this calls the system.inc file and replaces it with the default set.

    So I'm curious where the existing certificates are stored? DB, cache files or if someone has a curl POST script handy in order to push it into the Certificate Manager and setting as WebUI default.

    Currently I'm doing this all local on the lets encrypt container so would be ideal to include pfsense right inside the same script, however if I have to do this locally on pfsense, via shell, thats fine too. Just want a way to automate it. every Post I've come across relies on point and click methods (I'm a *nix guy).

    Any help would be appreciated

    Cheers,

    Update

    I've uploaded my code to github
    https://github.com/tylerhoadley/letsencrypt-automation



  • You were close.
    Everything is in the one and only config file : /conf/config.xml  :)



  • Ok that makes sense. so everytime its reloads, its that function what writes out the systems config replacing the static confs.

    i was looking at this file prior to posting and was trying for a scp or curl method. now however ill have to craft a script to replace the lines in cert manager portion, then reload the config. ill have some google'in to do.

    if anyone has code snippit to do this replace, and reload, that would be great. i can post  my end method once i get it setup and tested.

    thanks



  • @tylerhoadey:

    i was looking at this file prior to posting and was trying for a scp or curl method. now however ill have to craft a script to replace the lines in cert manager portion, then reload the config. ill have some google'in to do.

    Any luck with scripting an update to config.xml?



  • It's not only the scripting …

    Much has been said about implementing LetEncrypt into pfSense, like https://redmine.pfsense.org/issues/5434#note-2
    ( Use Google to find them alll )

    Install LetEncrypt  on your own (web) server on the net, just to see what happens ..... it is somewhat scarry, this huge pile of dependencies ...
    Remember : pfSense is just a "firewall".



  • I've been looking at using https://github.com/Neilpang/acme.sh. Haven't done a dry run yet but in planning I think copying certs into config.xml is my missing puzzle piece.



  • Sorry, no update… haven't had the time to sit down and build the regex replace. figured I would get back to this when the 90 day cert is due for renewal (and I'll be able to test it all from start to finish)

    30 days or so left, so will slot some time out to get it done.



  • I did come across this which seems like a pretty straightforward way of editing the config: https://forum.pfsense.org/index.php?topic=56956.msg304026#msg304026

    I need to dig in a little more to see how the certs are stores in config.xml and what I can key off of to make the update like this.



  • Ok, spend a bit of time this AM getting the pattern down, and reading advance sed substitution. It hasn't been fully tested but believe the guts of the task are done, with hopefully minor tweaks.

    Just to ensure everyone reading this is aware, I will be running this from a dedicated Lets Encrypt server I have that renews and copies my certs to few different servers and services, apache, Haproxy etc, I use ssh keys to seamlessly copy/execute from LE server to pfsense.

    the script below is using variables so adjust to your needs.

    
    #!/bin/bash
    
    # Letencrypt certificate filename (and path)
    FILE=test.crt
    
    # pfsense server name or IP
    SERVER=pfsense.example.com
    
    #remove newlines and Cert tags from certificate file
    
    CERT=`grep -v 'CERT' $FILE | grep '^[^<]' | tr -d '\n'`
    
    # copy the template to a sed script file
    cp pattern.template pattern.sub
    
    # replace the placeholder string in the pattern file with certificate information
    
    sed -i 's/REPLACEHOLDER/$CERT/g' pattern.sub
    
    # scp the pattern file to the pfsense system
    scp pattern.sub $SERVER:/tmp/
    
    # execute sed replace against the config.xml and reload the configuration
    
    ssh $SERVER 'sed -f /tmp/pattern.sub /conf/config.xml'
    ssh $SERVER 'rm /tmp/config.cache'
    
    

    as you can read above, I'm using sed -f to minimize the special characters and to match multi line substitution. seem to be simpler to format and test. I've also had to use a template approach as I don't want to keep matching older certs to insert new certs. that could be a pain, and I like to keep it as simple as I can (or at least for now).

    create the  pattern.template file

    *Note LetsEncrypt is the name of the certificate in pfsense certificate manager. aka the "Descriptive name" field. I recommend you keep this 100% unique within the configuration when you do the initial import. this approach assumes you have imported at least once in order to replace it

    
    /.*LetsEncrypt.*/{
    		N
    		/<crt>.*<\/crt>/{
    		s//<crt>REPLACEHOLDER<\/crt>/
    		P
    		D
    		}
    }</crt></crt> 
    

    Please don't use this directly against your "true" config.xml unless you have done further testing. I've uploaded this before I've done more than just quick testing against copied files.

    I have not done true Production testing against my firewall yet… although thats next on my list.

    Use at own risk.

    *** I have an update (escape characters were driving me crazy on the $CERT sed replace... I have changed it to use awk and have also inserted backslashes in to the CERT= variable) will post update tomorrow.



  • Like I posted yesterday, I believe I'm really close to having this down. Had to overcome escape characters issues with the sed script and the certs character '/' being passed, but now I see a bigger problem that I didn't see coming. Hoping a PFSense expert can help me out here.

    The issue I see is that in the config.xml, the cert and chain look like there hashed or in a different cert format.
    example.

    in the config.xml has my cert and chain starting with LS and ending in ==

    when I replace the cert in the file, its in the original pem format.

    What format does the config.xml present the cert and chain in?

    If I can prepare the cert for this format and insert… Then I have a working model script that will do this automation.

    *** base64 encoding
    -- php snippit from system_certmanager.php --
    $exp_data = base64_decode($a_cert[$id]['crt']);



  • Working model. tested and confirm certificate usage. * had to include the key, as Lets encrypt changes the default key on new generation/request.

    it requires php5 (CLI) on lets encrypt server.

    Remotescript.sh

    
    #!/bin/bash
    
    PHP=`which php`
    
    # Letencrypt certificate and chain filename (and path)
    # 
    # CHANGE THESE TO YOUR CERT/CHAIN PATH 
    # 
    CRT=/etc/letsencrypt/live/DNS/fullchain.pem
    KEY=/etc/letsencrypt/live/DNS/privkey.pem
    
    # pfsense server name or IP (using ssk keys for passwordless login)
    #
    SERVER=192.168.2.1
    
    # use php base64_encode function to convert certificate and chain
    #
    ENCRT=`$PHP -r '$cert = file_get_contents( $argv[1] , true);  echo base64_encode("$cert");' $CRT`
    ENKEY=`$PHP -r '$key = file_get_contents( $argv[1] , true);  echo base64_encode("$key");' $KEY`
    
    # replace the placeholder string in the pattern template with certificate encoded information.
    # redirect it out to the sub file so it can be scp to the pfsense server
    #
    cat pattern.template | awk '$1=$1' FS="CRTPLACEHOLDER" OFS="$ENCRT"  | awk '$1=$1' FS="KEYPLACEHOLDER" OFS="$ENKEY" > pattern.sub
    
    # scp the pattern file to the pfsense system
    #
    scp pattern.sub $SERVER:/tmp/
    
    # execute sed replace against the config.xml and reload the configuration
    #
    ssh $SERVER 'cp /conf/config.xml /tmp/config.xml && sed -f /tmp/pattern.sub < /tmp/config.xml > /conf/config.xml && rm /tmp/config.cache && /etc/rc.restart_webgui'
    
    

    the above code, reads in the cert and chain files, encodes them with php (which makes sense when dealing with the special characters in the certs). we then use a sed template to replace the Placeholder with the base64 encoding. once that pattern.sub is ready, we scp and run it with sed.

    create the template file, Change LetsEncrypt to the description name you used to import. I would recommend keeping this unique or understand this script is looking for any'NAME'any with <crt…crt>in the next line.

    pattern.template

    
    /.*LetsEncrypt.*/{
    		N
    		/<crt>.*<\/crt>/{
    		s//<crt>CRTPLACEHOLDER<\/crt>/
                    }
                    N
                    /<prv>.*<\/prv>/{
                    s//<prv>KEYPLACEHOLDER<\/prv>/
                    P
                    D
                    }
    
    }</prv></prv></crt></crt> 
    

    Of course I know there is some clean up and maybe a few things that could be done more efficiently, so please feel free to improve and optimize. this will get me by for now but would be open to removing php base64_encode function out and use base64 command line (more of a time issue right now to research)… pretty sure the sed -f command could be cleaner too with inplace rewriting on the file.</crt…crt>



  • Just curios why you're using a dedicated Letsencrypt server? There is a an sh script which runs perfectly on pfsense with no dependencies. I used this myself with success. If someone could add the key insertion script into the cron job we'd be all set. Check it out at:
    https://github.com/Neilpang/acme.sh

    Regarding the resistance to supporting Lets Encrypt in pfsense I don't see how not using a letsencrypt cert makes anyone safer. If the skeptics are right that illegitimate LE certs are easy to obtain (which I'm not convinced of), that affects us all whether we use LE Certs or not. Fact is the LE CA is trusted (by default) whether they like it or not. Someone correct me if I'm missing something.



  • In my environment it makes sense to use a dedicated server (which I will convert to a docker container). My ssl termination all happens behind pfsense. Don't get me wrong pfsense I know has lots of features, and have use haproxy amoung other services on it prior. However I like using pfsense for true firewall and nat rules (with dedicate resources for that purpose). I use other scale-able systems in the background to ensure services availability. and in my case a dedicated process in order to renew the LE certs. this keeps my systems clean of other dependencies.

    I am glad to hear you were able to get it running on pfsense. Like I said pfsense is feature rich and I know a lot of chatter out there on getting this component native or as an install package (and I am for it) Thats what makes pfsense fit for many use cases and setups.



  • Firstly: Thank you @tylerhoadey for the sed/php scripting you shared 8)
    Secondly: Thank you @dig1234 for pointing to NeilDang's acme.sh

    https://gitlab.com/snippets/26220

    Now just my modifications to make it run on the pfSense it self:

    1. install acme.sh, I used:
    curl https://get.acme.sh | sh
    
    
    1. mkdir /root/le and put script.sh and pattern.template (below) in there.
      Script.sh added some test for file existence, and pattern.template I somehow found the need to remove the tabs/spaces to make it work on pfSense/FreeBSD's sed.

    2. chmod +x script.sh ;)

    3. I disabled the webConfigurator's use of port 80, but allowed port 80 on the rules to use the standalone mechanism

    5) ```
    /root/.acme.sh/acme.sh --issue --standalone -d <domainname></domainname>

    
    6) The important part here is to now to add this certificate in the Certificate manager with the name atleast containing "LetsEncrypt"
    Use  /root/.acme.sh/<domainname>/fullchain.cer for the certifiacte
    and the private key would be /root/.acme.sh/<domainname>/<domainname>.key. The string LetsEncrypt in what is used by the sed pattern below to find the right certificate to update.
    
    7)```
    /root/le/script.sh <domainname></domainname>
    

    My next steps would be to  run /root/.acme.sh/.acme.sh –renewAll weekly/monthly from cron, followed by /root/le/script.sh <domainname>script.sh

    #!/bin/sh
    #
    # Thanks to tylerhoadey
    #  https://forum.pfsense.org/index.php?action=profile;u=282215
    
    PHP=`which php`
    
    # Letencrypt certificate and chain filename (and path)
    #
    # CHANGE THESE TO YOUR CERT/CHAIN PATH
    #
    DNS=$1
    CRT=/root/.acme.sh/${DNS}/fullchain.cer
    KEY=/root/.acme.sh/${DNS}/${DNS}.key
    
    # use php base64_encode function to convert certificate and chain
    #
    ENCRT=`$PHP -r '$cert = file_get_contents( $argv[1] , true);  echo base64_encode("$cert");' $CRT`
    ENKEY=`$PHP -r '$key = file_get_contents( $argv[1] , true);  echo base64_encode("$key");' $KEY`
    
    # replace the placeholder string in the pattern template with certificate encoded information.
    # redirect it out to the sub file so it can be scp to the pfsense server
    #
    cat pattern.template | awk '$1=$1' FS="CRTPLACEHOLDER" OFS="$ENCRT"  | awk '$1=$1' FS="KEYPLACEHOLDER" OFS="$ENKEY" > pattern.sub
    
    # scp the pattern file to the pfsense system
    #
    cp pattern.sub /tmp/
    
    # execute sed replace against the config.xml and reload the configuration
    #
    cp /conf/config.xml /tmp/config.xml && \
     sed -f /tmp/pattern.sub < /tmp/config.xml > /root/le/config.xml-tmp && \
     cp /root/le/config.xml-tmp /conf/config.xml && \
     rm /tmp/config.cache && /etc/rc.restart_webgui
    
    

    pattern.template

    
    /.*LetsEncrypt.*/{
    N
    /<crt.* {<br="">s//<crt>CRTPLACEHOLDER<\/crt>/
    }
    N
    /<prv.* {<br="">s//<prv>KEYPLACEHOLDER<\/prv>/
    P
    D
    }
    }</prv></prv.*></crt></crt.*> 
    ```</domainname></domainname></domainname></domainname>


  • I altered your script to work on the PFSense server itself. It makes use of dehydrated (which was formerly known as letsencrypt.sh .. but they updated the name of the repo not yet)

    It uses dehydrated to generate new certificates…. The certificate must be manually inserted in the cert manager with a known prefix once .... In my case sslcertificate- <domain>.... and your script automagically replaces the cert and privkey in the config.xml

    In this way i am able to use a cron job to generate new certfificates with dehydrated/ldehydrated. And use  your tool to renew the certs and privkeys in the confix.xml

    https://github.com/lukas2511/dehydrated is the dehydrated generated tool
    And your script is the tool that updates the certs on teh PFSense Server itself.
    It's available on https://github.com/robinvanleeuwen/update-ssl-certs-on-pfsense.git

    The Certs have to be made manually once  with a placeholder in the name for the first time to recognize it to be  updatyed : for example  sslcertificate- <mydomin>to correctly replace it with the pattern.template.... But it works....

    I  just editted your files and uploaded it to my own github repo.... If you want your name mentioned or some other kind of info about you, please let me know. I'll be happy to add it since your script was my first initial an best starting point!....

    Tnx!</mydomin></domain>



  • Just curious, did you see the new LE pFsense package? Would seem to replace any need for scripts. You can find it under Available Packages..



  • https://github.com/rigtersys/update-certificates-on-other-servers/blob/master/README.md

    It's a simple script to generate new certs on pfsense and rsync them to places behind your PFSense / HAProxy setup. I Needed since i placed a catchall on /.well-known/acme-challenge/* on all my IP's in ha-proxy, so i could not generate certrs inside my network with LE-certbot. I now let dehydrated genererate the certs and ssh/rsync them to the server i ran the script from…. It's a bit hacky tool... but maybe someone can get some hints/tips from it... If you have any suggestions how to improve it, let me know! so it can better all suit our needs... :)



  • @hvisage said in Automating Certificate imports with letencrypt script:

    enewAll weekly/monthly from cron, followed by

    Hi @hvisage thats script still alive? it work on recent vertions?? i'm stating to automate the copy of my acme generated certificates on one pfsense/dns server to other servers web, mail on my network, i'm looking for some solutions scripting for that end...
    ofther runn my script it give to me some error related permission over config.xml and config.cache so thats permissions cant be changed by a regular user.
    has ben pass 2 years of this post but never is late to make the work fine ... thanks


Log in to reply