How to Install Certificates from PFsense to other servers?

lrossi

@victorlclopes thank you so much for your reply. I ended up using the same solution you are suggesting on your post.

I wish i found your post back then because you explained everything in a very simple and systematic manner. Kudos to you.

I tried the HA Proxy solution but i couldn’t make it work for the life of me. I even followed the videos from Lawrence on youtube without success. That’s when i found about the “scp” and have been using it since.

In addition to what you suggested, I wanted to automate the process of downloading the certificate from the pfsense firewall and installing it in the local server because i knew that at some point i was going to forget to manually check if there was a new certificate.

So, i created a scrip that runs as a cron job on the local server once a day. The scripts checks if there is a new certificate available in the pfsence firewall and only attempts to download such certificate if there is a new one.

Then, it sends me an email letting me know that a new certificate was installed. It doesn’t send me any emails unless a new certificate is found and installed.

I wanted to share this solution before but i was waiting to see if the script worked before posting it here. You just beat me by a couple of days because my certificate was successfully updated this morning. Although the email portion failed for some reason, i need to hunt down that bug.

If anybody is interested, my script is as follows, i figured this could be my contribution to the community. Just make sure to adjust the variables to match your enviroment:

#!/bin/bash
#script to download the SSL certficicates
#from pfsense for the indicated domain
#
#by lrossi 07-18-2021

##################################################################
# modify the variables for host, cert, key
# localcert, localkey and TMP_output
# to meet your needs

# change host to match the name of the ssh
# connection to the pfsense firewall
# as defined in the file ~/.ssh/config
host='pfsense'

# location of the ssl certificate and
# keyfile in pfsense firewall
cert='/cf/conf/acme/URLOFVERTIFICATE.com.crt'
key='/cf/conf/acme/URLOFVERTIFICATE.com.key'

# location of the ssl certificate and
# key file in local server
localcert='/etc/apache2/ssl/URLOFVERTIFICATE.com.crt'
localkey='/etc/apache2/ssl/URLOFVERTIFICATE.com.key'

#name and location of the log file in the local server
TMP_OUTPUT="/var/log/pfsense-certificates.log"

#information for email notifications
EMAIL_SUBJECT_PREFIX="SSL Certificate -"
TO_EMAIL_ADDRESS="EMAIL-OF-ADMIN@@DOMAIN.com"
FROM_EMAIL_ADDRESS="EMAIL-OF-LOCAL-SERVER@DOMAIN.com"
FROM_NAME="HOSTNAME OF LOCAL SERVER"

# end of variables

# start of script

# DO NOT MODIFY BELOW THIS LINE unless you understand the code
#####################################################################

#create a log file if one is not found.
if [ ! -f $TMP_OUTPUT ]; then
        touch $TMP_OUTPUT
fi

echo '==============================='  >$TMP_OUTPUT
echo '==============================='  >>$TMP_OUTPUT
echo 'Verification of SSL Certificate'  >>$TMP_OUTPUT
echo 'from PFSense'                     >>$TMP_OUTPUT
echo $(date)                            >>$TMP_OUTPUT
echo '==============================='  >>$TMP_OUTPUT
echo ''                                 >>$TMP_OUTPUT

#download the online certificate and key only if certificate is newer
if cmp -s $localcert <(ssh $host "cat $cert")
then
	echo 'the online certificate'		>>$TMP_OUTPUT
	echo 'has not been updated.'		>>$TMP_OUTPUT
	echo 'nothing to do'			>>$TMP_OUTPUT
	echo ''					>>$TMP_OUTPUT
	echo '==============================='	>>$TMP_OUTPUT
	echo '==============================='	>>$TMP_OUTPUT

else
	echo 'There is a new certificate'	>>$TMP_OUTPUT
	echo ''					>>$TMP_OUTPUT
	echo 'Downloading new certificate'	>>$TMP_OUTPUT
	scp $host:$cert $localcert		>>$TMP_OUTPUT
	wait
	
	echo ''					>>$TMP_OUTPUT
	echo 'Downloading new Key'		>>$TMP_OUTPUT
	scp $host:$key $localkey		>>$TMP_OUTPUT
	wait
	
	echo ''					>>$TMP_OUTPUT
	echo 'Restarting the apache webserver'	>>$TMP_OUTPUT
	systemctl restart apache2		>>$TMP_OUTPUT
	wait
	
	echo''					>>$TMP_OUTPUT
	echo 'New certificate has been installed'	>>$TMP_OUTPUT
	echo ''					>>$TMP_OUTPUT
	echo '================================='	>>$TMP_OUTPUT
	echo '================================='	>>$TMP_OUTPUT
	
	cat "$TMP_OUTPUT" | /usr/bin/mail -s "$EMAIL_SUBJECT_PREFIX Updated and installed" -a "FROM: $FROM_NAME <$FROM_EMAIL_ADDRESS>" $TO_EMAIL_ADDRESS
        wait
fi

exit 0;

mwebb

Think my automated solution may help others. I have a preference to issue and renew only one certificate and reuse on multiple machines. I use ACME pkg to issue/renew a single "Let's Encrypt" certificate with multiple sites and wildcards with DNS api to namecheap. As part of the ACME renewal I automate the copy of the certificate to the XMLRPC primary on other pfsense clusters (that do not have ACME installed) and call the following custom script to update it there.

The method laid out below needs the certificates and script to be on the target machine, and assumes ACME pkg to be on the source machine. In my haste to get this working in my Lab I set up a root level trust between them (not good practice) would be better to set this up a user with limited access. To eliminate the need for the ssh trust, the certificates can rather be copied to another secure machine on the network and the script can be run once a week with cron by the individual target hosts. (ACME certificates are typically renewed every 60 days and are valid for 90.)

The custom script combines only the essential script code that I found in the ACME pkg libraries which calls code from built-in pfsense libraries to update the certificate. The good thing about custom is that it is compact, however, there is a small chance that a future pfsense update that changes the certificate store management could break it. A way around this is to install ACME pkg on all the target machines (to benefit from updates) but then the ACME certificate list must have the same certificate name in the ACME inventory but the item must NOT be active (i.e. configured but not requesting renewal), and then call /usr/local/pkg/acme/acme_command.sh directly.

#!/usr/local/bin/php -f
<?php


/* 
custom_cert_import.sh

Script to copy and import a certificate from one pfsense machine to another.

- ACME package install is optional, but recommended on source host. Works with certs with multi sites and wildcards.
- Intended to be part of system to automate the publishing of one certificate from one machine to many.
- If certificate <CERT_NAME> exists in in the store it is replaced
- The method assumes that the certificate <CERT_NAME> is already on in the target system certificate store and selected for use by Web Configurator and/or Packages (e.g. HAPROXY).

Suggested use: 
1. For first use establish ssh trust between source and target host
> ssh-copy-id -i ~/.ssh/id_rsa.pub root@<REMOTE_HOST_IP1>

2. For first use manually copy script to target that is an XMLRPC primary in another pfsense CARP cluster (e.g. non-ACME host)
> scp -i /root/.ssh/id_rsa custom_cert_import.sh root@<REMOTE_HOST_IP1>:/<PATH1>/ 

3. Add the following lines to ACME post-renewal host commands on the source (ACME) host
> scp -i /root/.ssh/id_rsa /cf/conf/acme/<CERT_NAME>.* root@<REMOTE_HOST_IP1>:/<PATH1>/ 
> ssh root@<REMOTE_HOST_IP1> php /<PATH1>/pfsense_custom_cert_import.sh importcert <CERT_NAME> <DOMAIN> /<PATH2>/<CERT_NAME>.key /<PATH2>/<CERT_NAME>.crt /<PATH2>/<CERT_NAME>.ca /<PATH2>/<CERT_NAME>.fullchain
> ssh root@<REMOTE_HOST_IP1> /etc/rc.filter_synchronize  		<-------- refresh remote system firewall to start using updated certificate
> ssh root@<REMOTE_HOST_IP1> /usr/local/etc/rc.d/haproxy.sh restart	<-------- refresh remote system package e.g. HAPROXY 
> ssh root@<REMOTE_HOST_IP2> /usr/local/etc/rc.d/haproxy.sh restart

<REMOTE_HOST_IP1>	target host 1 (XMLRPC primary in target cluster)
<REMOTE_HOST_IP2>	target host 2 (XMLRPC secondary in target cluster)
<PATH1>			path to script on remote host
<PATH2> 		path to certificates on remote host
<CERT_NAME>		the cert description that is used to identify the certificate in the pfsense certificate store
<DOMAIN>		the fqdn of the certificate DN
 
The code for this script has been copied from the two files below from a system with ACME package installed.
The code calls functions that are already part of a standard pfsense deployment.

/usr/local/pkg/acme/acme_command.sh
/usr/local/pkg/acme/acme.inc 

*/


// ******************************************************
// Code below is from /usr/local/pkg/acme/acme_command.sh
// ******************************************************

require_once("functions.inc");

$command = $argv[1];

if ($command == "importcert") {
	$certificatename = $argv[2];
	$domain = $argv[3];
	$CERT_KEY_PATH = $argv[4];
	$CERT_PATH = $argv[5];
	$CA_CERT_PATH = $argv[6];
	$CERT_FULLCHAIN_PATH = $argv[7];
	echo "\nIMPORT CERT $certificatename, $CERT_KEY_PATH, $CERT_PATH";
	storeCertificateCer($certificatename, $CERT_KEY_PATH, $CERT_PATH, $CERT_FULLCHAIN_PATH);
	/*deleted code*/
	$changedesc = "Services: Acme: ";
	$changedesc .= "Storing signed certificate: " . $certificatename;
	write_config($changedesc);
	/*deleted code*/
	return;
}


// ***************************************************
// Code below is from /usr/local/pkg/acme/acme.inc
// ***************************************************

function storeCertificateCer($certificatename, $keyfile, $cerfile, $fullchainfile) {
	global $config;
	$certupdated = false;
		
	$key = file_get_contents($keyfile);
	$crt = file_get_contents($cerfile);
	$fullchain = "";
	if(!empty($fullchainfile)) {
		$fullchain = file_get_contents($fullchainfile);
		preg_match_all("/-+BEGIN CERTIFICATE(.+?)END CERTIFICATE-+/s", $fullchain, $certificatematches);
		$first = true;
		foreach($certificatematches[0] as $cacert) {
			if ($first == true) {
				$first = false;
				continue;
			}
			saveCACertificateToStore($cacert);
		}
	}
	if (is_array($config['cert'])) {
		foreach ($config['cert'] as &$cert) {
			if ($cert['descr'] == $certificatename) {
				syslog(LOG_NOTICE, "Acme, storing new certificate: {$certificatename}");
				echo "\nupdate cert!";
				cert_import($cert, $crt, $key);
				$certupdated = true;
				break;
			}
		}
	}
		
	return $certupdated;
}

function saveCACertificateToStore($crt) {
	global $config;
	$crt_enc = base64_encode($crt);
	if (!is_array($config['ca'])) {
		$config['ca'] = array();
	}
	$a_ca =& $config['ca'];
	foreach($a_ca as $ca) {
		if ($ca['crt'] == $crt_enc) {
			return;
		}
	}
	$subject = cert_get_subject($crt, false);
	$cert = array();
	$cert['refid'] = uniqid();
	$cert['descr'] = "Acmecert: {$subject}";
	ca_import($cert, $crt);
	$a_ca[] = $cert;
	syslog(LOG_NOTICE, "Acme, storing new CA: {$subject}");
}

?>

Gertjan

@mwebb said in How to Install Certificates from PFsense to other servers?:

can be run once a week with cron by the individual target hosts. (ACME certificates are typically renewed every 60 days and are valid for 90.)

Use the "action list" :

?

mwebb

@Gertjan I presented 2 methods. The 1st method is simple and helps to understand the process. 2nd method is better practice but more work. Sorry can't figure out how to add screenshots. I don't use my cert on the webgui (so don't need and didn't show the webgui command). The point of my post was more to share the precious script that gives us a way to use built-in pfsense code to refresh certificates - the secret sauce of the whole thing. There are many different ways to use it.

First method suggests to use the ACME action list on the source server like you have shown but I caution that this method requires root access from source host to the target pfsense hosts to run the script. 2nd method is to use ACME global setting (general settings - write certificates) to copy the certificate to /cf/conf/acme (no other ACME actions needed) or to another machine (with ACME action needed), then install cron pkg on each of the targets and run the actions weekly on the target hosts which only need to be able to read the certificate from the source host or other machine.

Bronko

@victorlclopes
Thanks for your how-to, used it since some month ago...

But I have an issue regarding to copy certificates to more then one server in Action list:
Screenshot from 2024-06-04 11-33-11.png

As to see I have tried both, an one liner or two separate commands (last temporary disabled for one liner test).

Both is working directly from pfsense CLI without interaction (based idea).

In case of renewal (last night) system log shows with then enabled command, but second server (10.0.0.103) never connected by ssh given by the logs there...

Does anyone have any experience or hints here?

Gertjan

@Bronko

If the first part of the third action worked : what about using a forth - the one you've disabled ?

Be aware : we don't know nothing about the shell session used to fire up the actions.
Do what the first and second action imply : instead of just 'scp', use the full path of the command scp. After all, whatever the environnement is, its not the CLI.

Of write a shell script, and then fire up a shell that executes the script for you.
With some nice log lines like "command 1 done", "command 2 done" etc

Bronko

@Gertjan said in How to Install Certificates from PFsense to other servers?:

If the first part of the third action worked : what about using a forth - the one you've disabled ?

This is what I have tried to describe. Two separate scp commands doesn't worked for the second (here the forth) one.

Be aware : we don't know nothing about the shell session used to fire up the actions.
Do what the first and second action imply : instead of just 'scp', use the full path of the command scp. After all, whatever the environnement is, its not the CLI.

Of write a shell script, and then fire up a shell that executes the script for you.
With some nice log lines like "command 1 done", "command 2 done" etc

You are absolutely right, this is what I have to do...
Will be back in two month.... SMILEY!

Bronko

Still doesn't worked for the second copy in my script:

Screenshot from 2024-08-04 12-03-36.png

upcoming execution is logged:

Aug 4 03:16:45 	php 	38555 	Acme, Running /home/USER/acme_post_scp.sh
Aug 4 03:16:45 	php 	38555 	Acme, Running /usr/local/etc/rc.d/haproxy.sh restart
Aug 4 03:16:45 	php 	38555 	Acme, Running /etc/rc.restart_webgui

the running itself:

Aug 4 03:16:46 	php-cgi 	39623 	rc.restart_webgui: Creating rrd update script
Aug 4 03:16:46 	php-cgi 	43383 	haproxy: started new pid:47385
Aug 4 03:16:46 	php-cgi 	43383 	haproxy: reload old pid:80851

The simple echos in script doesn't find in no way into pfsense system logs, only in shell directly, but the first certificate copy to 10.0.0.100 was successful. Still no ssh connection at 10.0.0.103 sysloged there (as above).

darcey

@Bronko said in How to Install Certificates from PFsense to other servers?:

The simple echos in script doesn't find in no way into pfsense system logs, only in shell directly

Use logger command. Does the second remote host log an ssh attempt?

Bronko

@darcey said in How to Install Certificates from PFsense to other servers?:

Use logger command. Does the second remote host log an ssh attempt?

Thanks for logger. As mentioned above, second host never loged an ssh attempt.

Gertjan

@Bronko said in How to Install Certificates from PFsense to other servers?:

As mentioned above, second host never loged an ssh attempt.

You are not actually use the 'GUI' as shown above to create that shells script file, right ?

Bronko

@Gertjan said in How to Install Certificates from PFsense to other servers?:

@Bronko said in How to Install Certificates from PFsense to other servers?:

As mentioned above, second host never loged an ssh attempt.

You are not actually use the 'GUI' as shown above to create that shells script file, right ?

Nope, to be save regarding file handling installed Filer package for that...

Gertjan

@Bronko

You 'chmod x' the script file as executable ?

(noop, you won't escape from the console or better, SSH )

Bronko

@Gertjan said in How to Install Certificates from PFsense to other servers?:

Be aware : we don't know nothing about the shell session used to fire up the actions.

given by that, used the GUI for file creation and chmod 755 (check picture)

Bronko

@Bronko said in How to Install Certificates from PFsense to other servers?:

But I have an issue regarding to copy certificates to more then one server in Action list:

I have found the missing step:

If you don't use the standard 'admin' group member of 'admins' like me to login into pfsense, you have to extend
/root/.ssh/known_hosts by your target hosts from /home/USER/.ssh/known_hosts given by the fact, Actions list jobs running in root context... My fault.

Thanks for all your response.

mwebb

@Bronko Suggest to test if .ssh subfolders are persistent after reboot of each machine. FreeBSD typically purges them at reboot. Might need to run a script at boot time to recover them or run script each time you copy the scripts with following options to recreate / ignore the known hosts automatically: scp -o UserKnownHostsFile=/dev/null -o StrictHostKeychecking=no -i /root/.ssh/id_rsa <user>@<cert store host>:/<script>

Bronko

@mwebb said in How to Install Certificates from PFsense to other servers?:

Suggest to test if .ssh subfolders are persistent after reboot

at pfsense they are persistent