How to Install Certificates from PFsense to other servers?

mwebb

Think my automated solution may help others. I have a preference to issue and renew only one certificate and reuse on multiple machines. I use ACME pkg to issue/renew a single "Let's Encrypt" certificate with multiple sites and wildcards with DNS api to namecheap. As part of the ACME renewal I automate the copy of the certificate to the XMLRPC primary on other pfsense clusters (that do not have ACME installed) and call the following custom script to update it there.

The method laid out below needs the certificates and script to be on the target machine, and assumes ACME pkg to be on the source machine. In my haste to get this working in my Lab I set up a root level trust between them (not good practice) would be better to set this up a user with limited access. To eliminate the need for the ssh trust, the certificates can rather be copied to another secure machine on the network and the script can be run once a week with cron by the individual target hosts. (ACME certificates are typically renewed every 60 days and are valid for 90.)

The custom script combines only the essential script code that I found in the ACME pkg libraries which calls code from built-in pfsense libraries to update the certificate. The good thing about custom is that it is compact, however, there is a small chance that a future pfsense update that changes the certificate store management could break it. A way around this is to install ACME pkg on all the target machines (to benefit from updates) but then the ACME certificate list must have the same certificate name in the ACME inventory but the item must NOT be active (i.e. configured but not requesting renewal), and then call /usr/local/pkg/acme/acme_command.sh directly.

#!/usr/local/bin/php -f
<?php


/* 
custom_cert_import.sh

Script to copy and import a certificate from one pfsense machine to another.

- ACME package install is optional, but recommended on source host. Works with certs with multi sites and wildcards.
- Intended to be part of system to automate the publishing of one certificate from one machine to many.
- If certificate <CERT_NAME> exists in in the store it is replaced
- The method assumes that the certificate <CERT_NAME> is already on in the target system certificate store and selected for use by Web Configurator and/or Packages (e.g. HAPROXY).

Suggested use: 
1. For first use establish ssh trust between source and target host
> ssh-copy-id -i ~/.ssh/id_rsa.pub root@<REMOTE_HOST_IP1>

2. For first use manually copy script to target that is an XMLRPC primary in another pfsense CARP cluster (e.g. non-ACME host)
> scp -i /root/.ssh/id_rsa custom_cert_import.sh root@<REMOTE_HOST_IP1>:/<PATH1>/ 

3. Add the following lines to ACME post-renewal host commands on the source (ACME) host
> scp -i /root/.ssh/id_rsa /cf/conf/acme/<CERT_NAME>.* root@<REMOTE_HOST_IP1>:/<PATH1>/ 
> ssh root@<REMOTE_HOST_IP1> php /<PATH1>/pfsense_custom_cert_import.sh importcert <CERT_NAME> <DOMAIN> /<PATH2>/<CERT_NAME>.key /<PATH2>/<CERT_NAME>.crt /<PATH2>/<CERT_NAME>.ca /<PATH2>/<CERT_NAME>.fullchain
> ssh root@<REMOTE_HOST_IP1> /etc/rc.filter_synchronize  		<-------- refresh remote system firewall to start using updated certificate
> ssh root@<REMOTE_HOST_IP1> /usr/local/etc/rc.d/haproxy.sh restart	<-------- refresh remote system package e.g. HAPROXY 
> ssh root@<REMOTE_HOST_IP2> /usr/local/etc/rc.d/haproxy.sh restart

<REMOTE_HOST_IP1>	target host 1 (XMLRPC primary in target cluster)
<REMOTE_HOST_IP2>	target host 2 (XMLRPC secondary in target cluster)
<PATH1>			path to script on remote host
<PATH2> 		path to certificates on remote host
<CERT_NAME>		the cert description that is used to identify the certificate in the pfsense certificate store
<DOMAIN>		the fqdn of the certificate DN
 
The code for this script has been copied from the two files below from a system with ACME package installed.
The code calls functions that are already part of a standard pfsense deployment.

/usr/local/pkg/acme/acme_command.sh
/usr/local/pkg/acme/acme.inc 

*/


// ******************************************************
// Code below is from /usr/local/pkg/acme/acme_command.sh
// ******************************************************

require_once("functions.inc");

$command = $argv[1];

if ($command == "importcert") {
	$certificatename = $argv[2];
	$domain = $argv[3];
	$CERT_KEY_PATH = $argv[4];
	$CERT_PATH = $argv[5];
	$CA_CERT_PATH = $argv[6];
	$CERT_FULLCHAIN_PATH = $argv[7];
	echo "\nIMPORT CERT $certificatename, $CERT_KEY_PATH, $CERT_PATH";
	storeCertificateCer($certificatename, $CERT_KEY_PATH, $CERT_PATH, $CERT_FULLCHAIN_PATH);
	/*deleted code*/
	$changedesc = "Services: Acme: ";
	$changedesc .= "Storing signed certificate: " . $certificatename;
	write_config($changedesc);
	/*deleted code*/
	return;
}


// ***************************************************
// Code below is from /usr/local/pkg/acme/acme.inc
// ***************************************************

function storeCertificateCer($certificatename, $keyfile, $cerfile, $fullchainfile) {
	global $config;
	$certupdated = false;
		
	$key = file_get_contents($keyfile);
	$crt = file_get_contents($cerfile);
	$fullchain = "";
	if(!empty($fullchainfile)) {
		$fullchain = file_get_contents($fullchainfile);
		preg_match_all("/-+BEGIN CERTIFICATE(.+?)END CERTIFICATE-+/s", $fullchain, $certificatematches);
		$first = true;
		foreach($certificatematches[0] as $cacert) {
			if ($first == true) {
				$first = false;
				continue;
			}
			saveCACertificateToStore($cacert);
		}
	}
	if (is_array($config['cert'])) {
		foreach ($config['cert'] as &$cert) {
			if ($cert['descr'] == $certificatename) {
				syslog(LOG_NOTICE, "Acme, storing new certificate: {$certificatename}");
				echo "\nupdate cert!";
				cert_import($cert, $crt, $key);
				$certupdated = true;
				break;
			}
		}
	}
		
	return $certupdated;
}

function saveCACertificateToStore($crt) {
	global $config;
	$crt_enc = base64_encode($crt);
	if (!is_array($config['ca'])) {
		$config['ca'] = array();
	}
	$a_ca =& $config['ca'];
	foreach($a_ca as $ca) {
		if ($ca['crt'] == $crt_enc) {
			return;
		}
	}
	$subject = cert_get_subject($crt, false);
	$cert = array();
	$cert['refid'] = uniqid();
	$cert['descr'] = "Acmecert: {$subject}";
	ca_import($cert, $crt);
	$a_ca[] = $cert;
	syslog(LOG_NOTICE, "Acme, storing new CA: {$subject}");
}

?>

Gertjan

@mwebb said in How to Install Certificates from PFsense to other servers?:

can be run once a week with cron by the individual target hosts. (ACME certificates are typically renewed every 60 days and are valid for 90.)

Use the "action list" :

?

mwebb

@Gertjan I presented 2 methods. The 1st method is simple and helps to understand the process. 2nd method is better practice but more work. Sorry can't figure out how to add screenshots. I don't use my cert on the webgui (so don't need and didn't show the webgui command). The point of my post was more to share the precious script that gives us a way to use built-in pfsense code to refresh certificates - the secret sauce of the whole thing. There are many different ways to use it.

First method suggests to use the ACME action list on the source server like you have shown but I caution that this method requires root access from source host to the target pfsense hosts to run the script. 2nd method is to use ACME global setting (general settings - write certificates) to copy the certificate to /cf/conf/acme (no other ACME actions needed) or to another machine (with ACME action needed), then install cron pkg on each of the targets and run the actions weekly on the target hosts which only need to be able to read the certificate from the source host or other machine.

Bronko

@victorlclopes
Thanks for your how-to, used it since some month ago...

But I have an issue regarding to copy certificates to more then one server in Action list:
Screenshot from 2024-06-04 11-33-11.png

As to see I have tried both, an one liner or two separate commands (last temporary disabled for one liner test).

Both is working directly from pfsense CLI without interaction (based idea).

In case of renewal (last night) system log shows with then enabled command, but second server (10.0.0.103) never connected by ssh given by the logs there...

Does anyone have any experience or hints here?

Gertjan

@Bronko

If the first part of the third action worked : what about using a forth - the one you've disabled ?

Be aware : we don't know nothing about the shell session used to fire up the actions.
Do what the first and second action imply : instead of just 'scp', use the full path of the command scp. After all, whatever the environnement is, its not the CLI.

Of write a shell script, and then fire up a shell that executes the script for you.
With some nice log lines like "command 1 done", "command 2 done" etc

Bronko

@Gertjan said in How to Install Certificates from PFsense to other servers?:

If the first part of the third action worked : what about using a forth - the one you've disabled ?

This is what I have tried to describe. Two separate scp commands doesn't worked for the second (here the forth) one.

Be aware : we don't know nothing about the shell session used to fire up the actions.
Do what the first and second action imply : instead of just 'scp', use the full path of the command scp. After all, whatever the environnement is, its not the CLI.

Of write a shell script, and then fire up a shell that executes the script for you.
With some nice log lines like "command 1 done", "command 2 done" etc

You are absolutely right, this is what I have to do...
Will be back in two month.... SMILEY!

Bronko

Still doesn't worked for the second copy in my script:

Screenshot from 2024-08-04 12-03-36.png

upcoming execution is logged:

Aug 4 03:16:45 	php 	38555 	Acme, Running /home/USER/acme_post_scp.sh
Aug 4 03:16:45 	php 	38555 	Acme, Running /usr/local/etc/rc.d/haproxy.sh restart
Aug 4 03:16:45 	php 	38555 	Acme, Running /etc/rc.restart_webgui

the running itself:

Aug 4 03:16:46 	php-cgi 	39623 	rc.restart_webgui: Creating rrd update script
Aug 4 03:16:46 	php-cgi 	43383 	haproxy: started new pid:47385
Aug 4 03:16:46 	php-cgi 	43383 	haproxy: reload old pid:80851

The simple echos in script doesn't find in no way into pfsense system logs, only in shell directly, but the first certificate copy to 10.0.0.100 was successful. Still no ssh connection at 10.0.0.103 sysloged there (as above).

darcey

@Bronko said in How to Install Certificates from PFsense to other servers?:

The simple echos in script doesn't find in no way into pfsense system logs, only in shell directly

Use logger command. Does the second remote host log an ssh attempt?

Bronko

@darcey said in How to Install Certificates from PFsense to other servers?:

Use logger command. Does the second remote host log an ssh attempt?

Thanks for logger. As mentioned above, second host never loged an ssh attempt.

Gertjan

@Bronko said in How to Install Certificates from PFsense to other servers?:

As mentioned above, second host never loged an ssh attempt.

You are not actually use the 'GUI' as shown above to create that shells script file, right ?

Bronko

@Gertjan said in How to Install Certificates from PFsense to other servers?:

@Bronko said in How to Install Certificates from PFsense to other servers?:

As mentioned above, second host never loged an ssh attempt.

You are not actually use the 'GUI' as shown above to create that shells script file, right ?

Nope, to be save regarding file handling installed Filer package for that...

Gertjan

@Bronko

You 'chmod x' the script file as executable ?

(noop, you won't escape from the console or better, SSH )

Bronko

@Gertjan said in How to Install Certificates from PFsense to other servers?:

Be aware : we don't know nothing about the shell session used to fire up the actions.

given by that, used the GUI for file creation and chmod 755 (check picture)

Bronko

@Bronko said in How to Install Certificates from PFsense to other servers?:

But I have an issue regarding to copy certificates to more then one server in Action list:

I have found the missing step:

If you don't use the standard 'admin' group member of 'admins' like me to login into pfsense, you have to extend
/root/.ssh/known_hosts by your target hosts from /home/USER/.ssh/known_hosts given by the fact, Actions list jobs running in root context... My fault.

Thanks for all your response.

mwebb

@Bronko Suggest to test if .ssh subfolders are persistent after reboot of each machine. FreeBSD typically purges them at reboot. Might need to run a script at boot time to recover them or run script each time you copy the scripts with following options to recreate / ignore the known hosts automatically: scp -o UserKnownHostsFile=/dev/null -o StrictHostKeychecking=no -i /root/.ssh/id_rsa <user>@<cert store host>:/<script>

Bronko

@mwebb said in How to Install Certificates from PFsense to other servers?:

Suggest to test if .ssh subfolders are persistent after reboot

at pfsense they are persistent