Unbound Does Not Check Operating System Trust Store for DNS Over TLS

f1d094

Hello,

We have two instances of pfSense, PRIMARY and REMOTE. The local instance hosts overrides for all our services on DOMAIN.local , providing the internal IPs to hosts in the internal network. Both PRIMARY and REMOTE use DNS Over TLS, which works great for external hosts but systems on the REMOTE network never receive the overrides for DOMAIN.local if we set the DNS Resolver to forward for DOMAIN.local to the PRIMARY pfsense instance. On REMOTE we have DOMAIN.local configured as a forwarding resolver, have the name in the appropriate field, etc. The DOMAIN.local internal ROOT-CA was manually imported into REMOTE, and the box was checked for "Add this Certificate Authority to the Operating System Trust Store".

This problem has existed since we rolled out the remote location several years ago...I simply never persued it and instead just manually copied over all the hostname overrides. Since then, our company has grown as has the host list and keeping it in sync has proven a hassle. Yesterday I decided to fix it...or not, as the case may be.

Inspection of logs revealed that unbound considers our internal ROOT-CA as "self signed":

Oct 1 11:24:30 	unbound 	47674 	[47674:1] notice: ssl handshake failed 192.168.100.1 port 853
Oct 1 11:24:30 	unbound 	47674 	[47674:1] error: ssl handshake cert error: self-signed certificate in certificate chain
Oct 1 11:24:30 	unbound 	47674 	[47674:1] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed

Testing on-system ' openssl s_client -connect 192.168.100.1:853 -status ' shows no complaints about self-signed certificate.

Additional searching finds that our certificate is indeed in /usr/local/etc/ssl/certs/ and ' openssl storeutl -noout -text -certs /usr/local/etc/ssl/certs/57b2ad3e.crt ' displays the correct information.

Review of /var/unbound/unbound.conf shows that it is getting all of its CAs from /etc/ssl/cert.pem:

# TLS Configuration
tls-cert-bundle: "/etc/ssl/cert.pem"
tls-port: 853
tls-service-pem: "/var/unbound/sslcert.crt"
tls-service-key: "/var/unbound/sslcert.key"

Review of /etc/ssl/cert.pem shows our ROOT-CA is not in there:

openssl storeutl -noout -text -certs /etc/ssl/cert.pem | grep ROOT-CA

I then added tls-system-cert: yes to the custom options for unbound:

server:
tls-system-cert: yes
private-address: ::/0
local-zone: ip6.arpa. refuse
local-zone: "use-application-dns.net" always_nxdomain
log-queries: yes
server:include: /var/unbound/pfb_dnsbl.*conf

Listing of /etc/ssl/certs shows that the certificate is linked there:

lrwxr-xr-x  1 root wheel 45 Sep 30 18:45 57b2ad3e.0 -> ../../../usr/local/etc/ssl/certs/57b2ad3e.crt

But this only resulted in new errors for the DNS service:

Oct 1 11:29:23 	unbound 	64682 	[64682:0] notice: ssl handshake failed 192.168.100.1 port 853
Oct 1 11:29:23 	unbound 	64682 	[64682:0] error: ssl handshake cert error: self-signed certificate in certificate chain
Oct 1 11:29:23 	unbound 	64682 	[64682:0] error: and additionally crypto error:0A000086:SSL routines::certificate verify failed
Oct 1 11:29:23 	unbound 	64682 	[64682:0] error: and additionally crypto error:80000002:system library::No such file or directory
Oct 1 11:29:23 	unbound 	64682 	[64682:0] error: and additionally crypto error:16000069:STORE routines::unregistered scheme

It seems maybe like the default location for where unbound expects the system certificates to be doesn't exist: ' crypto error:80000002:system library::No such file or directory '...but I was unable to google this to find out where to make my own symlink and I am also unfamiliar how I would go about manually adding our ROOT-CA to /etc/ssl/cert.pem.

Thoughts?

f1d094

Current workaround: Manually editing /etc/ssl/cert.pem

Listing /etc/ssl shows that certs.pem actually comes from /usr/local/etc/ssl:

drwxr-xr-x   4 root wheel     7 Sep 30 18:43 .
drwxr-xr-x  33 root wheel   189 Sep 30 18:43 ..
lrwxr-xr-x   1 root wheel    32 Sep  9 11:54 cert.pem -> ../../usr/local/etc/ssl/cert.pem
drwxr-xr-x   2 root wheel   153 Oct  1 11:59 certs
-rw-r--r--   1 root wheel  4590 Sep 30 18:43 netgate-ca.pem
-rw-r--r--   1 root wheel 11928 Sep 30 18:43 openssl.cnf
drwxr-xr-x   2 root wheel    44 Oct  1 11:57 untrusted

Quick inspection of /usr/local/etc/ssl/cert.pem shows that it is just -text dumps of all of the trusted certs, with some comments at the top and bottom...along with a total in the last two lines:

# tail -n 3 /usr/local/etc/ssl/cert.pem

##  Number of certificates: 152
##  End of file.

I was unable to find a "correct" way to update this file, so instead I simply hacked together a solution. I made a backup of the original file, made a new copy without the last two lines, added the -text outout off all the certs in /usr/local/etc/ssl/ to the end of the new copy, each separated by three blank lines, then re-added the suffix with an updated total. I have no idea if the total matters, it is in a comment, but I did it anyway. YMMV

Here is my quick and diry script in case I need to do it again:

#!/bin/sh

fubar() {
   echo "$1"
   exit 1
}

TARGET="/usr/local/etc/ssl/cert.pem"
TEMPFOOT="/tmp/blarg"

# Make datestamped backup and hackfile
BACKUP="$TARGET.$(stat -f '%Sm' -t '%Y-%m-%d' $TARGET).backup"
HACKED="$TARGET.$(date +%Y-%m-%d).hacked"

# If already a symlink, fubar
[ -n "$(find $TARGET -type l)" ] && fubar "$TARGET already a symlink"

echo "TARGET => $TARGET"
echo "BACKUP => $BACKUP"
echo "HACKED => $HACKED"
echo

# Make list of certs to include
CERTLIST=$(find /usr/local/etc/ssl/certs/ -type f)
[ -n "$CERTLIST" ] || fubar "oops, no certs in CERTLIST"

# Num lines in TARGET
LINES=$(cat $TARGET|wc -l|tr -d [:blank:])

# Length minus last 2
CUTLINE=$(($LINES-2))

# Capture last 2 to TEMPFOOT
tail -n 2 $TARGET > $TEMPFOOT || fubar "failed to create $TEMPFOOT"
echo "Old footer:"
cat $TEMPFOOT
echo

# Parse out the files line and increment it in TEMPFOOT
NUMFILES=$(head -n 1 $TEMPFOOT|rev|cut -d ' ' -f1|rev)  # Number to increment is at end of first line
INCR=$(echo $CERTLIST|wc -w|tr -d [:blank:])            # Increment by number of certs
NEWNUM=$(($NUMFILES+$INCR))                             # New total
sed -i foo "s/$NUMFILES/$NEWNUM/" $TEMPFOOT || fubar "sed TEMPFOOT failed"

echo "New footer:"
cat $TEMPFOOT
echo

# Start the new file with everything up to cutline
echo "Starting $HACKED"
sed "$CUTLINE,$ d" $TARGET > $HACKED || fubar "sed TARGET failed"

# Add 3 blank lines between every cert in CERTLIST
for CERT in "$CERTLIST"; do
   echo >> $HACKED
   echo >> $HACKED
   echo >> $HACKED
   openssl x509 -in "$CERT" -text >> $HACKED || fubar "Failed to add $CERT"
   echo "Added $CERT"
done

# Append the updated footer from TEMPFOOT
cat $TEMPFOOT >> $HACKED
echo "Added new footer"

# Confirm then do
read -p "Hit any key to move TARGET to BACKUP and symlink HACKED to TARGET (or hit ctrl+c to exit) " FOO

mv $TARGET $BACKUP || fubar "can't move $TARGET to $BACKUP"
ln -s $HACKED $TARGET || fubar "can't symlink $HACKED to $TARGET"

echo "Done"
echo
echo "Use 'openssl storeutl -noout -text -certs $TARGET |grep <WHATEVER>' to verify the right certs are in $TARGET"
echo

After a restart of the service, there are no cert errors and Unbound on REMOTE now sees all the overrides for DOMAIN.local as expected.

johnpoz

@f1d094 pretty sure you can just add the CA in the cert manager and click the add to trust store.

https://docs.netgate.com/pfsense/en/latest/certificates/ca.html

f1d094

@johnpoz - Yes, as stated in my OP, I clicked "add to trust store" and it is indeed added. Using certctl list |grep <mydomain> finds it, as does testing with openssl s_client.

The problem is that unbound does not find it. Unbound relies solely upon /etc/ssl/cert.pem which is not updated.

johnpoz

@f1d094 ah my bad.. glad got it sorted.. not sure why unbound wouldn't just use the trust store.

f1d094

@johnpoz In the unbound docs there are two different config settings for where to look for certs:

tls-cert-bundle - This is a single bundle file, set by pfSense by default to "/etc/ssl/cert.pem" and this file is not updated
tls-system-cert - This is not set, is binary (yes/no) and defaults to 'no'

Setting 'tls-system-cert' to 'yes' only results in more errors (see above)...it appears to be looking in the wrong place and, as a 'yes/no' only option, it isn't configurable as to where to look.

Seems like a bug. 'tls-system-cert' should know to look in /usr/local/etc/ssl/certs/

Gertjan

@f1d094

Just a thought : unbound, the process, is chrooted. So, afaik, it can't 'see' or 'access' the pfSense file system, only what has been set up in the chroot.

Not related :

That server:include is ancient. You use pfblocker with the 'unbound' mode ?
If possible, use python mode.

f1d094

@Gertjan said in Unbound Does Not Check Operating System Trust Store for DNS Over TLS:

That server:include is ancient. You use pfblocker with the 'unbound' mode ?

Umm...yes? I'm generally in the camp of "If it ain't broke, leave it that way." It is running with whatever config we've had setup since forever.

What is this new hotness and why do I want it?

Gertjan

@f1d094 said in Unbound Does Not Check Operating System Trust Store for DNS Over TLS:

What is this new hotness and why do I want it?

When pfBlockerng is set like this :

it will assemble all DNSBL files together, and place the result into one big /var/unbound/pfb_dnsbl.conf, where unbound reads the file upon process start.
This worked well in the past., when the Internet was one big happy place with very view 'sites' (host names) to block. Then came the 'everyone wanted to be there' moment, and a lot of pure 'BS' sites were created. The ones you don't want to access. This list became bigger huge.
You can continue to use this mode, but do read this :

first.
Then read a couple of these : kea2unbound.
and know you understand why I told you to switch to Python mode

When a huge text file is read in by unbound at start, it could take minutes before it had parsed the entire file. during all this time, DNS is down.
Python mode offers a faster way to access the 'list' with all the blocked DNSBL entries, and allowing unbound to start faster if a restart was needed.
Python mode offers also more details to report (more graphs ans stats).

When switching to Python mode, the line "server:include:/var/unbound/pfb_dnsbl*.conf" will normally disappear. If it doesn't, no big deal. Just check if there are no "pfb_dnsbl*.conf" files anymore in /var/unbound/. If there are none, you can remove the line yourself.

f1d094