Unbound DNS Resolver Periodically Non-Responsive
-
I have had a few instances recently where Unbound has just stopped resolving for me. The last time it happened, I installed service watchdog to get alerts and restart the service as needed.
I got an alert at 6:30 AM this morning that the service had stopped and been restarted, and all was well after that this morning. When I came home tonight and logged on, the service was non responsive (not stopped) again, and I had to reboot the firewall to get the network back up and running.
I am on 26.03.1, I am running pfBlockerNG and that was updated to a new version within the last few weeks. See below for the logs from Unbound when it stopped this morning - any ideas?
-
For 'some reason', unbound was asked to use a certificate :
I'm not sure where it it got the file name "sslcert.key" from - not form its config file, /var/unbound/unbound.conf.
The file /var/unbound/sslcert.key should exist - check that for yourself.
Normally, you don't need to check/use this :
as even today, most LAN devices by default won't use DNS over TLS = port 853.
Some thoughts :
@pdavis said in Unbound DNS Resolver Periodically Non-Responsive:
I am running pfBlockerNG
Instead of updating the DNSBL feeds as 'often as possible', try updating them 'ones a week'.
After all, a DNSBL update will restart unbound, the Resolver.
Your mission, as an pfSense admin, is doing everything in your power so services won't get restarted only when strictly needed, like during a system powerup.@pdavis said in Unbound DNS Resolver Periodically Non-Responsive:
I installed service watchdog
Search the forum (here) posts about this watchdog service.
As soon as you think : "Wait, I don't want this to happen, I have to delete that thing ASAP" you understood what is going on.
I use pfBlocker, I use 26.03.1 (and earlier version) and unbound never dies on me. It restarts a coupe of times per week, but that's me messing around, not pfSense itself. -
@Gertjan Thank you for this. The file /var/unbound/sslcert.key does exist.
I did update the cron job to update my dnsbl less frequently in pfblockerng - I will see how that goes and update.
And thank you for the tip about service watchdog! It hasn't spammed me to death yet, and those posts seem older, but I will remove it.
-
You saw it for yourself : the two files exist.
Yet, unbound, duing startup, compliances they are not there :
As you've selected :
these two certificate files are needed when "Enable SSL/TLS Service" is checked.
The certificate you've selected (one line lower) is a 'valid' certificate ?
Maybe the issue isn't 'file not found' but 'file (== certificate) is invalid. -
@Gertjan Could be - it looks like there is a permissue in the most recent occurrence. I am going to turn off the Respond to incoming SSL/TLS queries from local clients setting.
-
@pdavis said in Unbound DNS Resolver Periodically Non-Responsive:
Could be - it looks like there is a permissue in the most recent occurrence. I am going to turn off the Respond to incoming SSL/TLS queries from local clients setting.
I think you're on to something.
When I check :
I start to see these errors also in my unbound/resolver log :
Right now, I have the option checked, and unbound resolver (DNS) is running.
It's something like this : when something restarts unbound, like pfBlockerng can do, unbound starts to produce theunbound 1669 [1669:4] error: Error in SSL_CTX use_PrivateKey_file crypto error:8000000D:system library::Permission denied
lines.
unbound aborts after being restarted, leaving the system without DNS.
I haven't investigated more, but this seems bad - not seen this behavior before.
I'm using pfSense 26.03.1.unbound itself, the binary FreeBSD/pfSense package, was upgraded a couple of days ago to version 1.25.1.
pfSense itself, 26.031 - is the same for the last 2 months or so.[2[26.03.1-RELEASE][root@pfSense.bhf.tld]/var/unbound: unbound-control -c /var/unbound/unbound.conf status version: 1.25.1 .... -
@Gertjan I am going to log a bug with Netgate. Thank you for your help with this!
-
I'm not seeing this in my testing.
- What does the
<unbound>section look like in/cf/conf/config.xml? - Is the cert that's selected in the DNS Resolver page a cert that was imported?
- Does the issue continue to happen after creating and using a new self-signed CA and server cert using the GUI?
- Does the issue continue to happen after removing the
.key,.crt, and.pemfiles in/var/unbound?
- What does the
-
Yeah I don't normally have that enabled - its pretty pointless in my environment, I have nothing that would be asking unbound over dot.
But I just enabled it - and unbound started up just fine.
-
@marcosm said in Unbound DNS Resolver Periodically Non-Responsive:
What does the <unbound> section look like in /cf/conf/config.xml?
See below:
<unbound>
<dnssec></dnssec>
<active_interface>lan,opt1,_lloclan,_llocopt1,lo0</active_interface>
<outgoing_interface>all</outgoing_interface>
<custom_options></custom_options>
<hideidentity></hideidentity>
<hideversion></hideversion>
<dnssecstripped></dnssecstripped>
<port></port>
<tlsport></tlsport>
<sslcertref>62125c664dd75</sslcertref>
<system_domain_local_zone_type>transparent</system_domain_local_zone_type>
<forward_tls_upstream></forward_tls_upstream>
<enable></enable>
<python_order>pre_validator</python_order>
<python_script>pfb_unbound</python_script>
<python></python>
</unbound>Is the cert that's selected in the DNS Resolver page a cert that was imported?
No, created within pfSense
Does the issue continue to happen after creating and using a new self-signed CA and server cert using the GUI?
Haven't tried this
Does the issue continue to happen after removing the .key, .crt, and .pem files in /var/unbound?
Haven't tried this
-
@pdavis Do you have actually have something using unbound via dot? For that to happen you would pretty much need a downstream resolver pointing to your unbound.. And then again its not a very good one if it doesn't validate the cert being used, it sure not going to trust some selfsigned cert used for the web gui.
I am not aware of application or OS that would use dot, they would normally be doh. You can make unbound serve up via doh, but it takes manual config. I believe I posted how to do that some time back. Was just an experiment - I don't actually use it.
And sure there are some dns clients you can do dot queries with - Are you using one of those, all the ones I have seen would be via a manual query, not something that is automated for day to day dns use, etc.
Unless you specially have setup a downstream name server or actually have clients using dot to query this unbound. Why don't you just uncheck the option for it to listen for them?
-
@johnpoz Not to my knowledge. Honestly I don't know when that setting got enabled, but I have turned it off.
It must have been in place for a long time though, so I'm still confused as to why this issue just started recently (timing seems to coincide with pfBlockerNG update).
I will report if I see any other issues with unbound, thank you for the assistance!
-
@pdavis possible the cert its using just expired recently? Haven't tried doing it with an expired cert. To be honest not even sure why that option is exposed in the general page.. For sure should be under advanced. Its not a common sort of setup. Your typical client is not going to use dot. Now if was used to enable doh. But should still be in advanced - and should come with a warning/note about using a cert client can validate, etc.
One of the big "advantages" if you will of dot or doh is validation you are talking to who you think your talking for dns with validation of the cert. This is never going to work just defaulting to using the cert the gui is using most likely never going to do that - because the default cert of the pfsense web gui is just a self signed cert.
You can setup doh for example with custom option box settings
#interface: 192.168.2.253@443 #tls-service-pem: "10year-doh.crt" #tls-service-key: "10year-doh.key"Where you call out the interface, and cert and key. Those are the commented out entries in mine for when I was playing with setting it up to do doh.
Personally I don't see much point in using doh or dot internally.. I mean its your local network, you should be pretty damn sure you are talking to who you think your talking to for dns, and highly unlikely you have a bad actor on your local network sniffing your dns traffic ;) If you did you have bigger things to worry about other than them being able to sniff your dns..
Only point I could see in setting up doh internally would be removing a warning your client might throw up when its not using doh. Which is just fear mongering and cosmetic if you ask me. And would bring up prob another issue if you allow 443 to pfsense address - this could expose your normal web admin gui, which defaults to using 443. Or conflicts of running multiple services on the same port if you don't change the web gui port.. So yeah this would be a more advanced configuration.
Unless you have specific need to run dot for your clients, if me I would just uncheck the box and move on. If there is some sort of issue or bug related to it - nice to find it and fix it sure, but should be done with people actually have a need for running it. Which I don't really see could be very many people.. Its a pretty niche sort of configuration.
I would be more inclined to vote they just remove it as an exposed option in the unbound gui - to keep users that really have no use for it clicking on it ;)
-
@pdavis said in Unbound DNS Resolver Periodically Non-Responsive:
Honestly I don't know when that setting got enabled
Afaik, "Enable SSL/TLS Service" can be checked or unchecked without any consequences.
Not the consequences, a failing unbound, shown above.
The resolver works fine with or without it.
If it's activatedBtw : activating this option is (should be !) fool-proof.
Normally, any** certicate listed under System > Certificates > Certificates will do just fine.** Not that I know of, but there are of course exceptions. The default pfSense WebGUI certificate that was created when pfSense was installed always worked for me.
Also : no other server type process should use (== "listen") on the LAN(s) interfaces with the same port = TCP 853. Normally, this isn't the case.I activated - see posts above - this "Enable SSL/TLS Service" myself also, as I was thinking : "useless, I agree, but why not ?!" , and a coupe of hours (a day later) I found my 'unbound' down with the cert errors as shown above in my log Resolver file)
My initial thought were also : is this linked to pfBlockerng as it can restart unbound (the resolver).I activated "Enable SSL/TLS Service" just a couple of hours ago, and started tailling the /var/log/resolver.log for error messages. I restarted the resolver several times, using all my known methods, like 'dashboard', Apply on the settings page, be reloading pfBLockerng and now I'm waiting on the pfBlockerng DNSBL cron event. Sure enough, I didn't saw any issues.
For the record : pfSense uses a dual staged unbound restart process.
First, it makes a /test/ folder in /var/unbound/, copies all files/settings from /var/unbound/ into /var/unbound/test/ and then chroot starts unbound in this folder /var/unbound/test/
If that worked out fine, pfSense knowns the settings in /var/unbound/ are good, and then the real unbound instance is started. It's something like that.Also :
The unbound config file contains the full path to the file.
Somewhat strange, as the error shows :
where the file with the error is
/sslcert.key
That's not "/var/unbound/sslcert.key"
Maybe ok, if unbound, if already chrooted into /var/unbound/ will read /sslcert.key just fine.
But "/sslcert.key" isn't in the 'root' of pfSense itself = /The error message itself is also lacking info. "error: error" isn't very useful.
"error: file can't be stat" or "error:no access rights" etc would be more useful.The 'unbound' error message is logged here.
after calling "SSL_CTX_use_PrivateKey_file(ctx, key, SSL_FILETYPE_PEM)", when this function returns false.
But SSL_CTX_use_PrivateKey_file() is not a unbound (project) function, its from openssl. OMG ...Btw : unbound really became huge ...
-
@Gertjan said in Unbound DNS Resolver Periodically Non-Responsive:
Normally, any** certicate listed under System > Certificates > Certificates will do just fine.
Sure - the service should come up with any valid cert.. The problem being coming up with some self cert that does not actually validate on the client pretty much defeats the so called advantages of dot or doh. You are sure you are talking to who you wanted to talk to because the cert validates.
You would have to jump through multiple hoops to get clients to validate some selfsigned cert. Even if you create a CA and sign the cert, all your clients would have to trust this CA.
If your client is not actually validating the dot or doh server its talking to - why are you using it?
C:\tools\doggo $ doggo.exe @tls://192.168.9.253 nas.home.arpa time=2026-07-09T09:04:59.968-05:00 level=ERROR msg="error in lookup" error="tls: failed to verify certificate: x509: certificate signed by unknown authority" NAME TYPE CLASS TTL ADDRESS NAMESERVER C:\tools\doggo $ doggo.exe @tls://192.168.9.253 nas.home.arpa --tls-hostname=sg4860-5cf44fa8eafc7 time=2026-07-09T09:07:33.151-05:00 level=ERROR msg="error in lookup" error="tls: failed to verify certificate: x509: certificate signed by unknown authority" NAME TYPE CLASS TTL ADDRESS NAMESERVER C:\tools\doggo $ doggo.exe @tls://192.168.9.253 --tls-hostname=sg4860-5cf44fa8eafc7 nas.home.arpa time=2026-07-09T09:08:14.342-05:00 level=ERROR msg="error in lookup" error="tls: failed to verify certificate: x509: certificate signed by unknown authority" NAME TYPE CLASS TTL ADDRESS NAMESERVER C:\tools\doggo $ doggo.exe @tls://192.168.9.253 nas.home.arpa --skip-hostname-verification NAME TYPE CLASS TTL ADDRESS NAMESERVER nas.home.arpa. A IN 3600s 192.168.9.10 192.168.9.253:853Notice it fails even when I give it the hostname on the selfsigned. It only returns an answer when I tell it not to check the cert. That sg4860-5cf44fa8eafc7, is the cn listed in the selfsigned.
Common Name: sg4860-5cf44fa8eafc7
Subject Alternative Names: sg4860-5cf44fa8eafc7
Organization: pfSense webConfigurator Self-Signed CertificateNow if I change to unbound to use a cert my machine trusts.
C:\tools\doggo $ doggo.exe @tls://192.168.9.253 nas.home.arpa NAME TYPE CLASS TTL ADDRESS NAMESERVER nas.home.arpa. A IN 3600s 192.168.9.10 192.168.9.253:853 C:\tools\doggo $ doggo.exe @tls://sg4860.home.arpa nas.home.arpa NAME TYPE CLASS TTL ADDRESS NAMESERVER nas.home.arpa. A IN 3600s 192.168.9.10 sg4860.home.arpa:853 C:\tools\doggo $ doggo.exe @tls://192.168.2.253 nas.home.arpa time=2026-07-09T09:13:55.083-05:00 level=ERROR msg="error in lookup" error="tls: failed to verify certificate: x509: certificate is valid for 192.168.9.253, not 192.168.2.253" NAME TYPE CLASS TTL ADDRESS NAMESERVER C:\tools\doggo $ doggo.exe @tls://192.168.2.253 nas.home.arpa --tls-hostname=sg4860.home.arpa NAME TYPE CLASS TTL ADDRESS NAMESERVER nas.home.arpa. A IN 3600s 192.168.9.10 192.168.2.253:853 C:\tools\doggo $ doggo.exe @tls://192.168.2.253 nas.home.arpa --tls-hostname=192.168.9.253 NAME TYPE CLASS TTL ADDRESS NAMESERVER nas.home.arpa. A IN 3600s 192.168.9.10 192.168.2.253:853The name and IP are in my certs as san, etc. Now have not dug deep enough into this to know if its just checking the CN or SANs but when using the selfsigned, that is the CN on the cert. But from the error I show believe where it mentions a non trusted CA, I assume its using my machines trusted CAs, since I have my CA from pfsense installed.
Any sane dot or doh client should not only validate the CN/SANs on the cert match up, but should actually validate signed by a CA the client trusts.
Notice in the first query I show
verify certificate: x509: certificate signed by unknown authority
Notice when I query 192.168.2.253, using the same cert - but its not in the CN or SAN
verify certificate: x509: certificate is valid for 192.168.9.253, not 192.168.2.253
If you really want to serve dns vs dot or doh, you should really take the time to configure it in a sane manner. Ie validation of the cert being used to encrypt the traffic.
If you are not going to take the time to do this, or if your not actually doing dot or doh - why would you setup your unbound to even listen on those ports, and tell it to use X cert, etc. Just something else to complex up your setup that could fail and cause you grief.
So - noticed my version of doggo was a bit behind, the newer version gives more error info..
I created a new CA (my machine doesn't trust)
$ doggo.exe @tls://sg4860.home.arpa nas.home.arpa nameserver=sg4860.home.arpa:853 error="tls: failed to verify certificate: x509: certificate signed by unknown authority\ntls: failed to verify certificate: x509: certificate signed by unknown authority" C:\tools\doggo-windows-x86_64 $ doggo.exe @tls://192.168.9.253 nas.home.arpa nameserver=192.168.9.253:853 error="tls: failed to verify certificate: x509: certificate signed by unknown authority\ntls: failed to verify certificate: x509: certificate signed by unknown authority"Even I put that CN and SANs for both the fqdn and IP.
If I switch back to the cert signed by CA my machine trusts.
$ doggo.exe @tls://192.168.9.253 nas.home.arpa NAME TYPE CLASS TTL ADDRESS NAMESERVER nas.home.arpa. A IN 3600s 192.168.9.10 192.168.9.253:853 $ doggo.exe @tls://sg4860.home.arpa nas.home.arpa NAME TYPE CLASS TTL ADDRESS NAMESERVER nas.home.arpa. A IN 3600s 192.168.9.10 sg4860.home.arpa:853 -
@johnpoz The cert I am using is not expired, it is for the Webgui and is valid until 2032
-
@pdavis unable to duplicate the problem here, but then again I don't use pfblocker in the way that it would interact with unbound. I only use it to manage some native aliases that I use in my rules.
Again going to ask - do you actually have a use case for dot? If not then that check box shouldn't be checked.
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 26.03.1 | Lab VMs 2.8.1, 26.03.1 -
This morning, I didn't find any issues.
unbound restarted several times over the last 24 hours.Maybe a openssl glitch ?
Makes me think about what Wietse said :NOTE: By turning on TLS support in 'Postfix', you not only get the ability to encrypt mail and to authenticate remote SMTP clients or servers. You also turn on hundreds of thousands of lines of OpenSSL library code. Assuming that OpenSSL is written as carefully as Wietse's own code, every 1000 lines introduces one additional bug into Postfix.
Swap "postfix" for "unbound" and "SMTP" for "DNS" and we're good.
@johnpoz said in Unbound DNS Resolver Periodically Non-Responsive:
do you actually have a use case for dot? If not then that check box shouldn't be checked.
I agree. That phrase should be printed on the top of every pfSense GUI page
But, imho, that's not the question here.
I know what that check box does.
DNS-over-HTTPS doesn't mention something like a disclaimer that it's usage has 'risks'. -
@johnpoz I totally agree with what you are saying, and have turned it off. No issues since - I will update if there are!
-
Going to do the same thing, I stop the testing.
