Unbound stops resolving, and Status/DNS Resolver tables become empty

LDS

pfSense 2.9 CE - but I saw it on 2.8.1 CE as well.

I have Unbound configured as recursive resolver (not forwarder). DNSSEC is enabled, but not SSL/TLS. Both IPv4 and IPv6 are active (my network is fully dual stack). DHCP is not enabled on the pfSense. RA advertises only the router.

Unbound answers only on two intefaces (DMZ and LAN), plus localhost. On the LAN, there is an internal Bind 9 DNS resolving internal domain addresses, which forwards first external ones to Unbound on the pfSense. Unbound has too a domain override to query the Bind server for the internal domain.

In the Advanced settings page, these settings are active:

Hide Identity
Hide Version
Query Name Minimization
Prefetch Suppor
Keep Probing
Message Cache Size is set to 10MB

There are explicit ACLs for DMZ and LAN subnets.

pfSense has Quad9 DNS as extrernal DNS resolvers:

9.9.9.9
149.112.112.112
2620:fe::f
2620:fe::9

DNS Resolution Behaviour is set to "Use local DNS, fallback to remote"

Sometimes, Unbound stop answering - it returns "REFUSED" only. When this happens, the tables under StatusDNS/Resolver become empty. Under normal log settings, I see nothing in the logs.

Restarting the service from pfSense GUI has no effect. I need to restart the whole pfSense system.

Hardware is a Supermicro E200-9A (Intel C3558) with 8 GB memory.

What could I do trying to pin down this issue?

SteveITS

@LDS Since you are forwarding, have you disabled DNSSEC in the DNS Resolver settings? That can cause issues if also forwarding. (TL,DR: trust Quad9 to do it)

LDS

As I wrote above "I have Unbound configured as recursive resolver (not forwarder). DNSSEC is enabled" :)

Unbound does not forwards, it's set as a recursive resolver, exactly to take advantage of DNSSEC.

The internal Bind 9 DNS server forwards to pfSense's Unbound first, and switches to recursion if Unbound fails.

SteveITS

@LDS Oh, you've set Quad9 for just pfSense, and are not forwarding to them in unbound? Then DNSSEC should be OK.

Unbound can be set to forward, in the "DNS Query Forwarding" section so I wasn't clear.

Do the DNS Resolver logs show anything? The service is running?

LDS

@SteveITS

Thank you for your time.

Yes, I set the Quad9 DNS in the pfSense General Settings page as a fallback, setting pfSense to query localhost first and then falling back to Quad9 DNS - just to ensure pfSense is able to keep on resolving adresseses if Unbound fails, since some firewall rules do use FQDN aliases. "DNS Query Forwarding" is NOT enabled in Unbound.

The service keeps running but only answers REFUSED. In the log, only INFO messages. I tried to post the relevant log here, but then the post is marked as spam and not submitted :(

Gertjan

@LDS said in Unbound stops resolving, and Status/DNS Resolver tables become empty:

I have Unbound configured as recursive resolver (not forwarder). DNSSEC is enabled, but not SSL/TLS.

In that case, your DNS was fine since the very first moment you started pfSense.
You can safely remove these :

9.9.9.9
149.112.112.112
2620:fe::f
2620:fe::9

as they are never used. Unbound won't used 4. Like never.
pfSense itself, some process, like when you execute a 'host google;com' on the command line, will use /etc/resolv.conf to find out who is or are the local DNS servers. Normally, that 127.0.0.1 and ::1 port 53 and it's unbound that listens on that port, and handles the request.
If you have listed 9.9.9.9 - 149.112.112.112 - 2620:fe::f - 2620:fe::9 in the GUI, these will also be in the /etc/resolv.conf and the OS that handles the DNS request will use some round robin scheme to use any of the IPs avaible, thus "127.0.0.1 ::1 9.9.9.9 - 149.112.112.112 - 2620:fe::f - 2620:fe::9". If it uses one of the last 4 , dnssec won't be used / makes no sense.
Unbound will only talk to the 13 root server, the TLD it got from these 13, and then the final domain name servers. 9.9.9.9 - 149.112.112.112 - 2620:fe::f - 2620:fe::9 will never be in hat path, as these are resolvers.

Btw : The "13", the TLDs don't use port 853, TLS, or DoH. it's "port 53 UDP and TCP if needed".
Maybe in the future ? But cost of switching from plain "port 53 UDP and TCP if needed" to purely TLS (TLS will be session based TCP) and encryption is way to much, even today.
There is are redit ot stackoverflow ? posts that handle the why part with more detailed.

@LDS said in Unbound stops resolving, and Status/DNS Resolver tables become empty:

Sometimes, Unbound stop answering

First, get a post-it and write on it : "!! set unbound log level back to normal !!" and stick it on top of your pfSense box.
Now which to a higher unbound log level like level 2 or go wild, level 3 :

as soon as unbound seems to fail, console or SSH access : option 8 :

dig @127.0.0.1 google.com +trace

The resolver log will show a lot of details.

Test if the unbound process is running :

ps aux | grep 'unbound'

Test who uses "port 53" (all interfaces) :

sockstat | grep ':53'

LDS

@Gertjan said in Unbound stops resolving, and Status/DNS Resolver tables become empty:

In that case, your DNS was fine since the very first moment you started pfSense.
You can safely remove these :
9.9.9.9
149.112.112.112
2620:fe::f
2620:fe::9
as they are never used. Unbound won't used 4. Like never.

I know Unbound won't use them, but pfSense itself? How this setting works, exactly?

I believed it meant that if the DNS on localhost doesn't answer, pfSense (and only pfSense) switches the the DNS set above. I'd like pfSense still wotks if Unbound becomes unresponsive, so services like ACME certificate renewal keeps on working. I know it won't resolve for other clients, and that's OK.

Unbound will only talk to the 13 root server, the TLD it got from these 13, and then the final domain name servers. 9.9.9.9 - 149.112.112.112 - 2620:fe::f - 2620:fe::9 will never be in hat path, as these are resolvers.

Of course, and that's fine, since I set it up as a recursive resolver. I just need pfSense itself keeps working, without Unbound.

First, get a post-it and write on it : "!! set unbound log level back to normal !!" and stick it on top of your pfSense box.

:-)

OK, I'll switch the logging level - just I didn't find a repeatable way of triggering the issue .

as soon as unbound seems to fail, console or SSH access : option 8 :
dig @127.0.0.1 google.com +trace
The resolver log will show a lot of details.

Test if the unbound process is running :
ps aux | grep 'unbound'

The only thing I'm sure is Unbound is still running, queries don't time out, they return "REFUSED" quickly.

Test who uses "port 53" (all interfaces) :
sockstat | grep ':53'

OK, thank you.

Gertjan

@LDS said in Unbound stops resolving, and Status/DNS Resolver tables become empty:

I believed it meant that if the DNS on localhost doesn't answer, pfSense (and only pfSense) switches the the DNS set above. I'd like pfSense still wotks if Unbound becomes unresponsive, so services like ACME certificate renewal keeps on working. I know it won't resolve for other clients, and that's OK.

That's how I understand how things work.
One 'minor' observation from me : unbound stops working situation !?
That only happens (for me) when the admin is being stupid again. And I know he can do stupid things., as I'm the admin. So, basically, when I'm not 'doing things' with pfSense, aka : not at work, it always works. And it's like this for the last decade (more).
I'm using a pfSense Plus, on 4100, with an UPS (as 'electricity' isn't the admin, but can mess up things also).
Btw : I'm m pretty sure your and my unbound binaries are 100 % the same. The only things that's different : you settings and mine.

LDS

One 'minor' observation from me : unbound stops working situation !?

It's random, as far as I can see. I'll also run a memory test as soon as I can - but right now the only service with issues is Unbound.

Until a couple of months ago I was not using Unbound - the LAN DNS server was the only resolver. Then I added a DMZ for some services, and started to use pfSense's Unbound so DMZ hosts didn't need to access the LAN DNS server. And it worked well until it didn't.

I'm using a pfSense Plus, on 4100, with an UPS (as 'electricity' isn't the admin, but can mess up things also).

There's an APC UPS feeding the pfSense - and apcupsd running on the pfSense itself to control it. I guess my Supermicro board is not much different from the Netgate 6100.

Btw : I'm m pretty sure your and my unbound binaries are 100 % the same. The only things that's different : you settings and mine.

Of course - maybe it's just a setting - if I only knew which one :-)

Gertjan

@LDS

It's a pity there isn't a pfSense DNS Monitoring tool, only the 'higher than default' log file which produces loads of log info that gets rotated away very fast. The default log info only shows some stat when unbound is stopped.
When these 'usage' stats are shown, it means that unbound was told to stop. Told to stop means : One level up, pfSense, told it to stop. And this stop is part of a restart sequence, so stop and the immediate start. This sequence exists because 'some' system event happened, most probably an interface invent.

I'm using this.
My own home made monitoring make it feel to unbound and other processes that I'm watching over their shoulder. Like a teacher @school, watching the kids really up close all the time.
This is an excellent way of getting the most out of them.
And I can see when ever I need to check up with pfSense wherever I am.