Random DNS Resolver failure with Quad9 over SSL

Gertjan

I agree.
Restarting unbound wouldn't change the WAN IP.
The connection gets restarted, a new (TCP) TLS stream gets negotiated, and that one seems to work.
If the previous TCP was really bad, then a TCP timeout would occur, which implies a re-connection. But that doesn't seem to happen ... so it was connected and kept 'alive', but very slow ?

Like you, I'd like to be able to point my finger at the source of the issue.
I presume that (from more to less) :
Quand9 has a lots of clients.
Not all, but still a lot use DNS over TLS and not the default UDP port 53 access.
There are pfSense users that use Quand9 with TLS, I'm pretty sure you are probably not the only one.
We all use the same (bit by bit) unbound program **

Only our pfSense (unbound, interface) settings - and your ISP differ.

So, what different between 'you' and 'me' ?

If possible, what happens when you use the none-TLS classic connection ?

Your pfSense version ?

If unbound (recent version) couldn't work with Quad9-TLS, we would have find that with one simple Google search ...

** With one difference : arm or x86-64 code. If you use an Intel processor, with have both the same binary.

bmeeks

@Gertjan said in Random DNS Resolver failure with Quad9 over SSL:

We all use the same (bit by bit) unbound program **

This is not always true. It depends on which version of pfSense someone is running. The unbound daemon in recent pfSense Plus updates is much more current than the daemon running in 2.7.2 CE. There were bugs in the older unbound versions that were addressed upstream.

One thing I've noticed here is posters seldom include their pfSense version (and thus the unbound or DNS Resolver version they are running). What works fine for someone using the DNS Resolver in pfSense Plus 24.11 might indeed be problematic under some conditions for a user running 2.7.2 CE.

My own impression by following this sub-forum is that many, many users shoot themselves in the foot by monkeying with the default DNS settings in pfSense. Unless you truly understand how DNS operates (the difference between resolvers and forwarders, how forwarding really works, what the root servers are, etc.) you should never monkey with the DNS settings on an out-of-the-box pfSense install. If you don't change a single thing it will just work. Unless you are a DNS Jedi master, once you start monkeying with forwarding, TLS, DNSSEC, etc., you can expect issues.

SteveITS

@bmeeks said in Random DNS Resolver failure with Quad9 over SSL:

more current than the daemon running in

Your post rang a bell...there was actually this, but it was fixed in 2.7.0/23.09:
https://redmine.pfsense.org/issues/14056

FWIW I've been using SSL at home to Quad9 without noticeable issues on 24.03 and now 24.11.

bmeeks

@SteveITS said in Random DNS Resolver failure with Quad9 over SSL:

@bmeeks said in Random DNS Resolver failure with Quad9 over SSL:

more current than the daemon running in

Your post rang a bell...there was actually this, but it was fixed in 2.7.0/23.09:
https://redmine.pfsense.org/issues/14056

FWIW I've been using SSL at home to Quad9 without noticeable issues on 24.03 and now 24.11.

Yeah, I didn't go back and research the history, but I do recall some early bugs in the DNS Resolver that got fixed with later releases. My comment was intended to remind folks there are multiple versions of some of these "problematic" binaries out there (unbound and kea), and something that works just fine on a recent 24.11 Plus installation may indeed behave differently on an older version.

Gertjan

@bmeeks said in Random DNS Resolver failure with Quad9 over SSL:

got fixed with later releases

wait .. no one here is using older, buggy versions, right ?

I presume @digitalgimpus uses 2.7.2. I've been using 2.7.2 half a year or so, and did some Quad9 testing. Worked just fine IRC, and I'm using pfSense with a load of hotel clients behind it, so if there was an issue they would have told me about it. After all, if a free service doesn't work, that would be considered as inadmissible (here in Europe), they would have asked for a a refund right immediately.

digitalgimpus

So here's how I have things configured, I don't think there's anything particularly unique going on here.

I've tried just the IPv4 hosts, I've tried IPv6 only, I've tried both.

nattygreg

Unbound does time out from time to time, use service watchdog to restart it. you will not see anymore glitch.

digitalgimpus

@nattygreg I have watchdog running. It sees the process as still up when this happens.

There are other times where unbound crashes or whatever and watchdog restarts successfully. If unbound was crashing it would be less problematic. The problem here is a prolonged dns outage because watchdog doesn't know to do anything.

Gertjan

@digitalgimpus

What is your pfSense version ?

Then here is a solution : do not do this (uncheck !) :

as every time a lease comes in on any LAN type interface, unbound gets restarted.
If you have have many LAN devices, and/or devices that are using wifi, this can get create a "x times a minute" unbound restart and as unbound needs anything from 5 to xx seconds to restart, you will get the impression unbound isn't running at all. And correct, its restarting all the time.
And the moment it is running, the service watchdog find it wasn't running a second ago, and adds its doses of restarts.
This issue is very known and very old, and a solution will be coming very soon now : pfSense 2.8.0 (pfSense Plus already has the solution).

michmoor

@keyser i agree here with you and I'm glad I'm not the only one. When i use Quad9, DNS breaks. No rhyme or reason. It can be a week or 2 weeks after i make the change. All external DNS resolution fails. At first it was weird where certain sites wouldn't resolve such as anything related to Microsoft. Ok strange but then after some time, everything external stopped resolving. This is only with Quad9.

I have since switched to Cloudflare - no issues.

digitalgimpus

@Gertjan said in Random DNS Resolver failure with Quad9 over SSL:

as every time a lease comes in on any LAN type interface, unbound gets restarted.
If you have have many LAN devices, and/or devices that are using wifi, this can get create a "x times a minute" unbound restart and as unbound needs anything from 5 to xx seconds to restart, you will get the impression unbound isn't running at all. And correct, its restarting all the time.
And the moment it is running, the service watchdog find it wasn't running a second ago, and adds its doses of restarts.
This issue is very known and very old, and a solution will be coming very soon now : pfSense 2.8.0 (pfSense Plus already has the solution).

I don't think this is the problem here.

If I use my ISP's DNS, or Google or CloudFlare, this isn't an issue. Only Quad9 requires a manual restart.

If this was the culprit, it should happen with all upstream providers regardless.

Gertjan

@digitalgimpus said in Random DNS Resolver failure with Quad9 over SSL:

I don't think this is the problem here

No need to think ^^ Fact check.

[25.03-BETA][root@pfSense.bhf.tld]/root: grep "start" /var/log/resolver.log
....
<30>1 2025-02-26T09:46:43.449085+01:00 pfSense.bhf.tld unbound 22263 - - [22263:0] info: start of service (unbound 1.22.0).
<30>1 2025-02-26T10:02:58.437287+01:00 pfSense.bhf.tld unbound 44152 - - [44152:0] info: start of service (unbound 1.22.0).
<30>1 2025-02-26T15:19:51.097535+01:00 pfSense.bhf.tld unbound 10684 - - [10684:0] info: start of service (unbound 1.22.0).
<30>1 2025-03-03T00:15:23.627116+01:00 pfSense.bhf.tld unbound 65579 - - [65579:0] info: start of service (unbound 1.22.0).

If your resolver(unbound) restarts a coupe of times a day, you'll be ok.
Several times per hour or even more : that's less optimal, or plain bad. Read again what has been said above ...
That said, using the watchdog and "Register DHCP leases in the DNS resolver" introduces race conditions. Many have tried and they all lost. See forum : hundreds or more posts about this subject)

nattygreg

@Gertjan did you patch unbound they are patches for that, that will stop inbound from restarting every time it gives out a lease. Do that first. Then set it up in watchdog. Yes a few times per day, it will restart, I know because it says connection loss when I’m streaming and it depends also sometimes unbound stops and will not restart therefore throwing everyone off the network.

Use the patches you can download patch in packages and then it will show recommended patches just install all.

Gertjan

@nattygreg

Read ... please.
I'm using 23.05-Beta, which is the latest and greatest.
All know patches are included in that included of pfSense. Maybe not the ones discovered after 5 February 2025.

To stop unbound from restating when new leases coming in, are when leases are renewed, uncheck "Register DHCP leases in the DNS resolver".
After all, by default, that option is not checked (by Netgate).

This situation is known since ... can't remember, 2012 ?!!

@nattygreg said in Random DNS Resolver failure with Quad9 over SSL:

because it says connection loss when I’m streaming

When the resolver restarts this will not influence or even break any connections already established.
After all the resolver (unbhund) handles DNS, which exists for us, humans. Your TV, phone, Pad, PC, etc etc uses ethernet traffic - not "host names". Only when a connection has to be created with a host name, for example, "www.youtube.com", then that "www.youtube.com" is translated ones into an IP address. pfSense's unbound and your device will then keep that resolved host name for a while (cached).
I see no reason why streaming stops when unbound restarts.
I fired up a Youtube and a netflix stream on my PC, and stopped unbound on pfSense for half a minute. Nothing stopped ...
And even when the netflix or youtube needed to resolve a publicity server host name, it will wait a bit before everything comes crashed down.

@nattygreg said in Random DNS Resolver failure with Quad9 over SSL:

sometimes unbound stops and will not restart therefore throwing everyone off the network

Nobody goes of the network, actually, the network works just fine.
Only resolving aka DNS doesn't work anymore. So, use the ancient method : use I addresses and things works very well.
I know, that tedious, people don't use numbers any more, and with Ipv6 it close to impossible.

@nattygreg said in Random DNS Resolver failure with Quad9 over SSL:

Use the patches you can download patch in packages and then it will show recommended patches just install all.

so : nope - no patches exist for me.
I did create my own patches, they are listed at the top of the page.

IMHO, the issue "Random DNS Resolver failure with Quad9 over SSL" can't be resolved with a patch.

nattygreg

@Gertjan you’re right I don’t have this problem since I only connect to quad9 over TLS. I read you said SSL, but from what I have read I could be wrong dns connection are over TLS, HTTPS, again I could wrong the ports are 53, 853, 443 if you are able to connect by SSL maybe and I said maybe using wrong to connect to Quad9

nattygreg

@digitalgimpus I had that same issue, when it happens look at the dns status under ping it would say zero, remove that dns and use another one

digitalgimpus

@Gertjan said in Random DNS Resolver failure with Quad9 over SSL:

@digitalgimpus said in Random DNS Resolver failure with Quad9 over SSL:

I don't think this is the problem here

No need to think ^^ Fact check.

Double checked. No restarts. No evidence of restarts.

Which isn't surprising. If it failed to come back up, watchdog would catch that and I'd see email's at a minimum.

Gertjan

@digitalgimpus said in Random DNS Resolver failure with Quad9 over SSL:

Double checked. No restarts. No evidence of restarts.

So all your LAN(s) device(s) have a static IP setup ? You don't use DHCP anywhere ?

This :

means that on every (new) lease (renewal) the resolver (unbound) gets restarted.

digitalgimpus

@Gertjan I'm well aware what it means.

And once again: If this was a problem on restart, that would be obvious and consistent in the logs. It would also be predictable and self remedying. It obviously isn't that.

In fact, the fix for the problem is to restart, which is why it's perplexing you think that's the problem.

bmeeks

@digitalgimpus:
Might your issue be related to this problem discovered by the OPNsense users?

https://forum.opnsense.org/index.php?topic=44414.0

I have not investigated this, and the failure reported over there seems to be more immediate and permanent (as in not random), but there still might be some relation. It seemed to only be impacting users over there attempting to use Quad9 with TLS (DoT).

Also found a related thread on a different forum here with a potential solution: https://discourse.pi-hole.net/t/cant-get-quad-9-dot-to-work-using-pi-hole-and-unbound/75683/3.

Another GitHub issue posted on the NLnetLabs unbound repo: https://github.com/NLnetLabs/unbound/issues/1247. This one sounds related as well.

And one more from the unbound GitHub repo issues list (this one closed, but the fix is not in pfSense yet): https://github.com/NLnetLabs/unbound/issues/1202.

So, it looks like unbound may have some internal TLS issues that seem to really manifest themselves with Quad9's servers when using DoT. Possible short-term solution is disable use of Quad9 and try to duplicate what you desire using Cloudflare until unbound's Quad9 TLS issues are resolved and the patched binary is pulled into pfSense.