Random DNS Resolver failure with Quad9 over SSL

digitalgimpus

@Gertjan said in Random DNS Resolver failure with Quad9 over SSL:

@digitalgimpus said in Random DNS Resolver failure with Quad9 over SSL:

I don't think this is the problem here

No need to think ^^ Fact check.

Double checked. No restarts. No evidence of restarts.

Which isn't surprising. If it failed to come back up, watchdog would catch that and I'd see email's at a minimum.

Gertjan

@digitalgimpus said in Random DNS Resolver failure with Quad9 over SSL:

Double checked. No restarts. No evidence of restarts.

So all your LAN(s) device(s) have a static IP setup ? You don't use DHCP anywhere ?

This :

means that on every (new) lease (renewal) the resolver (unbound) gets restarted.

digitalgimpus

@Gertjan I'm well aware what it means.

And once again: If this was a problem on restart, that would be obvious and consistent in the logs. It would also be predictable and self remedying. It obviously isn't that.

In fact, the fix for the problem is to restart, which is why it's perplexing you think that's the problem.

bmeeks

@digitalgimpus:
Might your issue be related to this problem discovered by the OPNsense users?

https://forum.opnsense.org/index.php?topic=44414.0

I have not investigated this, and the failure reported over there seems to be more immediate and permanent (as in not random), but there still might be some relation. It seemed to only be impacting users over there attempting to use Quad9 with TLS (DoT).

Also found a related thread on a different forum here with a potential solution: https://discourse.pi-hole.net/t/cant-get-quad-9-dot-to-work-using-pi-hole-and-unbound/75683/3.

Another GitHub issue posted on the NLnetLabs unbound repo: https://github.com/NLnetLabs/unbound/issues/1247. This one sounds related as well.

And one more from the unbound GitHub repo issues list (this one closed, but the fix is not in pfSense yet): https://github.com/NLnetLabs/unbound/issues/1202.

So, it looks like unbound may have some internal TLS issues that seem to really manifest themselves with Quad9's servers when using DoT. Possible short-term solution is disable use of Quad9 and try to duplicate what you desire using Cloudflare until unbound's Quad9 TLS issues are resolved and the patched binary is pulled into pfSense.

digitalgimpus

@bmeeks The first two look like something with the cert bundle included with the distro vs application to me, which would explain losing connectivity all at once, they likely updated a cert and the root cert wasn't in their bundle. That doesn't seem to be the case here. A restart wouldn't fix a problem like that.

The second two seem potentially more related, though not sure exactly how to verify if that's the case offhand.

bmeeks

@digitalgimpus said in Random DNS Resolver failure with Quad9 over SSL:

The second two seem potentially more related, though not sure exactly how to verify if that's the case offhand.

I agree. If the problem is something in the unbound binary, then you are stuck until a subsequent pfSense update comes out with a new unbound package bundled within. The binary is not fixable via any sort of patch in pfSense. Binary portions of packages are compiled against the base OS kernel and get more or less "locked" to the pfSense kernel version and thus they must both be updated together.

Once in a blue moon you can get by with manually installing an updated binary package, but the chances are high that doing so can wreck the pfSense installation by overwriting shared system libraries with newer versions that might be required for the updated binary package.

Gertjan

@digitalgimpus said in Random DNS Resolver failure with Quad9 over SSL:

The second two seem potentially more related, though not sure exactly how to verify if that's the case offhand

This second case, doesn't that mean this isn't a "quad9", but will also affect all the others like 1.1.1.1 over TLS to name one.
Which means the subject changes to a global "Unbound won't do any forwarding over TLS anymore".
I'll add to that : I really presume a lot of people are forwarding over TLS so this will impact them all.

What is your unbound version ?
Run "unbound -V" to see it.

digitalgimpus

@Gertjan I'm running 1.18.0.

I've tried switching temporally to 1.1.1.1 a few days ago and that was stable. So it's clearly not impacting everything.

I've also tried IPv4 vs IPv6 no difference.

SteveITS

@digitalgimpus FWIW 24.11 is using Version 1.22.0. I've not noticed any problems using Quad9 with SSL/TLS enabled.

digitalgimpus

@SteveITS Yea, hopefully in 2035 when 2.8 drops, I can give that a go. ;)

My suspicion is it's an unbound bug of some kind.

bmeeks

@digitalgimpus said in Random DNS Resolver failure with Quad9 over SSL:

I'm running 1.18.0.

Yes, it is helpful in the future for posters having issues with DNS or DHCP to post the pfSense branch they are using (CE or Plus and which version). The underlying binary components of the DNS Resolver and the DHCP server are quite different between the current 2.7.2 CE branch and the latest 24.11 Plus branches, for example. A number of unbound bugs were fixed upstream in the newer version released with pfSense Plus 24.11 as compared to the much older version bundled back with pfSense 2.7.2 CE.

Ditto for kea, the new DHCP server that came out first with 2.7.2 CE. The kea binary and its connections to the DNS Resolver are quite different from (and much more feature-rich) than the original kea still bundled with pfSense CE.

When a poster does not state their pfSense version (and thus, by extension, the version of unbound or kea) they are running, it is easy for responders to make false assumptions. For instance, "it's working fine for me" might be true if you are using the latest unbound on pfSense Plus 24.11, but something may well be broken on the older unbound that is bundled with pfSense CE 2.7.2. This is a natural consequence of the growing divergence between features and versions of packages included in pfSense 2.7.2 CE and those of pfSense Plus 24.11 (and soon, 25.03).