Major DNS Bug 23.01 with Quad9 on SSL

SteveITS

@nononono This has come up a few times here actually. You're referring to disabling DNSSEC when forwarding?

https://support.quad9.net/hc/en-us/articles/4433380601229-Setup-pfSense-and-DNS-over-TLS
"Disable Enable DNSSEC Support if enabled.
DNSSEC is already enforced by Quad9, and enabling DNSSEC at the forwarder level can cause false DNSSEC failures."

It was working in 22.05 and earlier for me but problematic in 23.01.

cmcdonald

I am working on replicating this in order to investigate. My usual setup is to just let Unbound do the recursive resolving for me without forwarding to another service.

nononono

@steveits the only reliable fix has been disabling SSL - so far I have tested:

DNSSEC enabled SSL/TLS disabled, runs no failure
DNSSEC enabled SSL/TLS enabled, completely unusable
DNSSEC disabled SSL/TLS enabled, intermittent crashes, nothing useful in logs, an unbound restart will get it working but it periodically will fail and stop serving any DNS responses
DNSSEC disabled SSL/TLS disabled, runs no failure

Running pfsense on a Netgate 7100

Cylosoft

@nononono said in Major DNS Bug 23.01 with Quad9 on SSL:

@steveits the only reliable fix has been disabling SSL - so far I have tested:

DNSSEC enabled SSL/TLS disabled, runs no failure

DNSSEC enabled SSL/TLS enabled, completely unusable

DNSSEC disabled SSL/TLS enabled, intermittent crashes, nothing useful in logs, an unbound restart will get it working but it periodically will fail and stop serving any DNS responses

DNSSEC disabled SSL/TLS disabled, runs no failure

Running pfsense on a Netgate 7100

We have been working through this on about a dozen pfSense boxes. I've found the same thing. Quad9 says not to use DNSSEC. So the last 2 options are the way.

We switched half of them to Cloudflare DNS with TLS. The other half TLS disabled, but still on Quad9. It looks like the switch to Cloudflare is going to be the temp fix. They have Malware filtered options that are pretty similar to Quad9.

SteveITS

@cmcdonald If it helps, IIRC:

no issues for the first couple hours or so after install
suddenly random issues connecting to I think multiple sites on my phone
LinkedIn app wasn't loading data or images
went to my PC, linkedin.com wouldn't resolve
no issues since disabling DNSSEC on Feb. 18

Obviously Quad9 thinks it's a potential issue since they recommend disabling it. But it's probably been enabled on this router since Plus 21.x sometime. I had a fairly early 2100. (for clarity, I did have to reinstall 22.05 due to the EFI bug, restore config, and updated to 23.01)

@nononono Hmmm, I just checked and do have "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers" enabled. If you have unbound crashing/stopping, that's not what I was experiencing then.

p1erre

@nononono as mentioned in the docs here you have to disable DNSEC for DNS over TLS and forwarding mode. I've had the same issue like you (with different forwarding server), but disabling DNSEC helped me. DNSEC enabled with forwarding server was woring in 22.05 and earlier versions.

Cylosoft

@p1erre said in Major DNS Bug 23.01 with Quad9 on SSL:

@nononono as mentioned in the docs here you have to disable DNSEC for DNS over TLS and forwarding mode. I've had the same issue like you (with different forwarding server), but disabling DNSEC helped me. DNSEC enabled with forwarding server was woring in 22.05 and earlier versions.

I had hoped that was the fix also, but with Quad9 we saw on every device that at some point with-in 24 hours they would stop returning results. The DNS server was up and would respond, it just wouldn't look things up anymore.

We are about 48 hours into half a dozen devices on Quad9 with TLS off and no issues. Also another half dozen are having no issues with Cloudlfare DNS and TSL on.

Cylosoft

@cmcdonald said in Major DNS Bug 23.01 with Quad9 on SSL:

I am working on replicating this in order to investigate. My usual setup is to just let Unbound do the recursive resolving for me without forwarding to another service.

I hope you can sort it out. It's going to be tough to duplicate. It happens in the 18 hour to 24 hour of being up. It responds just doesn't return results unless they are cached. And sometimes it would just start working again on it's own. I haven't found a great way to test other than end users complaining.

bigsy

@nononono Same issue here on a NG2100 running 23.01.

Running fine with DNSSEC disabled & SSL/TLS enabled on 22.05 but I've had to disable SSL/TLS on 23.01 to avoid intermittent DNS failures with Quad9.

This is on IPv4/IPv6 with the patch applied for redmine #13851.

jimp

Check the output of sockstat | grep unbound when it works and when it doesn't.

I thought they fixed it but a while back unbound had an issue where it couldn't reuse SSL connections on the same open sockets so in some cases they kept piling up.

EDIT: This is what I was thinking of, but it's been fixed/closed for a couple years now: https://github.com/NLnetLabs/unbound/issues/47

Isotope1842

This bug is still a problem, for Cloudflare DNS users as well. DNS stopped working on all clients immediately after updating to 23.01 this morning (Mar 15, 2023).

After finding this thread, I UNchecked only the following setting in the DNS Resolver settings, and DNS started working again:

This is clearly a bug because the current pfSense documentation itself advises that this setting be checked:

https://docs.netgate.com/pfsense/en/latest/recipes/dns-over-tls.html

jimp

That document is about configuring that specific feature, it's not "advising" that setting be checked in a general fashion for everyone.

It's working for many people and breaking for a few, but it's still not clear why.

Isotope1842

@jimp Bullet number four of the previously included screenshot explicitly directs the user to check that setting. The pfSense documentation is actually more than advising. It is directing.

jimp

@isotope1842 said in Major DNS Bug 23.01 with Quad9 on SSL:

@jimp Bullet number four of the previously included screenshot explicitly directs the user to check that setting. The pfSense documentation is actually more than advising. It is directing.

The page you quoted is a document about configuring DNS over TLS -- it's saying to check that if you want DNS over TLS. That's what that entire document is for.

It's not a part of the general setup or DNS resolver docs and so on. It's a recipe for users who are interested in that feature and want to know how to set it up.

There is nothing saying users should be following all of those recipes, they're there for reference for things users may want to do.

Isotope1842

@jimp The context in which the instructions are provided is exactly for users who want to use an upstream DNS over TLS provider. The original poster here was reporting a bug in using Quad9 and I reported the same bug when using Cloudflare.

See the top of the same linked documentation:

"Pick a DNS over TLS upstream provider, such as a private upstream DNS server or a public service like Cloudflare, Quad9, or Google public DNS."

Following the instructions prior to 23.01 worked. Immediately after upgrading to 23.01, DNS fails until unchecking a single setting.

jimp

Yes, and? They still work for that and for many providers and users who want to enable that feature.

But you're trying to imply this is something the docs have told everyone they should be doing which isn't true. They don't advise everyone to do it, just people who are interested in that feature.

But none of this is helpful. We still need more information about how and why it's failing. We have yet to be able to reproduce this in a lab environment, and there are plenty of us running DNS over TLS without problems.

Isotope1842

@jimp Steps to reproduce the problem:

Install pfSense 22.05 on a netgate device.
Configure DNS over TLS with Cloudflare.
Upgrade netgate device to pfSense 23.01.
Observe broken DNS for downstream clients.

jimp

@isotope1842 said in Major DNS Bug 23.01 with Quad9 on SSL:

@jimp Steps to reproduce the problem:

Install pfSense 22.05 on a netgate device.

Configure DNS over TLS with Cloudflare.

Upgrade netgate device to pfSense 23.01.

Observe broken DNS for downstream clients.

It is not that simple.

I have multiple lab VMs using DNS over TLS to Cloudflare and Quad9 that successfully resolve and have no problems.

stephenw10

Mmm, I assume you are seeing the same intermittent behaviour as other users? It's not failing for every query with that configuration?

Isotope1842

@jimp I just re-checked that single setting. DNS appears to continue to work. Curious to see whether it starts to fail again at some point.