Major DNS Bug 23.01 with Quad9 on SSL

[n8rfe]

Just reporting in here that I have the same issues with 23.01 and Quad 9 DNS when using unbound DNS

My settings were as follows on 22.05 since 22.05 was released with no issues with internet and DNS resolution.

• pfsense ubound DNS resolver enabled

• Using forwarding to Quad 9 DNS servers

• DNSSEC unticked

• Use SSL/TLS for outgoing DNS Queries to Forwarding Servers TICKED

Upgraded to 23.01 this weekend just gone 2nd April 2023. No other settings have changed on LAN network and no other firmware's have been upgraded on any devices on network so I am confident that the only change was the upgrade to pfsense from 22.05 TO 23.01.

Everything worked fine yesterday 3rd April until this morning when I now get multiple DNS resolution issues and cannot reach websites.

I have now changed the following single setting.

• Use SSL/TLS for outgoing DNS Queries to Forwarding Servers UNTICKED

Websites are now resolving correctly.

I will continue to monitor DNS resolution over the coming days to see if the issue reappears and report back.

[joedan]

@johnpoz said in Major DNS Bug 23.01 with Quad9 on SSL:

In your home? how many devices / users - you have a lot shit banging its head trying to resolve stuff your blocking.. That seems really high for a home..

Yes I have around 35 devices, a few that are quite bad including a Swann Security DVR that is taking up 25% of that DNS traffic calling out dropbox domains even though I don't have that function setup.

Just wanted to loop back and advise I did resolve my issue however it involved going back about 6 backup iterations and manually reapplying all my config changes one at a time and am now back on DNS over TLS (Cloudflare) with 23.01, and all recommended patches applied. I did also move ISP's during that time of panic so unfortunately have nothing further to contribute here because I was under pressure to get things working again quickly and made significant change in such a short time.

I was never using Quad9 though which appears to be an ongoing issue with other posters.

[Gertjan]

@joedan said in Major DNS Bug 23.01 with Quad9 on SSL:

a few that are quite bad including a Swann Security DVR that is taking up 25% of that DNS traffic calling out dropbox domains even though I don't have that function setup

Introduction : read this first.

If a device asks ever x seconds to resolve an identical host name, then, at first, your uplink isn't used : pfSense will just serve the same info edit : from local DNS cache, as log as this is TTL permitted end edit to the device again, and again. You're just burning some watt hours at your place.
But if the TTL is short, like mere minutes or less, then, yeah, the WAN gate is opened, and you start to activate a whole lot more of software/hardware on both sides

The thing is : as I said in the other thread : if you were looking at the WAN gate of quad9 (because, lets imagine, you work over there) : what would you do if you saw that some IP was clearly misbehaving ? => You throttle ?! As "just scale up the system up once more" without any cash flow in return will kill your job.
So, on your home site : your entire DNS just got delayed.

And that's of course a 'pfSense DNS bug'

[stephenw10]

I agree this could be Quad9 throttling responses. But still it would have been the same in 22.05.

[JeGr]

@skogs said in Major DNS Bug 23.01 with Quad9 on SSL:

I find this to be non-ideal, but at least functional.

Not wanting to start a big base line debate but why is letting a resolver do resolving non-ideal? Letting your box do DNS resolving on its own without any upstream forwarder that can be censored is a much more resilient setup (as we see) then having a single point of failure like a big DNS forwarder that throttles or censors DNS lookups (or in case of quad9 is jurisdically forced to do so). Just to try to hide your DNS queries from your ISP?

Just asking because I'm interested.

[johnpoz]

@jegr said in Major DNS Bug 23.01 with Quad9 on SSL:

Just to try to hide your DNS queries from your ISP?

hahah - which isn't really happening anyway. So I do a dns query for www.amazon.com, which I want to hide from my isp ;)

So my isp doesn't see the dns query, but they see me go to IP address 1.2.3.4 which pretty obvious is amazon.com - and here is the big one.. They still see what you are asking for - its just not via dns..

Until such time that esni, which is dead - long live ech.. Is everywhere - you are just sending your sni in the clear in your handshake - so who exactly are you hiding your dns query from? And for what purpose?

[n8rfe]

@n8rfe said in Major DNS Bug 23.01 with Quad9 on SSL:

Just reporting in here that I have the same issues with 23.01 and Quad 9 DNS when using unbound DNS

My settings were as follows on 22.05 since 22.05 was released with no issues with internet and DNS resolution.

• pfsense ubound DNS resolver enabled

• Using forwarding to Quad 9 DNS servers

• DNSSEC unticked

• Use SSL/TLS for outgoing DNS Queries to Forwarding Servers TICKED

Upgraded to 23.01 this weekend just gone 2nd April 2023. No other settings have changed on LAN network and no other firmware's have been upgraded on any devices on network so I am confident that the only change was the upgrade to pfsense from 22.05 TO 23.01.

Everything worked fine yesterday 3rd April until this morning when I now get multiple DNS resolution issues and cannot reach websites.

I have now changed the following single setting.

• Use SSL/TLS for outgoing DNS Queries to Forwarding Servers UNTICKED

Websites are now resolving correctly.

I will continue to monitor DNS resolution over the coming days to see if the issue reappears and report back.

So further update.

It appears that the outages I was experiencing were nothing to do with pfsense at all.

My Virgin Media ISP in the UK had a UK wide intermittent outage that has been affecting customers for the past 48 hours.

Please ignore my post above. TLS DNS is working just fine for me on 23.01

[Gertjan]

@n8rfe said in Major DNS Bug 23.01 with Quad9 on SSL:

Please ignore my post above. TLS DNS is working just fine for me on 23.01

I'm pretty sure that even @nononono will post back in the future here with a "solved" statement.

The initial subject of the thread "Major DNS Bug 23.01 with Quad9 on SSL " boils down to .... nothing.

[Cylosoft]

@gertjan said in Major DNS Bug 23.01 with Quad9 on SSL:

@n8rfe said in Major DNS Bug 23.01 with Quad9 on SSL:

Please ignore my post above. TLS DNS is working just fine for me on 23.01

I'm pretty sure that even @nononono will post back in the future here with a "solved" statement.

The initial subject of the thread "Major DNS Bug 23.01 with Quad9 on SSL " boils down to .... nothing.

I don't think I'd go as far as saying it's nothing. I see people having issues here, Reddit, Twitter. Settings that work in 22 don't work in 23. The defaults with forwarding don't work. With 22 you could turn on forwarding and turn on TLS and it worked. Now not so much. We have done several dozen 22 to 23 updates for customers now with different ISPs and different setups and every time we end up having to sort out DNS issues.

I think the biggest issue is we have a few customers that need to have DHCP entries included in DNS. It worked fine with 22. With 23 the restarts eventually either crash unbound entirely or it just stops responding to DNS for periods of time. We have had to setup a separate DNS server for these customers.

[JeGr]

@cylosoft said in Major DNS Bug 23.01 with Quad9 on SSL:

I think the biggest issue is we have a few customers that need to have DHCP entries included in DNS. It worked fine with 22. With 23 the restarts eventually either crash unbound entirely or it just stops responding to DNS for periods of time. We have had to setup a separate DNS server for these customers.

Sorry but it's documented for far longer then 22.x that DHCP registration with unbound is a "avoid!" situation due to it restarting every time a dhcp client registers its name. Either use dnsmasq (the forwarder) or don't use DHCP registration. That is literally written in the docs for far longer then "it came up with 23.01": https://docs.netgate.com/pfsense/en/latest/services/dns/resolver-config.html
Besides, a fully dynamic DHCP address shouldn't ever be needed as a name at all. A client doesn't need to have dns resolution of its name. And if it does need it because it runs services under a FQDN/hostname, it should have a static/DHCP-set static IP so it won't accidentally change and thus would have a dns name, that can be read at runtime by unbound by the "static DHCP" option which is fine and doesn't cause problems.

[Cylosoft]

@jegr said in Major DNS Bug 23.01 with Quad9 on SSL:

@cylosoft said in Major DNS Bug 23.01 with Quad9 on SSL:

I think the biggest issue is we have a few customers that need to have DHCP entries included in DNS. It worked fine with 22. With 23 the restarts eventually either crash unbound entirely or it just stops responding to DNS for periods of time. We have had to setup a separate DNS server for these customers.

Sorry but it's documented for far longer then 22.x that DHCP registration with unbound is a "avoid!" situation due to it restarting every time a dhcp client registers its name. Either use dnsmasq (the forwarder) or don't use DHCP registration. That is literally written in the docs for far longer then "it came up with 23.01": https://docs.netgate.com/pfsense/en/latest/services/dns/resolver-config.html

Regardless of that. It works with 22. It doesn't with 23. So to our customers 23 doesn't work.

[Gertjan]

@cylosoft said in Major DNS Bug 23.01 with Quad9 on SSL:

I see people having issues here, Reddit, Twitter.

Sure.
For me, issues can be found here : Issues exist as those can be reproduced.

@cylosoft said in Major DNS Bug 23.01 with Quad9 on SSL:

Settings that work in 22 don't work in 23

What settings ?

@cylosoft said in Major DNS Bug 23.01 with Quad9 on SSL:

The defaults with forwarding don't work. With 22 you could turn on forwarding and turn on TLS and it worked.

So this (last change July 2022) DNS Forwarder Configuration or the TLS equivalent Configuring DNS over TLS or How to Enable DNS over TLS on pfSense with Cloudflare are not correct ?
They looks ok to me, I've used that page to set up forwarding to 1.1.1.1 and the IPv6 equivalent, with the cert host names.
https://1.1.1.1/help confirmed me all was ok - I was forwarding.

I'm pretty sure my ISP isn't DNS bugging on me.
My pfSense can create enough entropy (see above) so my pfSense can start many (like a lot) TCP connections over TLS but I found out ones what happens with a web server (and mail etc) if entropy isn't available any more. TLS goes down fast....
Maybe this entropy thing is a non issue. I don't know.

@cylosoft said in Major DNS Bug 23.01 with Quad9 on SSL:

that need to have DHCP entries included in DNS.

That one is known for years now.
Fast solution : DHCP Mac static lease.

Next best :

@cmcdonald said in Major DNS Bug 23.01 with Quad9 on SSL:

I'm hopeful it will make it in 23.05.

@cylosoft said in Major DNS Bug 23.01 with Quad9 on SSL:

Reddit, Twitter.

No, thanks.
I'm not a #metoo guy. I prefer the #menot.
I don't like issues for myself, neither for others.
If I find one, I'll try to find out the 'why' part'.

Btw : I was posting in the scope of this thread

So we all know what our choices are asap.

[JeGr]

@cylosoft We have clients that use the option just fine. So is it a 23.01 problem or a "your configuration/setup" problem?
Besides - that isn't the context of this thread where it's about a forwarding DNS problem that seems to disappear when I read those responses and seems more related to problems of Quad9.

But if that's the biggest problem you see, I'd say let's open a separate thread for the DHCP-register / crash problem. Last time I looked at those reports here, most cases where related to pfBlocker+Unbound in combination though. You're running pfB in those configs, too? Just trying to analyze the context/root of the problem :)

Cheers

[Cylosoft]

@jegr said in Major DNS Bug 23.01 with Quad9 on SSL:

@cylosoft We have clients that use the option just fine. So is it a 23.01 problem or a "your configuration/setup" problem?
Besides - that isn't the context of this thread where it's about a forwarding DNS problem that seems to disappear when I read those responses and seems more related to problems of Quad9.

But if that's the biggest problem you see, I'd say let's open a separate thread for the DHCP-register / crash problem. Last time I looked at those reports here, most cases where related to pfBlocker+Unbound in combination though. You're running pfB in those configs, too? Just trying to analyze the context/root of the problem :)

Cheers

Yeah I've gone down this road. I understand all the context. I've been on a bunch of these threads posting including this one. We have "fixed" all of our customer configurations except the DHCP. I think the main issue is that the defaults don't work with forwarding and TLS. So generally speaking everyone with forwarding likely has a similar config and now is going down slightly different variations trying to find stable.

We had a lot of customers with Quad9, TLS, forwarding. As in 24+ customers when we paused the v23 upgrades. We thought it was Quad9 originally. But switching to CF DNS didn't fix it. We also have some customers running Quad9 with TLS on a stand alone DNS server taking most of the DNS traffic and the PF box doing Quad9 with TLS. The PF box will stop and the stand alone DNS box has no issues while using the same IP, and literally traffic going out through the PF box.

Turning off forwarding or disable TLS with forwarding seems to be a fix. But for those that want forwarding and TLS we have had to disable DNSSEC, enable serve expired, and increase the message cache size. We had several customers where unbound would completely crash about once a week up until we increased message cache size.

[stephenw10]

Hmm, something odd there. I use DoT forwarding to Google DNS on my own firewall and I've not seen any issues.

[SteveITS]

@gertjan said in Major DNS Bug 23.01 with Quad9 on SSL:

What settings ?

https://forum.netgate.com/post/1091473

If you're trying to test with forwarding enabled, leave it on for a few days. It doesn't happen immediately. In the past couple months several have said they have to turn off TLS as well but that was not my experience, nor is that noted in the Quad9 KB...however I do not recall to where they were forwarding.

[Cylosoft]

@stephenw10 said in Major DNS Bug 23.01 with Quad9 on SSL:

Hmm, something odd there. I use DoT forwarding to Google DNS on my own firewall and I've not seen any issues.

It's not easy to troubleshoot. Especially when customers have limited patience because it will essentially take the internet down for users. It doesn't show up right away. Most seem to be with-in a day or two. But a few customers it would take a week. We haven't sorted out if it's volume based or if it's time based or what.

If you were doing home network volume and had 15 min outages of DNS every couple of days I'm not sure most people would notice. Spin up a docker of uptime kuma or something similar and have it generate DNS queries and see if you really are up 100%.

We did add DNS check monitoring and more logging to a bunch of customer locations so we can try to sort things out better.

[SteveITS]

@steveits I'm just going to point out, if this takes a while to appear then on any router where unbound IS restarting regularly, that may hide it. At my home the pfBlocker overnight update restarts it because DNSBL is on.

[joedan]

Well unfortunately this is still an issue for me despite my efforts to change ISP and restore / rebuild my configuration.

Although I am not using Quad9 but Cloudflare DNS over TLS (without DNSSEC enabled).

So back to plain old port 53 forwarding to Cloudflare.

Whilst I had some quiet time I did run some dnsperf load testing with an average of 81 queries per second from an Ubuntu VM and managed to recreate what I intermittently see when this issue occurs. DNS over TLS breaks fairly quickly and completely stops DNS queries. I have to restart unbound to fix it.

Errors in the syslog are as before when it occurred randomly..

[54217:3] debug: tcp error for address 1.1.1.1 port 853
[54217:0] debug: outnettcp got tcp error -1

I did bump up some of the unbound cache settings, buffers and queries per thread but it still borks up quite quickly.

Running the same forwarding to Cloudflare without DNS over TLS I can run around 165 queries per second, it eventually slows down but out of the 3 or 4 times I only managed to break it once where DNS just stopped, otherwise it appeared to recover.

If there is anything I can adjust or commands run, I am happy to give it a go if this test can lead us closer to figuring out what’s going on?

[Gertjan]

@joedan

Interesting.

The common factor is "1.1.1.1".
'Normal' DNS uses port 53, most probably most traffic is UDP. Request and answer will always fit in just one Ethernet packet.
When using port 853, it uses DNS over TLS, so traffic can only be TCP. I wonder if just one ethernet pack can handle the entire DNS sequence.
Still, both methods used would be a fraction of your your WAN throughput.

When you visit with your PC a web site, or your mail client drop a mail into your SMTP mail server, or when it retrieves a mail, all traffic will use TLS also.
Small difference : this traffic isn't generated by pfSense, pfSense just passes the 'packets' not knowing that it is 'TLS' stuff

Did someone do this test :
Instead of having pfSense being the forwarder, contact "1.1.1.1" on behalf of the entire LAN, what about setting up the DNS (statically) so they all get their DNS from "1.1.1.1" using TLS (port 853) ? I understood that Windows 11 can do DNS over TLS (Windows 10 needs an extra program to be able to do DNS over TLS).
If the issue isn't local or 'pfSense' but somewhere on the link (ISP) or with "1.1.1.1", the issue should be the same.
If the issue goes away : that a big finger pointing to pfSense::Unbound.

I tend to think : no way .... as me using unbound 1.17.1 (pfSense 23.01) as a forwarder too 1.1.1.1:853 works just fine. Maybe my LANs are to small / not enough devices hammering out enough DNS requests ( a hotel, with many clients connected on the captive portal all day long ) to make the issue show up ?

@steveits said in Major DNS Bug 23.01 with Quad9 on SSL:

It was working in 22.05 and earlier for me but problematic in 23.01.

Humm, I didn't test that.
I followed a long and tedious seminar years ago, to learn about DNSSEC, as I wanted to make my own DNS server DNSSEC 'ready'. It was pure pain. I really had the impression that they just made world's most used Internet service (DNS) also the most complicated one.
But it worked out : https://dnsviz.net/d/brit-hotel-fumel.fr/dnssec/
I admit, I don't know what what unbound does if it was asked to do 'dnnsec' while forwarding.
Does it, at first, just get the DNS request and forward "what is the A of 'www.google.com'" to "1.1.1.1" and waits for an answer. And then, in paralellel ( ?) it also gets the RSIG DNSSEC records of the TLD of dot com and then the RSIG records of the DNS name server of google.com to re create from the root key (preloaded and initialized unbound startup - its the upper "DNSKEYalg=8, id=20326 - 2048 bits" key you can see at the DNSSEC trust chain, see my link) to bottom the validity of the chain ? ?
I don't know if 1.1.1.1 will honor these RSIG DNSSEC requests.
No ... wait ....
These RSIG will get send over '53' to the corresponding zone ...they can't be send to 1.1.1.1.
And if they ware, 1.1.1.1 would surely ditch them.

What I do know : this means unbound is still resolving .... while forwarding, and that doesn't make sens.
Also : the root key "20326 " is already known at unbound start, it doesn't change often ( key roll over is a admin's DNS nightmare )
But when unbound contacts the TLD domain server (like dot com) it will ask for a NS record of the google.com domain name server AND the zone RSIG at signs the dot come zone at the same moment.
The same thing will happen when the domain name server is asked for a A for the ending google.com. The entire trust chain is calculated from top to bottom, and if ok, the answer, the A record is made available for the requesting client.

So, yeah, doing DNSSEC while forwarding is like putting gasoline in a Tesla.
You would still be able to drive the Tesla, but thinks would become very stinky at the long run.
I even wonder where you would put the gasoline.
The slights spark would even explode the car as their is not a gas tank to hold it.

It doesn't make sense to use DNSSSEC when forwarding, as all these RSIG requests will totally annihilate the gain of time. All DNSSEC traffic is 'non TLS' (DNSSEC isn't available over TLS).
When forwarding, you chose to trust the upstream DNS - "no matter what".

I will re activate forwarding to 1.1.1.1 over TLS (853) WITH DNSSEC enabled.
All bets are open
Will useless DNS traffic sky rocket ? Will my pfSense take fire ?

Extra : and will unbound really do these RSIG lookup directly to the implicated TLDs and domain name servers, completely bypassing 1.1.1.1 ? I have to check that.

....
Sorry for the complicated rent.
All I know is that I don't know everything