Major DNS Bug 23.01 with Quad9 on SSL
-
@cmcdonald If it helps, IIRC:
- no issues for the first couple hours or so after install
- suddenly random issues connecting to I think multiple sites on my phone
- LinkedIn app wasn't loading data or images
- went to my PC, linkedin.com wouldn't resolve
- no issues since disabling DNSSEC on Feb. 18
Obviously Quad9 thinks it's a potential issue since they recommend disabling it. But it's probably been enabled on this router since Plus 21.x sometime. I had a fairly early 2100. (for clarity, I did have to reinstall 22.05 due to the EFI bug, restore config, and updated to 23.01)
@nononono Hmmm, I just checked and do have "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers" enabled. If you have unbound crashing/stopping, that's not what I was experiencing then.
-
-
@p1erre said in Major DNS Bug 23.01 with Quad9 on SSL:
@nononono as mentioned in the docs here you have to disable DNSEC for DNS over TLS and forwarding mode. I've had the same issue like you (with different forwarding server), but disabling DNSEC helped me. DNSEC enabled with forwarding server was woring in 22.05 and earlier versions.
I had hoped that was the fix also, but with Quad9 we saw on every device that at some point with-in 24 hours they would stop returning results. The DNS server was up and would respond, it just wouldn't look things up anymore.
We are about 48 hours into half a dozen devices on Quad9 with TLS off and no issues. Also another half dozen are having no issues with Cloudlfare DNS and TSL on.
-
@cmcdonald said in Major DNS Bug 23.01 with Quad9 on SSL:
I am working on replicating this in order to investigate. My usual setup is to just let Unbound do the recursive resolving for me without forwarding to another service.
I hope you can sort it out. It's going to be tough to duplicate. It happens in the 18 hour to 24 hour of being up. It responds just doesn't return results unless they are cached. And sometimes it would just start working again on it's own. I haven't found a great way to test other than end users complaining.
-
@nononono Same issue here on a NG2100 running 23.01.
Running fine with DNSSEC disabled & SSL/TLS enabled on 22.05 but I've had to disable SSL/TLS on 23.01 to avoid intermittent DNS failures with Quad9.
This is on IPv4/IPv6 with the patch applied for redmine #13851.
-
Check the output of
sockstat | grep unbound
when it works and when it doesn't.I thought they fixed it but a while back unbound had an issue where it couldn't reuse SSL connections on the same open sockets so in some cases they kept piling up.
EDIT: This is what I was thinking of, but it's been fixed/closed for a couple years now: https://github.com/NLnetLabs/unbound/issues/47
-
This bug is still a problem, for Cloudflare DNS users as well. DNS stopped working on all clients immediately after updating to 23.01 this morning (Mar 15, 2023).
After finding this thread, I UNchecked only the following setting in the DNS Resolver settings, and DNS started working again:
This is clearly a bug because the current pfSense documentation itself advises that this setting be checked:
https://docs.netgate.com/pfsense/en/latest/recipes/dns-over-tls.html
-
That document is about configuring that specific feature, it's not "advising" that setting be checked in a general fashion for everyone.
It's working for many people and breaking for a few, but it's still not clear why.
-
@jimp Bullet number four of the previously included screenshot explicitly directs the user to check that setting. The pfSense documentation is actually more than advising. It is directing.
-
@isotope1842 said in Major DNS Bug 23.01 with Quad9 on SSL:
@jimp Bullet number four of the previously included screenshot explicitly directs the user to check that setting. The pfSense documentation is actually more than advising. It is directing.
The page you quoted is a document about configuring DNS over TLS -- it's saying to check that if you want DNS over TLS. That's what that entire document is for.
It's not a part of the general setup or DNS resolver docs and so on. It's a recipe for users who are interested in that feature and want to know how to set it up.
There is nothing saying users should be following all of those recipes, they're there for reference for things users may want to do.
-
@jimp The context in which the instructions are provided is exactly for users who want to use an upstream DNS over TLS provider. The original poster here was reporting a bug in using Quad9 and I reported the same bug when using Cloudflare.
See the top of the same linked documentation:
"Pick a DNS over TLS upstream provider, such as a private upstream DNS server or a public service like Cloudflare, Quad9, or Google public DNS."
Following the instructions prior to 23.01 worked. Immediately after upgrading to 23.01, DNS fails until unchecking a single setting.
-
Yes, and? They still work for that and for many providers and users who want to enable that feature.
But you're trying to imply this is something the docs have told everyone they should be doing which isn't true. They don't advise everyone to do it, just people who are interested in that feature.
But none of this is helpful. We still need more information about how and why it's failing. We have yet to be able to reproduce this in a lab environment, and there are plenty of us running DNS over TLS without problems.
-
@jimp Steps to reproduce the problem:
- Install pfSense 22.05 on a netgate device.
- Configure DNS over TLS with Cloudflare.
- Upgrade netgate device to pfSense 23.01.
- Observe broken DNS for downstream clients.
-
@isotope1842 said in Major DNS Bug 23.01 with Quad9 on SSL:
@jimp Steps to reproduce the problem:
- Install pfSense 22.05 on a netgate device.
- Configure DNS over TLS with Cloudflare.
- Upgrade netgate device to pfSense 23.01.
- Observe broken DNS for downstream clients.
It is not that simple.
I have multiple lab VMs using DNS over TLS to Cloudflare and Quad9 that successfully resolve and have no problems.
-
Mmm, I assume you are seeing the same intermittent behaviour as other users? It's not failing for every query with that configuration?
-
@jimp I just re-checked that single setting. DNS appears to continue to work. Curious to see whether it starts to fail again at some point.
-
@isotope1842 said in Major DNS Bug 23.01 with Quad9 on SSL:
@jimp I just re-checked that single setting. DNS appears to continue to work. Curious to see whether it starts to fail again at some point.
Since you hit it once it's likely to fail again at some point, but nobody has yet to be able to pinpoint exactly when/why it happens.
I've been periodically checking my lab systems and they all just keep resolving no matter what I do. But they are lab systems so the load is considerably lower than it would be in a live environment.
-
@isotope1842 There are a few threads on this topic, or variations thereof, and in another one someone posted their problem seemed likely to happen when opening a group/folder of bookmarks/favorites at once...implying a higher number of simultaneous requests might trigger it.
I was also unable to replicate my issue by simply (re)checking the DNSSEC option, but I left it off as recommended.
-
After playing alot more - it might be an issue with Quad9's TLS DNS limiting responses more than anything.
While there is nothing helpful in the pfsense logs, Quad9 just appears to stop replying and then start responding again - almost as if there is a limit being imposed by Quad9 on requests - but of course pfsense must have some role as it never occurred before 23.01
This network is a very high traffic network, so maybe others that see the same thing manage high traffic networks as well - either way the only long term solution has been doing TLS DNS through Cloudflare
As another point - DHCP lease registrations is definitely not fixed as claimed in 23.01, unbound still likes to reboot too much to consider enabling it - as such at the most I am still only registering the clients that need it via static mapping.
-
Hmm, that would explain why we haven't been able to replicate it in test setups without the loading a production box has.
-
There were memory leaks and a segfault that were fixed. There is still ongoing work to eliminate the unbound reloads associated with lease registration events. I'm hopeful it will make it in 23.05.
-
@cmcdonald said in Major DNS Bug 23.01 with Quad9 on SSL:
There were memory leaks and a segfault that were fixed. There is still ongoing work to eliminate the unbound reloads associated with lease registration events. I'm hopeful it will make it in 23.05.
That is great to hear! Hopefully that finally is solved
-
@nononono said in Major DNS Bug 23.01 with Quad9 on SSL:
DHCP lease registrations is definitely not fixed as claimed in 23.01
Where is that stated? What it says is a "crash" was fixed when unbound restarting
https://redmine.pfsense.org/issues/11316?#note-79
Where does it say that unbound is not going to restart on dhcp registrations.. The issue with unbound restarting every time there is something going on with dhcp is ok for some users.. but if you have clients renewing lever few minutes then your still going to have issues with unbound and dhcp - even if they fixed some crash that could happen.
-
@johnpoz said in Major DNS Bug 23.01 with Quad9 on SSL:
What it says is a "crash" was fixed when unbound restarting
It may be referring to https://docs.netgate.com/pfsense/en/latest/releases/23-01.html "A long-standing difficult-to-reproduce crash in Unbound...It is now safe again to enable DHCP registration alongside Unbound Python mode in pfBlockerNG." ...which is talking about pfBlocker vs DHCP, not that DHCP registration won't restart Unbound anymore. (I've seen others post about this sentence as well...)
-
@steveits said in Major DNS Bug 23.01 with Quad9 on SSL:
someone posted their problem seemed likely to happen when opening a group/folder of bookmarks/favorites at once...implying a higher number of simultaneous requests might trigger it.
Playing around with various settings, and I don't know if this has anything to do with the quoted comment, but I raised incoming and outgoing tcp buffers in resolver adv settings from default 10 to 20. I believe (or imagine) that it has reduced the frequency of unbound 'hangs' on my system.
-
-
Just thought I would comment on this as I am seeing the same symptoms. It only just started happening recently and thought it was load related but not much has changed on my end except upgrading PFSense and associated packages.
I have turned parameter off this for now... Use SSL/TLS for outgoing DNS Queries to Forwarding Servers as I need the stability.
Summary:
- Using Cloudflare SSL/TLS on 23.01 (ipv4 / ipv6 resolvers)
- Build: 23.01-RELEASE (amd64) built on Fri Feb 10 20:06:33 UTC 2023
- System Patches package with recommended fixes applied including a manual patch for redmine #13851
- Machine Core(TM) i5-8365U, 16GB Ram, 128GB SSD
- Internet Connection 100/20
- Running a number of wireguard gateways (3 servers, both ipv4 and 6)
- Running a number of packages including...
According to PFBlockerNG graph, my dns reply stats are around 10-20K per hour during the busiest times.
Over the last week or two, uptime Kuma randomly alerted me to dns issues which is the same time my family complained they couldn't access any websites or their game would fail stating their was no internet. This would happen randomly and would take a couple of minutes to clear up again.
Fortunately I had graylog setup and unbound in debug mode for the most recent errors above.
Looking at the logs I note the following messages around the time where DNS briefly cut out...
Looks like PfSense is not processing DNS requests quick enough for me. The sample above occurred whilst the internet link wasn't even loaded, neither was my machine.
I have turned this setting off and will monitor... Use SSL/TLS for outgoing DNS Queries to Forwarding Servers
I haven't done any further debugging and I hope this helps someone out, if not I apologise for this long post.
-
@joedan said in Major DNS Bug 23.01 with Quad9 on SSL:
10-20K per hour during the busiest times.
In your home? how many devices / users - you have a lot shit banging its head trying to resolve stuff your blocking.. That seems really high for a home..
That is what like 240K to 480K queries in 24 hours.. I see like 50k in 24 hours, and that is with 28 active devices doing dns.. Many of them banging their heads looking for shit that is blocked.
-
I also had occasional dns failures when using quad9 dns. I simply turned off forwarding mode and have had no issues with root servers doing the work.
I find this to be non-ideal, but at least functional. -
Just reporting in here that I have the same issues with 23.01 and Quad 9 DNS when using unbound DNS
My settings were as follows on 22.05 since 22.05 was released with no issues with internet and DNS resolution.
-
pfsense ubound DNS resolver enabled
-
Using forwarding to Quad 9 DNS servers
-
DNSSEC unticked
-
Use SSL/TLS for outgoing DNS Queries to Forwarding Servers TICKED
Upgraded to 23.01 this weekend just gone 2nd April 2023. No other settings have changed on LAN network and no other firmware's have been upgraded on any devices on network so I am confident that the only change was the upgrade to pfsense from 22.05 TO 23.01.
Everything worked fine yesterday 3rd April until this morning when I now get multiple DNS resolution issues and cannot reach websites.
I have now changed the following single setting.
- Use SSL/TLS for outgoing DNS Queries to Forwarding Servers UNTICKED
Websites are now resolving correctly.
I will continue to monitor DNS resolution over the coming days to see if the issue reappears and report back.
-
-
@johnpoz said in Major DNS Bug 23.01 with Quad9 on SSL:
In your home? how many devices / users - you have a lot shit banging its head trying to resolve stuff your blocking.. That seems really high for a home..
Yes I have around 35 devices, a few that are quite bad including a Swann Security DVR that is taking up 25% of that DNS traffic calling out dropbox domains even though I don't have that function setup.
Just wanted to loop back and advise I did resolve my issue however it involved going back about 6 backup iterations and manually reapplying all my config changes one at a time and am now back on DNS over TLS (Cloudflare) with 23.01, and all recommended patches applied. I did also move ISP's during that time of panic so unfortunately have nothing further to contribute here because I was under pressure to get things working again quickly and made significant change in such a short time.
I was never using Quad9 though which appears to be an ongoing issue with other posters.
-
@joedan said in Major DNS Bug 23.01 with Quad9 on SSL:
a few that are quite bad including a Swann Security DVR that is taking up 25% of that DNS traffic calling out dropbox domains even though I don't have that function setup
Introduction : read this first.
If a device asks ever x seconds to resolve an identical host name, then, at first, your uplink isn't used : pfSense will just serve the same info edit : from local DNS cache, as log as this is TTL permitted end edit to the device again, and again. You're just burning some watt hours at your place.
But if the TTL is short, like mere minutes or less, then, yeah, the WAN gate is opened, and you start to activate a whole lot more of software/hardware on both sidesThe thing is : as I said in the other thread : if you were looking at the WAN gate of quad9 (because, lets imagine, you work over there) : what would you do if you saw that some IP was clearly misbehaving ? => You throttle ?! As "just scale up the system up once more" without any cash flow in return will kill your job.
So, on your home site : your entire DNS just got delayed.And that's of course a 'pfSense DNS bug'
-
I agree this could be Quad9 throttling responses. But still it would have been the same in 22.05.
-
@skogs said in Major DNS Bug 23.01 with Quad9 on SSL:
I find this to be non-ideal, but at least functional.
Not wanting to start a big base line debate but why is letting a resolver do resolving non-ideal? Letting your box do DNS resolving on its own without any upstream forwarder that can be censored is a much more resilient setup (as we see) then having a single point of failure like a big DNS forwarder that throttles or censors DNS lookups (or in case of quad9 is jurisdically forced to do so). Just to try to hide your DNS queries from your ISP?
Just asking because I'm interested.
-
@jegr said in Major DNS Bug 23.01 with Quad9 on SSL:
Just to try to hide your DNS queries from your ISP?
hahah - which isn't really happening anyway. So I do a dns query for www.amazon.com, which I want to hide from my isp ;)
So my isp doesn't see the dns query, but they see me go to IP address 1.2.3.4 which pretty obvious is amazon.com - and here is the big one.. They still see what you are asking for - its just not via dns..
Until such time that esni, which is dead - long live ech.. Is everywhere - you are just sending your sni in the clear in your handshake - so who exactly are you hiding your dns query from? And for what purpose?
-
@n8rfe said in Major DNS Bug 23.01 with Quad9 on SSL:
Just reporting in here that I have the same issues with 23.01 and Quad 9 DNS when using unbound DNS
My settings were as follows on 22.05 since 22.05 was released with no issues with internet and DNS resolution.
-
pfsense ubound DNS resolver enabled
-
Using forwarding to Quad 9 DNS servers
-
DNSSEC unticked
-
Use SSL/TLS for outgoing DNS Queries to Forwarding Servers TICKED
Upgraded to 23.01 this weekend just gone 2nd April 2023. No other settings have changed on LAN network and no other firmware's have been upgraded on any devices on network so I am confident that the only change was the upgrade to pfsense from 22.05 TO 23.01.
Everything worked fine yesterday 3rd April until this morning when I now get multiple DNS resolution issues and cannot reach websites.
I have now changed the following single setting.
- Use SSL/TLS for outgoing DNS Queries to Forwarding Servers UNTICKED
Websites are now resolving correctly.
I will continue to monitor DNS resolution over the coming days to see if the issue reappears and report back.
So further update.
It appears that the outages I was experiencing were nothing to do with pfsense at all.
My Virgin Media ISP in the UK had a UK wide intermittent outage that has been affecting customers for the past 48 hours.
Please ignore my post above. TLS DNS is working just fine for me on 23.01
-
-
@n8rfe said in Major DNS Bug 23.01 with Quad9 on SSL:
Please ignore my post above. TLS DNS is working just fine for me on 23.01
I'm pretty sure that even @nononono will post back in the future here with a "solved" statement.
The initial subject of the thread "Major DNS Bug 23.01 with Quad9 on SSL " boils down to .... nothing.
-
@gertjan said in Major DNS Bug 23.01 with Quad9 on SSL:
@n8rfe said in Major DNS Bug 23.01 with Quad9 on SSL:
Please ignore my post above. TLS DNS is working just fine for me on 23.01
I'm pretty sure that even @nononono will post back in the future here with a "solved" statement.
The initial subject of the thread "Major DNS Bug 23.01 with Quad9 on SSL " boils down to .... nothing.
I don't think I'd go as far as saying it's nothing. I see people having issues here, Reddit, Twitter. Settings that work in 22 don't work in 23. The defaults with forwarding don't work. With 22 you could turn on forwarding and turn on TLS and it worked. Now not so much. We have done several dozen 22 to 23 updates for customers now with different ISPs and different setups and every time we end up having to sort out DNS issues.
I think the biggest issue is we have a few customers that need to have DHCP entries included in DNS. It worked fine with 22. With 23 the restarts eventually either crash unbound entirely or it just stops responding to DNS for periods of time. We have had to setup a separate DNS server for these customers.
-
@cylosoft said in Major DNS Bug 23.01 with Quad9 on SSL:
I think the biggest issue is we have a few customers that need to have DHCP entries included in DNS. It worked fine with 22. With 23 the restarts eventually either crash unbound entirely or it just stops responding to DNS for periods of time. We have had to setup a separate DNS server for these customers.
Sorry but it's documented for far longer then 22.x that DHCP registration with unbound is a "avoid!" situation due to it restarting every time a dhcp client registers its name. Either use dnsmasq (the forwarder) or don't use DHCP registration. That is literally written in the docs for far longer then "it came up with 23.01": https://docs.netgate.com/pfsense/en/latest/services/dns/resolver-config.html
Besides, a fully dynamic DHCP address shouldn't ever be needed as a name at all. A client doesn't need to have dns resolution of its name. And if it does need it because it runs services under a FQDN/hostname, it should have a static/DHCP-set static IP so it won't accidentally change and thus would have a dns name, that can be read at runtime by unbound by the "static DHCP" option which is fine and doesn't cause problems. -
@jegr said in Major DNS Bug 23.01 with Quad9 on SSL:
@cylosoft said in Major DNS Bug 23.01 with Quad9 on SSL:
I think the biggest issue is we have a few customers that need to have DHCP entries included in DNS. It worked fine with 22. With 23 the restarts eventually either crash unbound entirely or it just stops responding to DNS for periods of time. We have had to setup a separate DNS server for these customers.
Sorry but it's documented for far longer then 22.x that DHCP registration with unbound is a "avoid!" situation due to it restarting every time a dhcp client registers its name. Either use dnsmasq (the forwarder) or don't use DHCP registration. That is literally written in the docs for far longer then "it came up with 23.01": https://docs.netgate.com/pfsense/en/latest/services/dns/resolver-config.html
Regardless of that. It works with 22. It doesn't with 23. So to our customers 23 doesn't work.
-
@cylosoft said in Major DNS Bug 23.01 with Quad9 on SSL:
I see people having issues here, Reddit, Twitter.
Sure.
For me, issues can be found here : Issues exist as those can be reproduced.@cylosoft said in Major DNS Bug 23.01 with Quad9 on SSL:
Settings that work in 22 don't work in 23
What settings ?
@cylosoft said in Major DNS Bug 23.01 with Quad9 on SSL:
The defaults with forwarding don't work. With 22 you could turn on forwarding and turn on TLS and it worked.
So this (last change July 2022) DNS Forwarder Configuration or the TLS equivalent Configuring DNS over TLS or How to Enable DNS over TLS on pfSense with Cloudflare are not correct ?
They looks ok to me, I've used that page to set up forwarding to 1.1.1.1 and the IPv6 equivalent, with the cert host names.
https://1.1.1.1/help confirmed me all was ok - I was forwarding.I'm pretty sure my ISP isn't DNS bugging on me.
My pfSense can create enough entropy (see above) so my pfSense can start many (like a lot) TCP connections over TLS but I found out ones what happens with a web server (and mail etc) if entropy isn't available any more. TLS goes down fast....
Maybe this entropy thing is a non issue. I don't know.@cylosoft said in Major DNS Bug 23.01 with Quad9 on SSL:
that need to have DHCP entries included in DNS.
That one is known for years now.
Fast solution : DHCP Mac static lease.Next best :
@cmcdonald said in Major DNS Bug 23.01 with Quad9 on SSL:
I'm hopeful it will make it in 23.05.
@cylosoft said in Major DNS Bug 23.01 with Quad9 on SSL:
Reddit, Twitter.
No, thanks.
I'm not a #metoo guy. I prefer the #menot.
I don't like issues for myself, neither for others.
If I find one, I'll try to find out the 'why' part'.Btw : I was posting in the scope of this thread
So we all know what our choices are asap.