Major DNS Bug 23.01 with Quad9 on SSL
-
@robbiett
Thanks for the command, I'm going to test I later.
But I did enable "Serve Expired" and now the lookup stays on 0 msec on 2nd lookup of the same domain.
-
@moonknight problem with cnn.com is they have the TTL set to 60 seconds..
;; QUESTION SECTION: ;cnn.com. IN A ;; ANSWER SECTION: cnn.com. 60 IN A 151.101.67.5 cnn.com. 60 IN A 151.101.195.5 cnn.com. 60 IN A 151.101.131.5 cnn.com. 60 IN A 151.101.3.5So if you forward to somewhere the ttl you can cache is going to be something shorter then 60 seconds, could be 59, could be 2..
There is no sane reason for them to have the ttl set so freaking low - other than they want lots of queries.. They charge their customers maybe by queries - that is hosted on aws dns..
;; AUTHORITY SECTION: cnn.com. 3600 IN NS ns-1086.awsdns-07.org. cnn.com. 3600 IN NS ns-1630.awsdns-11.co.uk. cnn.com. 3600 IN NS ns-47.awsdns-05.com. cnn.com. 3600 IN NS ns-576.awsdns-08.net.So what you can do on your side is yeah allow for serving expired, and you could also set your min ttl.. I do both, have min ttl of 3600, and serve expired..
-
Thanks for the information :)
I set "Minimum TTL for RRsets and Messages" to 3600 and seems to work :) -
@moonknight
Yep, that is the one. I have mine set at 2400 for reasons I read in a technical paper that I have long since forgotten. -
Thank you very much @robbiett :)
-
@robbiett said in Major DNS Bug 23.01 with Quad9 on SSL:
for reasons I read in a technical paper that I have long since forgotten
The hallmark of so many IT decisions carried forward decades into the future.
-
@steveits said in Major DNS Bug 23.01 with Quad9 on SSL:
The hallmark of so many IT decisions carried forward decades into the future.
Yeah, I resemble those remarks - especially as I am so many decades in. Started with a BBC Micro only to see ARM come around again. Next came the Apple IIe and off I went down that path, only to find myself down the line with BSD again.
I have forgotten so much along the way!
ļø -
I've been playing with these two settings for a few months now.
"Minimum TTL for RRsets and Messages"
"Serve Expired"
They have no effect on this issue.
Perhaps however there is a relationship with the use of IPv6. I previously said that I had no more problems on 23.05, however at that moment I did not check that IPv6 worked for me. Due to another bug, IPv6 did not work for me and maybe that's why there was no issue. The other day I restored IPv6 and without ASLR the same issues began as before. Now testing with ASLR disabledā¦. -
@w0w said in Major DNS Bug 23.01 with Quad9 on SSL:
"Minimum TTL for RRsets and Messages"
"Serve Expired"
They have no effect on this issue.I don't think any of us were suggesting that. Of course, if you can get an answer from the DNS cache it does dodge the issue for that particular query and those 2 settings do improve cache performance and make a hit more likely.
Regarding your thoughts on IPv6, that could indeed be part of the issue but unproven. I run IPv6 and most of my traffic tends to use it. I have not tried disabling it as part of the DNS diagnostics but happy to do so if NetGate thinks there is merit in it.
ļø -
@robbiett said in Major DNS Bug 23.01 with Quad9 on SSL:
Regarding your thoughts on IPv6, that could indeed be part of the issue but unproven. I run IPv6 and most of my traffic tends to use it. I have not tried disabling it as part of the DNS diagnostics but happy to do so if NetGate thinks there is merit in it.
Thing thoughts on my side.
If a ISP delivers a broken IPv4, they wouldn't exist.
If their IPv4 works fine, and they offer a broken IPv6, then thinks become complicated.Best choice : forget about pfSense, use the ISP router.
IPv6 might be ok then, but "with Quad9 on SSL " will be a no no. Exception : set it up for every LAN device manually.Or use pfSense, and disable IPv6 on the WAN side.
It's as easy as :
Your LAN devices will still use IPv6 among them, that's ok.
Btw : when this thread started, I disables my perfectly working unbound resolve mode, to activate "Quad9 on SSL using IPv6 and IPv4".
I apply the ASLR (see above) trick.
Up until today (3 weeks now ?) : I did not see any issue what so ever.
I've about 50 LAN devices connected, and a bunch of hotel clients on the captive portal.
No one ever yelled : "Internet doesn't work". -
@gertjan said in Major DNS Bug 23.01 with Quad9 on SSL:
Thing thoughts on my side.
If a ISP delivers a broken IPv4, they wouldn't exist.
If their IPv4 works fine, and they offer a broken IPv6, then thinks become complicated.Best choice : forget about pfSense, use the ISP router.
IPv6 might be ok then, but "with Quad9 on SSL " will be a no no. Exception : set it up for every LAN device manually.Or use pfSense, and disable IPv6 on the WAN side.
I would have a few issues with that:
- My ISP does not provide a router; even if they did it would not have the CPU power, interfaces and features that I, along with many others, require
- Forgetting about pfSense would be an odd thing to do with a NetGate router
- I have no issue with IPv6 and it is provided universally to UK broadband users - I am also dependant on it
- No link to IPv6 has been proven and NetGate has not offered it as a concern or potential cause of the forwarded DNS-over-TLS issues
ļø -
@robbiett said in Major DNS Bug 23.01 with Quad9 on SSL:
My ISP does not provide a router;
That's actually a good thing. No need to to support one neither. They have to concentrate on the "IPv4 RFC"and the same way on "RFC IPv6" and you'll be a happy user.
@robbiett said in Major DNS Bug 23.01 with Quad9 on SSL:
Forgetting about pfSense would be an odd thing to do with a NetGate router
Humm. Didn't thinks about that one,
for you
Anyway : I can't remove my ISP router as nothing else exists to terminate their fiber cable. This means that this ISP router doesn't only have to hand out GUA's for its LAN devices, as pfSense is just another LAN device, but it should also hand over entire /64 prefixes to pfSense, and that's already much more an issue these days.@robbiett said in Major DNS Bug 23.01 with Quad9 on SSL:
I have no issue with IPv6 and ....
You, maybe not.
Neither your ISP, maybe.
For IPv6 to work, also the entire IPv6 peering needs to work also.@robbiett said in Major DNS Bug 23.01 with Quad9 on SSL:
No link to IPv6 ....
You're right.
I'm probably 'out of subject'.
I brought it up because, if I use the same unbound (pfSense 23.01) using a Netgate device (4100) and the same "forward" settings to Quad9 over TLS, for 3 weeks, why can't I see any issue ? While others do. -
@robbiett said in Major DNS Bug 23.01 with Quad9 on SSL:
I don't think any of us were suggesting that.
I don't suggest this also
Seriously, I didn't notice any significant difference. Those random DNS errors that were there have not gone away, but those lags when the response comes with a significant delay too. In general, I'm just saying that if there was no main issue, I would never have noticed and the rest would not have been useful to set up anything if everything worked the same as in previous versions.@gertjan said in Major DNS Bug 23.01 with Quad9 on SSL:
Up until today (3 weeks now ?) : I did not see any issue what so ever.
I've about 50 LAN devices connected, and a bunch of hotel clients on the captive portal.@gertjan said in Major DNS Bug 23.01 with Quad9 on SSL:
for 3 weeks, why can't I see any issue ? While others do.
Maybe because it's not a "working/not working" question. When the page didnāt open once or the pictures didnāt load or didnāt load right away .... Well, I donāt know if I were a client of the network, itās unlikely that I would run to complain to someone about something not working. 99.9% everything works, but the feeling of something wrong remains, when the picture on the site opens with a delay as if it is being viewed by special services of all countries in the world
Even more, the more clients, the less chance that the issue will appear, they constantly update the cache, well, purely theoretically, this is possible if everyone visits the same resources. -
@w0w My 2 cents: Something was certainly wrong when link to NYTimes returns url failed to load because server doesnāt respond. And not just that site but many sites. And DNS log is full of āservfailā entries. And disabling pfBlocker does not change the problem.
Whatever was wrong is now āfixedā for me after performing McDonaldās change to unbound config. As an aside, I do not have ipV6 enabled on WAN interface.
-
We had an opportunity to squeeze in some additional fixes before we cut the next RC (and hopefully RELEASE) build. The next build of 23.05 will include a fix for this.
We will continue to monitor upstream developments and adjust accordingly.
Need help fast? https://www.netgate.com/support
-
@cmcdonald said in Major DNS Bug 23.01 with Quad9 on SSL:
The next build of 23.05 will include a fix for this.
Very fine work, very fine work indeed.
ļø -
@robbiett Updated to the latest release a few minutes ago. Not listening to the suggestions to disable IPv6 as I have been running IPv6 for a few years now without major issue until recently. DNS over TLS has been a major benefit as our ISP redirects port 53 DNS to their own servers and redirects mistyped domains or NXDOMAINS to their ad pages. This isn't cool in any way shape or form.
So here are the results that I am seeing:
First a domain that hasn't been queried since Unbound restarted:
Notice the 127.0.0.1 takes 451 ms to complete a TLS handshake and do the upstream lookup. This is expected as DoT requires the creation of a verified TLS connection then the DNS lookup. If you add up ping time and essentially triple that, you should be at about the right time.
Here is the same query run a few seconds later:
Unbound is answering from its cache as you can see by the 0 ms time on the 127.0.0.1 IP.
Unbound is configured to do preemptive lookups for the most queried data in its cache so once Unbound is up and running, it should answer the most active DNS queries in 0 ms. So far, DNSSEC is working as is pfBlocker.
I still get a few of these errors:
debug: outnettcp got tcp error -1
But so far, the issues I was experiencing earlier have stopped.
Thanks for all the hard work on this!
-
@cmcdonald
So no need to disable ASLR anymore? Is it disabled now by option when compiled? -
Run the elfctl with the 'elf' (full path) on the command line without any other parameters :
See example here : https://forum.netgate.com/topic/178413/major-dns-bug-23-01-with-quad9-on-ssl/139?_=1684213270230 -
G Gertjan referenced this topic on
-
File '/usr/local/sbin/unbound' features: noaslr 'Disable ASLR' is set. noprotmax 'Disable implicit PROT_MAX' is unset. nostackgap 'Disable stack gap' is unset. wxneeded 'Requires W+X mappings' is unset. la48 'amd64: Limit user VA to 48bit' is unset.Yep.
Didn't think of it myself, thanks
