Major DNS Bug 23.01 with Quad9 on SSL

RobbieTT

I should probably add that even with the ASLR unset I still get weird looking results when I attempt an individual DNS Lookup on a domain name that I know hasn't been cached:

2023-05-14 at 10.43.36.png

If I understand the pfSense diagnostics screen, when the internal DNS resolver has to use forwarding to answer a query I would expect a similar time to answer the query as the fastest responding name server (2629:fe::fe at 7ms in this example) plus the almost negligible processing delay from checking the cache. Yet it actually takes a snooze-worthy 168ms.

Why does the DNS resolver take 168ms for a simple forwarded (uncached) query when the forwarder itself has an answer from an upstream provider in just 7ms or, in other words, around 24 times slower than expected?

️

MoonKnight

@robbiett

Have been wondering about the same for some time now. It doesn't make sense

And if you do the same lookup just seconds after the first time "The query time" is on 0.
Wait 1 minute then back to 60 msec.

I have been having this behavior since 23.01 and maybe on 22.05 also .

RobbieTT

@moonknight said in Major DNS Bug 23.01 with Quad9 on SSL:

@robbiett
And if you do the same lookup just seconds after first time "The query time" is on 0.
Wait 1 minute then back to 60 msec.

I don't suffer the second part of your observation. Once my query is cached it stays cached until it is removed or reset - it obeys the settings I have given it.

If you stop the resolver for a moment and run the command:

unbound-control -c /var/unbound/unbound.conf dump_cache

...you can poke around and see what is in your cache.

️

MoonKnight

@robbiett
Thanks for the command, I'm going to test I later.
But I did enable "Serve Expired" and now the lookup stays on 0 msec on 2nd lookup of the same domain.

johnpoz

@moonknight problem with cnn.com is they have the TTL set to 60 seconds..

;; QUESTION SECTION:
;cnn.com.                       IN      A

;; ANSWER SECTION:
cnn.com.                60      IN      A       151.101.67.5
cnn.com.                60      IN      A       151.101.195.5
cnn.com.                60      IN      A       151.101.131.5
cnn.com.                60      IN      A       151.101.3.5

So if you forward to somewhere the ttl you can cache is going to be something shorter then 60 seconds, could be 59, could be 2..

There is no sane reason for them to have the ttl set so freaking low - other than they want lots of queries.. They charge their customers maybe by queries - that is hosted on aws dns..

;; AUTHORITY SECTION:
cnn.com.                3600    IN      NS      ns-1086.awsdns-07.org.
cnn.com.                3600    IN      NS      ns-1630.awsdns-11.co.uk.
cnn.com.                3600    IN      NS      ns-47.awsdns-05.com.
cnn.com.                3600    IN      NS      ns-576.awsdns-08.net.

So what you can do on your side is yeah allow for serving expired, and you could also set your min ttl.. I do both, have min ttl of 3600, and serve expired..

MoonKnight

@johnpoz

Thanks for the information :)
I set "Minimum TTL for RRsets and Messages" to 3600 and seems to work :)

RobbieTT

@moonknight
Yep, that is the one. I have mine set at 2400 for reasons I read in a technical paper that I have long since forgotten.

MoonKnight

Thank you very much @robbiett :)

SteveITS

@robbiett said in Major DNS Bug 23.01 with Quad9 on SSL:

for reasons I read in a technical paper that I have long since forgotten

The hallmark of so many IT decisions carried forward decades into the future.

RobbieTT

@steveits said in Major DNS Bug 23.01 with Quad9 on SSL:

The hallmark of so many IT decisions carried forward decades into the future.

Yeah, I resemble those remarks - especially as I am so many decades in. Started with a BBC Micro only to see ARM come around again. Next came the Apple IIe and off I went down that path, only to find myself down the line with BSD again.

I have forgotten so much along the way!

️

w0w

I've been playing with these two settings for a few months now.
"Minimum TTL for RRsets and Messages"
"Serve Expired"
They have no effect on this issue.
Perhaps however there is a relationship with the use of IPv6. I previously said that I had no more problems on 23.05, however at that moment I did not check that IPv6 worked for me. Due to another bug, IPv6 did not work for me and maybe that's why there was no issue. The other day I restored IPv6 and without ASLR the same issues began as before. Now testing with ASLR disabled….

RobbieTT

@w0w said in Major DNS Bug 23.01 with Quad9 on SSL:

"Minimum TTL for RRsets and Messages"
"Serve Expired"
They have no effect on this issue.

I don't think any of us were suggesting that. Of course, if you can get an answer from the DNS cache it does dodge the issue for that particular query and those 2 settings do improve cache performance and make a hit more likely.

Regarding your thoughts on IPv6, that could indeed be part of the issue but unproven. I run IPv6 and most of my traffic tends to use it. I have not tried disabling it as part of the DNS diagnostics but happy to do so if NetGate thinks there is merit in it.

️

Gertjan

@robbiett said in Major DNS Bug 23.01 with Quad9 on SSL:

Regarding your thoughts on IPv6, that could indeed be part of the issue but unproven. I run IPv6 and most of my traffic tends to use it. I have not tried disabling it as part of the DNS diagnostics but happy to do so if NetGate thinks there is merit in it.

Thing thoughts on my side.
If a ISP delivers a broken IPv4, they wouldn't exist.
If their IPv4 works fine, and they offer a broken IPv6, then thinks become complicated.

Best choice : forget about pfSense, use the ISP router.
IPv6 might be ok then, but "with Quad9 on SSL " will be a no no. Exception : set it up for every LAN device manually.

Or use pfSense, and disable IPv6 on the WAN side.
It's as easy as :

Your LAN devices will still use IPv6 among them, that's ok.

Btw : when this thread started, I disables my perfectly working unbound resolve mode, to activate "Quad9 on SSL using IPv6 and IPv4".
I apply the ASLR (see above) trick.
Up until today (3 weeks now ?) : I did not see any issue what so ever.
I've about 50 LAN devices connected, and a bunch of hotel clients on the captive portal.
No one ever yelled : "Internet doesn't work".

RobbieTT

@gertjan said in Major DNS Bug 23.01 with Quad9 on SSL:

Thing thoughts on my side.
If a ISP delivers a broken IPv4, they wouldn't exist.
If their IPv4 works fine, and they offer a broken IPv6, then thinks become complicated.

Best choice : forget about pfSense, use the ISP router.
IPv6 might be ok then, but "with Quad9 on SSL " will be a no no. Exception : set it up for every LAN device manually.

Or use pfSense, and disable IPv6 on the WAN side.

I would have a few issues with that:

My ISP does not provide a router; even if they did it would not have the CPU power, interfaces and features that I, along with many others, require
Forgetting about pfSense would be an odd thing to do with a NetGate router
I have no issue with IPv6 and it is provided universally to UK broadband users - I am also dependant on it
No link to IPv6 has been proven and NetGate has not offered it as a concern or potential cause of the forwarded DNS-over-TLS issues

️

Gertjan

@robbiett said in Major DNS Bug 23.01 with Quad9 on SSL:

My ISP does not provide a router;

That's actually a good thing. No need to to support one neither. They have to concentrate on the "IPv4 RFC"and the same way on "RFC IPv6" and you'll be a happy user.

@robbiett said in Major DNS Bug 23.01 with Quad9 on SSL:

Forgetting about pfSense would be an odd thing to do with a NetGate router

Humm. Didn't thinks about that one, for you
Anyway : I can't remove my ISP router as nothing else exists to terminate their fiber cable. This means that this ISP router doesn't only have to hand out GUA's for its LAN devices, as pfSense is just another LAN device, but it should also hand over entire /64 prefixes to pfSense, and that's already much more an issue these days.

@robbiett said in Major DNS Bug 23.01 with Quad9 on SSL:

I have no issue with IPv6 and ....

You, maybe not.
Neither your ISP, maybe.
For IPv6 to work, also the entire IPv6 peering needs to work also.

@robbiett said in Major DNS Bug 23.01 with Quad9 on SSL:

No link to IPv6 ....

You're right.
I'm probably 'out of subject'.
I brought it up because, if I use the same unbound (pfSense 23.01) using a Netgate device (4100) and the same "forward" settings to Quad9 over TLS, for 3 weeks, why can't I see any issue ? While others do.

w0w

@robbiett said in Major DNS Bug 23.01 with Quad9 on SSL:

I don't think any of us were suggesting that.

I don't suggest this also
Seriously, I didn't notice any significant difference. Those random DNS errors that were there have not gone away, but those lags when the response comes with a significant delay too. In general, I'm just saying that if there was no main issue, I would never have noticed and the rest would not have been useful to set up anything if everything worked the same as in previous versions.

@gertjan said in Major DNS Bug 23.01 with Quad9 on SSL:

Up until today (3 weeks now ?) : I did not see any issue what so ever.
I've about 50 LAN devices connected, and a bunch of hotel clients on the captive portal.

@gertjan said in Major DNS Bug 23.01 with Quad9 on SSL:

for 3 weeks, why can't I see any issue ? While others do.

Maybe because it's not a "working/not working" question. When the page didn’t open once or the pictures didn’t load or didn’t load right away .... Well, I don’t know if I were a client of the network, it’s unlikely that I would run to complain to someone about something not working. 99.9% everything works, but the feeling of something wrong remains, when the picture on the site opens with a delay as if it is being viewed by special services of all countries in the world
Even more, the more clients, the less chance that the issue will appear, they constantly update the cache, well, purely theoretically, this is possible if everyone visits the same resources.

JonH

@w0w My 2 cents: Something was certainly wrong when link to NYTimes returns url failed to load because server doesn’t respond. And not just that site but many sites. And DNS log is full of ‘servfail’ entries. And disabling pfBlocker does not change the problem.

Whatever was wrong is now ‘fixed’ for me after performing McDonald’s change to unbound config. As an aside, I do not have ipV6 enabled on WAN interface.

cmcdonald

We had an opportunity to squeeze in some additional fixes before we cut the next RC (and hopefully RELEASE) build. The next build of 23.05 will include a fix for this.

We will continue to monitor upstream developments and adjust accordingly.

RobbieTT

@cmcdonald said in Major DNS Bug 23.01 with Quad9 on SSL:

The next build of 23.05 will include a fix for this.

Very fine work, very fine work indeed.

️

N0m0fud

@robbiett Updated to the latest release a few minutes ago. Not listening to the suggestions to disable IPv6 as I have been running IPv6 for a few years now without major issue until recently. DNS over TLS has been a major benefit as our ISP redirects port 53 DNS to their own servers and redirects mistyped domains or NXDOMAINS to their ad pages. This isn't cool in any way shape or form.

So here are the results that I am seeing:

First a domain that hasn't been queried since Unbound restarted:

Notice the 127.0.0.1 takes 451 ms to complete a TLS handshake and do the upstream lookup. This is expected as DoT requires the creation of a verified TLS connection then the DNS lookup. If you add up ping time and essentially triple that, you should be at about the right time.

Here is the same query run a few seconds later:

Unbound is answering from its cache as you can see by the 0 ms time on the 127.0.0.1 IP.

Unbound is configured to do preemptive lookups for the most queried data in its cache so once Unbound is up and running, it should answer the most active DNS queries in 0 ms. So far, DNSSEC is working as is pfBlocker.

I still get a few of these errors:

debug: outnettcp got tcp error -1

But so far, the issues I was experiencing earlier have stopped.

Thanks for all the hard work on this!