Concurrent/Rapid DNS queries fail randomly after 23.01 upgrade

wujj123456

One of my common workflow is to open a bookmark folder, which is usually 7-10 sites. After 23.01 upgrade, if I open lots of domains at once, many of them would fail to get the name resolved. The domains fail to resolve spreads randomly and this happens both in Edge and Firefox (with trr.mode=5). It takes a few seconds or longer before I can retry to get the failed domains resolved correctly, but other domains resolve fine in the meantime. At the same time, the failed domains would also fail on other browser too until some time later, so they seem to be negatively cached. Otherwise, DNS is generally reliable so long as I don't do this kind of rapid queries. There are no additional logs or crashes in unbound when this happens.

I am quite confident this only happened after 23.01 upgrade, since that's my daily route and the list of sites hasn't changed, nor did I change any pfsense settings after upgrade until I started debugging this.

My settings:

DNS servers: quad9 and cloudflare. Both IPv4 and IPv6 ones.
DNS resolver: Enable DNSSEC support, Enable Forwarding Mode, Use SSL/TLS for outgoing DNS Queries to Forwarding Servers, Custom options "server:include: /var/unbound/pfb_dnsbl.*conf".

Here are the few things I've tried.

I've tried various combinations of DNS servers and the issue remains.
I have pfblockerNG installed but disabling it doesn't help. The other three packages I have doesn't mess with DNS (Status_Traffic_Totals, squid, sudo).
Switching from DNS resolver to forwarder solves the issue.

Anyone running into the same? Is there any specific settings I should provide or try that could be relevant to this issue?

johnpoz

@wujj123456 said in Concurrent/Rapid DNS queries fail randomly after 23.01 upgrade:

Enable DNSSEC support, Enable Forwarding Mode, Use SSL/TLS for outgoing DNS Queries to Forwarding Servers

This has been shown in multiple threads to be a bad config.. If your going to forward there is little reason to have dnssec enabled - its only going to lead to failures to query..

https://support.quad9.net/hc/en-us/articles/4409274993677-DNS-Forwarders-Best-Practices

Disable DNSSEC Validation

Since Quad9 already performs DNSSEC validation, DNSSEC being enabled in the forwarder will cause a duplication of the DNSSEC process, significantly reducing performance and potentially causing false BOGUS responses.

wujj123456

@johnpoz Thank you for the help. Looks like the upgrade just exposed my bad configuration. Not sure how I missed it but it's also called out in the doc:

DNSSEC is not generally compatible with forwarding mode, with or without DNS over TLS.

What's interesting though, is that disabling forward mode while enabling DNSSEC fully resolves my issue, but disabling DNSSEC and leaving forward mode and DNS over TLS doesn't. If I disable "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers", it works well too.

The 4 DNS servers I have are 2620:fe::fe, 2606:4700:4700::1112, 1.1.1.2, 9.9.9.9 and they all support DNS over TLS. DNS Lookup also can query all of them with forward mode + DNS over TLS. I've verified in state page that port is 53 without DNS over TLS and 853 with the feature.

Guess I will give up DNS over TLS for now, but if there are more suggestion that I should try, please let me know.

SteveITS

@wujj123456 There have been others saying they need to disable TLS as well. I have not seen that myself but am at home with probably lower volume…your comment on a fast set of lookups is interesting. Perhaps some sort of connection limit?

I did hit the DNSSEC issue on 23.01, not an issue on prior versions.

Gertjan

@wujj123456 said in Concurrent/Rapid DNS queries fail randomly after 23.01 upgrade:

but disabling DNSSEC and leaving forward mode and DNS over TLS doesn't.

Been using 1.1.1.1 (host name one.one.one.one) and 2606:4700:4700::1111 (host name one.one.one.one), port 853 = TLS. Just to test, for 3 weeks.
DNSSEC disabled of course.
I didn't detect any issue.

Went back to default : resolving (= not forwarding), DNSSEC actif.
Empty

as : if all choices work well, the one with no settings is the one to prefer

See here.