Frequent DNS timeouts

oopohj5Oo8shieZe1ree · Feb 24, 2023, 6:15 PM

On my pfSense 23.01 and pfBlockerNG 3.2 network I have frequent yet seemingly random DNS timeouts. They are most notable in web browsers when accessing a domain that hasn't been accessed recently. The browser hangs doing DNS resolution and sometimes fails outright. This happens on multiple devices, operating systems, browsers, and applications.

A few days ago I deleted my pfBlockerNG configuration and reinstalled. Using the setup wizard I created the default configuration and left it as is. I'm still experiencing DNS timeouts.

I'm at a loss for how to troubleshoot this. Any suggestions would be very welcome.

(Note: for my personal browser I've disabled custom DNS resolution to ensure the browser is going through pfSense and not a third-party DNS provider. This doesn't seem to help though.)

johnpoz · Feb 25, 2023, 4:35 AM

@oopohj5oo8shieze1ree if I had to guess I would guess unbound is restarting a lot, when it restarts yeah dns isn't going to work.

Look in your log - is unbound restarting?

SteveITS · Feb 25, 2023, 4:42 AM

@oopohj5oo8shieze1ree Also if you are forwarding, ensure DNSSEC is unchecked.

oopohj5Oo8shieZe1ree · Feb 25, 2023, 3:37 PM

@johnpoz I've looked in the general log file, pfBlockerNG log files, and the DNS resolver log file and I don't see unbound restarting.

I did notice a couple of these:

debug: outnettcp got tcp error -1

And occasionally:

/rc.linkup: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1677281515] unbound[36369:0] error: bind: address already in use [1677281515] unbound[36369:0] fatal error: could not open ports'

oopohj5Oo8shieZe1ree · Feb 25, 2023, 3:44 PM

@steveits I'm not exactly sure what you mean by forwarding.

In the DNS resolver settings both DNSSEC and DNS Query Forwarding are turned on. But I'm not running the DNS Forwarding service.

Should I disable DNSSEC in the DNS Resolver settings, and is that safe to do so?

SteveITS · Feb 25, 2023, 3:56 PM

@oopohj5oo8shieze1ree said in Frequent DNS timeouts:

both DNSSEC and DNS Query Forwarding are turned on

Yep that's it. 23.01 seems more sensitive/has problems in that configuration. Uncheck the DNSSEC option. you're already trusting the DNS servers to which you forward.

For instance per https://support.quad9.net/hc/en-us/articles/4433380601229-Setup-pfSense-and-DNS-over-TLS
"DNSSEC is already enforced by Quad9, and enabling DNSSEC at the forwarder level can cause false DNSSEC failures"

oopohj5Oo8shieZe1ree · Feb 25, 2023, 4:10 PM

@steveits Thanks for the help.

I also read the pfSense documentation and came to the same conclusion. I've disabled DNSSEC. I'll report back after a couple of days whether or not my issue has been resolved.

SteveITS · Feb 25, 2023, 4:14 PM

@oopohj5oo8shieze1ree Here’s hoping. It did for me and several others so far, despite not being a problem in prior versions.

oopohj5Oo8shieZe1ree · Feb 27, 2023, 6:09 PM

Unfortunately disabling DNSSEC has not fixed my issue. I'm still getting DNS timeouts from time to time :(

SteveITS · Feb 27, 2023, 6:27 PM

@oopohj5oo8shieze1ree There was a post today that 'Disabling "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers"' helped that person.

https://forum.netgate.com/post/1090876

SteveITS · Feb 27, 2023, 6:32 PM

Also there's a fix for Unbound not correctly binding to "All" interfaces on IPv6.

https://forum.netgate.com/topic/176989/problems-with-pfsense-ipv6-dns-function-does-it-exist/36

oopohj5Oo8shieZe1ree · Feb 27, 2023, 7:58 PM

@steveits Thanks for pointing me to the other threads.

I'm thinking of just giving up on using forwarding. I need to figure out if my ISP limits access to DNS servers when not forwarding.

oopohj5Oo8shieZe1ree · Mar 6, 2023, 9:07 PM

After turning off DNS forwarding, resolution was nearly instantaneous for a couple of days. But the random timeouts have returned.

I don't see anything in the logs to indicate something is failing.

Can someone point me to a DNS debugging guide or something that will help me figure out what the root cause is here.

Thank you.

SteveITS · Mar 6, 2023, 9:16 PM

@oopohj5oo8shieze1ree There are a few here:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/index.html#dns

Also take a look through https://forum.netgate.com/category/19/dhcp-and-dns as there are other posts for 23.01.

thundergate · Mar 15, 2023, 5:06 AM

@oopohj5oo8shieze1ree

Hi. Do have nearly the same issues.

But for me I don't use and DNS forwarding or anything else. Just pfSense Unbound in combination with pfBlockerNG.

Don't have any DNS fails at all. But looks like name resolution does hang after some amount of time. After that it looks like it is cached again and resolution works fine.
But I do have this issues nearly every day.

Something like a cleared unbound cache - what's not the case.

Gertjan · Mar 15, 2023, 9:12 AM

@thundergate said in Frequent DNS timeouts:

cleared unbound cache

This can only happens when the resolver -unbound is told to stop, or restart, which is a controlled stop, to be started right afterwards.
It can take several seconds to do so.
The cache will be lost, but subsequent DNS resolving won't take long, typical is a fraction of a second.

If unbound restarts happen very often, you can start to 'feel' the absence of the DNS sub system.

So, ask your pfSense how often it restarts :

grep "Restart" /var/log/resolver.log

If it's just couple of times a day (lesser == better) : this is not your issue.

oopohj5Oo8shieZe1ree · Mar 15, 2023, 4:20 PM

@steveits After switching from forwarding to normal resolving I let things sit for a bit to see what would happen. It looks like unbound is restarting a lot:

Mar 15 08:01:04 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1.
Mar 15 08:08:33 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1.
Mar 15 08:12:20 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1.
Mar 15 08:12:45 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1.
Mar 15 08:13:39 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1.
Mar 15 08:29:45 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1.
Mar 15 08:34:44 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1.
Mar 15 08:35:41 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1.
Mar 15 08:42:09 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1.
Mar 15 08:42:34 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1.
Mar 15 08:49:47 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1.
Mar 15 08:52:13 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1.

Is there a known workaround for this?

SteveITS · Mar 15, 2023, 4:59 PM

@oopohj5oo8shieze1ree The most common cause for restarts is having DHCP set to register DHCP leases in DNS, which triggers a restart after each and every DHCP lease. Options are to not do that, or to make the lease long enough that it renews in "days" not "hours." (renewal is 1/2 of the lease duration)

oopohj5Oo8shieZe1ree · Mar 15, 2023, 5:09 PM

@steveits I believe I have that turned off (in Services -> DHCP Server -> Dynamic DNS ->
Enable registration of DHCP client names in DNS). However, it does appear to be registering DHCP host names with the DNS server regardless of this setting.

I've increased the lease time and will report back.

Thank you.

johnpoz · Mar 15, 2023, 5:11 PM

@oopohj5oo8shieze1ree unbound starting that often is going to be problematic that is for sure..