DNS Resolver (Unbound) is Slow

[mpfrench]

Ref: Netgate 1100 running 23.01-RELEASE (arm64)
pfBlockerNG 3.2.0_3

I ran pfBlocker and configured several DNS blocking lists in addition to several IP blocks. With the exception that the 1100 would not load the UT1_Adult DNSBL, the system functioned well. To compensate, I set the DNS to forward to upstream servers that will filter adult content, i.e., 1.1.1.3 and 1.0.0.3.

I began to notice that my system acts a bit sluggish when web browsing and I've traced the problem to the way Unbound functions.

Steve Gibson developed a very useful utility for measuring DNS performance, DNSBenchmark. See https://www.grc.com/dns/benchmark.htm for a description.

I ran several tests with the 1100 feeding only one computer, comparing Unbound's performance with DNSmasq's (DNS Forwarder's) performance using Steve Gibson's utility. For all tests, both DNSmasq and Unbound were set to forward queries to 1.1.1.3 and 1.0.0.3.

First, here are the graphical and tabular results running DNSmasq. The address 192.168.13.1 is the address of the 1100's LAN while 192.168.1.1 is the Google Fiber Network Box LAN which is also set to forward to 1.1.1.3 and 1.0.0.3.

login-to-view

Here is the tabular data.
20230410_DNSmasq.txt

Next, I ran with the DNS Resolver (Unbound) and pfBlocker set to filter several DNSBLs. Again, running Unbound as a forwarder to 1.1.1.3 and 1.0.0.3.

login-to-view

Here is the tabular data.
20230410_Unbound_DNSBL.txt

The first thing I noticed was the extremely long times Unbound took to provide query answers and that it appeared that Unbound did not cache anything.

Thinking that the DNSBLs might be causing the relatively poor performance, I set pfBlocker not to perform DNSBL filtering and reran the tests.

login-to-view

Here is the tabular data.
20230410_Unbound_DNSBL.txt

Unbound's performance did not improve when DNSBLs were eliminated. Consequently, there is something in the underlying configuration, outside any pfsense GUI settings, that is causing this problem.

I run Unbound on a server in my system to provide local name resolution and forward all else to the 1100 in my normal configuration and do not see this behavior from Unbound on my server.

It looks as though the underlying Unbound settings need a tuneup.

[johnpoz]

@mpfrench did you uncheck dnssec when you changed unbound to foward.. That for sure is going to ask a lot of stuff doesn't need to ask for when forwarding...

Did you make sure unbound didn't restart during this process - for all we know it restarted 6 times during you testing because of dhcp requests on your network, etc..

As to your cached - no it wouldn't have a cache for www.cnn.com unless you had already asked for www.cnn.com before, and you hadn't restarted it between.. You ask 1.1.1.1 on the internet - yeah more than likely what you asked for is cached, because billy before you and kevin and kim, and barb all asked for it before you..

To be honest if your concerned that xyz returns an answer 10ms faster than abc and somehow think that is slowing down your internet.. You are looking down the wrong rabbit hole.

[mpfrench]

@mpfrench
DNSSEC was not running. I know that there is a bug in its pfsense implementation.

I have no control over whether or not Unbound restarts during testing. It is what it is. However, it does not appear to be restarting since the reported times would be much larger if it were restarting.

The Unbound configuration needs a tuneup.

[johnpoz]

@mpfrench said in DNS Resolver (Unbound) is Slow:

The Unbound configuration needs a tuneup.

MIne is working just fine - if you don't like yours, then change it..

[Gertjan]

@mpfrench said in DNS Resolver (Unbound) is Slow:

First, here are the graphical and tabular results running DNSmasq. The address 192.168.13.1 is the address of the 1100's LAN

The device ("PC") is using a 192.168.13.0/24, right ?

What I don't understand :

@mpfrench said in DNS Resolver (Unbound) is Slow:

is the Google Fiber Network Box LAN which is also set to forward to 1.1.1.3 and 1.0.0.3

So the pfSense WAN IP is something like 192.168.1.x, right ?
pfSense is a LAN device for this Google box.

That that device also uses a DNS, ok, but why would it be used / mentioned on your PC ?
I've run the same app, and no where my ISP router was included in these tests.

All devices on all my LANs should (could) use their gateway IP as the 'DNS server'. On all these LAN IPs unbound listens, and then forwards to 1.1.1.1 etc (over TLS, as I'm testing that mode for a couple of days).

Your second image :

Next, I ran with the DNS Resolver (Unbound) and pfBlocker set to filter several DNSBLs. Again, running Unbound as a forwarder to 1.1.1.3 and 1.0.0.3.

Again, If I get it right : your test device is connected to has IP 192.168.13.x and is connected to 192.168.13.1 (pfSense LAN), The pfSense WAN is connected to the Google device (iyt's LAN is 192.168.1.1) so : how is is possible that that device, further down the road, is faster as pfSense, closer to you ?

By any change : is your test device (PC° using 2 interfaces, like one wired NIC and one Wifi, connected, so it can 'see' and use both the Google dns and the pfSense dns

@mpfrench said in DNS Resolver (Unbound) is Slow:

pfBlockerNG 3.2.0_3

It's upgrade time since a couple of days. 3.2.0_4 is out.

@mpfrench said in DNS Resolver (Unbound) is Slow:

I know that there is a bug in its pfsense implementation.

Not a bug
If "DNSSEC" combined with forwarding would not work or create some confictual situation, then it would have been signaled here https://nlnetlabs.nl/documentation/unbound/unbound.conf/ as mutual exclusive.
It isn't.
Does it make sense, that is the question.
With extreme few words :
If you're forwarding, you want fast results, and you don't care the if it correct.
When resolving, you want to take the answer from the authoritative name server, and because you are talking with, ask also if DNSSE info is available. If so, a top to bottom check is executed to be sure the obtained answer isn't spoofed. This is an example of the entire check - this domain name is checked from top to bottom.

Btw : If you were forwarding to for example to 1.1.1.1 (they are resolving, as 101.1.1 is a resolver) : they are also doing DNSSEC. Knowing that, you only have to trust them and the link between you and them, and you'll be good.

Btw : you are testing the 1100 and you concluded : it can do better.
Ok, why not.
You could have tested this one also : Netgate 1000
Take also a spin with this one : 8200.
You'll see a trend, I'll bet on that.

You will also discover that your direct uplink, the line between you and your ISP make spart of the result.
And also the peering of your ISP to all these DNS server endpoints.
And the current load on all these "lines".
And the current work load of all these DNS servers.
So, your answer will be "it's x msec" with a big fault tolerance factor.

@mpfrench said in DNS Resolver (Unbound) is Slow:

The Unbound configuration needs a tuneup.

Take the one where you can stop thinking about it : the default values, when you installed pfSense.
This means : no forwarding what so ever - just resolving.
From a maintenance perspective : just perfect. And yeah, resolving take some time, you'll burn dozens of milliseconds. But ones in the local cache, and you take care of not restarting unbound, this cache will grow and stay up to date :

login-to-view

@mpfrench said in DNS Resolver (Unbound) is Slow:

and that it appeared that Unbound did not cache anything

There is no place for words like "appeared".
Why didn't you check instead of writing that line ?

/usr/local/sbin/unbound-control -c /var/unbound/unbound.conf dump_cache | wc -l

This is : use the "unbound-control" with the current unbound config file to dump the current cache to the command line.
Then count the line in that 'file'.
I've about 3200 lines in that cache file.
So, it isn't 'empty'.
And better : ones the A record of facebook.com is in there, it will keep it up to date for me, and renew the request when the TTL expires.

@mpfrench said in DNS Resolver (Unbound) is Slow:

I have no control over whether or not Unbound restarts during testing

But you can check :

grep 'start' /var/log/resolver.log

Or have a look at the Status > System Logs > System > DNS Resolver page and look up the stop and start moments.

Gibson said :

login-to-view

and that included also many implicit things.
Like (stupid, I know) : "Do not reset your router during testing".

But also : do not use other tools that might influence the DNS testing - or overall Internet access availability.
This means that you should test on a default, non forwarding pfSense without any pfSense packages (activated). That includes, and is not limited to : pfBlockerng.

For me, the fastest DNS in the neighborhood is

login-to-view

Take note : I've stopped and started manually unbound just before the test.
So the local cache was empty.
"Everything" had to be looked up against 1.1.1.1 over TLS.

[RobbieTT]

@mpfrench

Your resolver on your 1100 is on 192.168.1.1 and this looks to be really fast. The red bar is so small compared to the others that you can hardly see it. The tool also ranks the results with the best performing listed at the top, which (unsurprisingly) is your DNS Resolver, principally due to its working cache:

login-to-view

Is there some confusion somewhere or did I misread your question?

️

[Gertjan]

@robbiett said in DNS Resolver (Unbound) is Slow:

Your resolver on your 1100 is on 192.168.1.1

Doesn't match with :

@mpfrench said in DNS Resolver (Unbound) is Slow:

The address 192.168.13.1 is the address of the 1100's LAN while 192.168.1.1 is the Google Fiber Network Box LAN

Or am I reading things differently ?

[RobbieTT]

@gertjan

No, re-reading it your interpretation makes sense. What does not make sense is running 2 local hardware DNS servers together when testing. One tends to step on the other.

️

[RobbieTT]

@mpfrench

I ran the test you used, just to see what it did with my setup:

login-to-view

So my DNS Resolver is pretty quick.

I can make the graph look awful by including the uncached results:

login-to-view

But in reality my DNS queries rarely require going outside of the cache thanks primarily to prefetch, expired and the regular cache. When they do I have DNSSEC, DoT and the extra round-robin exchanges that go with them. As a result, these queries are much slower but as they happen <4% of the time the real-world impact is just imperceptible. The average results are just dominated by the cache performance.

As for Mr Gibson's test conclusions:

login-to-view

Some of it may be a little misleading but it does highlight the performance advantage of the DNS resolver in my system. I could improve the non-cache test speed by removing DoT and undertaking pure port 53 UDP resolutions but outside of this DNS test it would not be noticeable and expose more data than needed.

️