Unbound 1.8.1 only single thread processing DNS requests

imcdona

I ran into the DNS over TLS memory leak in Unbound 1.7.3 which is included in pfSense. Per the instructions here I updated Unbound to 1.8.1

pkg update; pkg upgrade unbound

After the upgrade I found that Unbound appears to only be using a single thread. Note that only "thread 0" has any stats

>unbound-control -c /var/unbound/unbound.conf stats_noreset

thread0.num.queries=1997
thread0.num.queries_ip_ratelimited=0
thread0.num.cachehits=21
thread0.num.cachemiss=1976
thread0.num.prefetch=3
thread0.num.zero_ttl=7
thread0.num.recursivereplies=1943
thread0.requestlist.avg=18.0273
thread0.requestlist.max=55
thread0.requestlist.overwritten=0
thread0.requestlist.exceeded=0
thread0.requestlist.current.all=25
thread0.requestlist.current.user=18
thread0.recursion.time.avg=0.360357
thread0.recursion.time.median=0.16633
thread0.tcpusage=0
thread1.num.queries=0
thread1.num.queries_ip_ratelimited=0
thread1.num.cachehits=0
thread1.num.cachemiss=0
thread1.num.prefetch=0
thread1.num.zero_ttl=0
thread1.num.recursivereplies=0
thread1.requestlist.avg=0
thread1.requestlist.max=0
thread1.requestlist.overwritten=0
thread1.requestlist.exceeded=0
thread1.requestlist.current.all=0
thread1.requestlist.current.user=0
thread1.recursion.time.avg=0.000000
thread1.recursion.time.median=0
thread1.tcpusage=0
thread2.num.queries=0
thread2.num.queries_ip_ratelimited=0
thread2.num.cachehits=0
thread2.num.cachemiss=0
thread2.num.prefetch=0
thread2.num.zero_ttl=0
thread2.num.recursivereplies=0
thread2.requestlist.avg=0
thread2.requestlist.max=0
thread2.requestlist.overwritten=0
thread2.requestlist.exceeded=0
thread2.requestlist.current.all=0
thread2.requestlist.current.user=0
thread2.recursion.time.avg=0.000000
thread2.recursion.time.median=0
thread2.tcpusage=0
thread3.num.queries=0
thread3.num.queries_ip_ratelimited=0
thread3.num.cachehits=0
thread3.num.cachemiss=0
thread3.num.prefetch=0
thread3.num.zero_ttl=0
thread3.num.recursivereplies=0
thread3.requestlist.avg=0
thread3.requestlist.max=0
thread3.requestlist.overwritten=0
thread3.requestlist.exceeded=0
thread3.requestlist.current.all=0
thread3.requestlist.current.user=0
thread3.recursion.time.avg=0.000000
thread3.recursion.time.median=0
thread3.tcpusage=0
total.num.queries=1997
total.num.queries_ip_ratelimited=0
total.num.cachehits=21
total.num.cachemiss=1976
total.num.prefetch=3
total.num.zero_ttl=7
total.num.recursivereplies=1943
total.requestlist.avg=18.0273
total.requestlist.max=55
total.requestlist.overwritten=0
total.requestlist.exceeded=0
total.requestlist.current.all=25
total.requestlist.current.user=18

Anyone have any ideas as to what may be going on here?

tibere86

@imcdona Looks like this bug was fixed in Unbound 1.8.2 released today.

Gertjan

Yep : see https://www.netgate.com/blog/pfsense-2-4-4-release-p1-now-available.html => Notable Bug Fixes => Updated Unbound to 1.8.1 to address issues with memory leaks, especially in DNS over TLS support.

See also the identical ( !! ) thread just a little bit further down in the same forum "DHCP DNS" for an free bonus.

tibere86

@imcdona On a side note, how do you access the Unbound stats? What command needs to be entered at the terminal?

imcdona

@tibere86 said in Unbound 1.8.1 only single thread processing DNS requests:

Looks like this bug was fixed in Unbound 1.8.2 released today.

@gertjan said in Unbound 1.8.1 only single thread processing DNS requests:

Yep : see https://www.netgate.com/blog/pfsense-2-4-4-release-p1-now-available.html => Notable Bug Fixes => Updated Unbound to 1.8.1 to address issues with memory leaks, especially in DNS over TLS support.
See also the identical ( !! ) thread just a little bit further down in the same forum "DHCP DNS" for an free bonus.

Wrong. They fixed the TLS memory issues. Unbound is still only processing requests from a single thread. The temporary fix is to add:

server:
so-reuseport: no

imcdona

@tibere86 said in Unbound 1.8.1 only single thread processing DNS requests:

@imcdona On a side note, how do you access the Unbound stats? What command needs to be entered at the terminal?

Run the following

unbound-control -c /var/unbound/unbound.conf stats_noreset

or for a list of other options run

unbound-control -c /var/unbound/unbound.conf

such as

unbound-control -c /var/unbound/unbound.conf dump_requestlist

bbrendon

Is there a bug open in redmine for this?
Is it considered a bug? Is the workaround required in 1.8.2?

And also, is there a 1.8.2 we can manually install?

tibere86

@imcdona Thanks!

jimp

@bbrendon said in Unbound 1.8.1 only single thread processing DNS requests:

Is there a bug open in redmine for this?

Not yet

Is it considered a bug?

Unbound considers it a problem with the default configuration on FreeBSD, so I'm not quite sure they consider it a bug.

Is the workaround required in 1.8.2?

No, since the default configuration changed. The workaround would still be safe to have, but it wouldn't make a difference since the default is 'no'.

And also, is there a 1.8.2 we can manually install?

No, it isn't even in FreeBSD ports yet, and since you can work around it with a simple config change, it probably wouldn't warrant an OOB update like the memory leak did.

lohphat

@imcdona Add those lines where?

johnpoz

In the custom options box.

TheNarc

@imcdona From what I can tell, the default value of so-reuseport is no, and that it would need to be set to yes to use multiple threads? But also, I've found information suggesting that FreeBSD's setsockopt implementation didn't support the option at all until version 12 (https://nlnetlabs.nl/pipermail/unbound-users/2018-November/011134.html). Of course, I won't argue with experimental evidence to the contrary, I'm just a bit confused.

lohphat

@johnpoz I tried that but didn't notice a change in behavior. I've seen other threads where unbound was crashing and restarting, could that be the cause of sluggish or timed-out DNS queries?

I've also noted that when PfBlockerNG-devel is enabled it places the line into the custom settings box but combines the lines. I was wondering if this is an issue? Are both formats allowed or is PfBlocerNG-devel not adding a line after "server:" where it should?

server:
so-reuseport: no
server:include: /var/unbound/pfb_dnsbl.*conf

RonpfS

You wont see change in behaviour, but when you reload/restart unbound, you will see different stats indicating that it's now multi-threaded .

Both

server:
so-reuseport: no

server: so-reuseport: no

are fine

lohphat

@ronpfs Is the missing white-space after "server:" also permitted?

server:include: /var/unbound/pfb_dnsbl.*conf

RonpfS

@lohphat said in Unbound 1.8.1 only single thread processing DNS requests:

Is the missing white-space after "server:" also permitted?

That's what I have in my settings :

server:qname-minimisation: yes
server:so-reuseport: no
server:include: /var/unbound/pfb_dnsbl.*conf

johnpoz

@ronpfs said in Unbound 1.8.1 only single thread processing DNS requests:

server:qname-minimisation: yes

This is now possible in the gui, no need for it custom just check the box in the gui.