Incorrect rrset-cache-size in unbound.conf

wagebox · Jan 31, 2015, 8:50 PM

Hello guys,

I've been playing with the unbound a little bit since it got introduced in the 2.2.2 release. I wanted to have an idea about memory consumption and limits so I sent some queries for a bunch of domains using dig. I have used the Alexa top 1 million domains list as a base and run a simple script to go through them by "digging" ANY for a while.

unbound is configured to use 100MB as message cache, which should result into having 200MB as rrset cache according to web configuration interface.

However, when checking the stats using unbound-control -c /var/unbound/unbound.conf stats (btw, some status page would be nice ;)), I have noticed, that rrset.cache.count never reaches significantly above 30k and mem.cache.rrset is stuck around 8MB.

unbound-control -c /var/unbound/unbound.conf stats:


$ unbound-control -c /var/unbound/unbound.conf stats
thread0.num.queries=18397
thread0.num.cachehits=5
thread0.num.cachemiss=18392
thread0.num.prefetch=0
thread0.num.recursivereplies=18388
thread0.requestlist.avg=2.89985
thread0.requestlist.max=22
thread0.requestlist.overwritten=0
thread0.requestlist.exceeded=0
thread0.requestlist.current.all=3
thread0.requestlist.current.user=2
thread0.recursion.time.avg=1.339847
thread0.recursion.time.median=0.246674
total.num.queries=18397
total.num.cachehits=5
total.num.cachemiss=18392
total.num.prefetch=0
total.num.recursivereplies=18388
total.requestlist.avg=2.89985
total.requestlist.max=22
total.requestlist.overwritten=0
total.requestlist.exceeded=0
total.requestlist.current.all=3
total.requestlist.current.user=2
total.recursion.time.avg=1.339847
total.recursion.time.median=0.246674
time.now=1422737153.945010
time.up=9341.711168
time.elapsed=533.959676
mem.total.sbrk=0
mem.cache.rrset=8913062
mem.cache.message=15570329
mem.mod.iterator=16532
mem.mod.validator=4045694
histogram.000000.000000.to.000000.000001=3
histogram.000000.000001.to.000000.000002=0
histogram.000000.000002.to.000000.000004=0
histogram.000000.000004.to.000000.000008=0
histogram.000000.000008.to.000000.000016=0
histogram.000000.000016.to.000000.000032=0
histogram.000000.000032.to.000000.000064=0
histogram.000000.000064.to.000000.000128=0
histogram.000000.000128.to.000000.000256=0
histogram.000000.000256.to.000000.000512=1
histogram.000000.000512.to.000000.001024=0
histogram.000000.001024.to.000000.002048=1
histogram.000000.002048.to.000000.004096=0
histogram.000000.004096.to.000000.008192=0
histogram.000000.008192.to.000000.016384=14
histogram.000000.016384.to.000000.032768=162
histogram.000000.032768.to.000000.065536=903
histogram.000000.065536.to.000000.131072=2939
histogram.000000.131072.to.000000.262144=5863
histogram.000000.262144.to.000000.524288=5532
histogram.000000.524288.to.000001.000000=1202
histogram.000001.000000.to.000002.000000=734
histogram.000002.000000.to.000004.000000=520
histogram.000004.000000.to.000008.000000=239
histogram.000008.000000.to.000016.000000=98
histogram.000016.000000.to.000032.000000=65
histogram.000032.000000.to.000064.000000=60
histogram.000064.000000.to.000128.000000=30
histogram.000128.000000.to.000256.000000=7
histogram.000256.000000.to.000512.000000=9
histogram.000512.000000.to.001024.000000=6
histogram.001024.000000.to.002048.000000=0
histogram.002048.000000.to.004096.000000=0
histogram.004096.000000.to.008192.000000=0
histogram.008192.000000.to.016384.000000=0
histogram.016384.000000.to.032768.000000=0
histogram.032768.000000.to.065536.000000=0
histogram.065536.000000.to.131072.000000=0
histogram.131072.000000.to.262144.000000=0
histogram.262144.000000.to.524288.000000=0
num.query.type.A=730
num.query.type.PTR=1
num.query.type.TXT=9
num.query.type.SRV=6
num.query.type.ANY=17651
num.query.class.IN=18397
num.query.opcode.QUERY=18397
num.query.tcp=0
num.query.tcpout=618
num.query.ipv6=0
num.query.flags.QR=0
num.query.flags.AA=0
num.query.flags.TC=0
num.query.flags.RD=18397
num.query.flags.RA=0
num.query.flags.Z=0
num.query.flags.AD=17649
num.query.flags.CD=0
num.query.edns.present=17649
num.query.edns.DO=0
num.answer.rcode.NOERROR=18178
num.answer.rcode.FORMERR=0
num.answer.rcode.SERVFAIL=186
num.answer.rcode.NXDOMAIN=29
num.answer.rcode.NOTIMPL=0
num.answer.rcode.REFUSED=0
num.answer.rcode.nodata=67
num.answer.secure=217
num.answer.bogus=0
num.rrset.bogus=1
unwanted.queries=0
unwanted.replies=1
msg.cache.count=59574
rrset.cache.count=30991
infra.cache.count=40580
key.cache.count=16413

So I checked the unbound.conf (IPs bogusified):


##########################
# Unbound Configuration
##########################

##
# Server configuration
##
server:

chroot: /var/unbound
username: "unbound"
directory: "/var/unbound"
pidfile: "/var/run/unbound.pid"
use-syslog: yes
port: 53
verbosity: 1
hide-identity: yes
hide-version: yes
harden-referral-path: no
harden-glue: yes
do-ip4: yes
do-ip6: yes
do-udp: yes
do-tcp: yes
do-daemonize: yes
module-config: "validator iterator"
unwanted-reply-threshold: 10000000
num-queries-per-thread: 512
jostle-timeout: 200
infra-host-ttl: 900
infra-cache-numhosts: 50000
outgoing-num-tcp: 10
incoming-num-tcp: 10
edns-buffer-size: 4096
cache-max-ttl: 86400
cache-min-ttl: 0
harden-dnssec-stripped: yes
msg-cache-size: 100m
num-threads: 1
msg-cache-slabs: 4
rrset-cache-slabs: 4
infra-cache-slabs: 4
key-cache-slabs: 4
rrset-cache-size: 8m
outgoing-range: 4096
#so-rcvbuf: 4m
auto-trust-anchor-file: /var/unbound/root.key
prefetch: yes
prefetch-key: yes
# Statistics
# Unbound Statistics
statistics-interval: 0
extended-statistics: yes
statistics-cumulative: yes

# Interface IP(s) to bind to
interface: 192.168.1.1
interface: 192.168.2.1
interface: 192.168.3.1

# Outgoing interfaces to be used
outgoing-interface: 1.2.3.4
outgoing-interface: 192.168.1.1

# DNS Rebinding
# For DNS Rebinding prevention
private-address: 10.0.0.0/8
private-address: 172.16.0.0/12
private-address: 169.254.0.0/16
private-address: 192.168.0.0/16
private-address: fd00::/8
private-address: fe80::/10
# Set private domains in case authoritative name server returns a Private IP address
private-domain: "lan.local"
domain-insecure: "lan.local"

# Access lists
include: /var/unbound/access_lists.conf

# Static host entries
include: /var/unbound/host_entries.conf

# dhcp lease entries
include: /var/unbound/dhcpleases_entries.conf

# Domain overrides
include: /var/unbound/domainoverrides.conf

# Unbound custom options
server: local-zone: "168.192.in-addr.arpa." nodefault
stub-zone: name: "168.192.in-addr.arpa."
stub-addr: 192.168.1.10

###
# Remote Control Config
###
include: /var/unbound/remotecontrol.conf

Despite having set the msg cache to 100MB in web config, the rrset cache is alwaus stuck at 8MB:

rrset-cache-size: 8m

Btw, maybe it would not be such a bad idea to add more controls, such sa number of threads, individually setting the msg and rrset caches and higher ceiling for maximum cache entries.

kejianshi · Jan 31, 2015, 8:56 PM

I'd like to be able to see whats in the hints.

wagebox · Jan 31, 2015, 9:49 PM

I don't believe it is related to root hints file - if that's what you are asking for. There is no entry specifying the root hints in unbound.conf, so the default internal list is used.

And there is no problem with dns resolution itself. This looks to be a web configurator problem. It's not setting the correct value for rrset-cache-size in unbound.conf.

kejianshi · Jan 31, 2015, 10:28 PM

haha - I know.

I was just saying, as long as you are asking for features, I'd like to be able to see the root hint.

doktornotor · Jan 31, 2015, 10:35 PM

@kejianshi:

I was just saying, as long as you are asking for features, I'd like to be able to see the root hint.


unbound-control -c /var/unbound/unbound.conf list_stubs | grep -v noprime

kejianshi · Jan 31, 2015, 10:39 PM

Thats also works… :P

doktornotor · Jan 31, 2015, 10:44 PM

Yeah… dunno why the status page vanished, was pretty nice in the 2.1.x package.

kejianshi · Feb 2, 2015, 7:38 PM

Dumb question and not even sure if it would make a difference…

But... After you changed the advanced setting in unbound, did you restart the service or reboot? I usually reboot after tinkering with anything not basic.

wagebox · Feb 2, 2015, 9:47 PM

For those who asked to see the root hints:


$ unbound-control -c /var/unbound/unbound.conf list_stubs | grep -v noprime
. IN stub prime M.ROOT-SERVERS.NET. L.ROOT-SERVERS.NET. K.ROOT-SERVERS.NET. J.ROOT-SERVERS.NET. I.ROOT-SERVERS.NET. H.ROOT-SERVERS.NET. G.ROOT-SERVERS.NET. F.ROOT-SERVERS.NET. E.ROOT-SERVERS.NET. D.ROOT-SERVERS.NET. C.ROOT-SERVERS.NET. B.ROOT-SERVERS.NET. A.ROOT-SERVERS.NET. 2001:dc3::35 2001:500:3::42 2001:7fd::1 2001:503:c27::2:30 2001:7fe::53 2001:500:1::803f:235 2001:500:2f::f 2001:500:2d::d 2001:500:2::c 2001:500:84::b 2001:503:ba3e::2:30 202.12.27.33 199.7.83.42 193.0.14.129 192.58.128.30 192.36.148.17 128.63.2.53 192.112.36.4 192.5.5.241 192.203.230.10 199.7.91.13 192.33.4.12 192.228.79.201 198.41.0.4

I have tried to change the settings, restart unbound and also reboot. The rrset-cache-size is never updated in unbound.conf.

So, I checked which of the php scripts is actually controlling unbound advanced options and found it's:

/usr/local/www/services_unbound_advanced.php

…only to find exactly nothing :) Simply put, there are no references to any functions or variables that seem to update the value of rrset-cache-size in unbound.conf. That piece of code is missing.

As this is the first time I'm looking at the inner workings of pfsense, please better check it, just to be sure I'm not missing something.

phil.davis · Feb 3, 2015, 3:01 AM

The back-end code is in /etc/in/unbound.inc
I added a comment to https://redmine.pfsense.org/issues/4367