DNS Resolver not caching correct?

johnpoz · Aug 31, 2019, 1:07 PM

@mrsunfire said in DNS Resolver not caching correct?:

(ns6218.hostgator.com) in 184 ms

So you see the authoritative ns are that far away from you - lot farther than 7ms ;)

At any given time you might of talked to something in the path that took longer than normal, maybe something in path was congested, maybe the ns took longer to answer, etc.

But that is how something is resolved - when none of it is cached.. now the once the ns for .com are cached no reason to ask roots for those, once the ns for whatever.com is cached no reason to talk to the gtld-servers.net ns - so you can just go ask the authoritative NS directly - but since its a lot farther than 7ms away from you - there can be delays... But in the big picture makes no matter. Since its only going to be once, and then its cached for the length of the TTL, and if you have prefetch setup and or zero answer - you will get an answer from unbound instantly and it will go refresh its stuff in the background..

So what does it matter if took 1000ms - that is not even going to be really noticed since I doubt a site couple hundred ms away from you is going to load instantly anyway.. Worse case you added a whole second to the page load time (ONCE)

There is little need of concerning your self if a site resolves in 30ms or 300ms, or even 1000ms.. Since once its cached - none of that matters any more. And in the big picture a fraction of a second in addition to the overall first load time of the page is meaningless.

perlenbacher · Aug 31, 2019, 2:13 PM

Hi johnpoz,

could you post your DNS Resolver, General Settings and Advanced Settings please.

It would be very handy for us all!

Thanks, Perlen

johnpoz · Aug 31, 2019, 3:11 PM

Sure here you go

Notice that I have disabled automatic ACLs so you will have to create your own to allow queries.

I have also changed from transparent to static for my zone.. Make sure you actually understand what settings do before changing them.. Any questions on what anything specifically does, just ask. Don't think this is some sort of guide to how you should set yours up.. These are my settings for my network and use case.. Most of them are just default.. Only a couple of changes really from out of the box settings. Which may or may not be good for your actual needs.

Generally speaking - out of the box should be fine for pretty much everyone.

As to the general settings - there are no dns set other than local... Here

perlenbacher · Aug 31, 2019, 4:15 PM

Very interesting, much appreciated. Thanks!

mrsunfire · Sep 1, 2019, 6:36 AM

For info: this is the result of cacheoutput after two days and I'm happy with it:

total.num.queries=21177
total.num.queries_ip_ratelimited=0
total.num.cachehits=15554
total.num.cachemiss=5623
total.num.prefetch=8627
total.num.zero_ttl=9232
total.num.recursivereplies=5623

Thanks a lot @johnpoz !

johnpoz · Sep 1, 2019, 8:13 AM

@mrsunfire said in DNS Resolver not caching correct?:

total.num.zero_ttl=9232

that seems like a lot of low TTLs - you might want to play with uping the min vs letting them set like 60 seconds and 5 min ttls

mrsunfire · Sep 1, 2019, 10:59 AM

So with what should I start? And what does this change? Isn't this dangerous?

johnpoz · Sep 1, 2019, 11:18 AM

In the advanced tab change the min ttl to say 1800 or 3600 vs letting them set such low ttls.. Could it cause issues - maybe. But I doubt it really..

Lets say you had something that was using a 60 second TTL, do the IPs even change - I had issue with some software where it checked a fqdn all the time, and the ttl was 60 seconds.. But upon check it multiple times the IP wasn't even changing - so what is the F'ing point of having dns query for it every 60 seconds.

Then you have them doing load balancing nonsense via dns

example
;scribe.logs.roku.com. IN A

;; ANSWER SECTION:
scribe.logs.roku.com. 60 IN A 52.21.150.70
scribe.logs.roku.com. 60 IN A 35.172.120.217
scribe.logs.roku.com. 60 IN A 34.233.159.203
scribe.logs.roku.com. 60 IN A 35.170.206.212
scribe.logs.roku.com. 60 IN A 52.200.248.59
scribe.logs.roku.com. 60 IN A 52.1.249.132
scribe.logs.roku.com. 60 IN A 34.226.55.236
scribe.logs.roku.com. 60 IN A 52.21.44.213

So while you might use 52.21.150.70 one time, next time is 25.172.120.217, etc. Only issue you could have is that 52.21.150.170 no longer works.

Have really never seen it be an issue.. But I wouldn't suggest you set the min to like 24 hours or anything.. If you find your having an issue getting somewhere, you can always just flush that specific entry from your cache.

This should lower the amount of both local queries and external queries.. Since now vs your local client asking for something every min with such a low ttl, it would only need to query 1 an hour for example.. Same goes for external queries. If your local device is not using a local cache - doesn't really matter what the ttl is for local queries - but would keep your dns from having to resolve it every freaking 60 seconds because some dumb iot device keeps asking for it every 30 freaking seconds, etc. because it has no local cache.

But hey if you got some local devices asking for whatever.something.tld every minute - you just got a 60x reduction in the number of external queries you have to do ;) if you change the min ttl to 1 hour. Multiply that by a few devices and few different fqdn being queried and in the course of 24 you could be doing 10s of 1000's of less queries.. Again prob not all that big of deal in the big picture.. But why do them if you don't really need to, etc.

mrsunfire · Sep 1, 2019, 6:19 PM

I don't know if I should play around with that because everything runs fast and good for now. But I see if my WAN goes down unbound restarts. Any chance to disable this function and keep it running?

Gertjan · Sep 2, 2019, 4:57 AM

@mrsunfire said in DNS Resolver not caching correct?:

But I see if my WAN goes down unbound restarts

It's possible that the restart of unbound wasn't needed. If the WAN IP stays the same, then maybe yes, it might not be needed.
I guess you should investigate why the WAN NIC goes down ? Missing an UPS ? Then add one.

There are many reasons that a WAN interface change has more effect, and then unbound should restart.
Loadbalancing. VPN usage (the tunnel is rebuild, and unbound should use the tunnel), etc.

However : there is no GUI setting that let you choose what to restart, or not.
Just stop ripping out the cable, or have people play with the power plug, and you'll be fine.

mrsunfire · Sep 2, 2019, 6:15 AM

The IP stays the same. I use cable and the inhouse coax is getting renewed. The old one sometimes has ingress and the connection goes down shortly. I have to say that pfSense then switches to my failover WAN2.

Any chance to disable unbound restart?

Gertjan · Sep 2, 2019, 6:21 AM

Do you mean your upstream WAN Ethernet goes really down ? The upstream device resets ?
It's a cable modem - your mentioned 'coax' ?

@mrsunfire said in DNS Resolver not caching correct?:

Any chance to disable unbound restart?

Not without you actually changing the code.

Btw : the monitoring process will also reset the interface if the upstream gateway becomes unreachable. You could change these settings.

mrsunfire · Sep 2, 2019, 6:26 AM

Yes the cable modem loses connection to the upstream gateway of my ISP. Then a gateway alarm appears and it switches to my WAN2.

Gertjan · Sep 2, 2019, 6:27 AM

@mrsunfire said in DNS Resolver not caching correct?:

Then a gateway alarm appears and it switches to my WAN2.

Ok. Pretty good reason to inform unbound about that event ^^

mrsunfire · Sep 2, 2019, 6:37 AM

Sure but why is it clearing the cache? An option to prevent that would be nice.

Gertjan · Sep 2, 2019, 7:43 AM

@mrsunfire said in DNS Resolver not caching correct?:

Sure but why is it clearing the cache? An option to prevent that would be nice.

Ah, now we are getting to the bottom of the subject.
It's unbound that needs to restart when the state of one of it's interfaces change.
I guess (my words) that unbound can't dynamic bind and unbind to IPs and/or ports.
This means : shutting down, Start again, thus reading the config (which is probably is rebuild because some interface came up = a viable or is missing now).
Side effect : cache is flushed / reset.

As far as I know, the cache isn't written when unbound stops to some cache file and read back in when starting.
Don't know if that is even possible - and I'm pretty sure this isn't done by pfSense.

Other side effect : people that use packages like pfBlockerNG with huge DNSBL lists will see something else : it will take long time (several tens of seconds) for unbound to start up because it has to read throughout all these lists and loading them.
If unbound restarts often - because, for example, it restarts when a DHCP lease comes in, they will experience DNS outages.

johnpoz · Sep 2, 2019, 10:09 AM

You could load the cache back

You have to dump it first
unbound-control dump_cache > unbound.dump

Then reload it
cat unbound.dump | unbound-control load_cache

Here is where you could run into a problem with that - petty sure the ttls would be frozen at that moment in time. I have not done much playing with doing something like that - just don't really see the point.. So your stuff resolves again... So first query would be a couple extra ms.. But if your ttl's are frozen at the moment of the dump, then counters restart on reload then you could be serving up expired ttls - and depending on the how long it was down, etc. They could be expired by very long time.. And still have lots of time left on them, so might be getting bad info for whatever the time left on the ttl was?

But sure - guess it could be coded to dump the cache every X minutes or something, and then on restart reload it. Would need easy way for users to flush other than restart.

Gertjan · Sep 2, 2019, 10:31 AM

@johnpoz said in DNS Resolver not caching correct?:

You could load the cache back
You have to dump it first
unbound-control dump_cache > unbound.dump
Then reload it
cat unbound.dump | unbound-control load_cache

Me ?
Ok.

First, a snapshot from what the cache is doing now :

unbound-control -c /var/unbound/unbound.conf stats_noreset | grep 'cache'

Note : adding " -c /var/unbound/unbound.conf '" isn't optional.

Now I dump to cache :

unbound-control -c /var/unbound/unbound.conf dump_cache > unbound.dump

For fun, look what's in this file : mine was nearly 3 Mega Bytes.

Now, I restart unbound in the GUI. A command line command exists, I'm; lazy and googled already to much this morning.

Another check :

unbound-control -c /var/unbound/unbound.conf stats_noreset | grep 'cache'

to see that unbound is running with an empty cache now.

Show time :

cat unbound.dump | unbound-control -c /var/unbound/unbound.conf load_cache

and re check again to see that "things" have been loaded :

unbound-control -c /var/unbound/unbound.conf stats_noreset | grep 'cache'

That seems to work - indeed

You're right - I can ^^

But pfSense can't / doesn't know how to do so.
(I can of course edit the related source files where unbound is restarted - it's just plain PHP .... but hey, all is fine for me already )

And yes, you're right. Mighty DNS gods would come down to earth when people start to load in old cache info and issues like that ....
There are already to much "issues" with DNS/unbound - or, take note : it works perfectly well out of the box.

johnpoz · Sep 2, 2019, 11:15 AM

No didn't mean for you to do it ;) Just stating how it "could" be done hehehehe

Guess it could be somewhat useful for someones who's unbound is restarting all the time.. I would look to why that is happening and stop it.. For example dhcp registration, or maybe pfblocker restarting every hour on a cron or something.

Or maybe your connecting is flipping over to backup, or going down whatever.. The dhcp is easy, just don't have it register dhcp leases. Your connecting going down or flipping might be harder to fix - but that shouldn't be happening on a regular basis that is for sure.

As to how often pfblocker restarts unbound - have to get with BBcan on that, off the top not sure when it might restart unbound.