Slow DNS after 22.05

Cool_Corona

Unbound on 2.5.2

johnpoz

@cool_corona @Kempain 300 some seconds unbound being up, doesn't bode well for caching name server ;)

Cool_Corona

@johnpoz How to track all unbound restarts?

tentpiglet

@johnpoz said in Slow DNS after 22.05:

Do you have IPv6?

No. Although I think pfSense assigns a link local address to my wan and lan interface, my ISP does not provide me with a ip6 address.

@gertjan said in Slow DNS after 22.05:

22.05 or 2.6.0 ?

22.05-RELEASE (amd64)
built on Wed Jun 22 18:56:13 UTC 2022
FreeBSD 12.3-STABLE

UNstable means :
You found it 'not running', and no process 'unbound' was present when you list processes ?
Or the process unbound was there, but didn't answer ?
An nslookup from a PC on a LAN failed ?

unstable means I'm using a browser on a client and I get a "DNS_PROBE_FINISHED_NXDOMAIN" error (google chome), and when I hit the reload button a few moments later, the page will be found.

edit : how many times a day unbound restarts ?

grep 'start of service' /var/log/resolver.log | wc -l

64

first entry was at 10:21am local time this morning, so that's over the past 2 hours.

johnpoz

@tentpiglet said in Slow DNS after 22.05:

DNS_PROBE_FINISHED_NXDOMAIN

And what specific were you looking for - NX is not a dns failure, that is what you were looking for doesn't exist.

Also - browser errors are not troubleshooting dns problem - for all we know your browser is using doh and has zero to do with yoru local dns.

tentpiglet

@johnpoz said in Slow DNS after 22.05:

And what specific were you looking for - NX is not a dns failure, that is what you were looking for doesn't exist.

Randomly, any site I would go to. For example, the last time it happened when I said "enough is enough" and spun up a pihole container on my Proxmox server, it was "www.facebook.com".

I'm pretty sure www.facebook.com exists.

My wife was also reporting random disconnects from an online game she was playing, as well as getting similar site not found errors on her phone while reading reddit. Since moving DNS over to the pihole container, she's experienced no further issues.

johnpoz

@tentpiglet said in Slow DNS after 22.05:

I'm pretty sure www.facebook.com exists.

A NX is a dns response that says that domain doesn't exist - why you got that error have no idea, but that is not a failure to talk to dns, that is whatever your dns was doing ended up with NX.. Maybe you would doing a query for www.facebook.com.org or www.facebook.com.somethingelse.tld

Your browser trying to help you auto complete something etc..

I would expect you have all kinds of issues if your local dns is restarting 64 times in less than 24 hours.. So yeah that would be problematic for sure..

kvhs

@johnpoz Like I described earlier I experienced this myself on duckduckgo.com when set as my primary search engine in the browser. Sometimes it would work othertimes not. Reloading a number of times would often work, but that could of course be an illusion if the cause was that some process was restarting.

What would be the preferred test to demonstrate the problem, given its intermittent nature?

Mikymike82

@johnpoz @provels @Gertjan , maybe this is not a widespread problem.... but this seems to be an issue thats more than a single user/unit problem.
Talking albout the so said thousends of other users that are not experiencing this problem is not helping in resolving this clear issue in version 22.05 as is clearly stated that even with a clean install i can clearly replicate the issue when upgrading to 22.05 from 22.01. The issue immideatly presents itself. The clients obviously can't resolve the domains hense the client(-s) getting the errors @tentpiglet was describing.

@Kempain @tentpiglet ; have you tried my suggestion for sesolving the issue (although maybe a temporary resolution), as stated i im running 1,5 week without problems at this moment

Mikymike82

btw, i'm not running in a virtualized environment.
Running bare metal on the "same" mircoserver hardware as the SG-7100. Supermicro board with Atom C3758.

Mikymike82

and further; as i am not having the problem any more, i was just replying to help others out.... but.... my solution is not to my satisfaction as this should not be the resolution but more a workaround for a problem on which we do not have a clear cause (yet).

johnpoz

@kvhs said in Slow DNS after 22.05:

preferred test to demonstrate the problem, given its intermittent nature?

How about making sure unbound isn't restarting every 300 seconds for starters.. This is only going cause trouble trying to actually find the issue.

Simple stats output could very enlightening to what might be going on as far as problems.

[22.05-RELEASE][admin@sg4860.local.lan]/root: unbound-control -c /var/unbound/unbound.conf stats_noreset | grep total.
total.num.queries=66013
total.num.queries_ip_ratelimited=0
total.num.cachehits=52460
total.num.cachemiss=13553
total.num.prefetch=27990
total.num.expired=24624
total.num.recursivereplies=13553
total.requestlist.avg=0.319452
total.requestlist.max=30
total.requestlist.overwritten=0
total.requestlist.exceeded=0
total.requestlist.current.all=0
total.requestlist.current.user=0
total.recursion.time.avg=0.086462
total.recursion.time.median=0.0408701
total.tcpusage=0
[22.05-RELEASE][admin@sg4860.local.lan]/root:

You can see average recursion time, median etc..

There are lots of things that might present themselves just looking the stats..

But your saying something is causing NX in your browser - ok do that specific query.. www.facebook.com shouldn't come back NX.. but why did your browser say that? Did you actually look for www.facebook.com or was it something else..

Lets see a dig +trace so we could see if your having a connection issue to something in the resolve path, but again a connection issue wouldn't cause a NX.. A NX is a specific response to what you asked for and some NS saying sorry that does not exist - be it root for the .tld, be a gltd server for the domain, or the authoritative NS for the domain telling you that record does not exist, etc.

I don't buy you were told www.facebook.com was NX.. maybe it was www.facbook.c0m or some other typo, etc. If www.facebook.com came back as NX, lets see the query showing that.. etc..

Its hard to get to the bottom of what is going on when users just say me too, or having a dns problem since went to 22.05 with zero information on what they are doing or trying to do or what the specific failure actually is - like I said for all we know their browser is using doh, or maybe they are trying to route through a vpn, and that vpn is going down, or maybe something they are trying to look up is blocking their vpn connection, etc. etc..

There are loads of things that could be going on..

The only thing I can say for sure - is I have seen zero issues with dns going from 22.01 to 22.05 - zero!! So if someone is having an issue, we need to info to figure it out - it sure is not something specific wrong in unbound that is generic in nature, or then everyone would be seeing the issue and the board would be a flame with posts complaining that dns broke on 22.05.. When clearly that is not the case.

tentpiglet

@johnpoz said in Slow DNS after 22.05:

Maybe you would doing a query for www.facebook.com.org or www.facebook.com.somethingelse.tld

Yeah... no, it wasn't that.

tentpiglet

@mikymike82 can you reiterate what your solution was, I've scrolled back but there's a lot of fluff in here so I can't seem to find it.

Mikymike82

@johnpoz For answering your question (from my experience), its not just "facebook.com", its everything, from apps to normal websites, random not resolving websites and apps not working. So not a specific client, website, browser etc..., mobile, desktop, laptops, narrowcasting etc.. everything thats trying to resolve an adress.
Again my "solution" seems to resolve the problem at hand... but not "normal" behaviour in my opinion.

johnpoz

@tentpiglet well where did it fail, www.facebook.com is a cname

;; ANSWER SECTION:
www.facebook.com.       30      IN      CNAME   star-mini.c10r.facebook.com.
star-mini.c10r.facebook.com. 30 IN      A       157.240.18.35

With a 30 second TTL, etc. where did you go ask after, are you doing qname forced strict? A NX is a specific response from a NS.. Its not a timeout or a servfail - its a specific response saying hey what your asking for doesn't exist..

even a typo of 4 wwww returns an answer not a NX

;; QUESTION SECTION:
;wwww.facebook.com.             IN      A

;; ANSWER SECTION:
wwww.facebook.com.      3600    IN      CNAME   star.facebook.com.
star.facebook.com.      3600    IN      CNAME   star.c10r.facebook.com.
star.c10r.facebook.com. 3600    IN      A       157.240.18.15

If I ask for some gibberish, the I get back NX, from the AUTHORITATIVE SOA for that domain..

;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 33101
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;lsjfdsldjf.facebook.com.       IN      A

;; AUTHORITY SECTION:
facebook.com.           3600    IN      SOA     a.ns.facebook.com. dns.facebook.com. 220198737 14400 1800 604800 300

So to troubleshoot a NX you are getting we need to know the specifics.. Its not a opps unbound is not running currently, or I trying to ask 1.2.3.4 but they are not answering..

Cool_Corona

@johnpoz

total.num.queries=77
total.num.queries_ip_ratelimited=0
total.num.cachehits=22
total.num.cachemiss=55
total.num.prefetch=0
total.num.expired=0
total.num.recursivereplies=55
total.requestlist.avg=0.272727
total.requestlist.max=4
total.requestlist.overwritten=0
total.requestlist.exceeded=0
total.requestlist.current.all=0
total.requestlist.current.user=0
total.recursion.time.avg=0.127636
total.recursion.time.median=0.0505173
total.tcpusage=0

What do you make of this. Unbound has restarted again...

johnpoz

@cool_corona said in Slow DNS after 22.05:

Unbound has restarted again...

yup - its going to be horrible as a caching resolver if it restarts ever few seconds..

Kempain

@mikymike82 said in Slow DNS after 22.05:

@Kempain @tentpiglet ; have you tried my suggestion for sesolving the issue (although maybe a temporary resolution), as stated i im running 1,5 week without problems at this moment

I'm actually not keen on forwarding my DNS unnecessarily but I can understand how it would resolve the issue since it would then bypass unbound.

I was planning on doing a fresh install until I heard you're still experiencing issues after doing that.
Still might to rule it out for me too though.

I've been monitoring unbound and it's been going strong for 13371 seconds now!
Memory usage of unbound doesn't seem to be particularly high although it is increasing slowly which is probably to be expected.

Just set logging to level 5 to try and capture more info although not sure the differences between 4/5 will help as it seems to be related identifying which client is having an issue and I know I'm experiencing it across device types.
This has meant that unbound has been restarted so will see how it goes...

Kempain

unbound-control -c /var/unbound/unbound.conf stats_noreset | grep total
total.num.queries=159
total.num.queries_ip_ratelimited=0
total.num.cachehits=61
total.num.cachemiss=98
total.num.prefetch=0
total.num.expired=0
total.num.recursivereplies=98
total.requestlist.avg=1.11224
total.requestlist.max=31
total.requestlist.overwritten=0
total.requestlist.exceeded=0
total.requestlist.current.all=0
total.requestlist.current.user=0
total.recursion.time.avg=1.462659
total.recursion.time.median=1.03385
total.tcpusage=0

Unbound has only just restarted so take those with a pinch of salt.

Interesting @tentpiglet mentioned 'DNS_PROBE_FINISHED_NXDOMAIN' errors in the browser because I've also been experiencing those at the same time as having DNS issues so I believe they are in some way related.