Slow DNS after 22.05

johnpoz

@provels But how can that be - we clearly all having issues, maybe your having an issue and just don't know it ;) hehehehe

Kempain

Managed to capture some nslookups when DNS bombed out.
It's really tricky to troubleshoot because it happens sporadically and randomly.

Interestingly requests to pfSense timed out twice, then it only returned an IPv6 address without IPv4, then it returned both IPv6 and IPv4. Maybe this adds some credence to the issue being related to IPv6 but who knows. IPv6 was returned first in this particular case so disabling IPv6 would have likely caused more harm.

I still want to check the unbound service to see how often that is restarting which I'll try do later to see if those 2 events line up.

pfSense DNS issue.png

johnpoz

@kempain said in Slow DNS after 22.05:

I still want to check the unbound service to see how often that is restarting which I'll try do later to see if those 2 events line up.

Also simple enough to just see how long its been running..

[22.05-RELEASE][admin@sg4860.local.lan]/root: unbound-control -c /var/unbound/unbound.conf status
version: 1.15.0
verbosity: 1
threads: 4
modules: 2 [ validator iterator ]
uptime: 333105 seconds
options: control(ssl)
unbound (pid 13952) is running...
[22.05-RELEASE][admin@sg4860.local.lan]/root:

So mine has been up 333K seconds, or getting close to 4 days.. Last time I posted it was like 10 days, another 6 days. I tend to restart it when doing something for some thread, where they need an option or something - think last one someone was asking how to block youtube, etc.

But you have no purposely restarted unbound/pfsense and your status shows that its only been up a short amount of time - something is wrong.

Kempain

Cheers @johnpoz

So looks like mine is restarting relatively regularly.

Cool_Corona

Unbound on 2.5.2

johnpoz

@cool_corona @Kempain 300 some seconds unbound being up, doesn't bode well for caching name server ;)

Cool_Corona

@johnpoz How to track all unbound restarts?

tentpiglet

@johnpoz said in Slow DNS after 22.05:

Do you have IPv6?

No. Although I think pfSense assigns a link local address to my wan and lan interface, my ISP does not provide me with a ip6 address.

@gertjan said in Slow DNS after 22.05:

22.05 or 2.6.0 ?

22.05-RELEASE (amd64)
built on Wed Jun 22 18:56:13 UTC 2022
FreeBSD 12.3-STABLE

UNstable means :
You found it 'not running', and no process 'unbound' was present when you list processes ?
Or the process unbound was there, but didn't answer ?
An nslookup from a PC on a LAN failed ?

unstable means I'm using a browser on a client and I get a "DNS_PROBE_FINISHED_NXDOMAIN" error (google chome), and when I hit the reload button a few moments later, the page will be found.

edit : how many times a day unbound restarts ?

grep 'start of service' /var/log/resolver.log | wc -l

64

first entry was at 10:21am local time this morning, so that's over the past 2 hours.

johnpoz

@tentpiglet said in Slow DNS after 22.05:

DNS_PROBE_FINISHED_NXDOMAIN

And what specific were you looking for - NX is not a dns failure, that is what you were looking for doesn't exist.

Also - browser errors are not troubleshooting dns problem - for all we know your browser is using doh and has zero to do with yoru local dns.

tentpiglet

@johnpoz said in Slow DNS after 22.05:

And what specific were you looking for - NX is not a dns failure, that is what you were looking for doesn't exist.

Randomly, any site I would go to. For example, the last time it happened when I said "enough is enough" and spun up a pihole container on my Proxmox server, it was "www.facebook.com".

I'm pretty sure www.facebook.com exists.

My wife was also reporting random disconnects from an online game she was playing, as well as getting similar site not found errors on her phone while reading reddit. Since moving DNS over to the pihole container, she's experienced no further issues.

johnpoz

@tentpiglet said in Slow DNS after 22.05:

I'm pretty sure www.facebook.com exists.

A NX is a dns response that says that domain doesn't exist - why you got that error have no idea, but that is not a failure to talk to dns, that is whatever your dns was doing ended up with NX.. Maybe you would doing a query for www.facebook.com.org or www.facebook.com.somethingelse.tld

Your browser trying to help you auto complete something etc..

I would expect you have all kinds of issues if your local dns is restarting 64 times in less than 24 hours.. So yeah that would be problematic for sure..

kvhs

@johnpoz Like I described earlier I experienced this myself on duckduckgo.com when set as my primary search engine in the browser. Sometimes it would work othertimes not. Reloading a number of times would often work, but that could of course be an illusion if the cause was that some process was restarting.

What would be the preferred test to demonstrate the problem, given its intermittent nature?

Mikymike82

@johnpoz @provels @Gertjan , maybe this is not a widespread problem.... but this seems to be an issue thats more than a single user/unit problem.
Talking albout the so said thousends of other users that are not experiencing this problem is not helping in resolving this clear issue in version 22.05 as is clearly stated that even with a clean install i can clearly replicate the issue when upgrading to 22.05 from 22.01. The issue immideatly presents itself. The clients obviously can't resolve the domains hense the client(-s) getting the errors @tentpiglet was describing.

@Kempain @tentpiglet ; have you tried my suggestion for sesolving the issue (although maybe a temporary resolution), as stated i im running 1,5 week without problems at this moment

Mikymike82

btw, i'm not running in a virtualized environment.
Running bare metal on the "same" mircoserver hardware as the SG-7100. Supermicro board with Atom C3758.

Mikymike82

and further; as i am not having the problem any more, i was just replying to help others out.... but.... my solution is not to my satisfaction as this should not be the resolution but more a workaround for a problem on which we do not have a clear cause (yet).

johnpoz

@kvhs said in Slow DNS after 22.05:

preferred test to demonstrate the problem, given its intermittent nature?

How about making sure unbound isn't restarting every 300 seconds for starters.. This is only going cause trouble trying to actually find the issue.

Simple stats output could very enlightening to what might be going on as far as problems.

[22.05-RELEASE][admin@sg4860.local.lan]/root: unbound-control -c /var/unbound/unbound.conf stats_noreset | grep total.
total.num.queries=66013
total.num.queries_ip_ratelimited=0
total.num.cachehits=52460
total.num.cachemiss=13553
total.num.prefetch=27990
total.num.expired=24624
total.num.recursivereplies=13553
total.requestlist.avg=0.319452
total.requestlist.max=30
total.requestlist.overwritten=0
total.requestlist.exceeded=0
total.requestlist.current.all=0
total.requestlist.current.user=0
total.recursion.time.avg=0.086462
total.recursion.time.median=0.0408701
total.tcpusage=0
[22.05-RELEASE][admin@sg4860.local.lan]/root:

You can see average recursion time, median etc..

There are lots of things that might present themselves just looking the stats..

But your saying something is causing NX in your browser - ok do that specific query.. www.facebook.com shouldn't come back NX.. but why did your browser say that? Did you actually look for www.facebook.com or was it something else..

Lets see a dig +trace so we could see if your having a connection issue to something in the resolve path, but again a connection issue wouldn't cause a NX.. A NX is a specific response to what you asked for and some NS saying sorry that does not exist - be it root for the .tld, be a gltd server for the domain, or the authoritative NS for the domain telling you that record does not exist, etc.

I don't buy you were told www.facebook.com was NX.. maybe it was www.facbook.c0m or some other typo, etc. If www.facebook.com came back as NX, lets see the query showing that.. etc..

Its hard to get to the bottom of what is going on when users just say me too, or having a dns problem since went to 22.05 with zero information on what they are doing or trying to do or what the specific failure actually is - like I said for all we know their browser is using doh, or maybe they are trying to route through a vpn, and that vpn is going down, or maybe something they are trying to look up is blocking their vpn connection, etc. etc..

There are loads of things that could be going on..

The only thing I can say for sure - is I have seen zero issues with dns going from 22.01 to 22.05 - zero!! So if someone is having an issue, we need to info to figure it out - it sure is not something specific wrong in unbound that is generic in nature, or then everyone would be seeing the issue and the board would be a flame with posts complaining that dns broke on 22.05.. When clearly that is not the case.

tentpiglet

@johnpoz said in Slow DNS after 22.05:

Maybe you would doing a query for www.facebook.com.org or www.facebook.com.somethingelse.tld

Yeah... no, it wasn't that.

tentpiglet

@mikymike82 can you reiterate what your solution was, I've scrolled back but there's a lot of fluff in here so I can't seem to find it.

Mikymike82

@johnpoz For answering your question (from my experience), its not just "facebook.com", its everything, from apps to normal websites, random not resolving websites and apps not working. So not a specific client, website, browser etc..., mobile, desktop, laptops, narrowcasting etc.. everything thats trying to resolve an adress.
Again my "solution" seems to resolve the problem at hand... but not "normal" behaviour in my opinion.

johnpoz

@tentpiglet well where did it fail, www.facebook.com is a cname

;; ANSWER SECTION:
www.facebook.com.       30      IN      CNAME   star-mini.c10r.facebook.com.
star-mini.c10r.facebook.com. 30 IN      A       157.240.18.35

With a 30 second TTL, etc. where did you go ask after, are you doing qname forced strict? A NX is a specific response from a NS.. Its not a timeout or a servfail - its a specific response saying hey what your asking for doesn't exist..

even a typo of 4 wwww returns an answer not a NX

;; QUESTION SECTION:
;wwww.facebook.com.             IN      A

;; ANSWER SECTION:
wwww.facebook.com.      3600    IN      CNAME   star.facebook.com.
star.facebook.com.      3600    IN      CNAME   star.c10r.facebook.com.
star.c10r.facebook.com. 3600    IN      A       157.240.18.15

If I ask for some gibberish, the I get back NX, from the AUTHORITATIVE SOA for that domain..

;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 33101
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;lsjfdsldjf.facebook.com.       IN      A

;; AUTHORITY SECTION:
facebook.com.           3600    IN      SOA     a.ns.facebook.com. dns.facebook.com. 220198737 14400 1800 604800 300

So to troubleshoot a NX you are getting we need to know the specifics.. Its not a opps unbound is not running currently, or I trying to ask 1.2.3.4 but they are not answering..