DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times

RickyBaker

been hunting non-stop but the network has "unfortunately" been very stable this week. This morning my wife said she was experienceng the DNS NX issue on her phone right when we woke up but when i fired up my phone I wasn't experiencing the problem. Went about my morning and a few minutes later, while on the head without a laptop, it happened to me. I fired up my ssh app and ssh'ed into the plex server and got this:

...but then i realized i was doing it on another machine that may not be experiencing the problem. I don't know why I didn't put that together before but the DNS issue USUALLY affects all devices at once but obviously not always. unfortunately my phone's local ssh session doesn't have the dig command. I'll look at installing it to increase my chance of catching it. Unfortunately I forgot that i only have a few minutes to screenshot the logs before they roll off and I missed it. I'm optimistic i'll catch it this weekend.

RickyBaker

@RickyBaker Still hunting, frustratingly the problem has def gotten less frequent an shorter in duration (but still ever present, my wife agrees, i'm not crazy). It's also happening more on individual devices where other devices work fine more often than before. It happened on my PC and when I ran the dig command on my plex debian box it was fine

In the log though I did find this around when I tried the dig command:

I also found this which looks shady to me:

Since it seems to be singular devices at a time now i'm slowing figuring out how to run dig commands on all the different OS's in my house. I have Android and linux and am following a tutorial for Windows now...

RickyBaker

@RickyBaker said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

following a tutorial for Windows now...
So i was trying to follow this tutorial but when I went to install it said it was already installed (I used the legacy windows version). I had tried to install it before but then typing the dig command still returned a "command not found". The installer suggested i remove the old one from add/remove programs but I couldn't find anything under BIND or ISC and the last installed program was discord a LONG time ago.

I tried to continue with the tutorial but it asked where BIND was installed to add it to the PATH (which I'm sure was my problem the first time around) but I don't know where it's installed and a windows search for BIND or ISC is expectedly noisy. any suggestions?

I'll keep plugging at it but it's an annoying speed bump that's really slowing down the troublshooting...

RickyBaker

Finally got one!!!

I pasted everything in the log back a few minutes here in case the totality of it is useful

https://pastebin.com/w2SGh8P0

@johnpoz Sorry for the delay in getting this I swear i was trying the whole time. thanks for the patience.

RickyBaker

@johnpoz got another one! though it does seem to be happening with a lot less frequency for some reason, i've just gotten better at catching them during the quick window of opportunity:

https://pastebin.com/WjyAukgm

RickyBaker

This one got a NXDOMAIN error:

SteveITS

@RickyBaker said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

This one got a NXDOMAIN error:

That is a URL not a hostname so it should fail. Remove the /apps/staff (as shown in the prior post).

Searching for "exceeded the maximum number of sends" looks like DNSSEC...:
https://community.ipfire.org/t/servfail-exceeded-the-maximum-number-of-sends/7645
https://www.reddit.com/r/pihole/comments/11hqrco/intermittent_servfail_when_using_unbound/

this one talks about not using UDP for DNS...?!
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=270824

This one talks about torrenting and DNSSEC:
https://www.reddit.com/r/opnsense/comments/1cinuyn/unbound_dns_issues_freezes_randomly/

Gertjan

@RickyBaker

About :

This very issue (or whatever it is) has its own thread on NLnetLabs (the author of unbound) exceeded the maximum nameserver nxdomains.
One of the authors of unbound is also answering.
Some tips are present.

Btw : this is DNS at its finest. I'll take this one home tonight, need to read it again.

Latest posts in that thread are just hours ago.

Here you go :

server:
	qname-minimisation: no
	aggressive-nsec: no

	infra-keep-probing: yes
	infra-cache-max-rtt: 2000
	infra-host-ttl: 0

	outbound-msg-retry: 32
	max-sent-count: 128

dono what the impact will be ....
I've never seen this "exceeded the maximum nameserver nxdomains" message myself.

RickyBaker

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

Here you go :

is the suggestion to throw this into the custom options section of the dns resolver? I'll check out all the links, was just looking to confirm the suggestion you had forwarded on...

Gertjan

@RickyBaker

Exact.
Like this :

RickyBaker

@Gertjan awesome, thanks for clarification. It's been added. I'll read up on all these threads while I wait for it to fail...

SteveITS

Just to link that other thread, which two of us linked above, to this one:
https://forum.netgate.com/topic/188297/sporadic-dns-issues-cryptic-error-in-logs/

RickyBaker

@SteveITS said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

this one talks about not using UDP for DNS...?!
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=270824

reading through these now, this one caught my eye because one of the only packages i have installed is UDP Broadcast Relay (in order to forward across the VLAN's I set up). Though i'll be honest, I really don't know much about UDP/TCP and not sure if this is the same ballpark as the Broadcast Relay. I can try the tcp-upstream: yes option after I feel confident the last iteration of changes didn't solve the issue.

RickyBaker

@SteveITS said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

https://www.reddit.com/r/opnsense/comments/1cinuyn/unbound_dns_issues_freezes_randomly/

This is another interesting theory, but I searched my log and I don't see anything referencing a tracker. Though I did just notice my enphase solar controller also just got a bunch of servfails...

RickyBaker

@SteveITS said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

https://community.ipfire.org/t/servfail-exceeded-the-maximum-number-of-sends/7645

just to dovetail all my thoughts on reading these links: this one seems very promising but doesn't have the total solution included. Though it also claims that DNSSEC was the culprit and I'm quite certain I was still experiencing the issue with DNSSEC disabled....

RickyBaker

Just to stay on top of things: I've yet to experience an outage like i'm used to (DNS_PROBE_FINISHED_NXDOMAIN). I was out of the house so its atypical but not unheard of (the length of time that is). However last night I tried to open an ebay link and the app wasn't able to bring up the item. It was an odd error in the app, but the really odd thing was that I was able to load webpages in chrome. Just documenting my journey, i'll be sure to grab any NXDOMAIN issues i catch in the coming days...

RickyBaker

@Gertjan just noticed an interesting thing: all 5 of my Nest Protects (Google Fire Alarms) are reporting not being connected to the internet (WiFi issue). It was last checked about 18 hours ago (a few hours after I made the changes, so maybe it checked a few times before it stopped retrying). The History showed that they have been connected and without error for as long as the History goes back so seems likely connected to these specific changes....But also no outages yet besides that weird ebay one that was not the same as the usual.

RickyBaker

@RickyBaker if I'm not mistaken this is WAY more servfail's that i was previously experiencing:

RickyBaker

@RickyBaker 3 instances of squeakydoor.nest.com servfail and 543 instances (probably a third as many fails) of time.google.com or some derivation of. I was not seeing this before the latest round of changes...

SteveITS

@RickyBaker The images above were lost due to the forum error...do you have forwarding enabled? This: