DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times

RickyBaker

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

If you set a debug you can get all kinds of info, the response who was asked, etc..

seems bad
When i attempted it on my unraid server the command wasn't found. when i did on pfsense itself and my plex server nslookup just seemed to hang looking for more input.

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

What browser are you using?

chrome but i can't imagine that's better

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

If logs are delayed or time is off in them

My logs were still stuck on 11:11:03 (last config update). i restarted the service and they are updating again.

mine doens't seem to have an answer section like yours
And this I believe is the corresponding failure in the log:

the previous fail in the log was from the browser...

I'm kind of swimming in all the different steps that were needed. Was this helpful? What have I discovered about my devices and their usage of the DNS?

johnpoz

@RickyBaker all those are failing.. You see servfail.. So no its never going to work.

Is 10.10.10.1 your actual IP, or are you pointing to the vip of pfblocker?

In your nslookup debug you never even asked for just www.msn.com - you just asked for www.msn.com.localdomain.

Put a . on the end with your nslookup.. You see how mine did search, with my home.arpa but then it dropped that and did my actual query. Your never did that.

What is asking for HTTPS record vs just A record? You see where you see query from 10.10.10.10 its doing both a A record query and a HTTPS query?

You might want to add these two options.. So easier to see what is query and what is reply.. And prob want to add the servfail option so might get some info to why it failed.

log-tag-queryreply: yes
log-servfail: yes

Add those to what you already have in your options box and save and apply.. This can give you more info..

So your not behind a vpn here, pfsense has no vpn client connection? You need to see in your debug for what your actually asking for www.msn.com.localdomain is never going to resolve.. Unless you had created that record locally.

And didn't we go over that 127.0.0.53, you need to know who exactly that is asking.. If you going to do a dig - do a directed query with the @ipaddress...

RickyBaker

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

So no its never going to work.

but it DOES SOMETIME work! that's why it's so infuriating

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

Is 10.10.10.1 your actual IP, or are you pointing to the vip of pfblocker?

i don't have pfblocker installed 10.10.10.1 is the ip address of my pfsense router.

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

In your nslookup debug you never even asked for just www.msn.com - you just asked for www.msn.com.localdomain.

I def did not intend to ask for www.msn.com.localdomain and I def did not type the words localdomain when I was running the sample you suggested. I merely enacted the samples you suggested as well as pointing my browser at www.msn.com. I'm guessing the https request is a browser feature that forces https, but that's just conjecture

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

So your not behind a vpn here, pfsense has no vpn client connection? You need to see in your debug for what your actually asking for www.msn.com.localdomain is never going to resolve.. Unless you had created that record locally.

I am not behind a VPN here (intentionally at least) and I have not created a record for msn.com locally (intentionally at least).

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

And didn't we go over that 127.0.0.53, you need to know who exactly that is asking.. If you going to do a dig - do a directed query with the @ipaddress...

yes, i knew there was a detail i forgot in that troubleshooting

i'm not 100% sure of the middle one and i have no idea what 127.0.0.53 is. Is there another test i should run to get more color?

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

log-tag-queryreply: yes
log-servfail: yes

johnpoz

@RickyBaker well lets see what happens with logging of servfail detaills. Because clearly its running and resolved your pfsense.localdomain name from from 10.10.10.1 when you did your nslookup.

Another thing I notice on your servfail your not getting the ede back..

You should be able to enable that with ede: yes in your custom box

See here

SteveITS

@RickyBaker

127.0.0.53

Your screenshot shows Ubuntu, that’s the local DNS resolver.
https://unix.stackexchange.com/questions/612416/why-does-etc-resolv-conf-point-at-127-0-0-53

RickyBaker

@SteveITS said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

Your screenshot shows Ubuntu, that’s the local DNS resolver.

does this mean that my plex server isn't using pfsense for dns resolving?

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

well lets see what happens with logging of servfail detaills.

tbc you want me to simply rerun those dig/nslookup sample tests you listed earlier right?

DNSKEY MIssing? also apparently way longer to complete

"unfortunately" i was not experiencing an outage at this time

johnpoz

@RickyBaker there you go - some actual useful info

So your having some sort of issue with dnssec.. I would expect that to fail with that query - that fqdn is test fqdn for making sure dnssec is working.. But we are seeing the servfail reason..

So now when normal queries fail we might get to the bottom of why your getting servfail vs an answer to what you ask for.

johnpoz

@RickyBaker said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

does this mean that my plex server isn't using pfsense for dns resolving?

No what it means is its asking the local cache at 127.0.0.53, your command shows that points to 10.10.10.1

Clearly went over this already like 6 days ago...

Gertjan

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

So your having some sort of issue with dnssec.

When looking up dnssec-failed.org, what would you expect ?

https://www.internetsociety.org/resources/deploy360/2013/dnssec-test-sites/

https://dnsviz.net/d/www.dnssec-failed.org/dnssec/

johnpoz

@Gertjan exactly - like I said ;)

RickyBaker

First, I would like to again apologize for my lack of knowledge. I promise I'm not trying to be difficult or annoying. This is all foreign terminology and concepts to me, but I'm trying my best and can't quantify how much I appreciate the time you're taking

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

So your having some sort of issue with dnssec.. I would expect that to fail with that query - that fqdn is test fqdn for making sure dnssec is working.. But we are seeing the servfail reason..

So now when normal queries fail we might get to the bottom of why your getting servfail vs an answer to what you ask for.

What do you mean by a normal query? How is this NOT a normal query? (ducks:)) What's the next step you'd like to see to further clarify?

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

No what it means is its asking the local cache at 127.0.0.53, your command shows that points to 10.10.10.1

Clearly went over this already like 6 days ago...

ahh that makes sense, sorry I missed that earlier. so does this mean i should be constantly trying new websites i don't ever visit to avoid it falling back to local cache? or is that a fundamental misunderstanding of the steps

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

When looking up dnssec-failed.org, what would you expect ?

thank you for the links. It somehow moved me closer AND farther away from understanding. I have AT&T fiber, why did it attempt a comcast run dnssec fail website. Is going to this website something built into the dig command? Also, correct me if I'm wrong, but I believe y'all had me re-enable DNSSEC just cause it was good practice. I can see how this failing is symptomatic of my greater problems but it's odd to me that whats manifesting itself is something I've been told is really optional and best practice, not required.

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

that fqdn is test fqdn for making sure dnssec is working.. But we are seeing the servfail reason..

All of this leaves me a little lost as to next steps. I keep going back to this line. I know what fqdn stands for, but this collection of words together just doesn't make sense to me, and I believe it's the key to understanding what I need to do next. as always, thanks for everything and further guidance would be greatly appreciated.

SteveITS

dnssec-failed.org

Just for reference I see SERVFAIL for it via Google or others.

>dig dnssec-failed.org @8.8.8.8

; <<>> DiG 9.16.44 <<>> dnssec-failed.org @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 64906
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; EDE: 9 (DNSKEY Missing): (No DNSKEY matches DS RRs of dnssec-failed.org)
;; QUESTION SECTION:
;dnssec-failed.org.             IN      A

;; Query time: 120 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon May 13 10:38:02 Central Daylight Time 2024
;; MSG SIZE  rcvd: 97

https://bluecatnetworks.com/blog/the-top-four-dns-response-codes-and-what-they-mean/
"a SERVFAIL is the DNS server telling you, “Hey, I can’t give you the answer for that query.”"

johnpoz

@SteveITS well yeah forwarding and trying to do dnssec is going to be problematic.. But that dnssec-failed.org should always fail.. It meant to fail.. As a way to validate your dnssec is working..

So yeah if you query any NS that is doing dnssec, google, quad9, etc.. then it would fail.. But if you query some NS that isn't doing dnssec than it would pass..

example

; <<>> DiG 9.16.50 <<>> @8.8.8.8 dnssec-failed.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 3602
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; EDE: 9 (DNSKEY Missing): (No DNSKEY matches DS RRs of dnssec-failed.org)
;; QUESTION SECTION:
;dnssec-failed.org.             IN      A

;; Query time: 95 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon May 13 10:54:20 Central Daylight Time 2024
;; MSG SIZE  rcvd: 97

But if say ask something not doing dnssec..

$ dig @4.2.2.2 dnssec-failed.org

; <<>> DiG 9.16.50 <<>> @4.2.2.2 dnssec-failed.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39041
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 8192
;; QUESTION SECTION:
;dnssec-failed.org.             IN      A

;; ANSWER SECTION:
dnssec-failed.org.      300     IN      A       96.99.227.255

;; Query time: 52 msec
;; SERVER: 4.2.2.2#53(4.2.2.2)
;; WHEN: Mon May 13 10:55:08 Central Daylight Time 2024
;; MSG SIZE  rcvd: 62

This is another example where it makes no sense to check to use dnssec if your forwarding.. Either where you forward is doing dnssec already. Most of the major players do, some have some different IPs you can query that don't.. But pretty much all of them do dnssec. If where you forward does not do dnssec, asking for it in unbound settings isn't going to do anything other then more than likely cause failures..

johnpoz

@RickyBaker said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

All of this leaves me a little lost as to next steps.

The next step is to wait till you fail again.. You were seeing servfail - but we didn't know why or what was the reason for it. Now that you have enabled logging of servfail details.. Next time you have a problem - we can hope to see why.. And then address that..

Also have you updated to 2.7.2 yet? This should be your next step to be honest..

RickyBaker

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

Also have you updated to 2.7.2 yet? This should be your next step to be honest..

no i have not but I can prioritize. i know it SHOULD be easy and smooth but i'm so nervous. especially with it not updating by itself.

@SteveITS said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

https://bluecatnetworks.com/blog/the-top-four-dns-response-codes-and-what-they-mean/

thanks this is a very useful article

RickyBaker

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

The next step is to wait till you fail again..

and what specific commands should i be running? I assume you don't need the resolvectl one, just "dig www.netgate.com" or www.msn.com?

johnpoz

@RickyBaker yeah and looking in the log.. So we can see what it logs for failure if the dig output doesn't show us as much detail on it, etc.

RickyBaker

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

yeah and looking in the log.. So we can see what it logs for failure if the dig output doesn't show us as much detail on it, etc.

great thanks for the clarification

RickyBaker

@RickyBaker spent all afternoon waiting with my computer for an outage, finally went to bed around 1130pm. Happened right away and resolved by the time i sprinted downstairs. stay tuned

RickyBaker

been hunting non-stop but the network has "unfortunately" been very stable this week. This morning my wife said she was experienceng the DNS NX issue on her phone right when we woke up but when i fired up my phone I wasn't experiencing the problem. Went about my morning and a few minutes later, while on the head without a laptop, it happened to me. I fired up my ssh app and ssh'ed into the plex server and got this:

...but then i realized i was doing it on another machine that may not be experiencing the problem. I don't know why I didn't put that together before but the DNS issue USUALLY affects all devices at once but obviously not always. unfortunately my phone's local ssh session doesn't have the dig command. I'll look at installing it to increase my chance of catching it. Unfortunately I forgot that i only have a few minutes to screenshot the logs before they roll off and I missed it. I'm optimistic i'll catch it this weekend.