DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times

RickyBaker · May 4, 2024, 8:41 PM

https://pastebin.com/SFR8BXb0

Woke up from a nap and experienced one of the longest internet outages of this whole saga. It was out at 3:14 when I tried to open venmo and was out for over 20 minutes before it came back. the above is the DNS resolver log but I think i have the log level dialed too high cause 2000 entries didn't even go back 2 minutes. I've changed it back to Log Level 1 but could someone check it out and see if there's any clues in there (or what log level I should have it at)? Or is there another log that I should also be monitoring? Is it possible the problem is purely something with the wifi and Ubiquiti?

Gertjan · May 6, 2024, 6:39 AM

@RickyBaker

Same thoughts here : a high level of log details actually the details your looking for, as there is only for 2 minutes worth of info.
If you have some disk space left, you can make the log files bigger.

login-to-view

If needed, you can make the log retention a bit smaller - I've "7", you can make it 5 or 4.

You can also make this, one

login-to-view

a bit bigger.

The actual goal is :
As soon as you find a situation where a device has no access anymore, you have to check :
Does the access without using DNS works ? For example, ping 8.8.8.8 from that device.
Also double check : does the device has a valid IP, gateway and dns set at that moment ?
Example :

ipconfig /all

and check the duration of the lease, the gateway, the DNS (both should point to the IP of pfSense.

Check on the device if "DNS" works :

C:\Users\Gauche>nslookup www.google.com
Serveur :   pfSense.bhf.tld
Address:  2a01:cb19:907:bedf:92ec:77ff:fe29:392c

Réponse ne faisant pas autorité :
Nom :    www.google.com
Addresses:  2a00:1450:4007:81a::2004
          142.250.201.164

Take note : for me, both IPv6 and IPv4 works.

Then (also) check on pfSense if resolving works :

dig @127.0.0.1 www.google.com +short

and then

dig @192.168.1.1 www.google.com +short

where 192.168.1.1 is your LAN interface.

Check if unbound is up and running :

[24.03-RELEASE][root@pfSense.bhf.tld]/root: ps ax | grep 'unbound'
74113  -  Ss       4:32.60 /usr/local/sbin/unbound -c /var/unbound/unbound.conf
....
....

and

[24.03-RELEASE][root@pfSense.bhf.tld]/root: sockstat | grep 'unbound'
unbound  unbound    74113 3   udp6   *:53                  *:*
unbound  unbound    74113 4   tcp6   *:53                  *:*
unbound  unbound    74113 5   udp4   *:53                  *:*
unbound  unbound    74113 6   tcp4   *:53                  *:*
unbound  unbound    74113 9   tcp4   127.0.0.1:953         *:*
...
...
...
...

With the unbound log details set to "1", it will still contains the number of restarts (a controlled stop and then a start :

grep "stopped" /var/log/resolver.log
.....

<30>1 2024-05-06T00:15:24.852356+02:00 pfSense.bhf.tld unbound 12814 - - [12814:0] info: service stopped (unbound 1.19.3).

Btw : the actual unbound version is 1.19.3 as I'm using 24.03.
pfSense 2.8.0 will be coming out soon.
Not that the version really matters (imho) as I was using 1.17.x also a long time, and don't recall having any issues.

@RickyBaker said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

Is it possible the problem is purely something with the wifi and Ubiquiti?

For me, an AP should be what it should do :; being a radio to wire signal converter.
True, an AP can do a lot more, and really braking the connection for you.
When testing connectivity issues, add APs and other gadgets later on, when you know the wired connection works well.
The same thing goes for L3 'smart, VLAN based' switches : only use the when the bare bone network works well.

johnpoz · May 6, 2024, 8:34 AM

@RickyBaker did you actually do some dns queries while you were having the issue, to both unbound and say external dns?

You should log queries and replies as well if your wanting to troubleshoot dns not working.

int your options box in unbound

server:
log-queries: yes
log-replies: yes

what is the response, timeout talking to unbound, servfail, nx?

fire up your fav dns tool, nslookup, dig, doggo, host, etc. and actual validate what is failing.. If you look some fqdn do you get a response. if so what is the response, did it work, did unbound return servfail, or nx domain ?

Does unbound answer local resources, like pfsense fqdn? Does something that is cached work, only new queries fail. You can view what is in your cache

[23.09.1-RELEASE][admin@sg4860.home.arpa]/root: unbound-control -c /var/unbound/unbound.conf dump_cache | grep forum.netgate.com
forum.netgate.com.      2452    IN      A       208.123.73.71
msg forum.netgate.com. IN A 32896 1 2452 3 1 1 3 6 
forum.netgate.com. IN A 0
[23.09.1-RELEASE][admin@sg4860.home.arpa]/root:

If that fails, then do a query directed to some external NS like quad9 or google - do those work?

RickyBaker · May 6, 2024, 5:35 PM

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

As soon as you find a situation where a device has no access anymore, you have to check :
Does the access without using DNS works ? For example, ping 8.8.8.8 from that device.
Also double check : does the device has a valid IP, gateway and dns set at that moment ?

this is really helpful, thank you, i will try to screenshot this post and enact it as well as possible the minute i notice an outage. BTW I tested a hard wired PC when i had an outage and also observed dns connectivity issues fwiw. but all of this is a very good framework for continuing the troubleshooting

johnpoz · May 6, 2024, 5:45 PM

@RickyBaker how are you knowing the dns is failing? Are you doing an actual query with a tool? like dig or nslookup?

Or your browser just doesn't load - for all you know your browser is using doh..

When you have the issue, can your client ping its gateway (pfsense) can you ping the internet via IP, 8.8.8.8 for example.

If you can not ping pfsense, then you have a local network issue most likely. If you can not ping the internet - maybe just your internet is out. If you can ping pfsense, can you do a query for pfsense name, this should always work even if the internet is down. Only reason it wouldn't is you can't actually talk to pfsense, or unbound is not running.

Doing some basic connectivity tests and dns queries should point to where your actual problem is.

RickyBaker · May 6, 2024, 7:27 PM

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

did you actually do some dns queries while you were having the issue, to both unbound and say external dns?

at one point when i finally realized i could use dig on the pfsense itself I ran the command you posted to 8.8.8.8 and it worked successfully but I need to test this more thoroughly (i.e. other linux devices not the pfsense) and try 8.8.8.8 as well as google.com. thanks for reminding.

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

int your options box in unbound

I'm sorry where would i find/set these options set?

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

what is the response, timeout talking to unbound, servfail, nx?

again, very sorry but how would I do this? I don't even KNOW what servfail, nx is? In fact, reading the rest of the suggestions i can tell this is an important framework for isolating the issue but it's just far beyond my grasp of the tools at play. I will google each individual term in hopes of understanding better but if there's a more specific you could include for me to enact and post that would be very helpful.

SteveITS · May 6, 2024, 7:37 PM

@RickyBaker Windows has nslookup...run "nslookup netgate.com" and see what it returns. Do the same with "dig netgate.com" on the router, and/or use Diagnostics/DNS Lookup to test there.

Servfail is an error. NXDOMAIN means the host doesn't exist.

At the time of the outage also "ping 8.8.8.8" to ensure your Internet is working, even if DNS is not.

In other words try to narrow down your problem.

re: options, he means the settings in Services/DNS Resolver under "Display Custom Options" which is blank by default.

RickyBaker · May 6, 2024, 8:15 PM

@SteveITS said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

s. Do the same with "dig netgate.com" on the router, and/or use Diagnostics/DNS Lookup to test there.

great, thanks for that explanation

johnpoz · May 6, 2024, 8:25 PM

@RickyBaker also when your dig, make sure its pointed to pfsense IP with an at.. Linux likes to use 127.0.0.53 which doesn't really tell you who got asked. So put the @ipaddress in your query..

example

user@UC:~$ dig www.netgate.com

; <<>> DiG 9.18.18-0ubuntu0.22.04.2-Ubuntu <<>> www.netgate.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14863
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;www.netgate.com.               IN      A

;; ANSWER SECTION:
www.netgate.com.        30      IN      CNAME   1826203.group3.sites.hubspot.net.
1826203.group3.sites.hubspot.net. 30 IN CNAME   group3.sites.hscoscdn00.net.
group3.sites.hscoscdn00.net. 30 IN      A       199.60.103.30
group3.sites.hscoscdn00.net. 30 IN      A       199.60.103.226

;; Query time: 12 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Mon May 06 15:21:41 CDT 2024
;; MSG SIZE  rcvd: 160

user@UC:~$

You don't what ns it actually asked.. so do

user@UC:~$ dig @192.168.2.253 www.netgate.com

; <<>> DiG 9.18.18-0ubuntu0.22.04.2-Ubuntu <<>> @192.168.2.253 www.netgate.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19274
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.netgate.com.               IN      A

;; ANSWER SECTION:
www.netgate.com.        3553    IN      CNAME   1826203.group3.sites.hubspot.net.
1826203.group3.sites.hubspot.net. 3553 IN CNAME group3.sites.hscoscdn00.net.
group3.sites.hscoscdn00.net. 3553 IN    A       199.60.103.30
group3.sites.hscoscdn00.net. 3553 IN    A       199.60.103.226

;; Query time: 4 msec
;; SERVER: 192.168.2.253#53(192.168.2.253) (UDP)
;; WHEN: Mon May 06 15:22:28 CDT 2024
;; MSG SIZE  rcvd: 160

user@UC:~$

Where 192.168.2.253 is the IP address of pfsense..

You can normally see where 127.0.0.53 is forwarding to with

user@UC:~$ resolvectl status
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (ens3)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.3.10
       DNS Servers: 192.168.3.10
        DNS Domain: home.arpa
user@UC:~$

RickyBaker · May 7, 2024, 1:36 PM

@johnpoz thanks, this is very helpful, i will race to enact during the next outing and repost here. thanks for your patience.

RickyBaker · May 8, 2024, 3:52 PM

@johnpoz couldn't get to a laptop fast enough yesterday to enact these troubleshooting steps but I was able to fire up a local session on juicessh on my phone:

Gertjan · May 8, 2024, 4:03 PM

@RickyBaker

and you stopped testing ...
That laptop moved that fast ??
No more issues right after the ping google test ?

You don't need to react 'fast', as the logs, example : unbound /var/log/resolver/log and the /var/log/system = system logs are there for days.

RickyBaker · May 8, 2024, 6:16 PM

@Gertjan i know i know. here's the dns log: https://pastebin.com/FCRuijbe i can't seem to find the right degree of logging. If I set it to 1 I get almost no information, if I set it to 2 or above 2000 lines doesn't go back 2 minutes. I'll be setting at 2 though cause this is useless.

I'll be totally honest, it happened at the worst possible time. I was watching both my kids and making dinner (trying to look up a recipe). the closest computer was floors away and I just didn't have time. I very quickly ran the ping on my phone before i burned the garlic:). I'm "hoping" for a more opportune break down tonight.

RickyBaker · May 8, 2024, 6:18 PM

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

int your options box in unbound

login-to-view
This look right?

Gertjan · May 9, 2024, 6:54 AM

@RickyBaker said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

If I set it to 1 I get almost no information

Leave it to 1.

This - most recent at the top :

May 6 14:05:33	unbound	41106	[41106:0] info: average recursion processing time 0.809539 sec
May 6 14:05:33	unbound	41106	[41106:0] info: server stats for thread 0: requestlist max 62 avg 2.13559 exceeded 0 jostled 0
May 6 14:05:33	unbound	41106	[41106:0] info: server stats for thread 0: 46864 queries, 34916 answers from cache, 11948 recursions, 0 prefetch, 0 rejected by ip ratelimiting
May 6 14:05:33	unbound	41106	[41106:0] info: service stopped (unbound 1.17.1).
May 6 13:04:22	unbound	41106	[41106:0] info: generate keytag query _ta-4f66. NULL IN
May 6 01:40:18	unbound	41106	[41106:0] info: generate keytag query _ta-4f66. NULL IN
May 5 14:44:33	unbound	41106	[41106:0] info: generate keytag query _ta-4f66. NULL IN
May 5 02:54:05	unbound	41106	[41106:0] info: generate keytag query _ta-4f66. NULL IN
May 4 15:34:00	newsyslog	97047	logfile turned over due to size>500K
May 4 15:34:00	newsyslog	97047	logfile turned over due to size>500K
May 4 15:33:17	unbound	41106	[41106:1] info: generate keytag query _ta-4f66. NULL IN
May 4 15:33:17	unbound	41106	[41106:0] info: start of service (unbound 1.17.1).
May 4 15:33:17	unbound	41106	[41106:0] notice: init module 1: iterator

is somewhat strange.
Up until May 4 15:34:17 you get hundreds of log line per second. Not an issue, but this will flood the logs. To make logs more useful, and if possible (disk size), make logs files way bigger then just "500K", for example 2Mbytes each.

Then there is the line from the system logger at 15h34 that says : time's up, file to big, rotating.
From that moment, no more unbound logs ....
Lines start to re appear the next day with the usual, hourly "info: generate keytag query _ta-4f66.NULL IN" It looks like unbound now logs at "level 1"

And the the log continues at May 6. ..... again a rather big gap in the logging.

I'm pretty sure unbound was working for you all this time.
Why it's not logging the "level 1" classic hourly "generate keytag..." is puzzling to me. The log system (syslog) fails ? Something else ?

I'm not sure if it was already asked, but :
What's the system you're running pfSense on, bare metal, VM ?
How much RAM ? Disk size ? pfSense 2.7.2, right ?

RickyBaker · May 9, 2024, 8:00 PM

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

Leave it to 1.

done, changed it back, here's today's log on Level 2 just in case there's anything useful in it: https://pastebin.com/x4ryj4AB I had an outage at about 630am.

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

make logs files way bigger then just "500K", for example 2Mbytes each.

sure, where would i set this option? I didn't see it in System->General Setup or Advanced Settings but that could be user error.

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

From that moment, no more unbound logs

that's super weird right?

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

The log system (syslog) fails ? Something else ?

genuinely don't know, how would I find out?

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

I'm not sure if it was already asked, but :
What's the system you're running pfSense on, bare metal, VM ?
How much RAM ? Disk size ?

it's a Dell Poweredge R210 II 1U Server Xeon E3-1240 3.3GHz 8GB 500GB I bought a number of years ago. It is only running pfsense

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

pfSense 2.7.2, right ?

no 2.7.0. I was unable to update from 2.7.0 to 2.7.2 when you suggested that earlier in the thread. I followed the instructions in this thread i can see you referenced at one point:: https://forum.netgate.com/topic/184670/issue-with-going-from-2-7-0-to-2-7-2 and it def got me closer but I can't seem to complete the upgrade. This is what i always see when i select option 13 when ssh'ed in:
login-to-view

johnpoz · May 9, 2024, 8:44 PM

@RickyBaker if you can not update to 2.7.2 and you have tried all the common tricks to get past whatever specific issue your running into.. There is always clean install option.

Take a backup of your config, install clean from scratch 2.7.2 an then restore config.. To be honest this shouldn't take much longer than upgrading it from previous version..

I have done this a few times over the years, recently last time was when they started allowing zfs, and then new version changed the layout, etc. and to get the new layout you really needed to just do a clean install.

Reason I am not yet on 24.03 is same reason, need to do clean install.. Because I want to move over to ssd.. And to do that I have to unplug all the cables, open the box install the ssd, etc. etc. And have just not yet gotten around to it. And I know if I just click upgrade, I will put it off until next version of + releases.. And really don't want to do that ;) Maybe next time my wife goes out for a few hours and I am not working will be good time to take the network down ;)

Gertjan · May 10, 2024, 6:08 AM

@RickyBaker said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

sure, where would i set this option? I didn't see it in System->General Setup or Advanced Settings but that could be user error.

The log file size can be set under Status > System Logs > Settings :

login-to-view

I've made mine 4 times bigger.
You can check the max needed size, and free space left.

@RickyBaker said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

ow would I find out?

I'm not sure.
I just had a look at my Resolver log (level 1 set) :

login-to-view

as you can see, only a couple of lines since May 8 ...

Only the DNSSEC related "info: generate keytag query _ta-4f66. NULL IN" shows up every xx hours or so.
So, unbound being silent for longer periods is normal after all.

johnpoz · May 10, 2024, 3:02 PM

How many days are we into this - and you say its a problem for like 10 minutes... And yet to get one actual test of what is happening other than you saying your internet stops working for like 10 minutes.

Other than a ping from your phone to 8.8.8.8, which for all we know used cell data at the time?

And your browser reports NX... for all we know its using doh to resolve and not even asking unbound - do we have some entries in your log of these queries and answers you set, coming from your devices in the logs - you turned on query logging did you not.

edit:
Here I turned on logging, you can see my client asked for www.yahoo.com and got an answer - the no error

Then a I asked for just some gibberish, see the NX response

May 10 09:58:56 	unbound 	49103 	[49103:2] info: 192.168.9.100 www.lsjhfldsjdlfsjdf.com. A IN NXDOMAIN 0.039479 0 126
May 10 09:58:55 	unbound 	49103 	[49103:2] info: 192.168.9.100 www.lsjhfldsjdlfsjdf.com. A IN
May 10 09:58:47 	unbound 	49103 	[49103:2] info: 192.168.9.100 www.yahoo.com. A IN NOERROR 0.379134 0 119
May 10 09:58:47 	unbound 	49103 	[49103:2] info: 192.168.9.100 www.yahoo.com. A IN

RickyBaker · May 10, 2024, 4:33 PM

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

Maybe next time my wife goes out for a few hours and I am not working will be good time to take the network down ;)

exactly. I have 2 young kids, if I get the house to myself for an hour, weathering the anxiety of clean installing pfsense isn't the most tantalizing prospect. But i can prioritize if we think it's possible it's causing the issue.

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

The log file size can be set under Status > System Logs > Settings :

really helpful thank you. i have mirrored your settings. Only the "Disk space currently used by logs" is different (understandably)

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

So, unbound being silent for longer periods is normal after all.

ok good to know, ty

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

How many days are we into this

yeah I know, and I'm sorry. in my defense, I'm rarely home and it happens rarely. It's only happened to me when I was home twice since i recieved the latest troubleshooting steps and they were, to save you the unsavory details, very inopportune times. I have screenshotted the posts with instructions and will be carrying around a laptop all weekend in hopes of "catching it" thank you for your patience.

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

for all we know its using doh to resolve and not even asking unbound - do we have some entries in your log of these queries and answers you set, coming from your devices in the logs - you turned on query logging did you not.

yeah i'm not so sure what DOH is, but I assume those steps you included will get to the bottom of it, and I hope to have those results before the end of the weekend.
Is this sufficient for turning on queries? login-to-view

@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

Then a I asked for just some gibberish, see the NX response

i see a fair number of NOERROR responses in the log but mostly it seems to be pimarily occupied by my IP cams. I tried to run your test (yahoo.com and slkdjfuiughdil.com) over the vpn but the log doesn't seem to be updating in the GUI. It's currently 11:29 but my log is last showing 11:11 and I just opened yahoo.com. I'll try not over the VPN tonight. login-to-view
It's been 5 minutes and still no update to the log. I did search for NXDOMAIN and there are 43 instances (of 3000 entries). Here's one:
login-to-view