DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times
-
@RickyBaker did you actually do some dns queries while you were having the issue, to both unbound and say external dns?
You should log queries and replies as well if your wanting to troubleshoot dns not working.
int your options box in unbound
server: log-queries: yes log-replies: yeswhat is the response, timeout talking to unbound, servfail, nx?
fire up your fav dns tool, nslookup, dig, doggo, host, etc. and actual validate what is failing.. If you look some fqdn do you get a response. if so what is the response, did it work, did unbound return servfail, or nx domain ?
Does unbound answer local resources, like pfsense fqdn? Does something that is cached work, only new queries fail. You can view what is in your cache
[23.09.1-RELEASE][admin@sg4860.home.arpa]/root: unbound-control -c /var/unbound/unbound.conf dump_cache | grep forum.netgate.com forum.netgate.com. 2452 IN A 208.123.73.71 msg forum.netgate.com. IN A 32896 1 2452 3 1 1 3 6 forum.netgate.com. IN A 0 [23.09.1-RELEASE][admin@sg4860.home.arpa]/root:If that fails, then do a query directed to some external NS like quad9 or google - do those work?
-
@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
As soon as you find a situation where a device has no access anymore, you have to check :
Does the access without using DNS works ? For example, ping 8.8.8.8 from that device.
Also double check : does the device has a valid IP, gateway and dns set at that moment ?this is really helpful, thank you, i will try to screenshot this post and enact it as well as possible the minute i notice an outage. BTW I tested a hard wired PC when i had an outage and also observed dns connectivity issues fwiw. but all of this is a very good framework for continuing the troubleshooting
-
@RickyBaker how are you knowing the dns is failing? Are you doing an actual query with a tool? like dig or nslookup?
Or your browser just doesn't load - for all you know your browser is using doh..
When you have the issue, can your client ping its gateway (pfsense) can you ping the internet via IP, 8.8.8.8 for example.
If you can not ping pfsense, then you have a local network issue most likely. If you can not ping the internet - maybe just your internet is out. If you can ping pfsense, can you do a query for pfsense name, this should always work even if the internet is down. Only reason it wouldn't is you can't actually talk to pfsense, or unbound is not running.
Doing some basic connectivity tests and dns queries should point to where your actual problem is.
-
@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
did you actually do some dns queries while you were having the issue, to both unbound and say external dns?
at one point when i finally realized i could use dig on the pfsense itself I ran the command you posted to 8.8.8.8 and it worked successfully but I need to test this more thoroughly (i.e. other linux devices not the pfsense) and try 8.8.8.8 as well as google.com. thanks for reminding.
@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
int your options box in unbound
I'm sorry where would i find/set these options set?
@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
what is the response, timeout talking to unbound, servfail, nx?
again, very sorry but how would I do this? I don't even KNOW what servfail, nx is? In fact, reading the rest of the suggestions i can tell this is an important framework for isolating the issue but it's just far beyond my grasp of the tools at play. I will google each individual term in hopes of understanding better but if there's a more specific you could include for me to enact and post that would be very helpful.
-
@RickyBaker Windows has nslookup...run "nslookup netgate.com" and see what it returns. Do the same with "dig netgate.com" on the router, and/or use Diagnostics/DNS Lookup to test there.
Servfail is an error. NXDOMAIN means the host doesn't exist.
At the time of the outage also "ping 8.8.8.8" to ensure your Internet is working, even if DNS is not.
In other words try to narrow down your problem.
re: options, he means the settings in Services/DNS Resolver under "Display Custom Options" which is blank by default.
-
@SteveITS said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
s. Do the same with "dig netgate.com" on the router, and/or use Diagnostics/DNS Lookup to test there.
great, thanks for that explanation
-
@RickyBaker also when your dig, make sure its pointed to pfsense IP with an at.. Linux likes to use 127.0.0.53 which doesn't really tell you who got asked. So put the @ipaddress in your query..
example
user@UC:~$ dig www.netgate.com ; <<>> DiG 9.18.18-0ubuntu0.22.04.2-Ubuntu <<>> www.netgate.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14863 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;www.netgate.com. IN A ;; ANSWER SECTION: www.netgate.com. 30 IN CNAME 1826203.group3.sites.hubspot.net. 1826203.group3.sites.hubspot.net. 30 IN CNAME group3.sites.hscoscdn00.net. group3.sites.hscoscdn00.net. 30 IN A 199.60.103.30 group3.sites.hscoscdn00.net. 30 IN A 199.60.103.226 ;; Query time: 12 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP) ;; WHEN: Mon May 06 15:21:41 CDT 2024 ;; MSG SIZE rcvd: 160 user@UC:~$You don't what ns it actually asked.. so do
user@UC:~$ dig @192.168.2.253 www.netgate.com ; <<>> DiG 9.18.18-0ubuntu0.22.04.2-Ubuntu <<>> @192.168.2.253 www.netgate.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19274 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;www.netgate.com. IN A ;; ANSWER SECTION: www.netgate.com. 3553 IN CNAME 1826203.group3.sites.hubspot.net. 1826203.group3.sites.hubspot.net. 3553 IN CNAME group3.sites.hscoscdn00.net. group3.sites.hscoscdn00.net. 3553 IN A 199.60.103.30 group3.sites.hscoscdn00.net. 3553 IN A 199.60.103.226 ;; Query time: 4 msec ;; SERVER: 192.168.2.253#53(192.168.2.253) (UDP) ;; WHEN: Mon May 06 15:22:28 CDT 2024 ;; MSG SIZE rcvd: 160 user@UC:~$Where 192.168.2.253 is the IP address of pfsense..
You can normally see where 127.0.0.53 is forwarding to with
user@UC:~$ resolvectl status Global Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported resolv.conf mode: stub Link 2 (ens3) Current Scopes: DNS Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported Current DNS Server: 192.168.3.10 DNS Servers: 192.168.3.10 DNS Domain: home.arpa user@UC:~$ -
@johnpoz thanks, this is very helpful, i will race to enact during the next outing and repost here. thanks for your patience.
-
@johnpoz couldn't get to a laptop fast enough yesterday to enact these troubleshooting steps but I was able to fire up a local session on juicessh on my phone:
-
and you stopped testing ...
That laptop moved that fast ??
No more issues right after the ping google test ?You don't need to react 'fast', as the logs, example : unbound /var/log/resolver/log and the /var/log/system = system logs are there for days.
-
@Gertjan i know i know. here's the dns log: https://pastebin.com/FCRuijbe i can't seem to find the right degree of logging. If I set it to 1 I get almost no information, if I set it to 2 or above 2000 lines doesn't go back 2 minutes. I'll be setting at 2 though cause this is useless.
I'll be totally honest, it happened at the worst possible time. I was watching both my kids and making dinner (trying to look up a recipe). the closest computer was floors away and I just didn't have time. I very quickly ran the ping on my phone before i burned the garlic:). I'm "hoping" for a more opportune break down tonight.
-
@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
int your options box in unbound
This look right? -
@RickyBaker said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
If I set it to 1 I get almost no information
Leave it to 1.
This - most recent at the top :
May 6 14:05:33 unbound 41106 [41106:0] info: average recursion processing time 0.809539 sec May 6 14:05:33 unbound 41106 [41106:0] info: server stats for thread 0: requestlist max 62 avg 2.13559 exceeded 0 jostled 0 May 6 14:05:33 unbound 41106 [41106:0] info: server stats for thread 0: 46864 queries, 34916 answers from cache, 11948 recursions, 0 prefetch, 0 rejected by ip ratelimiting May 6 14:05:33 unbound 41106 [41106:0] info: service stopped (unbound 1.17.1). May 6 13:04:22 unbound 41106 [41106:0] info: generate keytag query _ta-4f66. NULL IN May 6 01:40:18 unbound 41106 [41106:0] info: generate keytag query _ta-4f66. NULL IN May 5 14:44:33 unbound 41106 [41106:0] info: generate keytag query _ta-4f66. NULL IN May 5 02:54:05 unbound 41106 [41106:0] info: generate keytag query _ta-4f66. NULL IN May 4 15:34:00 newsyslog 97047 logfile turned over due to size>500K May 4 15:34:00 newsyslog 97047 logfile turned over due to size>500K May 4 15:33:17 unbound 41106 [41106:1] info: generate keytag query _ta-4f66. NULL IN May 4 15:33:17 unbound 41106 [41106:0] info: start of service (unbound 1.17.1). May 4 15:33:17 unbound 41106 [41106:0] notice: init module 1: iteratoris somewhat strange.
Up until May 4 15:34:17 you get hundreds of log line per second. Not an issue, but this will flood the logs. To make logs more useful, and if possible (disk size), make logs files way bigger then just "500K", for example 2Mbytes each.Then there is the line from the system logger at 15h34 that says : time's up, file to big, rotating.
From that moment, no more unbound logs ....
Lines start to re appear the next day with the usual, hourly "info: generate keytag query _ta-4f66.NULL IN" It looks like unbound now logs at "level 1"And the the log continues at May 6. ..... again a rather big gap in the logging.
I'm pretty sure unbound was working for you all this time.
Why it's not logging the "level 1" classic hourly "generate keytag..." is puzzling to me. The log system (syslog) fails ? Something else ?I'm not sure if it was already asked, but :
What's the system you're running pfSense on, bare metal, VM ?
How much RAM ? Disk size ? pfSense 2.7.2, right ? -
@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
Leave it to 1.
done, changed it back, here's today's log on Level 2 just in case there's anything useful in it: https://pastebin.com/x4ryj4AB I had an outage at about 630am.
@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
make logs files way bigger then just "500K", for example 2Mbytes each.
sure, where would i set this option? I didn't see it in System->General Setup or Advanced Settings but that could be user error.
@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
From that moment, no more unbound logs
that's super weird right?
@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
The log system (syslog) fails ? Something else ?
genuinely don't know, how would I find out?
@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
I'm not sure if it was already asked, but :
What's the system you're running pfSense on, bare metal, VM ?
How much RAM ? Disk size ?it's a Dell Poweredge R210 II 1U Server Xeon E3-1240 3.3GHz 8GB 500GB I bought a number of years ago. It is only running pfsense
@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
pfSense 2.7.2, right ?
no 2.7.0. I was unable to update from 2.7.0 to 2.7.2 when you suggested that earlier in the thread. I followed the instructions in this thread i can see you referenced at one point:: https://forum.netgate.com/topic/184670/issue-with-going-from-2-7-0-to-2-7-2 and it def got me closer but I can't seem to complete the upgrade. This is what i always see when i select option 13 when ssh'ed in:
-
@RickyBaker if you can not update to 2.7.2 and you have tried all the common tricks to get past whatever specific issue your running into.. There is always clean install option.
Take a backup of your config, install clean from scratch 2.7.2 an then restore config.. To be honest this shouldn't take much longer than upgrading it from previous version..
I have done this a few times over the years, recently last time was when they started allowing zfs, and then new version changed the layout, etc. and to get the new layout you really needed to just do a clean install.
Reason I am not yet on 24.03 is same reason, need to do clean install.. Because I want to move over to ssd.. And to do that I have to unplug all the cables, open the box install the ssd, etc. etc. And have just not yet gotten around to it. And I know if I just click upgrade, I will put it off until next version of + releases.. And really don't want to do that ;) Maybe next time my wife goes out for a few hours and I am not working will be good time to take the network down ;)
-
@RickyBaker said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
sure, where would i set this option? I didn't see it in System->General Setup or Advanced Settings but that could be user error.
The log file size can be set under Status > System Logs > Settings :
I've made mine 4 times bigger.
You can check the max needed size, and free space left.@RickyBaker said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
ow would I find out?
I'm not sure.
I just had a look at my Resolver log (level 1 set) :
as you can see, only a couple of lines since May 8 ...
Only the DNSSEC related "info: generate keytag query _ta-4f66. NULL IN" shows up every xx hours or so.
So, unbound being silent for longer periods is normal after all. -
How many days are we into this - and you say its a problem for like 10 minutes... And yet to get one actual test of what is happening other than you saying your internet stops working for like 10 minutes.
Other than a ping from your phone to 8.8.8.8, which for all we know used cell data at the time?
And your browser reports NX... for all we know its using doh to resolve and not even asking unbound - do we have some entries in your log of these queries and answers you set, coming from your devices in the logs - you turned on query logging did you not.
edit:
Here I turned on logging, you can see my client asked for www.yahoo.com and got an answer - the no errorThen a I asked for just some gibberish, see the NX response
May 10 09:58:56 unbound 49103 [49103:2] info: 192.168.9.100 www.lsjhfldsjdlfsjdf.com. A IN NXDOMAIN 0.039479 0 126 May 10 09:58:55 unbound 49103 [49103:2] info: 192.168.9.100 www.lsjhfldsjdlfsjdf.com. A IN May 10 09:58:47 unbound 49103 [49103:2] info: 192.168.9.100 www.yahoo.com. A IN NOERROR 0.379134 0 119 May 10 09:58:47 unbound 49103 [49103:2] info: 192.168.9.100 www.yahoo.com. A IN -
@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
Maybe next time my wife goes out for a few hours and I am not working will be good time to take the network down ;)
exactly. I have 2 young kids, if I get the house to myself for an hour, weathering the anxiety of clean installing pfsense isn't the most tantalizing prospect. But i can prioritize if we think it's possible it's causing the issue.
@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
The log file size can be set under Status > System Logs > Settings :
really helpful thank you. i have mirrored your settings. Only the "Disk space currently used by logs" is different (understandably)
@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
So, unbound being silent for longer periods is normal after all.
ok good to know, ty
@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
How many days are we into this
yeah I know, and I'm sorry. in my defense, I'm rarely home and it happens rarely. It's only happened to me when I was home twice since i recieved the latest troubleshooting steps and they were, to save you the unsavory details, very inopportune times. I have screenshotted the posts with instructions and will be carrying around a laptop all weekend in hopes of "catching it" thank you for your patience.
@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
for all we know its using doh to resolve and not even asking unbound - do we have some entries in your log of these queries and answers you set, coming from your devices in the logs - you turned on query logging did you not.
yeah i'm not so sure what DOH is, but I assume those steps you included will get to the bottom of it, and I hope to have those results before the end of the weekend.
Is this sufficient for turning on queries?
@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
Then a I asked for just some gibberish, see the NX response
i see a fair number of NOERROR responses in the log but mostly it seems to be pimarily occupied by my IP cams. I tried to run your test (yahoo.com and slkdjfuiughdil.com) over the vpn but the log doesn't seem to be updating in the GUI. It's currently 11:29 but my log is last showing 11:11 and I just opened yahoo.com. I'll try not over the VPN tonight.
It's been 5 minutes and still no update to the log. I did search for NXDOMAIN and there are 43 instances (of 3000 entries). Here's one:
-
@RickyBaker Those are be expected... That 10.10.10.6 is asking for the ptr (reverse) of 172.17.0.1 - is that an IP on your network, seems like a default sort of docker network to me.
But yeah that would return nx normally because your local dns doesn't have it setup.. But the ptr for say your pfsense IP should answer..
example
; <<>> DiG 9.16.50 <<>> -x 192.168.9.253 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58768 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;253.9.168.192.in-addr.arpa. IN PTR ;; ANSWER SECTION: 253.9.168.192.in-addr.arpa. 3600 IN PTR sg4860.home.arpa. ;; Query time: 9 msec ;; SERVER: 192.168.3.10#53(192.168.3.10) ;; WHEN: Fri May 10 11:43:43 Central Daylight Time 2024 ;; MSG SIZE rcvd: 81Your logs shouldn't be behind.... Maybe you have time off on pfsense? If that was the case you could be having issues with dnssec validation?
Here I did a query for something that was pretty sure would not be asked for often and in cache www.msn.com because don't go there.. The 192.168.3.10 is my pihole IP.. So client via my dig command pihole, which then asked unbound on pfsense to look it up.
You can see the time in the unbound log matches up with the time on my client. 11:48:02
And I can see that query in my pihole as well - at the same time..
If logs are delayed or time is off in them - then yeah you got other things going on.. Do you have log compression setup?
What for sure would like to validate is your browser is even asking unbound for stuff - so if in your browser go to say www.lsjdfldsjdflsjgibberishwhatever.com you should be able to see that get asked for in unbound, and an NX response - like you saw in my previous screenshot.
And normally - by the time you do it in your browser.. And then go open the resolver log in the pfsense gui, that entry should be listed. There sure shouldn't be minutes of delay before that is in your log, fractions of seconds, maybe a second? But if delayed being seen in the log the timestamp should be pretty freaking exact on..
-
@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
@RickyBaker Those are be expected... That 10.10.10.6 is asking for the ptr (reverse) of 172.17.0.1 - is that an IP on your network, seems like a default sort of docker network to me.
i can't say off the top of my head but 10.10.10.6 is my unraid server and that is where the dockers live so seems likely. There are also a lot from wpad.localdomain from the computer i'm connected with over VPN. here's a couple other ones:
10.10.10.12 is a hardwired PC i'm running wireshark on to capture any shenanigans@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
Your logs shouldn't be behind.... Maybe you have time off on pfsense? If that was the case you could be having issues with dnssec validation?
i even ssh'ed into the /var/log and it hasn't been updated since 11:11 but other logs have been:
very odd indeed. My pfsense time is up to date but i did notice that the last log update seems to oddly coincide with the last config change:
@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
If logs are delayed or time is off in them - then yeah you got other things going on.. Do you have log compression setup?
i just removed the compression per @Gertjan helpful suggestions
@johnpoz said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:
And normally - by the time you do it in your browser.. And then go open the resolver log in the pfsense gui, that entry should be listed. There sure shouldn't be minutes of delay before that is in your log, fractions of seconds, maybe a second? But if delayed being seen in the log the timestamp should be pretty freaking exact on..
it's less than perfect that I'm doing this over VPN. I'll run these tests in a couple hours the minute i get home. But i'm not seeing that for some reason.
