DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times

RickyBaker

My MyQ wifi garage door opener is offline now too, i'm rolling back those changes i guess...

RickyBaker

removed those Custom options, and restarted the resolver and the internet came back for my wife. Maybe i should've attempted to just restart the resolver first?

Current Custom options:

server:
log-queries: yes
log-replies: yes
log-tag-queryreply: yes
log-servfail: yes
ede: yes
#qname-minimisation: no
#aggressive-nsec: no
#infra-keep-probing: yes
#infra-cache-max-rtt: 2000
#infra-host-ttl: 0
#outbound-msg-retry: 32
#max-sent-count: 128

edit; she said it broke a minute later and i restarted the resolver and it seems to be working again...for now. My wifi garage door opener or Nest Protects are still not online

edit2: took a few minutes but my garage door opener is back online as well as 3/5 Nest Protects...

RickyBaker

Sorry for the slightly off topic q, but i, as @johnpoz suggested, feel that updating to 2.7.2 is my next logical step (but I can't do it from the gui or console). So I made a bootable usb stick with 2.7.2 and started the process....but then i forgot the options I selected when I first installed pfsense years ago. Is there a way to check on the gui or console which option i selected for ZFS or UFs? Or which drive I installed it to? I thought my Dell r210 II only had one drive in it but I was presented with 2 during the installation (maybe a partition) and I didn't remember which I did. I just want to make sure my installation is exactly the same as the original to avoid issues.

edit: safe to assume this means zfs on a single drive?

Also I read online there would be an upgrade option when I loaded up a flashable usb with psfense already installed on the target media but this was not a presented option....I also read online that this means the installation media can't read the pfsense install....how much should I be reading into this?

SteveITS

@RickyBaker ZFS was released with 2.6.0 so when did you install yours? Or check the dashboard Disk widget. In general you’d want ZFS.

Re upgrading, one can’t do that like you’re describing I think. However this is a thing: https://docs.netgate.com/pfsense/en/latest/backup/restore-during-install.html#recover-config-xml-from-existing-installation. Or just restore after.

The installer may show the usb stick? Can you tell by size?

Gertjan

@RickyBaker said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

Also I read online there would be an upgrade option when I loaded up a flashable usb with psfense already installed on the target media but this was not a presented option....I also read online that this means the installation media can't read the pfsense install....how much should I be reading into this?

As you can see, "just download pfSense" isn't an option anymore. It's gone. From now on, you load an installer, never used myself that one before.
This must be it : the online installer.

I also upvote the upgrade to 2.7.2.
You'll be using a far newer unbound version, probably "1.19.3."

About the install media : forget about CD, DVD etc.
Use : Prepare a USB Memstick.
If the stick isn't broken, it works. Tried this method several times on Netgate devices an bare bone stuff like your device.

@RickyBaker said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

So I made a bootable usb stick with 2.7.2 and started the process....but then i forgot the options I selected when I first installed pfsense years ago

Ah, ok, you have a 2.7.2 on USB ready.
When installing on a device, known, or not, try to know what de 'FreeBSD' name of the hard disk is. There can be one or more drives. I say drive name, not the partition names on that drive as they will will be lost.
If you can chose, go for the best file system : ZFS.
Enumerate also your NIC names : you can see them in the Interfaces >Interface Assignments menu.
And thats all there is to do.
Go for a manual default bare bone 192.168.1.1/24 LAN and DHCP WAN setup, if you can.
Make a backup of your current config first.
When installing, I never 'recover' the config (if found). I test the system first with the 'everything to default' settings. When that works out : LAN and WAN (and unbound ^^) ok, then you can import your backed up config later on using the GUI.
If any issues come back at that moment : you know it's your config ;)

RickyBaker

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

I test the system first with the 'everything to default' settings.

because of the sporadicness of the issue, it's going to be very difficult to test the default settings. Any suggestions? i don't want to set up all the downstream devices to deal with the new ip address et al.

and slightly off topic/dumb q: is there a way for me to view any and all drives pfsense sees in the GUI or the command line?

Gertjan

@RickyBaker

cat /etc/fstab

RickyBaker

@Gertjan anyway to see the sizes of these? I tried a bunch of ls commands that were all not included in pfsense (and apt wasn't either)

I found camcontrol devlist:

I dunno waht that middle ahci is or the sized of the hard drives but it really would appear i have 2 500gb hdd's in there (the 2 wdc above)...
geom disk list got me sizes:

I genuinely can't remember instaling a second harddrive in there but here we are. would you guys agree that's what it looks like?

RickyBaker

I have no idea what's happening. I booted from the installer USB drive and the installation seemed a little different. It didn't request ZFS or UFS (or the other 3 options this time) and I got through the selection of interfaces and right to the installation selection an got this error when attempting to access the Netgate servers:

When I switched back to booting off the included hard drive (my old configuration, everything came up alright, but I had no internet. and the homepage of pfsense read that it was unable to check for updates:

Is this a crazy coincidence of my ISP going down the exact minute I tried to install the new pfsense or could I possibly have done something to the internet connection while powering down the modem and trying to install the newer version? Soooo frustrating either way!

Edit: I needed to tether to the hotspot of my mobile to write this post, as this post loaded but said there were issues connecting to the negate servers. But I was able load other pages on my phone and laptop. It's almost like negate was alone blocked on my ISP, so weird!

Edit2: There seems to be no issue with internet or connecting to the netgate forum now. Maybe it was the worlds world coincidence

RickyBaker

since the homepage was again displaying that I was "up to date" even though I was on 2.7.0 I decided to reattempt all the suggestions in this thread: https://forum.netgate.com/topic/184670/issue-with-going-from-2-7-0-to-2-7-2/15

And it worked! no scary full reinstall. I'm on 2.7.2 and I will report back. Though after the weird issue with Netgate Servers being unable to be contacted I restarted my AT&T FIber ONT and haven't noticed an outage since (thought it was only a day).....In any event i'll be back here to report any developments

RickyBaker

@SteveITS said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

update: https://forum.netgate.com/topic/187506/kea-dhcp-feature-roadmap/6

So is it safe to switch over?

SteveITS

@RickyBaker if you need just basic dhcp and not the missing features it should be fine. Or just wait until it’s not a preview.

You can also switch back.

RickyBaker

OK just reporting back. It's been over a week since i upgraded to 2.7.2 and I've only had 2 nxdomain (both by my wife). The first was some very janky website that seemed to point to a xxx.xxx.local domain which i remember from earlier posts will always happen. And yesterday to an unknown website (to me) but other websites were opening and the rest of the internet seemed to be working without issue. Since preceding the upgrade I had also reset the AT&T Fiber ONT (and went without issue for about 2 days before the firmware upgrade). I am almost positive I had reset the ONT earlier in the troubleshooting and perhaps wish I'd tested for outages longer between the reset and upgrading the firmware.

BUT it seems that one of the 2 actions has solved my issue. At least until I hit submit on this post here...

edit: I guess i would like to know if it's possible my issue was ISP/ONT?

Gertjan

@RickyBaker said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

edit: I guess i would like to know if it's possible my issue was ISP/ONT?

This ONT thing is attached to your WAN interface, right ?
Does it have a web GUI ? If so, does it have some stats to show you ? Error counter ? Last reconnect moment ?
If your WAN has issues, it would not be "DNS only" but all kind of traffic that would be impacted.

Btw : on already three devices (PCs) I use regularly, I saw that my browser (Firefox) changed recently its DNS settings :

I alsway have set this to : "Off" which means firefox uses the systems (Micrsoft OS) DNS settings = pfSense.
But no, I found the settings were 'reset' back to "default protection" which probably means it does DNS over TLS to some obscure DNS server, bypassing pfSense ... but hitting pfBlockerng's "DNS over HTTPS/TLS/QUIC Blocking".

Exactly this : Home > pfSense Software > DHCP and DNS : the top most forum post : HEADS UP: Be aware of Trusted Recursive Resolver (TRR) in Firefox

RickyBaker

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

This ONT thing is attached to your WAN interface, right ?
Does it have a web GUI ? If so, does it have some stats to show you ? Error counter ? Last reconnect moment ?

yeah it's connected to the wan. it def has a GUI but I think i'd have to unplug it to access it. I had a reverse proxy set up for my comcast router but i'm not sure i ported it over when i switched ISP's.

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

If your WAN has issues, it would not be "DNS only" but all kind of traffic that would be impacted.

this was always my thought and why i'm assuming it's the firmware upgrade that did it and that I was just lucky for a day or 2 before that.

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

But no, I found the settings were 'reset' back to "default protection" which probably means it does DNS over TLS to some obscure DNS server, bypassing pfSense ... but hitting pfBlockerng's "DNS over HTTPS/TLS/QUIC Blocking".

that's so annoying, thanks for the heads up

RickyBaker

Ugh, albeit infrequent, the issue has occurred a couple times over the last couple days. Yesterday in bed my wife complained about it happening in the eBay app (and Chrome and the public libraries audiobook app) and I feverishly started opening random websites from my Google feed (swipe left on an android). I eventually got an nxdomain error but was able to open 5-6 links before AND after the error. It's very odd to me that my wife seems to experience the problem with much more consistency and degree than I do. Resetting the DNS Resolver restored usefulness immediately.

johnpoz

@RickyBaker exceeded max number of sends is the root of the problem it seems.

You might want to bump your logging level up to 5 in unbound.. This might give you more insight to the actual failure when you see the servfail and its reason.

I would prob set

do-ip6: no

As well..

Are you seeing a crazy amount of queries? Is something bombing unbound with queries before those failures?

bmeeks

Quite a long thread here on the unbound GitHub account that I think might be related to the OP's issue: https://github.com/NLnetLabs/unbound/issues/362.

From my reading of the long thread, unbound itself is producing the SERVFAIL messages at times inappropriately as a byproduct of trying to "protect" the client from abuse. The thread is primarily an argument about DNSBL anti-spam logic and resultant DNS lookups, but I think the underlying logic inside unbound itself may be part (or maybe even all) of the issue the OP is seeing randomly.

There are also a couple of suggested unbound config changes in that thread that might help mitigate the issue. Of note is the parameter to greatly increase the "maximum number of sends" limit.

For what it's worth, I found reports of this same behavior on "the other Sense" product. That leads me to strongly suspect the root cause is something in unbound that perhaps specific configurations or even specific queries triggers the error.

johnpoz

@bmeeks I would concur.. From the quick look into that error, also got a sense that ipv6 failing or issues with IPv6 could bring it on.. Which is why I would suggest telling unbound not to do IPv6, it sure isn't going to break anything that is for sure.

I have never ran into this issue, but then again I have the ipv6 set to no.. So maybe that is why I haven't seen it???

Something triggers it for sure.. What that something is the question ;)

I don't see how increasing the number of sends could cause any sort of neg issues.. Increasing that again don't see how it could hurt anything or make anything worse. I don't have it set to anything other than default.. But then again my clients don't actually ask pfsense, they ask a pihole in front of pfsense.. So maybe a massive amount of queries that trigger the max send is being held back by pihole. I have seen warnings where clients have been throttled in pihole because they have done more than 1000 queries in 60 seconds..

Sometimes blocking something from resolving can cause clients to go into panick mode or something - and whatever idiot logic the guy that wrote it thought that hey didn't get an answer, lets ask 1000 times in a minute and we might get an answer?? hehhehehe

Gertjan

@bmeeks said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

Quite a long thread here on the unbound GitHub account that I think might be related to the OP's issue: https://github.com/NLnetLabs/unbound/issues/362.

Mentioned that thread @NETlabs a month ago, some propositions were tested already AFAIK.

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

This very issue (or whatever it is) has its own thread on NLnetLabs (the author of unbound) exceeded the maximum nameserver nxdomains.

exceeded the maximum nameserver nxdomains = https://github.com/NLnetLabs/unbound/issues/362