DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times

[RickyBaker]

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

and now your looking at the resolver log file, in real time.

I'm doing this, but not sure what to be looking for. it's the same as the gui but faster moving. Lots of servfail and ".localdomain" and "exceeded maximum sends"

[RickyBaker]

I'm losing my mind:
login-to-view
login-to-view

My wife is going to kill me, she can't use her computer. She asked if we should call our ISP shudder but i don't think it's an issue with the ISP, right? Once again I go back to how sudden this problem came about without changes and how prevalent it has become....

Is there any indication the pfsense hardware is failing? Anything?

[RickyBaker]

My MyQ wifi garage door opener is offline now too, i'm rolling back those changes i guess...

[RickyBaker]

removed those Custom options, and restarted the resolver and the internet came back for my wife. Maybe i should've attempted to just restart the resolver first?

Current Custom options:

server:
log-queries: yes
log-replies: yes
log-tag-queryreply: yes
log-servfail: yes
ede: yes
#qname-minimisation: no
#aggressive-nsec: no
#infra-keep-probing: yes
#infra-cache-max-rtt: 2000
#infra-host-ttl: 0
#outbound-msg-retry: 32
#max-sent-count: 128

edit; she said it broke a minute later and i restarted the resolver and it seems to be working again...for now. My wifi garage door opener or Nest Protects are still not online

edit2: took a few minutes but my garage door opener is back online as well as 3/5 Nest Protects...

[RickyBaker]

Sorry for the slightly off topic q, but i, as @johnpoz suggested, feel that updating to 2.7.2 is my next logical step (but I can't do it from the gui or console). So I made a bootable usb stick with 2.7.2 and started the process....but then i forgot the options I selected when I first installed pfsense years ago. Is there a way to check on the gui or console which option i selected for ZFS or UFs? Or which drive I installed it to? I thought my Dell r210 II only had one drive in it but I was presented with 2 during the installation (maybe a partition) and I didn't remember which I did. I just want to make sure my installation is exactly the same as the original to avoid issues.

edit: safe to assume this means zfs on a single drive?
login-to-view

Also I read online there would be an upgrade option when I loaded up a flashable usb with psfense already installed on the target media but this was not a presented option....I also read online that this means the installation media can't read the pfsense install....how much should I be reading into this?

[SteveITS]

@RickyBaker ZFS was released with 2.6.0 so when did you install yours? Or check the dashboard Disk widget. In general you’d want ZFS.

Re upgrading, one can’t do that like you’re describing I think. However this is a thing: https://docs.netgate.com/pfsense/en/latest/backup/restore-during-install.html#recover-config-xml-from-existing-installation. Or just restore after.

The installer may show the usb stick? Can you tell by size?

[Gertjan]

@RickyBaker said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

Also I read online there would be an upgrade option when I loaded up a flashable usb with psfense already installed on the target media but this was not a presented option....I also read online that this means the installation media can't read the pfsense install....how much should I be reading into this?

As you can see, "just download pfSense" isn't an option anymore. It's gone. From now on, you load an installer, never used myself that one before.
This must be it : the online installer.

I also upvote the upgrade to 2.7.2.
You'll be using a far newer unbound version, probably "1.19.3."

About the install media : forget about CD, DVD etc.
Use : Prepare a USB Memstick.
If the stick isn't broken, it works. Tried this method several times on Netgate devices an bare bone stuff like your device.

@RickyBaker said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

So I made a bootable usb stick with 2.7.2 and started the process....but then i forgot the options I selected when I first installed pfsense years ago

Ah, ok, you have a 2.7.2 on USB ready.
When installing on a device, known, or not, try to know what de 'FreeBSD' name of the hard disk is. There can be one or more drives. I say drive name, not the partition names on that drive as they will will be lost.
If you can chose, go for the best file system : ZFS.
Enumerate also your NIC names : you can see them in the Interfaces >Interface Assignments menu.
And thats all there is to do.
Go for a manual default bare bone 192.168.1.1/24 LAN and DHCP WAN setup, if you can.
Make a backup of your current config first.
When installing, I never 'recover' the config (if found). I test the system first with the 'everything to default' settings. When that works out : LAN and WAN (and unbound ^^) ok, then you can import your backed up config later on using the GUI.
If any issues come back at that moment : you know it's your config ;)

[RickyBaker]

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

I test the system first with the 'everything to default' settings.

because of the sporadicness of the issue, it's going to be very difficult to test the default settings. Any suggestions? i don't want to set up all the downstream devices to deal with the new ip address et al.

and slightly off topic/dumb q: is there a way for me to view any and all drives pfsense sees in the GUI or the command line?

[Gertjan]

@RickyBaker

cat /etc/fstab

[RickyBaker]

@Gertjan anyway to see the sizes of these? I tried a bunch of ls commands that were all not included in pfsense (and apt wasn't either)

login-to-view

I found camcontrol devlist:
login-to-view
I dunno waht that middle ahci is or the sized of the hard drives but it really would appear i have 2 500gb hdd's in there (the 2 wdc above)...
geom disk list got me sizes:login-to-view

I genuinely can't remember instaling a second harddrive in there but here we are. would you guys agree that's what it looks like?

[RickyBaker]

I have no idea what's happening. I booted from the installer USB drive and the installation seemed a little different. It didn't request ZFS or UFS (or the other 3 options this time) and I got through the selection of interfaces and right to the installation selection an got this error when attempting to access the Netgate servers:
login-to-view
When I switched back to booting off the included hard drive (my old configuration, everything came up alright, but I had no internet. and the homepage of pfsense read that it was unable to check for updates:

login-to-view

Is this a crazy coincidence of my ISP going down the exact minute I tried to install the new pfsense or could I possibly have done something to the internet connection while powering down the modem and trying to install the newer version? Soooo frustrating either way!

Edit: I needed to tether to the hotspot of my mobile to write this post, as this post loaded but said there were issues connecting to the negate servers. But I was able load other pages on my phone and laptop. It's almost like negate was alone blocked on my ISP, so weird!

Edit2: There seems to be no issue with internet or connecting to the netgate forum now. Maybe it was the worlds world coincidence

[RickyBaker]

since the homepage was again displaying that I was "up to date" even though I was on 2.7.0 I decided to reattempt all the suggestions in this thread: https://forum.netgate.com/topic/184670/issue-with-going-from-2-7-0-to-2-7-2/15

And it worked! no scary full reinstall. I'm on 2.7.2 and I will report back. Though after the weird issue with Netgate Servers being unable to be contacted I restarted my AT&T FIber ONT and haven't noticed an outage since (thought it was only a day).....In any event i'll be back here to report any developments

[RickyBaker]

@SteveITS said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

update: https://forum.netgate.com/topic/187506/kea-dhcp-feature-roadmap/6

login-to-view

So is it safe to switch over?

[SteveITS]

@RickyBaker if you need just basic dhcp and not the missing features it should be fine. Or just wait until it’s not a preview.

You can also switch back.

[RickyBaker]

OK just reporting back. It's been over a week since i upgraded to 2.7.2 and I've only had 2 nxdomain (both by my wife). The first was some very janky website that seemed to point to a xxx.xxx.local domain which i remember from earlier posts will always happen. And yesterday to an unknown website (to me) but other websites were opening and the rest of the internet seemed to be working without issue. Since preceding the upgrade I had also reset the AT&T Fiber ONT (and went without issue for about 2 days before the firmware upgrade). I am almost positive I had reset the ONT earlier in the troubleshooting and perhaps wish I'd tested for outages longer between the reset and upgrading the firmware.

BUT it seems that one of the 2 actions has solved my issue. At least until I hit submit on this post here...

edit: I guess i would like to know if it's possible my issue was ISP/ONT?

[Gertjan]

@RickyBaker said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

edit: I guess i would like to know if it's possible my issue was ISP/ONT?

This ONT thing is attached to your WAN interface, right ?
Does it have a web GUI ? If so, does it have some stats to show you ? Error counter ? Last reconnect moment ?
If your WAN has issues, it would not be "DNS only" but all kind of traffic that would be impacted.

Btw : on already three devices (PCs) I use regularly, I saw that my browser (Firefox) changed recently its DNS settings :

login-to-view

I alsway have set this to : "Off" which means firefox uses the systems (Micrsoft OS) DNS settings = pfSense.
But no, I found the settings were 'reset' back to "default protection" which probably means it does DNS over TLS to some obscure DNS server, bypassing pfSense ... but hitting pfBlockerng's "DNS over HTTPS/TLS/QUIC Blocking".

Exactly this : Home > pfSense Software > DHCP and DNS : the top most forum post : HEADS UP: Be aware of Trusted Recursive Resolver (TRR) in Firefox

[RickyBaker]

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

This ONT thing is attached to your WAN interface, right ?
Does it have a web GUI ? If so, does it have some stats to show you ? Error counter ? Last reconnect moment ?

yeah it's connected to the wan. it def has a GUI but I think i'd have to unplug it to access it. I had a reverse proxy set up for my comcast router but i'm not sure i ported it over when i switched ISP's.

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

If your WAN has issues, it would not be "DNS only" but all kind of traffic that would be impacted.

this was always my thought and why i'm assuming it's the firmware upgrade that did it and that I was just lucky for a day or 2 before that.

@Gertjan said in DNS_PROBE_FINISHED_NXDOMAIN sporadically for anywhere from 30secs to 10min. works flawlessly at all other times:

But no, I found the settings were 'reset' back to "default protection" which probably means it does DNS over TLS to some obscure DNS server, bypassing pfSense ... but hitting pfBlockerng's "DNS over HTTPS/TLS/QUIC Blocking".

that's so annoying, thanks for the heads up

[RickyBaker]

Ugh, albeit infrequent, the issue has occurred a couple times over the last couple days. Yesterday in bed my wife complained about it happening in the eBay app (and Chrome and the public libraries audiobook app) and I feverishly started opening random websites from my Google feed (swipe left on an android). I eventually got an nxdomain error but was able to open 5-6 links before AND after the error. It's very odd to me that my wife seems to experience the problem with much more consistency and degree than I do. Resetting the DNS Resolver restored usefulness immediately.

login-to-view
login-to-view
login-to-view

[johnpoz]

@RickyBaker exceeded max number of sends is the root of the problem it seems.

You might want to bump your logging level up to 5 in unbound.. This might give you more insight to the actual failure when you see the servfail and its reason.

I would prob set

do-ip6: no

As well..

Are you seeing a crazy amount of queries? Is something bombing unbound with queries before those failures?

[bmeeks]

Quite a long thread here on the unbound GitHub account that I think might be related to the OP's issue: https://github.com/NLnetLabs/unbound/issues/362.

From my reading of the long thread, unbound itself is producing the SERVFAIL messages at times inappropriately as a byproduct of trying to "protect" the client from abuse. The thread is primarily an argument about DNSBL anti-spam logic and resultant DNS lookups, but I think the underlying logic inside unbound itself may be part (or maybe even all) of the issue the OP is seeing randomly.

There are also a couple of suggested unbound config changes in that thread that might help mitigate the issue. Of note is the parameter to greatly increase the "maximum number of sends" limit.

For what it's worth, I found reports of this same behavior on "the other Sense" product. That leads me to strongly suspect the root cause is something in unbound that perhaps specific configurations or even specific queries triggers the error.