Frequent DNS timeouts

thundergate

Do you have dhcp stuff in its log that might match up

Within DHCP I do have a lot of those messages (see screenshot):

johnpoz

@thundergate do those times match up? I see you have register dhcp off in your settings.. But maybe it didn't take?

Something is clearly restarting unbound, and a lot.. And the only thing comes to mind that would restart it that often would be dhcp registrations.

I would guess for whatever reason your setting of not to register dhcp is not actually working.. For whatever reason.

Quick test of that might be to just turn off all your dhcp services on pfsense.. Do your restarts stop? You don't need dhcp running 24/7 it can be off for a while. if you you have all your dhcp services off on pfsense, and your still seeing unbound restart like crazy like that - then you know its not dhcp registrations doing it. With the amount of restarts your seeing - I would think you should be able to tell in 10 minutes or so if that is the problem..

JonH

@thundergate said in Frequent DNS timeouts:

And my DNS Resolver log is full of entries.... Don't really know what is causing this issues?!

Do you use Service Watchdog? Is it possible that these restarts could be from the Watchdog restarting it? I removed unbound from my Watchdog monitoring because it was restarting it too often. It was a month ago and I've forgotten if my problems created a log like you posted.

Also note that my Resolver was not stopping, it was hanging and would simply 'fix itself' after 3-6 minutes or so. In my case not using Watchdog has been useful for me.

JonH

@thundergate said in Frequent DNS timeouts:

Within DHCP I do have a lot of those messages (see screenshot):

The other day i had similar entries in DHCP log for one IP. These started after I had removed power from one of my IoT devices that I was also blocking with a firewall rule.

This particular device is a bed that also monitors sleep patterns. I have rules that block it's access to 'the motherland'. It also uses an iPhone app so there is also this extra chatter. The app is unused so I deleted it. I also found entries in the States table for that IP and deleted the State for the specific IP. I also deleted the arp entry and rebooted pfSense and my wifi AP at the same time prior to repowering the device that was causing this issue.

That problem has now stopped.

thundergate

Sorry for my late feedback.

But after disabling and re-enabling some settings the issues are gone.

Don't know why - but at the moment no unbound restarts.

level4

Be aware that pfBlocker-NG cron/update also restarts Unbound, when (for instance) DNSBL lists are updated.

thundergate

Oh no... The stop/start of unbound started again.

What I could figure out is, that is has somehow be related to my Mac going into standby/hybernate mode. Than those unbound stop/start begins.

As it's a testing setup and my Mac is the only network device within the pfSense setup I can say, that it has to be something related to the Mac and pfSense / pfBlockerNG?!

johnpoz

@thundergate said in Frequent DNS timeouts:

Mac going into standby/hybernate mode.

Or is asking for dhcp all the time like my wife's iphone.. Would seem more like it - what would your mac going into standby have to do with pfblocker ???

See above where I posted my wife phone doing this

Mar 16 01:38:52 	dhcpd 	93450 	DHCPACK on 192.168.2.203 to 88:b2:91:98:d6:f0 via igb2
Mar 16 01:38:52 	dhcpd 	93450 	DHCPREQUEST for 192.168.2.203 from 88:b2:91:98:d6:f0 via igb2
Mar 16 01:37:41 	dhcpd 	93450 	DHCPACK on 192.168.2.203 to 88:b2:91:98:d6:f0 via igb2
Mar 16 01:37:41 	dhcpd 	93450 	DHCPREQUEST for 192.168.2.203 from 88:b2:91:98:d6:f0 via igb2
Mar 16 01:31:44 	dhcpd 	93450 	DHCPACK on 192.168.2.203 to 88:b2:91:98:d6:f0 via igb2
Mar 16 01:31:44 	dhcpd 	93450 	DHCPREQUEST for 192.168.2.203 from 88:b2:91:98:d6:f0 via igb2
Mar 16 01:30:01 	dhcpd 	93450 	DHCPACK on 192.168.2.203 to 88:b2:91:98:d6:f0 via igb2
Mar 16 01:30:01 	dhcpd 	93450 	DHCPREQUEST for 192.168.2.203 from 88:b2:91:98:d6:f0 via igb2
Mar 16 01:29:20 	dhcpd 	93450 	DHCPACK on 192.168.2.203 to 88:b2:91:98:d6:f0 via igb2
Mar 16 01:29:20 	dhcpd 	93450 	DHCPREQUEST for 192.168.2.203 from 88:b2:91:98:d6:f0 via igb2

If your unbound is restarting on dhcp then yeah that is going to be horrible.. That was when my wifes phone is on the charger, it shouldn't of been doing shit, let also be asking for dhcp..

Gertjan

@thundergate said in Frequent DNS timeouts:

related to my Mac going into standby/hybernate mode. Than those unbound stop/start begins

MAC directly connected by wire on the LAN port ?
In that case, when mac goes down, LAN port goes down == NIC event == unbound restarts.
Solution : use a switch.
Or is this not your case ?

johnpoz

@gertjan said in Frequent DNS timeouts:

MAC directly connected by wire on the LAN port ?

That would be odd, but yeah that could do it ass well ;)

thundergate

@gertjan said in Frequent DNS timeouts:

MAC directly connected by wire on the LAN port ?

Oh noooo. That's it. Thanks for the hint. I was looking into logs forever, but forget about that simple one.

Gertjan

@thundergate

Now you know it, look again at the main system log using the console access while your MAC system is shut down == LAN shut down also.
Switch on the MAC.
You'll see a NIC (LAN) uplink event in the system log (check also the hardware or dmesg log).
That triggers a whole lot activity on the system. Every system process using (listing) to the LAN interface will get restarted = DHCP server, NTP, the WebGUI to name a few, and also unbound.

phipac

So far the only thing that has worked for me regarding the DNS hanging is to not use DNS over TLS to quad9 or Cloudflare or whatever upstream servers you have set. I did not see anyone else mention that so far (though it's likely I missed it). After going back to plain old DNS on port 53, all pages load much faster and I don't have missing icons and images. No more timeouts seen on the Status/DNS Resolver page either.

SteveITS

@phipac said in Frequent DNS timeouts:

not use DNS over TLS

I posted it above, referencing a different thread. It didn't seem to be any problem for me at home, but others (in one or more threads, have lost track) have said it definitely made a difference. One theory offered was that a high number of DNS requests could perhaps influence it.

JonH

@phipac I switched from quad9 to cloudflare about a week ago. There was a noticeable improvement in reliability but I still experience hangs. Much fewer and lasting shorter periods. I may try openDNS next.

The theory @SteveITS mentioned seems plausible. Equally plausible is maybe I have a configuration error. I note Apple devices seem very chatty although I haven’t had the opportunity to check other os’s to compare. The hangs I experience usually happen while using iPad/iPhone’s. I haven’t noticed it on a Mac but cannot say for sure.

Gertjan

Clouadflare, quad9, OpenDNS using TLS :
Read the first phrase here : Support TLS de Postfix where Postfix could actually be 'anything'.

It's probably not 'thousands' but tens of thousands of lines, because back then, TLS wasn't even TLS 1.x, but the far more simple SSL. I'm not pointing the exisyte,nce of bug, as a decade or so later, most are ironed out. But the code complexity is huge. For those who doubt : have a look at what OpenSSL is. It's open source, and it's mind boggling.

edit : and keep in mind that all the TLS carefully constructed on your side has to be undone on the other side.
Running on hardware that handles millions identical tasks .... On these guys also just found out that their electricity bill just 3 folded. So, If I was to maintain these (free !!) services, I would start to 'throttle'. What would you do ? ;)

And then there is this. If entropy is missing on a system, TLS goes bad.
My words : entropy is the possibility of the system to generate random numbers. Belief it or not, these are hard to generate. And when the stock goes low, the system ... does what, wait ?
TLS eats entropy for breakfast.

All that said : I've been using 1.1.1.1 and the IPv6 counterpart for several weeks.
The only thing I noticed is that I noticed nothing. I forgot that I switched from DNSSEC Resolving to forwarding over TLS.
I'm not using a big system : a SG 4100 with 23.01.

JonH

@gertjan said in Frequent DNS timeouts:

I'm not using a big system : a SG 4100 with 23.01.

Me neither. SG-5100.

What is not clear to me in many posts is if pfBlocker is part of what folks are seeing. As many others said, I too had no problems with 22.xx and problems started after upgrading to 23.01 and new release of pfBlocker. The one change that comes to mind is using python mode for unbound.

I am now traveling and hope all the smoke settles over the next few weeks.

Gertjan

@jonh said in Frequent DNS timeouts:

that comes to mind is using python mode for unbound

That 'mode' doesn't restart unbound.

@thundergate said in Frequent DNS timeouts:

Also disabled python mode - and still all the unbound restarts.

Later on he discovered that unbound also restart when a LAN interface is take Up/Down.

So, unbound can get restarted for more then one reason.

I admit, my words, and what @thundergate wrote, are not a proof. With some luck, @BBcan177 can certify my words (but he probably won't even bother) as he can read that python script very well, as he wrote it.

pfBlockerng can restart unbound, as that is way to take in account new DNDBL info.
If it was possible to re download all the DNSBL feeds every five minutes, and (condition) one of these feeds had a changed content, then, yeah, unbound would get restarted every 5 minutes.
The golden rule always applies : the admin always rules, even if he doesn't know what he is doing ;)

One of the reason why I refresh my DNSBL feeds ones a week. If a DNSBL feedactually changed, then unbound gets restarted. The cron task has been set at 3 AM sunday, so I never detect an unbound restart = a 3 seconds outage.

Also, don't see this post as a 'your are silly' and 'I am smart'. It's just me pushing you to discover what the real reason is.
The logs will tell you.

phipac

@jonh Good information - thank you! I have not noticed a difference based on OS or manufacturer. I was seeing similar rates of hang on Windows, Linux, and Android.

Conducting a few more tests to see if I can narrow anything else down.

phipac

@gertjan Entropy. I hadn't considered that. I just noticed all of my ubuntu-based VMs flatlined to 256 entropy last fall. I think it lined up with the release of the 5.10 kernel but I can't be sure. It happens that I have pfsense running on proxmox, which also shows a flatline of 256 entropy. In pfsense, this is what I get from sysctl kern.random:

kern.random.fortuna.concurrent_read: 1
kern.random.fortuna.minpoolsize: 64
kern.random.rdrand.rdrand_independent_seed: 0
kern.random.use_chacha20_cipher: 1
kern.random.block_seeded_status: 0
kern.random.random_sources: 'Intel Secure Key RNG'
kern.random.harvest.mask_symbolic: PURE_VMGENID,PURE_RDRAND,[CALLOUT],[UMA],[FS_ATIME],SWI,[INTERRUPT],NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
kern.random.harvest.mask_bin: 01000000010000000101011111
kern.random.harvest.mask: 16843103
kern.random.initial_seeding.disable_bypass_warnings: 0
kern.random.initial_seeding.arc4random_bypassed_before_seeding: 0
kern.random.initial_seeding.read_random_bypassed_before_seeding: 0
kern.random.initial_seeding.bypass_before_seeding: 1