Slow DNS after 22.05

istacey

So far so good with DNS servers issued via DHCP to client devices.

Simple things like playing audio via Amazon Echo works, no intermittent problems with websites that I know are up.

Fingers crossed this is a sufficient work around.

kvhs

Hi! Many helpful posts here!

Just wanted to mention that I'm also seeing the intermittently slow resolution described above:
Loading of websites often require refreshes to either have the site name resolved or CDN for images or stylesheets. I'd like to emphasize the intermittent nature of the problem -- I have duckduckgo.com set as my default search engine (i.e. a very frequently visited site) and have gotten name resolution errors in the browser time and time again over the last weeks with no clear pattern for when it's happening.

I have a Netgate 2100 and upgraded from version 22.01 to 22.05 a few weeks ago. The problem started with the upgrade. I had not made changes to the DNS Resolver settings before so the default of using the DNS servers given via DHCP on WAN was reflected on the front page with three servers listed, 127.0.0.1 being the first. Client devices were given the pfSense IP as their DNS Server.

To remedy the situation I tried adding CloudFlare's 1.1.1.1 and 1.0.0.1 as DNS servers in System > General Setup and subsequently unchecked "Allow DNS server list to be overridden by DHCP/PPP on WAN or remote OpenVPN server" but the problem persisted.

Based on replies in this thread, I checked "Serve Expired" on Services > DNS Resolver > Advanced Settings. The problem still occurs from time to time although seeingly less frequent. Resolution appears slow.

Further, I tried disabling DNSSEC (unchecked "Enable DNSSEC Support" in Services > DNS Resolver > General Settings) and disabled hardening of DNSSEC data (unchecked "Harden DNSSEC Data" in Services > DNS Resolver > Advanced Settings). Failures still occur.

To circumvent these problems I temporarily disabled the DNS Resolver.

I'll be watching this thread, hoping a solution pops up.

worldHopp

@kvhs said in Slow DNS after 22.05:

Hi! Many helpful posts here!

Just wanted to mention that I'm also seeing the intermittently slow resolution described above:
Loading of websites often require refreshes to either have the site name resolved or CDN for images or stylesheets. I'd like to emphasize the intermittent nature of the problem -- I have duckduckgo.com set as my default search engine (i.e. a very frequently visited site) and have gotten name resolution errors in the browser time and time again over the last weeks with no clear pattern for when it's happening.

I have a Netgate 2100 and upgraded from version 22.01 to 22.05 a few weeks ago. The problem started with the upgrade. I had not made changes to the DNS Resolver settings before so the default of using the DNS servers given via DHCP on WAN was reflected on the front page with three servers listed, 127.0.0.1 being the first. Client devices were given the pfSense IP as their DNS Server.

To remedy the situation I tried adding CloudFlare's 1.1.1.1 and 1.0.0.1 as DNS servers in System > General Setup and subsequently unchecked "Allow DNS server list to be overridden by DHCP/PPP on WAN or remote OpenVPN server" but the problem persisted.

Based on replies in this thread, I checked "Serve Expired" on Services > DNS Resolver > Advanced Settings. The problem still occurs from time to time although seeingly less frequent. Resolution appears slow.

Further, I tried disabling DNSSEC (unchecked "Enable DNSSEC Support" in Services > DNS Resolver > General Settings) and disabled hardening of DNSSEC data (unchecked "Harden DNSSEC Data" in Services > DNS Resolver > Advanced Settings). Failures still occur.

To circumvent these problems I temporarily disabled the DNS Resolver.

I'll be watching this thread, hoping a solution pops up.

Following on from my original reply where it looked like restarting the service resolved... it didn't.

Just wanted to say I have had the same experience - tried many of the suggestions here. I have tried with the resolver/forwarder, with DNSSEC enabled/disabled. Tried pre-fetch keys, harden DNSSEC data.

I have given up with the slow or unresponsive DNS resolution since 22.05 and put my clients on Google DNS over TLS which is working perfectly.

Hopefully somebody can find a solution as I rather liked using the resolver on my SG2100.

lohphat

@worldhopp

In summary, my fixes have been stable.

Enable Serve Expired -- this helped with CDN lookups. This was not set in 22.01
Set minimum TTL to 300 seconds. This was not set in 22.01
Disable Experimental 0x 20 support -- this was working in 22.01 but caused instability in 22.05.

So far things have been stable for over a week. I tried with and without pfBlocker-devel and various attempts to use forwarding or not (it was necessary while I was searching for a fix but I'm back to resolving locally again).

So yes, it seems "something has changed" but there's no smoking gun.

kvhs

There are a number of bug fixes on Unbound since 1.15.0 which pfSense 22.05 uses, but I don't have enough knowledge of DNS to determine if those fixes are likely to fix these problems.

https://github.com/NLnetLabs/unbound/tags

I find this one solved in 1.16.0 interesting though: https://github.com/NLnetLabs/unbound/issues/670

bigbadvoodoodaddy

Having this issue with an SG-6100 after going from 22.01 to 22.05 also. So far the Enable Serve Expired seems to be resolving the issue, but time will tell

Kempain

Also seeing these intermittent DNS issues on my 5100 since updating to 22.05.
Haven't had a chance to troubleshoot yet but same issues outlined above.
Will try enabling Serve Expired tomorrow and see if that resolves.

Kempain

@lohphat said in Slow DNS after 22.05:

Set minimum TTL to 300 seconds. This was not set in 22.01

I enabled Serve Expired but this didn't seem to help in my case.
Experimental 0x 20 support was already disabled.

Is the min TTL setting: Minimum TTL for RRsets and Messages?

So far I'm just thinking of rolling back to 22.01.
It seems like whatever was updated in unbound is causing issues for a small subset of us.

bigbadvoodoodaddy

@kempain

I may have to roll back as well. The Enable Serve Expired (seemingly) does help a little, but I am still getting dns timeouts frequently. I have now also enabled cache-min-ttl (also known as Minimum TTL for RRsets and Messages) to 300 sec. My Experimental 0x20 support has never been enabled.

https://nlnetlabs.nl/documentation/unbound/unbound.conf/

Not sure if this is related (probably should be talking on unbound's GitHub at this point) but I'm seeing a bunch of "outnettcp got tcp error -1" in debug logs when turned up to logging level 4.

lohphat

@kvhs said in Slow DNS after 22.05:

I find this one solved in 1.16.0 interesting though: https://github.com/NLnetLabs/unbound/issues/670

This seems a reasonable trail to start following -- this may be an out of memory/heap issue.

Just curious, for those of us seeing issues are you also running IPv6? I am.

In the bug notes it seems that disabling IPv6 addressed the issue as less memory overhead is needed. I wonder if the unbound changes may necessitate bumping up memory allocation to prevent spurious lookup failures.

Kempain

@bigbadvoodoodaddy

Just enabled logging level 4 and also see a few 'outnettcp got tcp error -1' errors but no idea if it's related.

@lohphat

Also running IPv6.

Not sure I can actually rollback unless I can use config backup from 22.05 on 22.01.
Wondering if it would be better if I just wipe and reinstall 22.05, then restore config just in case something got messed up with the upgrade.

I believe I saw @johnpoz runs an SG-5100 too, and upgraded from 22.01 to 22.05 and doesn't have the same problems.

Gertjan

Add me to the list #nothingtodeclare

Running 22.05 one a Intel based box, a SG 4100.
I'm using IPv6, although tunnel based, using ipv6.he.net

unbound settings are native, that is, I'm not forwarding, unbound makes use of the "13 main Internet Root servers".
On the Services > DNS Resolver > Advanced Settings I have set :
Query Name Minimization
Prefetch Support
Prefetch DNS Key Support
Harden DNSSEC Data
Serve Expired
Keep Probing
Experimental Bit 0x20 Support
Other values are - I guess, default.

On the Services > DNS Resolver >General Settings page :

Network Interfaces : All
Outgoing Network Interfaces : All
DNSSEC : Enabled ( Remember : DNSSEC makes sense only when you are NOT forwarding )
Python Module : Ebaled ( As I'm using pfBlockerng-devel also)
Note : DHCP Registration NOT set, which means unbound doesn't get restarted on every DHCP lease event. All known important LAN devices have static MAC DHCP leases.
Static DHCP : enabled (as this one won't restart unbound)
Custom options : None.

Memory usage ? How often unbound restarts ? Requests handled ?
I have it all the hard numbers and graphs, so I can see if something is happening, and I can check if setting makes any changes.
Look here.

Remember : this is DNS. I can't have or tolerate a 'doesn't work'

Also : Netgate pfSense comes with a default DNS set up. This one works out of the box(teher might be one exception, read below) : why not using that setting and be done with it ?
And no, Netgate does not ask you to forward to any DNS requests to some company's remote resolver. pfSense has its own resolver.

Yes, I've tried forwardig, it did seem to work fine, but I never kept this mode for longer as a couple of days. I guess I don't need a remote resolver as unbound does a good job doing that for me.

Btw ;: I'm using 22.05 on a SG4100 for a couple of weeks now. Before that, I was using a bare bone Intel box using a quand Intel NIC setup. Never had any issues except for the major unbound bugs that touches everybody back then, and that was always corrected immediately.
I never had to go back a previous version, and that for the last 10+ years, since pfSense version 1.x

Networks usage : 3 LANs,, one major company LAN, one untrusted client "captive portal" LAN with a bunch of access points for the hotel clients, one DMZ type LAN.
No VLAN stuff

I'm using pfBlockerng-devel, it syncs feeds ones a week, with a minimal feeds list. I'm just blocking the major adds and bads hit list.

My ISP gives me a good (I guess) uplink with an static IPv4. It's still VDSL copper wire (about 24 Mbits sec down). This will be fibre in a very near future.

I tend to use pfSense functionality that I "know", that I can debug, that I trust, that I understand.

And one last thing :And please, I do not want to offend any one here :
I rent a 'big' bare bone server for for my web sites, mail and other stuff like Munin. I'm handling all my own DNS needs myself, using bind (named), about 20 domain names. My registrar's name server entries point to my own DNS name servers, a master and two or three DNS backup servers (small VPSs).
For 99,9 % of the time, I check regularly my DNS. For example, I use this site to mention just one.
Because I'm doing my own DNNSEC, I use this. Many other test sites exists.
The majority of the DNS tests are done remotely and locally using "dig".

And here it is : no one should be handling its own DNS, as this forces you to fully understand what DNS is, how it works, how to see issues and how to deal with them
I didn't saw another way to fully understand this 'DNS' thing.
But, suddenly, when you know all DNS, DNS will never be an issue any more.

Consider this : take a small 1 $/€ a month VPS, and a domain name (5 $/€ a year ?) a play with your own domain name. You'll be massacring loads of misunderstandings pretty fast.
This was called 'learning' back then ;)

Do not underestimate the number of times your local pfSense has no issue at at, but your simply visiting a site that has issues with it's DNS. Just wait it out. Don't start modifying your own setup as it was good already.
Even Facebook managed to completely disappear from the net, a year or so ago, because some guy really messed up.

johnpoz

@kempain said in Slow DNS after 22.05:

runs an SG-5100 too

I'm on a SG4860, and correct I have seen zero issues with dns running 22.05 since the day it dropped.

mynet

First post to add one more to the list.

SG-1100 with no problems prior to the update. I did have some issues with the update so am running a brand new, fresh installation. The only changes from stock are to admin password, host name, and adding google DNS servers (8.8.8.8 and 8.8.4.4)

I have had extensive problems with MacOS/iOS devices and very little if any with windows. I thought it was IPv6 related b/c no IPv6 on the windows machines but enabling it did not produce any problems (either that or I didn't leave it turned on long enough).

Having some flashbacks to dial-up modem days....

johnpoz

@mynet said in Slow DNS after 22.05:

and adding google DNS servers (8.8.8.8 and 8.8.4.4)

So your doing forwarding? And your forwarding via TLS? ie dot?

mynet

@johnpoz said in Slow DNS after 22.05:

So your doing forwarding? And your forwarding via TLS? ie dot?

No. I'm not trying (or trying not to) do anything beyond the basics. I mindlessly followed the initial configuration guide (https://docs.netgate.com/pfsense/en/latest/solutions/sg-1100/initial-configuration.html) which suggests those two addresses.

Beyond that, in System/General Settings/Advanced/DNS Server Settings, DNS server override is checked, and DNS Resolution Behavior is set to Use Local DNS and Fall Back to Remote DNS Servers.

In Services/DNS Resolver, Enable DNS Resolver is checked, and In Services/DNS Forwarder, Enable DNS Forwarder is unchecked.

All of this is what I had in the prior version which worked fine.

One thing that seems different for me compared to some others is that I haven't noticed any issues with a windows machine though I haven't used it extensively. As I said, I thought the issue might be IPv6, but I am not sure now.

johnpoz

@mynet said in Slow DNS after 22.05:

https://docs.netgate.com/pfsense/en/latest/solutions/sg-1100/initial-configuration.html

Have no idea why those are in there.. Out of the box those wouldn't even be used by clients, and would only be used by pfsense if unbound failed to work..

They are pointless to be in there unless resolving fails, and clients wouldn't work anyway.. Only pfsense would be able to resolve stuff and check on its updates, etc..

The guide should prob be reworked with more detail on what a resolver is, how it works out of the box and that you don't need to put anything in for dns, and most likely don't even want dhcp to override and set dns, etc. Only if resolving is not working should you look to doing that.

lohphat

@johnpoz Wouldn't initial DNS resolver IPs be useful if you were still installing packages during initial config in an environment where DNS is blocked/filtered (e.g. in a datacenter) and unbound was in a state where it's not able to resolve yet (e.g. a VPN config needs to be setup) until config is completed later?

Without initial external DNS, does unbound have an initial list of hard-coded IPs for the root servers to bootstrap DNS?

johnpoz

@lohphat said in Slow DNS after 22.05:

where DNS is blocked/filtered

If dns is blocked how are you talking to 8.8.8.8 ;)

Yes there are scenarios where you would have to forward, say an ISP that only allows access to its dns.. Believe if your on SAT connection for example your limited to specific NS you can talk to..

My point in reworking that guide is could give the impression that you need dns, when out of the box you don't need to set anything.. Because out of the box unbound is resolver.. So unless you have some specific restrictions in place there is no need to put in anything..

Finding the piece that completes this puzzle - is it boxes on ARM? Is it only ARM that forward via TLS? There is some piece to this puzzle missing that would complete it and make sense.

All I can say is I have had zero issues with unbound resolving.. but I am not on arm, and I don't do anything like forwarding. Pretty much unbound out of the box config, other than I set a min ttl of 1 hour and serve zero.. I also do min for qname, but allow for fallback, because if you don't there are lots of cname stuff that ends up having issues - stupid ass MS with like chained cnames 7 deep and shit ;) I think there is some technet domains that fail if you don't allow for fallback on the qname stuff, etc.

lohphat

@johnpoz said in Slow DNS after 22.05:

@lohphat said in Slow DNS after 22.05:

where DNS is blocked/filtered

If dns is blocked how are you talking to 8.8.8.8 ;)

...or an internal datacenter DNS server

We are in agreement that the docs do need a rework as do the info text blocks to document defaults and any dependencies on other settings.

I'd be all in for a single click "Return page (or package) settings to default values".