Slow DNS after 22.05
-
Here you go:
[22.05-RELEASE][root@pfSense.localdomain]/root: unbound-control -c /var/unbound/unbound.conf lookup spectrum.s3.amazonaws.com The following name servers are used for lookup of spectrum.s3.amazonaws.com. ;rrset 5502 4 0 7 0 s3.amazonaws.com. 5502 IN NS ns-1300.awsdns-34.org. s3.amazonaws.com. 5502 IN NS ns-1579.awsdns-05.co.uk. s3.amazonaws.com. 5502 IN NS ns-63.awsdns-07.com. s3.amazonaws.com. 5502 IN NS ns-771.awsdns-32.net. ;rrset 77561 1 0 8 0 ns-771.awsdns-32.net. 77561 IN A 205.251.195.3 ;rrset 77561 1 0 8 0 ns-771.awsdns-32.net. 77561 IN AAAA 2600:9000:5303:300::1 ;rrset 77561 1 0 8 0 ns-63.awsdns-07.com. 77561 IN A 205.251.192.63 ;rrset 77650 1 0 8 0 ns-63.awsdns-07.com. 77650 IN AAAA 2600:9000:5300:3f00::1 ;rrset 77561 1 0 8 0 ns-1579.awsdns-05.co.uk. 77561 IN A 205.251.198.43 ;rrset 77561 1 0 8 0 ns-1579.awsdns-05.co.uk. 77561 IN AAAA 2600:9000:5306:2b00::1 ;rrset 77561 1 0 8 0 ns-1300.awsdns-34.org. 77561 IN A 205.251.197.20 ;rrset 77561 1 0 8 0 ns-1300.awsdns-34.org. 77561 IN AAAA 2600:9000:5305:1400::1 Delegation with 4 names, of which 0 can be examined to query further addresses. It provides 8 IP addresses. 2600:9000:5305:1400::1 expired, rto 555896848 msec, tA 0 tAAAA 0 tother 0. 205.251.197.20 expired, rto 555896848 msec, tA 0 tAAAA 0 tother 0. 2600:9000:5306:2b00::1 expired, rto 555896848 msec, tA 0 tAAAA 0 tother 0. 205.251.198.43 expired, rto 555896848 msec, tA 0 tAAAA 0 tother 0. 2600:9000:5300:3f00::1 not in infra cache. 205.251.192.63 expired, rto 555896848 msec, tA 0 tAAAA 0 tother 0. 2600:9000:5303:300::1 expired, rto 555896848 msec, tA 0 tAAAA 0 tother 0. 205.251.195.3 not in infra cache. [22.05-RELEASE][root@pfSense.localdomain]/root: unbound-control -c /var/unbound/unbound.conf lookup lycraservice-pa.googleapis.com The following name servers are used for lookup of lycraservice-pa.googleapis.com. ;rrset 74731 4 0 2 0 googleapis.com. 74731 IN NS ns2.google.com. googleapis.com. 74731 IN NS ns1.google.com. googleapis.com. 74731 IN NS ns3.google.com. googleapis.com. 74731 IN NS ns4.google.com. ;rrset 74608 1 0 1 0 ns4.google.com. 74608 IN A 216.239.38.10 ;rrset 74608 1 0 1 0 ns4.google.com. 74608 IN AAAA 2001:4860:4802:38::a ;rrset 74608 1 0 1 0 ns3.google.com. 74608 IN A 216.239.36.10 ;rrset 74608 1 0 1 0 ns3.google.com. 74608 IN AAAA 2001:4860:4802:36::a ;rrset 74608 1 0 1 0 ns1.google.com. 74608 IN A 216.239.32.10 ;rrset 74608 1 0 1 0 ns1.google.com. 74608 IN AAAA 2001:4860:4802:32::a ;rrset 74608 1 0 1 0 ns2.google.com. 74608 IN A 216.239.34.10 ;rrset 74608 1 0 1 0 ns2.google.com. 74608 IN AAAA 2001:4860:4802:34::a Delegation with 4 names, of which 0 can be examined to query further addresses. It provides 8 IP addresses. 2001:4860:4802:34::a expired, rto 555896080 msec, tA 0 tAAAA 0 tother 0. 216.239.34.10 expired, rto 555896080 msec, tA 0 tAAAA 0 tother 0. 2001:4860:4802:32::a rto 376 msec, ttl 784, ping 0 var 94 rtt 376, tA 0, tAAAA 0, tother 0, EDNS 0 assumed. 216.239.32.10 not in infra cache. 2001:4860:4802:36::a rto 376 msec, ttl 784, ping 0 var 94 rtt 376, tA 0, tAAAA 0, tother 0, EDNS 0 assumed. 216.239.36.10 not in infra cache. 2001:4860:4802:38::a rto 376 msec, ttl 784, ping 0 var 94 rtt 376, tA 0, tAAAA 0, tother 0, EDNS 0 assumed. 216.239.38.10 not in infra cache. [22.05-RELEASE][root@pfSense.localdomain]/root: unbound-control -c /var/unbound/unbound.conf lookup streetviewpixels-pa.googleapis.com The following name servers are used for lookup of streetviewpixels-pa.googleapis.com. ;rrset 74647 4 0 2 0 googleapis.com. 74647 IN NS ns2.google.com. googleapis.com. 74647 IN NS ns1.google.com. googleapis.com. 74647 IN NS ns3.google.com. googleapis.com. 74647 IN NS ns4.google.com. ;rrset 74524 1 0 1 0 ns4.google.com. 74524 IN A 216.239.38.10 ;rrset 74524 1 0 1 0 ns4.google.com. 74524 IN AAAA 2001:4860:4802:38::a ;rrset 74524 1 0 1 0 ns3.google.com. 74524 IN A 216.239.36.10 ;rrset 74524 1 0 1 0 ns3.google.com. 74524 IN AAAA 2001:4860:4802:36::a ;rrset 74524 1 0 1 0 ns1.google.com. 74524 IN A 216.239.32.10 ;rrset 74524 1 0 1 0 ns1.google.com. 74524 IN AAAA 2001:4860:4802:32::a ;rrset 74524 1 0 1 0 ns2.google.com. 74524 IN A 216.239.34.10 ;rrset 74524 1 0 1 0 ns2.google.com. 74524 IN AAAA 2001:4860:4802:34::a Delegation with 4 names, of which 0 can be examined to query further addresses. It provides 8 IP addresses. 2001:4860:4802:34::a expired, rto 555896080 msec, tA 0 tAAAA 0 tother 0. 216.239.34.10 expired, rto 555896080 msec, tA 0 tAAAA 0 tother 0. 2001:4860:4802:32::a rto 376 msec, ttl 700, ping 0 var 94 rtt 376, tA 0, tAAAA 0, tother 0, EDNS 0 assumed. 216.239.32.10 not in infra cache. 2001:4860:4802:36::a rto 376 msec, ttl 700, ping 0 var 94 rtt 376, tA 0, tAAAA 0, tother 0, EDNS 0 assumed. 216.239.36.10 not in infra cache. 2001:4860:4802:38::a rto 376 msec, ttl 700, ping 0 var 94 rtt 376, tA 0, tAAAA 0, tother 0, EDNS 0 assumed. 216.239.38.10 not in infra cache. [22.05-RELEASE][root@pfSense.localdomain]/root: -
@lohphat try disabling the experimental bit 0x20 support, and also try enabling the "serve expired" option which helps a ton with cdn dns performance.
-
@lohphat said in Slow DNS after 22.05:
My new question is why the alias being resolved used while the test rule I use the alias in, isn't being used at all.
??
The Alias is defined.
So it get resolved regularly.
An alias can be used not only in a firewall rule, but also some where else.
Or even not used at all : as long as it is defined, it get resolved.No "help me" PM's please. Use the forum, the community will thank you.
Edit : and where are the logs ?? -
@johnpoz said in Slow DNS after 22.05:
@lohphat said in Slow DNS after 22.05:
DNS forwarding and DNSSEC enabled:
That is never a good idea, if your going to forward where you forward either does dnssec or it doesn't you asking for it does nothing - other than problematic.
Interesting side-effect if DNSSEC is disabled, I get a warning if I try to change the Advanced Settings:
The following input errors were detected: Harden DNSSEC Data option can only be enabled if DNSSEC support is enabled.Another example of where info text could be clearer when there are inter-dependencies of settings. I want DNSSEC, but it's not needed if you're forwarding, but then the setting on another page is dependent on it. I didn't get a warning that it was set when I disabled DNSSEC on the main page. Confusing.
Also: The setting default should be included in each setting text so that as you're tweaking over time, knowing the default setting is helpful to return settings to original.
-
@gertjan But that behavior is not obvious.
Getting DNS lookup failures in the log took me a while to trace where I had those references. I had long forgotten the alias I had created to test a f/w rule for DNS blocking.
-
@lohphat said in Slow DNS after 22.05:
Confusing
True.
But changing a setting on a first page that automatically change (disables) a setting on a second page is more confusing.
All this is IMHO of course.Right now, you are hinted to disable first the extra 'special' DNSSEC settings and the second page, and then, at last; disable DNSSEC all together on the main page.
And strange .... DNSSEC works so well for me the last several years already.
I'm using DNSSEC, as pfSense is setup out of the box to use it.
Because a good 'flat' classic Internet connection should not intervene with my outgoing traffic.
If this wasn't the case, I would change ISP ASAP.These :
I've checked years ago.
Never had the need to remove them (which means I rarely visit sites with DNSSEC issues, I guess) -
@gertjan After all the testing the setting which has seemed to solve my problem is to enable "Serve Expired".
That was not enabled when I was running pre 22.05 so I wonder what changed between 22.01 and 22.05 which changed the behavior in my environment.
So far, after re-enabling DNSSEC and the Experimental 0x 20 support, things are working again -- just "Serve Expired" seems to have been the issue.
-
@lohphat I would turn on prefetch as well.
You also might want to set a min ttl. You mention seems to be problem with cdn stuff - possible the ttl is so freaking small that if your having time problems with resolving - you could run into timeouts..
I looked at one example you gave - 300 seconds..
;; QUESTION SECTION: ;i.ytimg.com. IN A ;; ANSWER SECTION: i.ytimg.com. 300 IN A 142.250.191.150 i.ytimg.com. 300 IN A 142.250.191.182 i.ytimg.com. 300 IN A 142.250.191.214 i.ytimg.com. 300 IN A 142.250.191.246 i.ytimg.com. 300 IN A 142.251.32.22 i.ytimg.com. 300 IN A 142.250.190.22 i.ytimg.com. 300 IN A 142.250.190.54 i.ytimg.com. 300 IN A 142.250.190.86 i.ytimg.com. 300 IN A 142.250.190.118 i.ytimg.com. 300 IN A 142.250.190.150 i.ytimg.com. 300 IN A 172.217.0.182 i.ytimg.com. 300 IN A 172.217.1.118 i.ytimg.com. 300 IN A 172.217.2.54 i.ytimg.com. 300 IN A 172.217.4.54 i.ytimg.com. 300 IN A 172.217.5.22 i.ytimg.com. 300 IN A 172.217.4.86 -
Hi, +1 here on the issue with erratic behaviour on DNS lookups since 22.05 update.
I'm going to read the full chain here, but lots of similarities in the bits I've skim-read.
Lots of these entries in the logs for unbound
Jul 13 21:43:18 unbound 982 [982:1] error: recvfrom 22 failed: Protocol not available
I'm using Cloudflare DNS servers, not allowing my WAN connections DHCP settings to flow through and have things set to "use remote, ignore local". I don't have these DNS servers set in my DHCP settings.
DNS Forwarder is inactive, DNS Resolver is active.
-
Over the last few days the only change I've made in addition to "Serve Expired" and add a minimum TTL of 900 sec (setting the help text doesn't specify units, but I have a long-standing complaint on the lack of min detail in setting help text). I also turned off "Use Experimental 0x 20" for DNS spoofing; this too over several days proved unstable (and a change between 22.01 to 22.05 as it was working fine before).
So yes, something has significantly changed in unbound in the last release.
-
@lohphat said in Slow DNS after 22.05:
something has significantly changed
Yeah it did went from version 1.12 or .13.something to 1.15
I have had zero issues with resolving anything. And unbound currently has been running for
[22.05-RELEASE][admin@sg4860.local.lan]/root: unbound-control -c /var/unbound/unbound.conf status version: 1.15.0 verbosity: 1 threads: 4 modules: 2 [ validator iterator ] uptime: 899181 seconds options: control(ssl) unbound (pid 87400) is running... [22.05-RELEASE][admin@sg4860.local.lan]/root:900k seconds = like 10 days..
While not saying your not having issues - clearly it something with your connection or unique to your setup because if it was something wrong with unbound itself - then everyone running 22.05 would complaining..
-
@johnpoz I think you're probably right. The issue is most likely down to a combination of 22.05 running on my specific hardware (NG 3100 which uses ARM that someone said further up has quirks on occasion) with my specific setup (which isn't far from a few tweaks from vanilla).
What I'm hoping is, someone smarter than me will be able to point me in the right direction.
I'm going to try telling my devices to use an external DHCP server, effectively bypassing pfSense and see if that improves things.
-
@istacey said in Slow DNS after 22.05:
@johnpoz I think you're probably right. The issue is most likely down to a combination of 22.05 running on my specific hardware (NG 3100 which uses ARM that someone said further up has quirks on occasion) with my specific setup (which isn't far from a few tweaks from vanilla).
What I'm hoping is, someone smarter than me will be able to point me in the right direction.
I'm going to try telling my devices to use an external DHCP server, effectively bypassing pfSense and see if that improves things.
Hey!
Like I told before I am also having the same problem since 22.05 on my NG-3100 without changing anything else on the configuration. I´ve also tested different settings with the DNS Resolver, but with no success. After all I´m now using a DNS Resolver installed on my NAS - System, wich is set up as DNS Server in the DHCP - Settings.
With this everything is fine and works like before. But I´d like to change the settings back to the pfSense as DNS Resolver and hope the error will be find.
Greetings,
Markus
-
So far so good with DNS servers issued via DHCP to client devices.
Simple things like playing audio via Amazon Echo works, no intermittent problems with websites that I know are up.
Fingers crossed this is a sufficient work around.
-
Hi! Many helpful posts here!
Just wanted to mention that I'm also seeing the intermittently slow resolution described above:
Loading of websites often require refreshes to either have the site name resolved or CDN for images or stylesheets. I'd like to emphasize the intermittent nature of the problem -- I have duckduckgo.com set as my default search engine (i.e. a very frequently visited site) and have gotten name resolution errors in the browser time and time again over the last weeks with no clear pattern for when it's happening.I have a Netgate 2100 and upgraded from version 22.01 to 22.05 a few weeks ago. The problem started with the upgrade. I had not made changes to the DNS Resolver settings before so the default of using the DNS servers given via DHCP on WAN was reflected on the front page with three servers listed, 127.0.0.1 being the first. Client devices were given the pfSense IP as their DNS Server.
To remedy the situation I tried adding CloudFlare's 1.1.1.1 and 1.0.0.1 as DNS servers in System > General Setup and subsequently unchecked "Allow DNS server list to be overridden by DHCP/PPP on WAN or remote OpenVPN server" but the problem persisted.
Based on replies in this thread, I checked "Serve Expired" on Services > DNS Resolver > Advanced Settings. The problem still occurs from time to time although seeingly less frequent. Resolution appears slow.
Further, I tried disabling DNSSEC (unchecked "Enable DNSSEC Support" in Services > DNS Resolver > General Settings) and disabled hardening of DNSSEC data (unchecked "Harden DNSSEC Data" in Services > DNS Resolver > Advanced Settings). Failures still occur.
To circumvent these problems I temporarily disabled the DNS Resolver.
I'll be watching this thread, hoping a solution pops up.
-
@kvhs said in Slow DNS after 22.05:
Hi! Many helpful posts here!
Just wanted to mention that I'm also seeing the intermittently slow resolution described above:
Loading of websites often require refreshes to either have the site name resolved or CDN for images or stylesheets. I'd like to emphasize the intermittent nature of the problem -- I have duckduckgo.com set as my default search engine (i.e. a very frequently visited site) and have gotten name resolution errors in the browser time and time again over the last weeks with no clear pattern for when it's happening.I have a Netgate 2100 and upgraded from version 22.01 to 22.05 a few weeks ago. The problem started with the upgrade. I had not made changes to the DNS Resolver settings before so the default of using the DNS servers given via DHCP on WAN was reflected on the front page with three servers listed, 127.0.0.1 being the first. Client devices were given the pfSense IP as their DNS Server.
To remedy the situation I tried adding CloudFlare's 1.1.1.1 and 1.0.0.1 as DNS servers in System > General Setup and subsequently unchecked "Allow DNS server list to be overridden by DHCP/PPP on WAN or remote OpenVPN server" but the problem persisted.
Based on replies in this thread, I checked "Serve Expired" on Services > DNS Resolver > Advanced Settings. The problem still occurs from time to time although seeingly less frequent. Resolution appears slow.
Further, I tried disabling DNSSEC (unchecked "Enable DNSSEC Support" in Services > DNS Resolver > General Settings) and disabled hardening of DNSSEC data (unchecked "Harden DNSSEC Data" in Services > DNS Resolver > Advanced Settings). Failures still occur.
To circumvent these problems I temporarily disabled the DNS Resolver.
I'll be watching this thread, hoping a solution pops up.
Following on from my original reply where it looked like restarting the service resolved... it didn't.
Just wanted to say I have had the same experience - tried many of the suggestions here. I have tried with the resolver/forwarder, with DNSSEC enabled/disabled. Tried pre-fetch keys, harden DNSSEC data.
I have given up with the slow or unresponsive DNS resolution since 22.05 and put my clients on Google DNS over TLS which is working perfectly.
Hopefully somebody can find a solution as I rather liked using the resolver on my SG2100.
-
In summary, my fixes have been stable.
- Enable Serve Expired -- this helped with CDN lookups. This was not set in 22.01
- Set minimum TTL to 300 seconds. This was not set in 22.01
- Disable Experimental 0x 20 support -- this was working in 22.01 but caused instability in 22.05.
So far things have been stable for over a week. I tried with and without pfBlocker-devel and various attempts to use forwarding or not (it was necessary while I was searching for a fix but I'm back to resolving locally again).
So yes, it seems "something has changed" but there's no smoking gun.
-
There are a number of bug fixes on Unbound since 1.15.0 which pfSense 22.05 uses, but I don't have enough knowledge of DNS to determine if those fixes are likely to fix these problems.
https://github.com/NLnetLabs/unbound/tags
I find this one solved in 1.16.0 interesting though: https://github.com/NLnetLabs/unbound/issues/670
-
Having this issue with an SG-6100 after going from 22.01 to 22.05 also. So far the Enable Serve Expired seems to be resolving the issue, but time will tell
-
Also seeing these intermittent DNS issues on my 5100 since updating to 22.05.
Haven't had a chance to troubleshoot yet but same issues outlined above.
Will try enabling Serve Expired tomorrow and see if that resolves.
