All DNS lookups extremely slow
-
We have 2.6.0 pfsense installed and we have DNS resolver enabled, DNSSEC Support enabled and Enable Forward Mode also checked. We have been seeing this problem of extremely latent DNS lookups for every webpage.
Do I possibly have a setting that isn't configured properly? Any suggestions would be appreciated.
-
@rollin1 said in All DNS lookups extremely slow:
DNSSEC Support enabled and Enable Forward Mode also checked
That is not a good setting - if your going to forward, dnssec should be off. Where you forward is either doing dnssec or they are not. That setting is going to be nothing but problematic.
-
"DNSSEC Support" I assume your upstream supports this and that it is not a new change. You should likely not need this.
"Enable Forward Mode"
How is your ping time to your upstream DNS?
also a traceroute?What other packages?
VPN ? is the upstream DNS on the other end of the VPN?What kind of response time are you getting from say
Diagnostics - > DNS Lookup
then compare to a dig from the command line (but lookup a different site)Then try a nslookup or dig or other (as appropriate) from some connected client?
-
@rollin1 here don't just believe us..
https://docs.quad9.net/Quad9_For_Organizations/DNS_Forwarder_Best_Practices/
Disable DNSSEC Validation
Since Quad9 already performs DNSSEC validation, DNSSEC being enabled in the forwarder will cause a duplication of the DNSSEC process, significantly reducing performance and potentially causing false BOGUS responses.
edit: side note, 2.6 is no longer supported you should really look to updating. 2.7.2 is the current CE release.
-
@johnpoz
I know i definitely need to upgrade this place.@jrey
Ping times to both DNS servers is 3-4ms. I also disabled DNSSEC. No additional packages are installed. DNS lookups are slow with 127.0.0.1 initially and then its 0ms once the site is cached.I also selected the specific interfaces LAN and localhost under DNS Resolver. I selected WAN under Outgoing Network Interfaces. I unchecked Enable DNSSEC Support and Enable Forwarding Mode. Under advanced resolver options I have prefetch DNS Key Support checked and Harden DNSSEC Data checked. I removed the check for Harden before rebooting the pfsense vm.
I am definitely planning to get the upgrade taken care of this evening.
Thanks for the help
-
@rollin1 said in All DNS lookups extremely slow:
I have prefetch DNS Key Support checked and Harden DNSSEC Data checked.
Why would you think you need dnssec stuff if your not doing dnssec.. Those could also be removed. But I don't that would a problem - but there is little point to having anything on for dnssec, if your not doing dnssec.
What do you think is a long time for a response for asking for something? If you forward to something say googledns just example and they don't have what your asking for, they have to resolve it.. So your saying your only 3-4 ms away from where your forwarding? Are these your isp nameservers?
or do you live down the street from a google data center if your using google ;)
Here is a query to something I normally don't go too.. This was resolved by unbound, I do not forward.
; <<>> DiG 9.16.45 <<>> www.cnn.com @192.168.9.253 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1355 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;www.cnn.com. IN A ;; ANSWER SECTION: www.cnn.com. 3600 IN CNAME cnn-tls.map.fastly.net. cnn-tls.map.fastly.net. 3600 IN A 146.75.79.5 ;; Query time: 59 msec ;; SERVER: 192.168.9.253#53(192.168.9.253) ;; WHEN: Wed Dec 13 16:06:04 Central Standard Time 2023 ;; MSG SIZE rcvd: 92So a whole 59ms.. that is 0.059 of a second..
Now sure once I have cached that answer its lower..
; <<>> DiG 9.16.45 <<>> www.cnn.com @192.168.9.253 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23739 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;www.cnn.com. IN A ;; ANSWER SECTION: www.cnn.com. 3523 IN CNAME cnn-tls.map.fastly.net. cnn-tls.map.fastly.net. 3523 IN A 146.75.79.5 ;; Query time: 1 msec ;; SERVER: 192.168.9.253#53(192.168.9.253) ;; WHEN: Wed Dec 13 16:07:21 Central Standard Time 2023 ;; MSG SIZE rcvd: 92Now if you were getting responses that took say 2000 ms or 2 seconds.. I would say you have a problem but even low hundred your talking say like 0.3 seconds for 300 ms..
Here is asking googledns directly
; <<>> DiG 9.16.45 <<>> www.cnn.com @8.8.8.8 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51779 ;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;www.cnn.com. IN A ;; ANSWER SECTION: www.cnn.com. 227 IN CNAME cnn-tls.map.fastly.net. cnn-tls.map.fastly.net. 29 IN A 151.101.3.5 cnn-tls.map.fastly.net. 29 IN A 151.101.67.5 cnn-tls.map.fastly.net. 29 IN A 151.101.131.5 cnn-tls.map.fastly.net. 29 IN A 151.101.195.5 ;; Query time: 20 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Wed Dec 13 16:11:23 Central Standard Time 2023 ;; MSG SIZE rcvd: 140Wow 0.04 of seconds faster, but look at the TTL, so now I have to go ask them again in 30 seconds.. or at best 227 seconds.. Which then again will be most likely some other random ttl lower than what the authoritative ns is.. So save .04 seconds, but will be doing more dns queries... Users do have their reasons for forwarding, hey on a real shitty high latency connection forwarding might be a better solution.. But I find it better to directly talk to the authoritative ns..
But lets see these queries that are so painfully slow ;)
Also are you forwarding just via normal forwarding in the clear, or are you forwarding via tls, ie dot?
-
I changed their configuration yesterday after the suggestions on here
DNS Resolver is enabled
I decided to no longer ForwardAdvanced Settings
Increased Message Cache to 250MB
Increased Number of hosts to cache to 200000Lookup Response Times
domain 1
1st lookup
127.0.0.1 30 msec
8.8.8.8 53 msec
8.8.4.4 3 msec2nd lookup
127.0.0.1 0 msec
8.8.8.8 52 msec
8.8.4.4 3 msecdomain 2
1st lookup
127.0.0.1 43 msec
8.8.8.8 14 msec
8.8.4.4 4 msec2nd lookup
127.0.0.1 0 msec
8.8.8.8 13 msec
8.8.4.4 3 msecThe domain can be cached in pfSense but from any workstation or server on the LAN its a 5-10 second delay before anything is displayed
-
@rollin1 So that a lan client doing those looks you posted? Once it is cached in unbound - the client should get a response in 1 or 2ms tops.. Or you have something on on your network between the client and pfsense.
Lets say 500 ms even for a dns query does not equate to a website taking 5-10 to display? What browser are you using? You sure its even using your local dns and not doh?
Fire up say in firefox the web developers tools and you can see exactly something takes or what is delay.. Sure chrome or edge has sim tools?
Are you blocking stuff and it keeps trying a retrans before it gives up?
5 to 10 seconds for something to load doesn't seem like slow dns to me - you got something else going on..
-
@johnpoz
So that a lan client doing those looks you posted?
This is a lan workstation, logged into pfsense via web gui using Diagnostics - DNS Lookup
Once it is cached in unbound - the client should get a response in 1 or 2ms tops - I completely agree with youOr you have something on on your network between the client and pfsense.
I don't have anything between the client and pfsense.Lets say 500 ms even for a dns query does not equate to a website taking 5-10 to display?
I am not saying that it translates I was really trying to review everything and hopefully with other users have more pfsense configuration experience than myselfWhat browser are you using?
I have used all of them - Chrome, Firefox, EdgeYou sure its even using your local dns and not doh?
I have pointed clients to local dns servers and to google dns servers and the issue still exists either wayFire up say in firefox the web developers tools and you can see exactly something takes or what is delay.. Sure chrome or edge has sim tools?
I will also check into this tonight/tomorrow morning using Firefox web developer toolsAre you blocking stuff and it keeps trying a retrans before it gives up?
No filtering or blocking is being done
No VPN's configured5 to 10 seconds for something to load doesn't seem like slow dns to me - you got something else going on..
I also patched the clients 2nd ESXi host today. Afterword, I deployed a fresh install of PFSense and I updated 2.6.0 to 2.7.0 successfully running a copy of clients config. The production pfsense vm is running on a host that needs ESXi to be patched, so pfsense 2.7.0 and newer will run properly. -
@rollin1 said in All DNS lookups extremely slow:
I have pointed clients to local dns servers and to google dns servers and the issue still exists either way
Browsers these days like to ignore your client settings and use doh anyway, unless you jump through hoops to disable the use of doh.
But if it is using what you set in the client, and you pointed it directly to say googledns and still slow, kind of points to something other than dns being the source of the slow page load.
-
@johnpoz said in All DNS lookups extremely slow:
pointed clients to local dns servers and to google dns servers
clients local dns AND google
-
@jrey said in All DNS lookups extremely slow:
clients local dns AND google
This is never a good setup anyway - since you really have no idea exactly which the client will use when you point to more than one NS.
-
This is never a good setup anyway
that was my point.
I, like you, would much rather see the actual "dig" or "NSLookup" directly from whatever the client is, not just the times. just the times doesn't really say what the response was or if there even was a valid response. Without knowing what "domain 1" or "domain 2" are or what might be actually loading (a webpage? something else?)
This really does not feel like a DNS "slowness" issue.
you got something else going on
^^ yup
after the suggestions on here
..
increased Message Cache to 250MB
Increased Number of hosts to cache to 200000did I miss something on here?
-
nslookup will use what 'system' (PC) uses.
An application like Firefox, out of the box, will completely bypass the system DNS, and goes directly to the outside DoH server, doing its DoH thing. The pfSense resolver will never see the requests. -
@jrey said in All DNS lookups extremely slow:
did I miss something on here?
you don't need to adjust those to be honest, the defaults should be fine.. And that sure has nothing to do with whatever issue your having.
-
did I miss something on here?
My question was more about the OP stated changing them in reference to something said "on here".
the defaults should be fine
I'm 100% in agreement with this
-
@jrey said in All DNS lookups extremely slow:
reference to something said "on here".
No idea where he would of seen that - I have sure never stated anything of the sort. And don't recall any one saying it, if I would of seen it I would of questioned it.
-
@johnpoz
The pfsense vm is running 2.6.0 right now and its on ESXi 6.0 which will be updated to the most recent patch for esxi 6.0 this week. The pfsense vm also has the vmxnet3 nic's which I know usually work better than the e1000. I have to get the firmware/bios updated on the host and then I plan to patch. I know the host is running extremely old esxi software but they have to continue to keep 4 NT4 servers running for their manufacturing.I think the settings I found for adjusting things under Advanced Settings were from another support forum. I mistakenly lumped them with the suggestions I got from this group, sorry about doing that.
Anyone else have any ideas after updating the host and the pfsense version to 2.7.2 so they are running current software
I also changed all of my advanced settings back to the defaults for the DB size and the number of entries in the DB.
-
@rollin1 bumping those settings to those values - you must be handling a shit ton of dns to a bunch of different clients all going to lots of different places ;)
If I had a need for dns that those settings would make sense, prob wouldn't be running it on my router ;) And would have a dedicated NS properly sized for my needs.
-
@johnpoz
Those settings were never something I adjusted in any other pfsense install that I have done but the latency is too significant to not notice. What do you think if I change all the settings to forward vs look everything up? I have honestly wondered through out this whole issue if the problem could be with the vm itself. I mean after the DNSSEC settings were changed the lookup times improved but they are definitely not normal. What do you think about the following steps- Update ESXI 6 - Client has been running esxi 6 with no patches for a while
A. Once patched to most recent patch for ESXi 6 I can then deploy a fresh 2.7.2 pfsense vm
B. After patches the new 2.7.2 pfsense vm will actually run and not power off
C. Import the clients config on the new vm and see how everything runs
Does anyone see any issue with doing this? I basically want to rule out anything with the existing pfsense vm and their configuration is very basic (no vpn's, no complex network)
- Update ESXI 6 - Client has been running esxi 6 with no patches for a while
