Frequent DNS timeouts
-
On my pfSense 23.01 and pfBlockerNG 3.2 network I have frequent yet seemingly random DNS timeouts. They are most notable in web browsers when accessing a domain that hasn't been accessed recently. The browser hangs doing DNS resolution and sometimes fails outright. This happens on multiple devices, operating systems, browsers, and applications.
A few days ago I deleted my pfBlockerNG configuration and reinstalled. Using the setup wizard I created the default configuration and left it as is. I'm still experiencing DNS timeouts.
I'm at a loss for how to troubleshoot this. Any suggestions would be very welcome.
(Note: for my personal browser I've disabled custom DNS resolution to ensure the browser is going through pfSense and not a third-party DNS provider. This doesn't seem to help though.)
-
@oopohj5oo8shieze1ree if I had to guess I would guess unbound is restarting a lot, when it restarts yeah dns isn't going to work.
Look in your log - is unbound restarting?
-
@oopohj5oo8shieze1ree Also if you are forwarding, ensure DNSSEC is unchecked.
-
@johnpoz I've looked in the general log file, pfBlockerNG log files, and the DNS resolver log file and I don't see unbound restarting.
I did notice a couple of these:
debug: outnettcp got tcp error -1And occasionally:
/rc.linkup: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1677281515] unbound[36369:0] error: bind: address already in use [1677281515] unbound[36369:0] fatal error: could not open ports' -
@steveits I'm not exactly sure what you mean by forwarding.
In the DNS resolver settings both DNSSEC and DNS Query Forwarding are turned on. But I'm not running the DNS Forwarding service.
Should I disable DNSSEC in the DNS Resolver settings, and is that safe to do so?
-
@oopohj5oo8shieze1ree said in Frequent DNS timeouts:
both DNSSEC and DNS Query Forwarding are turned on
Yep that's it. 23.01 seems more sensitive/has problems in that configuration. Uncheck the DNSSEC option. you're already trusting the DNS servers to which you forward.
For instance per https://support.quad9.net/hc/en-us/articles/4433380601229-Setup-pfSense-and-DNS-over-TLS
"DNSSEC is already enforced by Quad9, and enabling DNSSEC at the forwarder level can cause false DNSSEC failures" -
@steveits Thanks for the help.
I also read the pfSense documentation and came to the same conclusion. I've disabled DNSSEC. I'll report back after a couple of days whether or not my issue has been resolved.
-
@oopohj5oo8shieze1ree Here’s hoping. It did for me and several others so far, despite not being a problem in prior versions.
-
Unfortunately disabling DNSSEC has not fixed my issue. I'm still getting DNS timeouts from time to time :(
-
@oopohj5oo8shieze1ree There was a post today that 'Disabling "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers"' helped that person.
-
Also there's a fix for Unbound not correctly binding to "All" interfaces on IPv6.
https://forum.netgate.com/topic/176989/problems-with-pfsense-ipv6-dns-function-does-it-exist/36
-
@steveits Thanks for pointing me to the other threads.
I'm thinking of just giving up on using forwarding. I need to figure out if my ISP limits access to DNS servers when not forwarding.
-
After turning off DNS forwarding, resolution was nearly instantaneous for a couple of days. But the random timeouts have returned.
I don't see anything in the logs to indicate something is failing.
Can someone point me to a DNS debugging guide or something that will help me figure out what the root cause is here.
Thank you.
-
@oopohj5oo8shieze1ree There are a few here:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/index.html#dnsAlso take a look through https://forum.netgate.com/category/19/dhcp-and-dns as there are other posts for 23.01.
-
Hi. Do have nearly the same issues.
But for me I don't use and DNS forwarding or anything else. Just pfSense Unbound in combination with pfBlockerNG.
Don't have any DNS fails at all. But looks like name resolution does hang after some amount of time. After that it looks like it is cached again and resolution works fine.
But I do have this issues nearly every day.Something like a cleared unbound cache - what's not the case.
-
@thundergate said in Frequent DNS timeouts:
cleared unbound cache
This can only happens when the resolver -unbound is told to stop, or restart, which is a controlled stop, to be started right afterwards.
It can take several seconds to do so.
The cache will be lost, but subsequent DNS resolving won't take long, typical is a fraction of a second.If unbound restarts happen very often, you can start to 'feel' the absence of the DNS sub system.
So, ask your pfSense how often it restarts :
grep "Restart" /var/log/resolver.logIf it's just couple of times a day (lesser == better) : this is not your issue.
-
@steveits After switching from forwarding to normal resolving I let things sit for a bit to see what would happen. It looks like unbound is restarting a lot:
Mar 15 08:01:04 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1. Mar 15 08:08:33 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1. Mar 15 08:12:20 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1. Mar 15 08:12:45 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1. Mar 15 08:13:39 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1. Mar 15 08:29:45 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1. Mar 15 08:34:44 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1. Mar 15 08:35:41 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1. Mar 15 08:42:09 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1. Mar 15 08:42:34 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1. Mar 15 08:49:47 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1. Mar 15 08:52:13 netgate unbound[20724]: [20724:0] notice: Restart of unbound 1.17.1.Is there a known workaround for this?
-
@oopohj5oo8shieze1ree The most common cause for restarts is having DHCP set to register DHCP leases in DNS, which triggers a restart after each and every DHCP lease. Options are to not do that, or to make the lease long enough that it renews in "days" not "hours." (renewal is 1/2 of the lease duration)
-
@steveits I believe I have that turned off (in Services -> DHCP Server -> Dynamic DNS ->
Enable registration of DHCP client names in DNS). However, it does appear to be registering DHCP host names with the DNS server regardless of this setting.I've increased the lease time and will report back.
Thank you.
-
@oopohj5oo8shieze1ree unbound starting that often is going to be problematic that is for sure..
-
Same for me - a lot of unbound restarts and I actually don't know why?!
-
@thundergate well no restarts here
[23.01-RELEASE][admin@sg4860.local.lan]/root: unbound-control -c /var/unbound/unbound.conf status version: 1.17.1 verbosity: 1 threads: 4 modules: 2 [ validator iterator ] uptime: 196553 seconds options: control(ssl) unbound (pid 56217) is running... [23.01-RELEASE][admin@sg4860.local.lan]/root:196K seconds - 54 hours... Which was when I restarted pfsense to fix my swap not showing on widget..
If unbound is restarting - especially that often, your not going to have a good time.. You need to figure out why its restarting, registration of dhcp is typical reason you would see restarts like that..
-
-
@thundergate no - it has been a known issue for years that registering dhcp restarts unbound. I only register static mappings
-
@johnpoz said in Frequent DNS timeouts:
known issue for years that registering dhcp restarts unbound
-
@johnpoz
Oh no - That's stupid?!But I do need those DHCP leases to be seen to know what device does make all those requests.... Cannot stand with IP addresses only.
Used OPNsense before - but didn't had those issues, if I remember correctly?
-
@thundergate Then until resolved, as I noted above make your lease time longer. It will restart on average every ( (lease duration/2) / # leases ).
-
@steveits Will have to look into it.
I'm quite disappointed. Never thought that such an error does exist within pfSense (and it does exist since a few years now).
Are you all not interested in name resolution and do only handle IPs?
For me unbound restarts every 2-5 minutes (doesn't look like it is the DHCP lease issue at all?!).
-
@thundergate said in Frequent DNS timeouts:
re you all not interested in name resolution and do only handle IPs?
All of the devices I have that I need to resolve or want to resolve via name I have reserved IP for ;)
-
@thundergate said in Frequent DNS timeouts:
DHCP lease issue at all?!).
well look in your dhcp log - does it match up or not.. My leases are 4 days long.. But 2 hour lease, with lots of devices yeah you could have a few an hour for sure..
-
@thundergate said in Frequent DNS timeouts:
not interested in name resolution and do only handle IPs?
Depends on the setup. Clients with Windows domains use Windows DNS so it's handled. Windows in general/SMB will discover an address by NetBIOS name anyway. Printers get static or reservations. So in most cases it isn't really needed.
-
@steveits Main point where I do need it is within my pfBlockerNG logs to see what device is doing which requests and so on...
-
@thundergate said in Frequent DNS timeouts:
But I do need those DHCP leases to be seen to know what device does make all those requests....
DHCP will still work.
The only thing that doesn't happen anymore is that their, mostly stupid host names, like AZERFDGHH, and you know what that is : the doorbell, and 6 devices that all are called 'Android' and four 'iPad' and whatever, will pollute your DHCP leases list
If you really want to control, and start even to pretend that you want to know what devices belongs to who : "who is what and when" on your network, then gives them logical names, names you chose, and not the build in device name.
And, oh, before I forget : a lot of devices don't even give a host name to enter into the DNS, but the resolver will get restarted anyway ....
That issue is also solved .... by the pfSense admin, you, of course.So : take a first step : list all the MAC addresses, and give them all names that you understand.
At the end, pfSense will contain a list with all the devices that you , and names that you can easily remember.
On the device side, for every device : you have nothing to do, as most use DHCP out of the box.
The day you find a device that was using a IP out of the DHCP server pool, you know on the spot that you have a new device on your network.Static DHCP lease are read into the resolver unbound upon start and will not change. Except the day you add a new device to your network, and create the "MAC IP host name" for it.
[23.01-RELEASE][admin@pfSense.what-a-mess.tld]/root: unbound-control -c /var/unbound/unbound.conf status version: 1.17.1 verbosity: 1 threads: 2 modules: 3 [ python validator iterator ] uptime: 111205 seconds options: control(ssl) unbound (pid 24788) is running...That's a bit more as 3 days for me, when I was testing UPS shutdown procedure.
@thundergate said in Frequent DNS timeouts:
For me unbound restarts every 2-5 minutes (doesn't look like it is the DHCP lease issue at all?!).
Actually, I hope for you that this is your issue.
If it's not : entering into the light the other X reasons why unbound gets restarted :
You WAN, or LAN or any other interface is bad, goes up and down all the time.
This will restart unbound, and many other processes also.
Evey x minutes ..
Not a good thing.
Or unbound is plain 'bad' : less plausible, as me and you use the same code : days without restart is possible.
Another reason : it has been seen that people wanted to update their pfblockerng feeds every hours or so. If the any of these lists actually changed => unbound gets restarted.And then this example : remember that stupid doorbell mentioned above : it was to cheap, it had a broken dhcp client, it was asking a new lease every minute .... The pfsense admin was posting here, as he had checked "DCP registration" and did not look into the logs to see that that doorbell was asking a new lease every xx seconds.
Also : people don't feel or notice radio waves. Device do, as they need it for the wifi connection. When the device is at the edge of reach ability, the link gets set going up and down every x seconds. On every linkup, a dhpc request is fired. Your phone has now become a pfsense unbound killer.
Now you know why the DHCP registration is, by default, not checked.
Now you now (parts) of what need to be checked once in a while, before you check it.I was hoping for a more permanent solution, years ago.
I'm not waiting any more I solved the issue for me, on my side. And DNS rocks, for me.No "help me" PM's please. Use the forum.
-
@gertjan said in Frequent DNS timeouts:
did not look into the logs to see that that doorbell was asking a new lease every xx seconds.
I have seen this - client just asks and asks and asks.. Even when they just got a lease good for hours or even days.
-
Thx for all the feedback.
I did turn of register DHCP leases and will now start to add them by myself. As far as I understand it's a 'one time job' and then the client does have a static lease/IP and that's it?!
-
@thundergate yup set it and shouldn't have to touch it again unless you want that device to have a different IP, or you want to hand out something specific to that device different than your normal scope etc..
Look at this POS device
Mar 16 01:38:52 dhcpd 93450 DHCPACK on 192.168.2.203 to 88:b2:91:98:d6:f0 via igb2 Mar 16 01:38:52 dhcpd 93450 DHCPREQUEST for 192.168.2.203 from 88:b2:91:98:d6:f0 via igb2 Mar 16 01:37:41 dhcpd 93450 DHCPACK on 192.168.2.203 to 88:b2:91:98:d6:f0 via igb2 Mar 16 01:37:41 dhcpd 93450 DHCPREQUEST for 192.168.2.203 from 88:b2:91:98:d6:f0 via igb2 Mar 16 01:31:44 dhcpd 93450 DHCPACK on 192.168.2.203 to 88:b2:91:98:d6:f0 via igb2 Mar 16 01:31:44 dhcpd 93450 DHCPREQUEST for 192.168.2.203 from 88:b2:91:98:d6:f0 via igb2 Mar 16 01:30:01 dhcpd 93450 DHCPACK on 192.168.2.203 to 88:b2:91:98:d6:f0 via igb2 Mar 16 01:30:01 dhcpd 93450 DHCPREQUEST for 192.168.2.203 from 88:b2:91:98:d6:f0 via igb2 Mar 16 01:29:20 dhcpd 93450 DHCPACK on 192.168.2.203 to 88:b2:91:98:d6:f0 via igb2 Mar 16 01:29:20 dhcpd 93450 DHCPREQUEST for 192.168.2.203 from 88:b2:91:98:d6:f0 via igb2 Mar 16 01:19:00 dhcpd 93450 DHCPACK on 192.168.2.203 to 88:b2:91:98:d6:f0 via igb2 Mar 16 01:19:00 dhcpd 93450 DHCPREQUEST for 192.168.2.203 from 88:b2:91:98:d6:f0 via igb2 Mar 16 01:18:23 dhcpd 93450 DHCPACK on 192.168.2.203 to 88:b2:91:98:d6:f0 via igb2 Mar 16 01:18:23 dhcpd 93450 DHCPREQUEST for 192.168.2.203 from 88:b2:91:98:d6:f0 via igb2 Mar 16 01:17:49 dhcpd 93450 DHCPACK on 192.168.2.203 to 88:b2:91:98:d6:f0 via igb2 Mar 16 01:17:49 dhcpd 93450 DHCPREQUEST for 192.168.2.203 from 88:b2:91:98:d6:f0 via igb2 Mar 16 01:13:43 dhcpd 93450 DHCPACK on 192.168.2.203 to 88:b2:91:98:d6:f0 via igb2 Mar 16 01:13:43 dhcpd 93450 DHCPREQUEST for 192.168.2.203 from 88:b2:91:98:d6:f0 via igb2 Mar 16 01:11:04 dhcpd 93450 DHCPACK on 192.168.2.203 to 88:b2:91:98:d6:f0 via igb2 Mar 16 01:11:04 dhcpd 93450 DHCPREQUEST for 192.168.2.203 from 88:b2:91:98:d6:f0 via igb2Thats my wife's shitty iphone, charging..
You have some device doing that - going to cause unbound to go crazy restarting like that..
A reservation doesn't stop them from asking.. But you can not resolve it, and not have to worry about registering dhcp dynamic clients in unbound.
I really should prob get the wifi just to turn off her wifi when she is charging it... I looked and it did it last night as well..
edit: looking at my wifi log, her phone is roaming between 2 different APs it keeps flipping back and forth - this is what is most likely causing the dhcp - maybe I can get here to move where she is charging it but the rssi shouldn't be switching between ap like that..
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 23.01 | Lab VMs CE 2.6, 2.7 -
@johnpoz said in Frequent DNS timeouts:
....
edit: looking at my wifi log, her phone is roaming between 2 different APs it keeps flipping back and forth - this is what is most likely causing the dhcp - maybe I can get here to move where she is charging it but the rssi shouldn't be switching between ap like that..Bigger issues are on the horizon.
iPhone 'decides' to backup their content "when they are charging, have wifi, feel happy, and who knows what other criteria have to be met". That is, when you have the 1 $/€ monthly Apple backup plan, which permits you to restore on a new iPhone with one click - no messages photos ( ! ) apps and settings lost if something happens with the current one.
Believe me, this 1$ solution is way better as what a lawyer will ask you ;)The wifi hopping : true : to much wifi is killing the wifi.
She could disable the "auto connect" on all overlapping home wifi SSID's except for one and you DHCP issue will be solved.Btw : here, where I work, I've 4 AP's using the same SSID, as its the wifi access with a captive portal for our hotel. I see this hopping a lot, as people tend to move in the building.
Our captive portal has its own network and its own DHCP server.
And don't want to see what their news are, as, for me, it's a non trusted network@thundergate said in Frequent DNS timeouts:
As far as I understand it's a 'one time job' and then the client does have a static lease/IP and that's it?!
Exact.
The device will say : "he, I'm aa:bb:cc:dd:ee:ff and do you have an IP for me" and pfSense will hand over the IP you've selected for it. And not an IP from the DHCP pool.
Most device will even tell ask for that same IP in the future.
Nice side effect : you will know from now on that your NAS has 192.168.1.10 from now on.
And unbound doesn't get restarted.No "help me" PM's please. Use the forum.
-
@gertjan said in Frequent DNS timeouts:
on all overlapping home wifi SSID's except for one and you DHCP issue will be solved.
She is not jumping ssids.. she is moving from 1 AP to another one.. From looking she is right at the cusp of the min rssi I had set.. Tmrw I will put the developers tool on her phone so I can see what she is seeing for the signal strengh.. But I bumped the min rssi a few dbm and it seems to have settled in to 1 AP now.
And it settles down after a bit.. I am not having any issues, I just noticed my wifes phone doing that and thought it was a perfect example what could cause unbound to restart if your registering dhcp, which I am not.. And here phone has a reservation..
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 23.01 | Lab VMs CE 2.6, 2.7 -
@johnpoz said in Frequent DNS timeouts:
She is not jumping ssids.
I understood that. The device is hopping around as the current SSID becomes less good as the surrounding available SSIDs, already known, so it hops over.
And the process repeats.No "help me" PM's please. Use the forum.
-
@gertjan it stop doing it once I changed the min rssi from -67 to -73
