Unbound seems to be restarting frequently
-
Even if it's something malicious, unbound shouldn't reboot all the time. lol
-
I am also seeing this issue with unbound restarting frequently on a clean 2.2 install. It is triggered by DHCPREQUEST events in dhcpd, but only when a host name is specified in the request. For example, a request to dhcpd like this:
Mar 2 15:43:41 dhcpd: DHCPREQUEST for 192.168.1.208 from xx:xx:xx:xx:xx:xx (ipcam_xxxxxxxxxxxx) via em1
…always results in a restart of unbound at the same exact time:
Mar 2 15:43:41 unbound: [55204:0] info: service stopped (unbound 1.5.1).
(followed by a bunch of info logging and then…)
Mar 2 15:43:41 unbound: [55204:0] notice: Restart of unbound 1.5.1.
Mar 2 15:43:41 unbound: [55204:0] notice: init module 0: iterator
Mar 2 15:43:41 unbound: [55204:0] info: start of service (unbound 1.5.1).However, a DHCPREQUEST without a host name does not trigger the restart (also this is a static mapping, the one above is from the DHCP pool):
Mar 2 15:47:46 dhcpd: DHCPREQUEST for 192.168.1.11 from xx:xx:xx:xx:xx:xx via em1
I do not have "DNS Query Forwarding" enabled in the unbound configuration. After reading this thread, I tried disabling DNSSEC and it had no effect on the restarting behavior. I do have "Register DHCP leases in the DNS Resolver" and "Register DHCP static mappings in the DNS Resolver" enabled which leads me to believe the problem lies in the DHCP -> DNS registration functionality.
I would not normally notice this behavior except that there are times where the frequent restarting of unbound sometimes seems to cause unbound to stop resolving DNS queries properly. This usually happens when there are a lot of devices going on or offline and requesting DHCP leases. A manual restart of unbound in the web GUI fixes it, although the restart behavior continues whenever a DHCPREQUEST with hostname is received.
-
I think people were seeing crashes when they had devices (such as cameras or laptops) that have both a wired and wireless connection and both were connected simultaneously and shouldn't have been…
You sure you don't have something like that going on?
-
It is unlikely that is the case with my installation. Of the wireless devices I have on a wired connection, most don't even have the wireless SSID configured. Also, my AP and all wireless devices are on a separate subnet, so anything on the 192.168.1 subnet is 100% wired.
-
For me, the reboots stopped when I removed the DNS settings from System -> General Setup.
Interestingly, I also have DNS Registration and Static DHCP enabled in resolver.. For the time being, I'm switching back to dnsmasq while I can sort it out.
-
My logs (and resultant DNS problems) indicate that previously known MAC addresses and DHCP leases frequently trigger a restart, although not necessarily 100% of the time. I found that I can reproduce the unbound restart behavior 100% of the time by bringing a machine online with a new (to the firewall) MAC and having it initiate a DHCP request. This is on a wired machine with no wireless card. With this, I ran a few testing scenarios:
Register DHCP leases in the DNS Resolver - enabled
Register DHCP static mappings in the DNS Resolver - enabled
Result when DHCP is requested - unbound restartSo I disabled both:
Register DHCP leases in the DNS Resolver - disabled
Register DHCP static mappings in the DNS Resolver - disabled
Result when DHCP is requested - NO unbound restartI enabled static only:
Register DHCP leases in the DNS Resolver - disabled
Register DHCP static mappings in the DNS Resolver - enabled
Result when DHCP is requested - NO unbound restartI went back to the original config with both enabled:
Register DHCP leases in the DNS Resolver - enabled
Register DHCP static mappings in the DNS Resolver - enabled
Result when DHCP is requested - unbound restartThis lends some weight to the theory that the restarts are related to the "Register DHCP leases in the DNS Resolver" setting. However, one amendment to my previous post is that my testing scenario does not set a host name, yet it still triggers a restart. This is the dhcpd log that will reliably trigger unbound to restart when Register DHCP in DNS is enabled and the MAC address is not previously known to the firewall (newest on top):
Mar 2 17:40:42 dhcpd: DHCPACK on 192.168.200.212 to xx:xx:xx:xx:xx:xx via em3
Mar 2 17:40:42 dhcpd: DHCPREQUEST for 192.168.200.212 (192.168.200.1) from xx:xx:xx:xx:xx:xx via em3
Mar 2 17:40:42 dhcpd: DHCPOFFER on 192.168.200.212 to xx:xx:xx:xx:xx:xx via em3
Mar 2 17:40:41 dhcpd: DHCPDISCOVER from xx:xx:xx:xx:xx:xx via em3 -
This sounds like a bug to me.
-
So your saying unbound restarts on a dhcp request… Yeah I have not seen this, but when I get home I will check.. If a bug then it would be very simple to duplicate.
-
With the proper settings you can break anything.
-
Well I know for sure I have both those options enabled and have not seen or atleast noticed unbound restarting.. And I would assume I would see lots of dhcp requests. When I get home I will take a double look and force a request, etc.
To be honest if this was a real bug you would think the forums would be blowing up with this happening.
-
For me it was configuring DNS servers in System -> General Setup, regardless if I had Forwarding Mode disabled/enabled. Simply having external servers caused the restarts for me.
dparring, what are your System -> General Setup DNS settings? If you remove them and only use local host, do the reboots stop?
-
CiscoKid85 is on to something. I had Google DNS configured under general settings (8.8.8.8/8.8.4.4). When I cleared those entries, saved, and redid the test, DHCP did NOT cause unbound to restart. Even stranger is that when I put the entries back in, the test no longer causes unbound to restart. However, when I ran my previous tests again, it started happening after I reanabled "Register DHCP leases in the DNS Resolver". Here is the sequence:
New DHCP causes unbound restart
+Removed 8.8.8.8/8.8.4.4 from general settings DNS
New DHCP does NOT cause unbound restart
+Added 8.8.8.8/8.8.4.4 back to general settings
New DHCP does NOT cause unbound restart
+Rebooted pfSense
New DHCP does NOT cause unbound restart
+Unchecked "Register DHCP leases in the DNS Resolver"
New DHCP does NOT cause unbound restart
+Checked "Register DHCP leases in the DNS Resolver"
New DHCP causes unbound restart
+Removed 8.8.8.8/8.8.4.4 from general settings DNS
New DHCP does NOT cause unbound restart
+Unchecked "Register DHCP leases in the DNS Resolver"
New DHCP does NOT cause unbound restart
+Checked "Register DHCP leases in the DNS Resolver"
New DHCP causes unbound restartConfusing right? If I had to guess, there is a bug related to enabling "Register DHCP leases in the DNS Resolver" that causes unbound to restart when dhcpd issues leases, and that making other changes to the DNS system somehow fixes it. There is another thread on the forum about unbound restarts where people seem to have resolved it by playing with similar settings, so it makes sense that this is only happening in certain difficult to reproduce scenarios. Also, unless you're digging in the logs or experiencing DNS outages, most people wouldn't even notice this is happening. I'd be interested to see if anyone else can produce similar findings by enabling and disabling the "Register DHCP leases in the DNS Resolver" setting.
-
One more data point as a direct continuation from the sequence above:
+Left general settings exactly the same (dns servers are blank) and hit save
New DHCP does NOT cause unbound restartMy best guess of the bug at this point:
Enabling "Register DHCP leases in the DNS Resolver" causes it
Something on the general settings page save sequence fixes it (logs indicate the save kicks off a dhcpd and unbound restart, possibly other actions) -
Even more craziness when I cleared out some of the old DHCP leases from my testing earlier:
+Deleted leases using the delete button in DHCP Leases page (this seems to trigger a dhcpd and unbound restart with each delete request)
New DHCP causes unbound restart
+Left general settings exactly the same (dns servers are blank) and hit save
New DHCP does NOT cause unbound restart -
Well dhcp doesn't seem to be restarting mine.. But now that take a closer look it does seem to have restarted a few time when seems odd that it did. But must not be happening enough for me to notice. Last time I looked at the log didn't see any craziness there.. But now there is more restarts than you would think should be there.
If its a combinations of things, and something removes the issue like save or no dns in general, etc. Then sure that makes sense that less people would see it that had the specific settings and combinations of things.
Will keep an eye on it more, have not noticed any issue with resolving anything. But does seem to have been restarting more than it should..
Mar 3 04:15:10 unbound: [26324:0] notice: Restart of unbound 1.5.1.
Mar 3 04:03:19 unbound: [26324:0] notice: Restart of unbound 1.5.1.
Mar 3 03:25:24 unbound: [26324:0] notice: Restart of unbound 1.5.1.
Mar 3 03:05:42 unbound: [26324:0] notice: Restart of unbound 1.5.1.
Mar 3 01:33:22 unbound: [26324:0] notice: Restart of unbound 1.5.1.
Mar 3 01:10:28 unbound: [26324:0] notice: Restart of unbound 1.5.1.
Mar 3 00:36:41 unbound: [26324:0] notice: Restart of unbound 1.5.1.
Mar 3 00:12:51 unbound: [26324:0] notice: Restart of unbound 1.5.1.If I look in the dhcp log there is lots of dhcp stuff going on with renews and such at 2 in the morning but no restart that matches up to then, etc. I don't see any dhcp traffic that matches up with these restart times.
-
Hmmm - Strange. Mine is also often showing "notice: Restart of unbound" now that I take a closer look, but I'm not noticing any performance issues.
-
When I upgraded to the current 2.2 release from the previous version I switched to using unbound but I've been finding that it stops working ever now and then - suddenly nothing on the network resolves. Manually restarting unbound fixes this … until the next time. I've just switched back to dnsmasq on my work system and rebooted in the hope that this will fix it.
I'm seeing this both on my work system and on my home firewall too. Looking at the status display it seems that unbound is still working - it doesn't show up as stopped, it just doesn't work. Sorry if these notes aren't very helpful but there does seem to be an issue here.
Both firewalls are pretty much vanilla systems, different hardware (DELL and Netgate) with similar configurations. The only non-standard thing about them both is that I have two WAN connections on each machine - other than that they are pretty boring configurations.
-
edmund, that is basically the same behavior I see. Unbound restarts frequently but it generally doesn't affect anything; only occasionally does it stop resolving. When that happens, the service shows as running but it just doesn't resolve properly. My nagios monitor reports it as "DNS CRITICAL - 0.129 seconds response time (No ANSWER SECTION found)" when it happens. A manual restart via the webgui, or even just waiting for unbound to restart itself will fix it. I don't know if this is directly related to the DHCP bug or if it is just a consequence of the service restarting so frequently. I'm running pfSense in a VM on ESXi with a single WAN connection.
Another followup to my testing from yesterday: Unbound eventually resumed its restart behavior a few hours after I had "solved" it by pushing the save button on the general settings page. According to the logs, it looked like it resumed after dhcp did a routine write of the leases file to disk. I still haven't changed any settings in the system 20 hours later, and unbound is restarting on some, but not all, DHCPREQUEST events. Interestingly, in the current system state my test scenario (new MAC requesting DHCP) currently doesn't trigger an unbound restart like it did before. So hitting save under general settings isn't a perfect fix but it seems to get it into a slightly more reliable state.
This is my best understanding of the issue so far:
- Enabling "Register DHCP leases in the DNS Resolver" reliably puts unbound into a state where it restarts on brand new DHCP leases
- Pressing the save button on the general settings screen seems to stop unbound from restarting on DHCP requests for a short time
- However, unbound still manages to get into a slightly unstable state in the course of normal dhcpd activities, possibly precipitated by dhcpd writing to the leases file
-
Just checking… did you did turn on "Harden Glue" and "Harden DNSSEC data"?
There have been a couple threads about Unbound ceasing to resolve if these were not enabled.
When I upgraded to the current 2.2 release from the previous version I switched to using unbound but I've been finding that it stops working ever now and then - suddenly nothing on the network resolves. Manually restarting unbound fixes this … until the next time. I've just switched back to dnsmasq on my work system and rebooted in the hope that this will fix it.
-
Just checking… did you did turn on "Harden Glue" and "Harden DNSSEC data"?
No - both unchecked, my general philosophy is not to check boxes unless there's a good reason and I didn't think that either of these were relevant. So unbound was running with the defaults. They get rather upset at work if the resolver goes walkabout so I'll leave unbound disabled here and see what the home configuration is doing when I get home tonight.
