Unbound frequently restarts on 2.2 - is this normal?
-
While troubleshooting another issue, I discovered frequent restarts (every hour) in the unbound log. Is this normal or is something wrong?
Feb 5 15:50:06 unbound: [13647:0] info: start of service (unbound 1.5.1).
Feb 5 15:50:06 unbound: [13647:0] notice: init module 0: iterator
Feb 5 15:50:06 unbound: [13647:0] notice: Restart of unbound 1.5.1.Feb 5 16:50:30 unbound: [13647:0] info: start of service (unbound 1.5.1).
Feb 5 16:50:30 unbound: [13647:0] notice: init module 0: iterator
Feb 5 16:50:30 unbound: [13647:0] notice: Restart of unbound 1.5.1.Thanks,
Marco2.2-RELEASE (i386)
nanobsd (2g) -
Check the system log for what's going on at the time when unbound gets restarted. Like, new DHCP lease on WAN.
-
Like, new DHCP lease on WAN.
Something is up with 2.2 and DHCP. I'm getting all kinds of DHCP resets that I never got before 2.2. It's causing all kinds of issues with apinger and unbound; typically manifested as me "loosing" internet connectivity for 3-5 minutes.
I've not been able to isolate what the heck is going on as there isn't much in the logs, but the common thread is DHCP address renewal (my WAN IP address hasn't changed since I initially got my FIOS connection). I don't know what is different but it's really been frustrating trying to nail down where the blame lies. I think most of it is with apinger which I'm about ready to just turn off, which is a shame since I like some of the RRD reports it contributes to. But I like stable internet even more :p
-
I have the same problem with unbound. There seems to be a direct connection between unbound stopping and restarting and DHCP leases. At 8:56:11 the unbound service stopped and then restarted:
Feb 11 08:56:11 unbound: [16927:0] info: start of service (unbound 1.5.1). Feb 11 08:56:11 unbound: [16927:0] notice: init module 1: iterator Feb 11 08:56:11 unbound: [16927:0] notice: init module 0: validator Feb 11 08:56:11 unbound: [16927:0] notice: Restart of unbound 1.5.1. Feb 11 08:56:11 unbound: [16927:0] info: 0.262144 0.524288 1 Feb 11 08:56:11 unbound: [16927:0] info: 0.065536 0.131072 1 Feb 11 08:56:11 unbound: [16927:0] info: 0.032768 0.065536 9 Feb 11 08:56:11 unbound: [16927:0] info: 0.016384 0.032768 2 Feb 11 08:56:11 unbound: [16927:0] info: 0.008192 0.016384 1 Feb 11 08:56:11 unbound: [16927:0] info: 0.004096 0.008192 1 Feb 11 08:56:11 unbound: [16927:0] info: 0.002048 0.004096 3 Feb 11 08:56:11 unbound: [16927:0] info: 0.001024 0.002048 1 Feb 11 08:56:11 unbound: [16927:0] info: 0.000000 0.000001 1 Feb 11 08:56:11 unbound: [16927:0] info: lower(secs) upper(secs) recursions Feb 11 08:56:11 unbound: [16927:0] info: [25%]=0.004096 median[50%]=0.0364089 [75%]=0.0546133 Feb 11 08:56:11 unbound: [16927:0] info: histogram of recursion processing times Feb 11 08:56:11 unbound: [16927:0] info: average recursion processing time 0.044537 sec Feb 11 08:56:11 unbound: [16927:0] info: server stats for thread 1: requestlist max 2 avg 0.3 exceeded 0 jostled 0 Feb 11 08:56:11 unbound: [16927:0] info: server stats for thread 1: 111 queries, 91 answers from cache, 20 recursions, 0 prefetch Feb 11 08:56:11 unbound: [16927:0] info: 0.524288 1.000000 1 Feb 11 08:56:11 unbound: [16927:0] info: 0.262144 0.524288 2 Feb 11 08:56:11 unbound: [16927:0] info: 0.131072 0.262144 1 Feb 11 08:56:11 unbound: [16927:0] info: 0.065536 0.131072 2 Feb 11 08:56:11 unbound: [16927:0] info: 0.032768 0.065536 6 Feb 11 08:56:11 unbound: [16927:0] info: 0.016384 0.032768 6 Feb 11 08:56:11 unbound: [16927:0] info: 0.008192 0.016384 1 Feb 11 08:56:11 unbound: [16927:0] info: 0.004096 0.008192 5 Feb 11 08:56:11 unbound: [16927:0] info: 0.002048 0.004096 4 Feb 11 08:56:11 unbound: [16927:0] info: lower(secs) upper(secs) recursions Feb 11 08:56:11 unbound: [16927:0] info: [25%]=0.0065536 median[50%]=0.0273067 [75%]=0.0600747 Feb 11 08:56:11 unbound: [16927:0] info: histogram of recursion processing times Feb 11 08:56:11 unbound: [16927:0] info: average recursion processing time 0.082688 sec Feb 11 08:56:11 unbound: [16927:0] info: server stats for thread 0: requestlist max 3 avg 0.857143 exceeded 0 jostled 0 Feb 11 08:56:11 unbound: [16927:0] info: server stats for thread 0: 136 queries, 108 answers from cache, 28 recursions, 0 prefetch Feb 11 08:56:11 unbound: [16927:0] info: service stopped (unbound 1.5.1).
At that exact time i get this in DHCP logs:
Feb 11 08:56:11 dhcpd: DHCPACK on 192.168.1.15 to 1c:af:f7:xx:xx:xx (DIR-600) via em0
Feb 11 08:56:11 dhcpd: DHCPREQUEST for 192.168.1.15 from 1c:af:f7:xx:xx:xx (DIR-600) via em0
Feb 11 08:56:11 dhcpd: Wrote 9 leases to leases file.
Feb 11 08:56:11 dhcpd: Wrote 0 new dynamic host decls to leases file.
Feb 11 08:56:11 dhcpd: Wrote 0 deleted host decls to leases file.Again at 09:57:00 unbound service restarts:
Feb 11 09:57:00 unbound: [16927:0] info: start of service (unbound 1.5.1). Feb 11 09:57:00 unbound: [16927:0] notice: init module 1: iterator Feb 11 09:57:00 unbound: [16927:0] notice: init module 0: validator Feb 11 09:57:00 unbound: [16927:0] notice: Restart of unbound 1.5.1. Feb 11 09:57:00 unbound: [16927:0] info: 0.524288 1.000000 1 Feb 11 09:57:00 unbound: [16927:0] info: 0.262144 0.524288 6
Corelated with this DHCP log:
Feb 11 09:57:00 router dhcpd: DHCPOFFER on 192.168.1.17 to 78:dd:08:xx:xx:xx (vaio) via em0 Feb 11 09:57:00 router dhcpd: Wrote 0 deleted host decls to leases file. Feb 11 09:57:00 router dhcpd: Wrote 0 new dynamic host decls to leases file. Feb 11 09:57:00 router dhcpd: Wrote 9 leases to leases file. Feb 11 09:57:00 router dhcpd: DHCPREQUEST for 192.168.1.17 (192.168.1.1) from 78:dd:08:xx:xx:xx (vaio) via em0 Feb 11 09:57:00 router dhcpd: DHCPACK on 192.168.1.17 to 78:dd:08:xx:xx:xx (vaio) via em0 Feb 11 09:57:00 router dhcpd: Request message from fe80::dc38:d6fa:xxx:xxx port 546, transaction ID 0x6B31FC00 Feb 11 09:57:00 router dhcpd: Sending Reply to fe80::dc38:d6fa:xxx:xxx port 546
And yet again at 10:57:17:
Feb 11 10:57:17 unbound: [16927:0] info: start of service (unbound 1.5.1). Feb 11 10:57:17 unbound: [16927:0] notice: init module 1: iterator Feb 11 10:57:17 unbound: [16927:0] notice: init module 0: validator Feb 11 10:57:17 unbound: [16927:0] notice: Restart of unbound 1.5.1. Feb 11 10:57:17 unbound: [16927:0] info: 2.000000 4.000000 1 Feb 11 10:57:17 unbound: [16927:0] info: 1.000000 2.000000 1 Feb 11 10:57:17 unbound: [16927:0] info: 0.524288 1.000000 4 Feb 11 10:57:17 unbound: [16927:0] info: 0.262144 0.524288 8 Feb 11 10:57:17 unbound: [16927:0] info: 0.131072 0.262144 9 Feb 11 10:57:17 unbound: [16927:0] info: 0.065536 0.131072 12 Feb 11 10:57:17 unbound: [16927:0] info: 0.032768 0.065536 23 Feb 11 10:57:17 unbound: [16927:0] info: 0.016384 0.032768 43 Feb 11 10:57:17 unbound: [16927:0] info: 0.008192 0.016384 41 Feb 11 10:57:17 unbound: [16927:0] info: 0.004096 0.008192 57 Feb 11 10:57:17 unbound: [16927:0] info: 0.002048 0.004096 43 Feb 11 10:57:17 unbound: [16927:0] info: 0.001024 0.002048 9 Feb 11 10:57:17 unbound: [16927:0] info: 0.000512 0.001024 1 Feb 11 10:57:17 unbound: [16927:0] info: 0.000000 0.000001 19 Feb 11 10:57:17 unbound: [16927:0] info: lower(secs) upper(secs) recursions Feb 11 10:57:17 unbound: [16927:0] info: [25%]=0.00389358 median[50%]=0.00949073 [75%]=0.029053 Feb 11 10:57:17 unbound: [16927:0] info: histogram of recursion processing times Feb 11 10:57:17 unbound: [16927:0] info: average recursion processing time 0.054781 sec Feb 11 10:57:17 unbound: [16927:0] info: server stats for thread 1: requestlist max 4 avg 0.231618 exceeded 0 jostled 0 Feb 11 10:57:17 unbound: [16927:0] info: server stats for thread 1: 329 queries, 57 answers from cache, 272 recursions, 0 prefetch Feb 11 10:57:17 unbound: [16927:0] info: 1.000000 2.000000 2 Feb 11 10:57:17 unbound: [16927:0] info: 0.524288 1.000000 9 Feb 11 10:57:17 unbound: [16927:0] info: 0.262144 0.524288 19 Feb 11 10:57:17 unbound: [16927:0] info: 0.131072 0.262144 23 Feb 11 10:57:17 unbound: [16927:0] info: 0.065536 0.131072 24 Feb 11 10:57:17 unbound: [16927:0] info: 0.032768 0.065536 44 Feb 11 10:57:17 unbound: [16927:0] info: 0.016384 0.032768 68 Feb 11 10:57:17 unbound: [16927:0] info: 0.008192 0.016384 66 Feb 11 10:57:17 unbound: [16927:0] info: 0.004096 0.008192 95 Feb 11 10:57:17 unbound: [16927:0] info: 0.002048 0.004096 45 Feb 11 10:57:17 unbound: [16927:0] info: 0.001024 0.002048 12 Feb 11 10:57:17 unbound: [16927:0] info: 0.000512 0.001024 1 Feb 11 10:57:17 unbound: [16927:0] info: 0.000000 0.000001 20 Feb 11 10:57:17 unbound: [16927:0] info: lower(secs) upper(secs) recursions Feb 11 10:57:17 unbound: [16927:0] info: [25%]=0.00534636 median[50%]=0.013281 [75%]=0.0431942 Feb 11 10:57:17 unbound: [16927:0] info: histogram of recursion processing times Feb 11 10:57:17 unbound: [16927:0] info: average recursion processing time 0.066407 sec Feb 11 10:57:17 unbound: [16927:0] info: server stats for thread 0: requestlist max 4 avg 0.441589 exceeded 0 jostled 0 Feb 11 10:57:17 unbound: [16927:0] info: server stats for thread 0: 516 queries, 88 answers from cache, 428 recursions, 0 prefetch Feb 11 10:57:17 unbound: [16927:0] info: service stopped (unbound 1.5.1).
DHCP logs:
Feb 11 10:57:17 dhcpd: Sending Reply to fe80::1863:4b13:xxxx:xxxx port 546 Feb 11 10:57:17 dhcpd: Confirm message from fe80::1863:4b13:xxxx:xxxx port 546, transaction ID 0x51830100 Feb 11 10:57:17 dhcpd: DHCPACK on 192.168.1.12 to 78:7e:61:xx:xx:xx (Anamas-iPhone) via em0 Feb 11 10:57:17 dhcpd: DHCPREQUEST for 192.168.1.12 from 78:7e:61:xx:xx:xx (Anamas-iPhone) via em0 Feb 11 10:57:17 dhcpd: Wrote 9 leases to leases file. Feb 11 10:57:17 dhcpd: Wrote 0 new dynamic host decls to leases file. Feb 11 10:57:17 dhcpd: Wrote 0 deleted host decls to leases file.
-
Do you have a cron job rc.backup_dhcpleases.sh that is running every hour?
It must be over-enthusiastic and restart unbound when the lease file is written. Perhaps it can just send unbound a message? Or perhaps it does not need to tell unbound anything - unbound should already be getting DHCP lease data on-the-fly as it happens?
-
Nope…
$ crontab -l
crontab: no crontab for root -
crontab -l
That does not work for me either.
cat /etc/crontab
and see what is there.
Or instal the Cron package and you an see the cron jobs in the webGUI. -
$ cat /etc/crontab
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin
HOME=/var/log
#minute hour mday month wday who commandpfSense specific crontab entries
Created: February 8, 2015, 6:13 pm
1,31 0-5 * * * root /usr/bin/nice -n20 adjkerntz -a
1 3 1 * * root /usr/bin/nice -n20 /etc/rc.update_bogons.sh
*/60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 sshlockout
1 1 * * * root /usr/bin/nice -n20 /etc/rc.dyndns.update
*/60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot
30 12 * * * root /usr/bin/nice -n20 /etc/rc.update_urltables
*/60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 webConfiguratorlockoutIf possible do not add items to this file manually.
If you do so, this file must be terminated with a blank line (e.g. new line)
-
I am going to have to assume this is the intended behaviour at this point. It doesn't cause any adverse effects for me.
I do have the OpenVPN client configured (in case this is somehow interacting with unbound) but it is constantly connected and not restarting.
Would be nice to get confirmation from one of the developers though…
-
No - Its not intentional (Unless mine is broken. Unbound has been running without service restart for a week)
-
While troubleshooting another issue, I discovered frequent restarts (every hour) in the unbound log. Is this normal or is something wrong?
Related to your ISP lease renewal ? Status: System logs: General:
…
() check_reload_status: Starting packages
() php-fpm[34310]: /rc.start_packages: Restarting/Starting all packages.
… -
Nope, nothing correlates in system logs. I do have Dynamic DNS setup though but that only updates once per day, so doubt it's related.
Cron jobs output:
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin
HOME=/var/log
#minute hour mday month wday who commandpfSense specific crontab entries
Created: February 7, 2015, 8:05 pm
1,31 0-5 * * * root /usr/bin/nice -n20 adjkerntz -a
1 3 1 * * root /usr/bin/nice -n20 /etc/rc.update_bogons.sh
*/60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 sshlockout
*/60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 webConfiguratorlockout
1 1 * * * root /usr/bin/nice -n20 /etc/rc.dyndns.update
*/60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot
30 12 * * * root /usr/bin/nice -n20 /etc/rc.update_urltablesIf possible do not add items to this file manually.
If you do so, this file must be terminated with a blank line (e.g. new line)
-
One thing is clear in my case : Every time the dhcp service writes leases to leases file the unbound service restarts…every single time.
-
Inq - I just noticed that one of my network devices (Apple TV) was going crazy asking for dhcp every few seconds. I did a clean install of 2.2 and didnt reboot some of my devices, which probably caused this issue.
I am now thinking that this issue could have been causing the frequent unbound resets. I will monitor and update this thread.
Thanks,
Marco -
It would be interesting and probably helpful to see your DHCP server setting and your unbound settings.
-
Ok, this issue was definitely caused by a misconfiguration on my Apple TV (connected to both wired and wireless simultaneously). This was causing the Apple TV to get stuck in a DHCP request loop (verified by DHCP logs).
In addition, I have the following setting in Unbound: Register DHCP leases in the DNS Resolver
Since I corrected the Apple TV issue, the hourly unbound restarts have stopped.
Thanks,
Marco -
Sounds like something I see lots of people doing with laptops. Plugging in wired while wireless is still on. Same with IP Cams.
-
@Inq:
One thing is clear in my case : Every time the dhcp service writes leases to leases file the unbound service restarts…every single time.
I noticed the same thing the other day, unbound restarting every time the dhcp service wrote the leases file, about once an hour. It hadn't been doing this until I changed the setting for "unwanted reply threshold" in the dns resolver advanced settings from disabled to 10 million. I changed it back to disabled yesterday and unbound has not restarted since. No idea what the relationship is, but there seems to be one.
edit: nevermind, changed another setting in dns resolver and now unbound is restarting at the same time the dhcp service writes the leases file again…
-
I'm experiencing the same problem with Unbound. Does this mean that it's normal for it to restart if the option to "Register DHCP leases in the DNS Resolver" is enabled?
-
edit: nevermind, changed another setting in dns resolver and now unbound is restarting at the same time the dhcp service writes the leases file again…
I see the same thing: Unbound is restarting on every DHCPREQUEST - DHCPACK sequence:
Status: System logs: DHCP log:04-06-2015 09:49:36 Local7.Info 192.168.1.1 Apr 6 09:49:37 dhcpd: DHCPREQUEST for 192.168.2.235 from 78:3a:84:xx:34:1f (Philippes-Air) via sis0 04-06-2015 09:49:36 Local7.Info 192.168.1.1 Apr 6 09:49:37 dhcpd: DHCPACK on 192.168.2.235 to 78:3a:84:xx:34:1f (Philippes-Air) via sis0 04-06-2015 09:49:36 User.Info 192.168.1.1 Apr 6 09:49:37 dhcpleases: Sending HUP signal to dns daemon(53865)
I guess, when "Sending HUP signal to dns daemon" is fired, Unbound restarts.
At that moment, on the "Status: System logs: Resolver", I see (reversed time order):04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: start of service (unbound 1.5.3). 04-06-2015 09:52:08 Daemon.Notice 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] notice: init module 0: iterator 04-06-2015 09:52:08 Daemon.Notice 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] notice: Restart of unbound 1.5.3. 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: 0.524288 1.000000 1 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: 0.262144 0.524288 2 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: 0.131072 0.262144 1 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: lower(secs) upper(secs) recursions 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: [25%]=0.262144 median[50%]=0.393216 [75%]=0.524288 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: histogram of recursion processing times 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: average recursion processing time 0.414164 sec 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: server stats for thread 1: requestlist max 4 avg 1 exceeded 0 jostled 0 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: server stats for thread 1: 4 queries, 0 answers from cache, 4 recursions, 0 prefetch 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: 0.524288 1.000000 4 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: 0.262144 0.524288 5 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: 0.131072 0.262144 3 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: 0.065536 0.131072 3 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: 0.032768 0.065536 3 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: 0.016384 0.032768 1 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: lower(secs) upper(secs) recursions 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: [25%]=0.08192 median[50%]=0.240299 [75%]=0.484966 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: histogram of recursion processing times 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: average recursion processing time 0.331678 sec 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: server stats for thread 0: requestlist max 28 avg 3.36842 exceeded 0 jostled 0 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: server stats for thread 0: 25 queries, 6 answers from cache, 19 recursions, 0 prefetch 04-06-2015 09:52:08 Daemon.Info 192.168.1.1 Apr 6 09:52:09 unbound: [53865:0] info: service stopped (unbound 1.5.3).
This means that unbound restart several times per minute … is this normal ???
Or is this what it is all about, the setting "DHCP Registration" on the "Services: DNS Resolver" page ?