Frequent unbound restarts

sotirone · Dec 20, 2019, 3:15 PM

@Gertjan said in Frequent unbound restarts:

Give all important devices static mac mappings, and the DHCP server daemon won't restart unbound any-more.

That is a workaround, not a solution. Furthermore, one of the goals of DHCP is to reduce manual intervention. Having to manually set up (possibly thousands of) static DHCP entries it not much better than setting static IPs directly on clients.

jasonArloUser · Dec 20, 2019, 4:29 PM

@Gertjan Oh I agree that Bind would be way too much effort here. But the solution is actually very close. All that needs to happen is the DHCP code that is currently sending a HUP signal just needs to call unbound-control to load the local zone, as that is all it can be effecting.

In any case, I've done what you did so I don't expect to feel the pain much anymore but if I get annoyed at adding devices to the static mapping maybe I'll try to submit a patch request. :)

Gertjan · Dec 21, 2019, 12:23 AM

I had a look at this issue, and if memory servers me well, it's not the dhcp daemon that kick unbound, but the dhcpleases process (the IPv4 version) that is just stopped when you uncheck :

This little program updates the /etc/hosts file and signals unbound to "reload".
unbound's default reaction will be : restart - what is causing all this.
I'll hope that unbound will be more intelligent one day, and 'watches' files so it reload these file(s) that needed to be reloaded.

I didn't look how Live Sync (unbound) is implemented ... but I tend to say it isn't done as 'bind' does it. But it's a beginning.
True is : pfSense should use it for its hosts file / DHCP changes - if possible.

Btw :
Same story, from years ago https://forum.netgate.com/topic/79375/unbound-frequently-restarts-on-2-2-is-this-normal
https://forum.netgate.com/topic/80517/unbound-seems-to-be-restarting-frequently
etc.

Today, it's nearly 2020 ..... and where is the doc about this "Live sync" : NOT here https://nlnetlabs.nl/documentation/unbound/ (and that's the source).

jt · Feb 4, 2020, 1:22 PM

do I understand correctly that the "Register DHCP leases in DNS resolver" functionality is totally unusable, forcing us to do a lot of manual work with registering devices' DHCP addresses as static leases? wtf? first FQDN alias resolution, now this.. pFsense is starting to be unusable.
i mean, how are you others deal with this? :/

Gertjan · Feb 4, 2020, 2:15 PM

"Register DHCP leases in DNS resolver" is usable.
It's just a setting that must be understood so you can can recognise DNS situations, and do something about it.
With many devices (dozens or hundreds) and/or a very short DHCP lease, unbound, the resolver, could get restarted rather often.
What about forcing a long DHCP lease like a day or 2 ?
If very heavy DNS packages like pfBlocker are used also, the start-up time of unbound gets impacted.
All of this related to what device you use to run pfSense. An I7-core with SSD will not sweat - a loaded SG-1100 could give noticeable DNS outages. Added to that : the cache is lost.

Also : only new leases, introducing new devices to the network, will restart unbound. As successful DHCP renewing doesn't.

This is one of the many reasons why "visitors" belong on their own network with "Register DHCP leases in DNS resolver" set to OFF. I don't care what their IP is, neither their host name.

@work, I have about 40 devices.
I tend to fix every device to a known IP4 and Pv6. I still tend to use the IPv4 as a device number, but I know that that concept will vanish when IPv4 fades out. I use MAC-Leases of course, and have to set up this ones for every new device. At the same moment, I choose a simple, short, representative name for the device.
These leases are also placed into the "hosts" file ( == "Registered").

@home : I don't care less. I do not need to know the name of the phone a visitor brought along with him. Neither the IP.
My couple of own devices are - as above - locked to a "MAC based lease".
So I'm not using "Register DHCP leases in DNS resolver".

So, IMHO : it's a close to a no-problem.
( but for others, it could be the next planet killer )

jt · Feb 4, 2020, 2:23 PM

@Gertjan thx for the reaction, and for the suggestions and ideas as well. very helpfull - i'm not by far a super-skilled pfsense admin, so much appreciate the ideas.

just one clarification, if you could: I haven't found a way to enable DHCP leases DNS registration for particular subnet - i only see a global checkbox in DNS Resolver settings. could you please point me to a place this can be configured per interface/subnet?

Gertjan · Feb 4, 2020, 2:50 PM

For every LAN type of network, you have a DHCP server with its dedicated settings :

For every DHCP, you can set and maintain, at the bottom of the page, the "DHCP Static Mappings for this Interface".

On the Status > DHCP Leases page you can also choose what lease you want to add as a Static lease.

jt · Feb 4, 2020, 2:54 PM

@Gertjan said in Frequent unbound restarts:

This is one of the many reasons why "visitors" belong on their own network with "Register DHCP leases in DNS resolver" set to OFF. I don't care what their IP is, neither their host name.

this is what I referred to. the above sounds like there should be an option to disable "Register DHCP leases in DNS resolver" feature on each network (subnet) separately.

your last screenshot refers to a different functionality - setting dhcp static leases.

Gertjan · Feb 4, 2020, 3:14 PM

@jt said in Frequent unbound restarts:

setting dhcp static leases.

These behave like the non static leases, and will never change.
It has it's own check box. I have that always checked.

True is : the "dhcpleases process", that collects DHCP server leases for any DHCP server process, will restart the DNS system (unbound or the forwarder, dnsmasq) if new leases come in. There is no "per interface" choice.

If you select :

you will see 'dhcpleases" process running :

ps ax | grep dhcpl

It's this process that 'HUPS' unbound - or the forwader dnsmasq.

I guess things were designed as such a long time ago. Without really knowing that it could one day concerns a looooot of devices on a network (10 Mbits half duplex 'blindly fast' in past ...).

As said earlier : unbound will get reprogrammed sooner or later so it will reread changes in config file on the fly some day.

jasonArloUser · Feb 4, 2020, 7:08 PM

@Gertjan unbound doesn't need to be reprogrammed. It already has the ability to reload configuration with the unbound-control command. The issue here is that the process that updates the leases doesn't use that, it sends the process a HUP. This is a bug as the lease program knows it can only be affecting the local zone so there is no possible reason to do anything other than reload that zone. Sending HUP and expecting unbound to somehow know that HUP means a reload of the local zone would be incorrect.

There are 2 simple ways to fix this issue: (a) fix the script to call unbounded-control as it should or (b) in the shell command that starts unbounded it is probably possible to set up a HUP handler that calls unbound-control. The second solution is wrong here because HUP should really reload everything but at least it would get things working without having to rewrite anything else.

Gertjan · Feb 5, 2020, 10:26 AM

@jasonArloUser said in Frequent unbound restarts:

@Gertjan unbound doesn't need to be reprogrammed. It already has the ability to reload configuration with the unbound-control command. The issue here is that the process that updates the leases doesn't use that, it sends the process a HUP. This is a bug as the lease program knows it can only be affecting the local zone so there is no possible reason to do anything other than reload that zone. Sending HUP and expecting unbound to somehow know that HUP means a reload of the local zone would be incorrect.

I know. Still waiting for this to happen.
Look at the word reload here. Or from the authors.

What I read here is : it just restarts, or something close to stop and start.
Cache is lost - all config files are read in again ... The only benefit is that the process isn't destroyed, and recreated.

This means that, for me, that when a relative small /etc/hosts file has to get read in a again, it also reads other config files like the "/var/unbound/pfb_dnsbl.*conf" from pfBlocker if it is installed and activated.

A Former User · Feb 13, 2020, 8:46 AM

Hi

Just wanted to add I came here after googling this event in the logs.

I have a feeling the restarts started or became worse when my ISP enabled IPv6. I did have the check box set for Register DHCP leases in the DNS Resolver, but with IPv4 noticed no issues.

When I enabled IPv6 I found web pages didn't zip in quite as fast as they did when it was IPv4 only, but were okay after the first visit, I put this down to the web browser using a fall back mechanism then caching the results of it for a while. A few weeks on and I saw the restarts in the log every second or so and came here.

So I've disabled the DCHP registration option, the resets have stopped, and web pages are loading without the extra latency, which makes sense as the results are staying in the DNS resolver cache and no delays from hitting pfSense when the service is restarting.

It's like I've had a speed upgrade.

Regards

Phil

RichMawdsley · Feb 16, 2020, 11:00 AM

I installed pfSense about a month ago and have been trying to track down my random internet drops for weeks. Finally saw the HUP signal message in the logs this morning and realised it happened on every DHCP request.. which lead me to this page.

I'm a little in shock over how absolutley stone age this is. A really basic ability causes it to restart the whole thing. What the hell?!

Static DHCP as a workaround is great and all, but as others have said, it's no where near a solution. It's manually doing the job for DHCP & DNS.

Anyone tested if this is fixed in any of the later v5 releases etc? Or is that not where the problem lay?

Gertjan · Feb 16, 2020, 3:47 PM

I never actuality tested (looked for) this : If a new lease comes in and the option is checked, then unbound is HUPped.
A lease renewal : no, as only the duration is updated, which is not a DNS 'thing'.

jt · Feb 17, 2020, 7:35 PM

how's the BIND implementation currently doing? would switching from unbound help here?

Gertjan · Feb 17, 2020, 8:16 PM

bind check the config file it's using, and parses them when it detect it was 'touched' by some other process.
What I do know is this : it won't ditch the cache when this happens.
It can also unbind and bind to new interfaces, when they come and leave.

But I can't tell if it would be better.
bind is huge. Setup has to been done manually, even if you use a GUI like pfSense uses.
bind needs users to look at the manual (huge also) for sure , otherwise DNS becomes a mess.

I don't bother, practically all my devices on my trusted LAN's have static MAC leases, for IPv4 and IPv6, so 'my' unbound doesn't restarts very often (less then ones a day, probably even less frequent).

Orbixx · Feb 21, 2020, 8:32 AM

I have this problem after adding pfBlockerNG with a significantly large DNSBL list. Will try the following to reduce/eliminate impact:

Reduce pfBlockerNG lists to a more reasonable size
Add more static IP leases where reasonable
Increase lease time

lawrencedol · Apr 17, 2020, 3:18 PM

[Post deleted. My problem is not related to this.]

stephenw10 · Apr 15, 2020, 9:00 PM

If you are getting disruption to VoIP calls that's clearly not directly DNS related. It may in fact be nothing to do with Unbound at all and in fact is just a symptom of something else that also causes Unbound to take far longer to reload.
Have you been seeing this before 2.4.5 or just since upgrading? If it's only in 2.4.5 you are probably hitting this:
https://redmine.pfsense.org/issues/10414

Try opening top -aSH and also pinging the firewall and then go to Status > Filter Reload in the gui and reload the filter.
If you see pings spike and processes shoot up to the top of the top table, pfctl, sshd, dpinger etc, then you almist certainly are hitting that.

Steve

lawrencedol · Apr 15, 2020, 9:04 PM

@stephenw10

Appreciate the feedback, thanks. I guess I am still digging on my issue because my son just confirmed to me that my specific problem is not yet resolved.