Frequent unbound restarts

mtarbox

I'm glad that someone else posted on this topic, albeit back in 2017.
I enabled TLD yesterday, and pfblocker is set to update at 0030.
Sometime around 0200 this morning my pfsense box started sending out multiple emails from the service watchdog that the dhcpd service and unbound had stopped and it was restarting those services. However I did not receive my daily email from the pfsense box, normally that comes in around 5-530am.
I'm unable to ssh into the pfsense box, and it was unresponsive this morning.
Unfortunately I start my work week today, so I can't even delve into it.
Luckily I do have a current backup, so I'm thinking I will follow the above, disable DHCP Registration and configure Host Overrides in DNS Resolver, and see if that solves the issues.

2.4.2
8gb ram

![Screenshot (20).png](/public/imported_attachments/1/Screenshot (20).png)
![Screenshot (20).png_thumb](/public/imported_attachments/1/Screenshot (20).png_thumb)

bbrendon

It seems like a good interim "hack" would be to patch the code to prevent unbound from restarting with dhcp udpates and create a cron job to do it on a more "controlled" basis.

Does anyone know which files in the php code needs changing? Is it under /usr/local/www ?

Gertjan

You will find what you're looking for here /etc/inc.
Check services.inc and system.inc.

It's far more easier to disable the "DHCP Registration" completly, and give all host that you want be able to resolve at any time a static fixed MAC DHCP lease entry.
When these host renew, they will receive the same IP every time, and will unbound not be told to restart.

Remember : the real issue is unbound itself : it reads config file when it start. If some host-IP information is known afterwards - unbound can only be made aware when it restarts. A cron solution would make your device "non resolvable" for a certain time, the time it takes before the cron job runs.

RonpfS

@gertjan said in Frequent unbound restarts:

unbound can only be made aware when it restarts

pfblockerNG-devel is changing unbound internal db (Resolver Live Sync) using unbound-control cmd. So maybe at some point DHCP server could be doing something similar.

sotirone

Is there really no other way to refresh other than restarting Unbound and reloading everything?

I mean this is not only a problem with the "Register DHCP leases in the DNS Resolver" but also with Remote OpenVPN clients that are using the DNS Resolver.

Every time a Remote Client connection is initiated or stopped, Unbound has to restart. If you are using pfBlocker-ng, that could mean upwards of 60 seconds of downtime.

I thought I read here on the forum that a fix was being developed but I guess I am mistaken.

Either Netgate or upstream needs to do something about this. Having to disable important features for the sake of uptime seems like quite a big problem for me.

Is there any way to escalate this?

RonpfS

DHCP could change the Unbound in-memory db as pfblockerNG does with Live Reload, instead of restarting Unbound.

jasonArloUser

Hi all

Is there any update on this? I'm trying to use PFBlockerNG-devel but this causes unbound restarts to take a noticeable amount of time and them happening every 10-30 minutes makes the whole package unusable.

I did a quick check on github and I see where the C code is sending a HUP to DNS. I could change this to call unbound-control like pfBlockerNG does (though I can't assume unbound is running the way PFBlocker can, so it will be slightly more complicated). Would such a pull request be accepted?

EDIT: Alternatively, we could change the unbound startup script to catch HUP and call unbound-control instead of restarting? That might be more robust.

Gertjan

@jasonArloUser said in Frequent unbound restarts:

I'm trying to use PFBlockerNG-devel but this causes unbound restarts

as per pfBlocker settings, it will reload restart unbound every :

@jasonArloUser said in Frequent unbound restarts:

to take a noticeable amount of time

Other options are : use less feeds. Or a bigger system.

@jasonArloUser said in Frequent unbound restarts:

on github and I see where the C code is sending a HUP to DNS

The DHCP daemon ?
Do what @RonpfS mentioned.
Give all important devices static mac mappings, and the DHCP server daemon won't restart unbound any-more.

jasonArloUser

That's what I'm doing but the fact is that the current implementation is wrong. When a new host enters the DNS local zone because it gets a DHCP release the correct thing to do is reload that zone, not restart the DNS service.

Gertjan

Note that I agree with you.
bind doesn't go wacka when an attached interface goes down. Or a zone needs to be reloaded. Etc.
But bind is huuuuuuge to setup (read : error prune )- and the code foot print is even bigger.
Putting a GUI in front of bind is daunting - a never ending story.

But ..... for me the case was solved a long time ago.
I just don't have unbound restarted any-more when a lease comes in.
Problem solved.

I had to list up all my IPv4 devices - they all use DHCP except for 2 servers, they are static.
Most of my device have a Static MAC entry, so the DNS info is already stored into the DNS.

New, other devices that enter my network could connect my 'known' devices like printers etc.
Other, already present devices don't need to connect to these new devices so an DNS entry isn't needed.

sotirone

@Gertjan said in Frequent unbound restarts:

Give all important devices static mac mappings, and the DHCP server daemon won't restart unbound any-more.

That is a workaround, not a solution. Furthermore, one of the goals of DHCP is to reduce manual intervention. Having to manually set up (possibly thousands of) static DHCP entries it not much better than setting static IPs directly on clients.

jasonArloUser

@Gertjan Oh I agree that Bind would be way too much effort here. But the solution is actually very close. All that needs to happen is the DHCP code that is currently sending a HUP signal just needs to call unbound-control to load the local zone, as that is all it can be effecting.

In any case, I've done what you did so I don't expect to feel the pain much anymore but if I get annoyed at adding devices to the static mapping maybe I'll try to submit a patch request. :)

Gertjan

I had a look at this issue, and if memory servers me well, it's not the dhcp daemon that kick unbound, but the dhcpleases process (the IPv4 version) that is just stopped when you uncheck :

This little program updates the /etc/hosts file and signals unbound to "reload".
unbound's default reaction will be : restart - what is causing all this.
I'll hope that unbound will be more intelligent one day, and 'watches' files so it reload these file(s) that needed to be reloaded.

I didn't look how Live Sync (unbound) is implemented ... but I tend to say it isn't done as 'bind' does it. But it's a beginning.
True is : pfSense should use it for its hosts file / DHCP changes - if possible.

Btw :
Same story, from years ago https://forum.netgate.com/topic/79375/unbound-frequently-restarts-on-2-2-is-this-normal
https://forum.netgate.com/topic/80517/unbound-seems-to-be-restarting-frequently
etc.

Today, it's nearly 2020 ..... and where is the doc about this "Live sync" : NOT here https://nlnetlabs.nl/documentation/unbound/ (and that's the source).

jt

do I understand correctly that the "Register DHCP leases in DNS resolver" functionality is totally unusable, forcing us to do a lot of manual work with registering devices' DHCP addresses as static leases? wtf? first FQDN alias resolution, now this.. pFsense is starting to be unusable.
i mean, how are you others deal with this? :/

Gertjan

"Register DHCP leases in DNS resolver" is usable.
It's just a setting that must be understood so you can can recognise DNS situations, and do something about it.
With many devices (dozens or hundreds) and/or a very short DHCP lease, unbound, the resolver, could get restarted rather often.
What about forcing a long DHCP lease like a day or 2 ?
If very heavy DNS packages like pfBlocker are used also, the start-up time of unbound gets impacted.
All of this related to what device you use to run pfSense. An I7-core with SSD will not sweat - a loaded SG-1100 could give noticeable DNS outages. Added to that : the cache is lost.

Also : only new leases, introducing new devices to the network, will restart unbound. As successful DHCP renewing doesn't.

This is one of the many reasons why "visitors" belong on their own network with "Register DHCP leases in DNS resolver" set to OFF. I don't care what their IP is, neither their host name.

@work, I have about 40 devices.
I tend to fix every device to a known IP4 and Pv6. I still tend to use the IPv4 as a device number, but I know that that concept will vanish when IPv4 fades out. I use MAC-Leases of course, and have to set up this ones for every new device. At the same moment, I choose a simple, short, representative name for the device.
These leases are also placed into the "hosts" file ( == "Registered").

@home : I don't care less. I do not need to know the name of the phone a visitor brought along with him. Neither the IP.
My couple of own devices are - as above - locked to a "MAC based lease".
So I'm not using "Register DHCP leases in DNS resolver".

So, IMHO : it's a close to a no-problem.
( but for others, it could be the next planet killer )

jt

@Gertjan thx for the reaction, and for the suggestions and ideas as well. very helpfull - i'm not by far a super-skilled pfsense admin, so much appreciate the ideas.

just one clarification, if you could: I haven't found a way to enable DHCP leases DNS registration for particular subnet - i only see a global checkbox in DNS Resolver settings. could you please point me to a place this can be configured per interface/subnet?

Gertjan

For every LAN type of network, you have a DHCP server with its dedicated settings :

For every DHCP, you can set and maintain, at the bottom of the page, the "DHCP Static Mappings for this Interface".

On the Status > DHCP Leases page you can also choose what lease you want to add as a Static lease.

jt

@Gertjan said in Frequent unbound restarts:

This is one of the many reasons why "visitors" belong on their own network with "Register DHCP leases in DNS resolver" set to OFF. I don't care what their IP is, neither their host name.

this is what I referred to. the above sounds like there should be an option to disable "Register DHCP leases in DNS resolver" feature on each network (subnet) separately.

your last screenshot refers to a different functionality - setting dhcp static leases.

Gertjan

@jt said in Frequent unbound restarts:

setting dhcp static leases.

These behave like the non static leases, and will never change.
It has it's own check box. I have that always checked.

True is : the "dhcpleases process", that collects DHCP server leases for any DHCP server process, will restart the DNS system (unbound or the forwarder, dnsmasq) if new leases come in. There is no "per interface" choice.

If you select :

you will see 'dhcpleases" process running :

ps ax | grep dhcpl

It's this process that 'HUPS' unbound - or the forwader dnsmasq.

I guess things were designed as such a long time ago. Without really knowing that it could one day concerns a looooot of devices on a network (10 Mbits half duplex 'blindly fast' in past ...).

As said earlier : unbound will get reprogrammed sooner or later so it will reread changes in config file on the fly some day.

jasonArloUser

@Gertjan unbound doesn't need to be reprogrammed. It already has the ability to reload configuration with the unbound-control command. The issue here is that the process that updates the leases doesn't use that, it sends the process a HUP. This is a bug as the lease program knows it can only be affecting the local zone so there is no possible reason to do anything other than reload that zone. Sending HUP and expecting unbound to somehow know that HUP means a reload of the local zone would be incorrect.

There are 2 simple ways to fix this issue: (a) fix the script to call unbounded-control as it should or (b) in the shell command that starts unbounded it is probably possible to set up a HUP handler that calls unbound-control. The second solution is wrong here because HUP should really reload everything but at least it would get things working without having to rewrite anything else.