Unbound lease include is missing or inconsistent

cybis

I recently switched to the Kea DHCP server on CE 2.8.1 where some of clients got assigned new IPs (compared to the previous ISC DHCP leases). This might have been caused by unbound crashes due to my pfBlocker DNSBL mode setting (see https://forum.netgate.com/post/1243986)

The changed IPs caused various home automation issues in my network which forced me to modify /var/lib/kea/dhcp4.leases and /var/unbound/leases/leases4.conf manually. I know, those files shouldn't be touched, but due to the fragmented IP leases/allocation and the fact that static mapping can't be inside the DHCP pool yet (see https://redmine.pfsense.org/issues/15659), I had no other solution.

The log shows now inconsistent unbound lease data:

Unbound lease include is missing or inconsistent: /var/unbound/leases/leases4.conf

Disabling and enabling the DHCP DNS registration and early DNS registration, which restarts both Kea and unbound, doesn't fix the inconsistency. Renaming/removing the 2 files mentioned above and restarting the services doesn't help either.

Is there any procedure one can follow or trigger to fix the inconsistency?

Gertjan

@cybis said in Unbound lease include is missing or inconsistent:

Is there any procedure one can follow or trigger to fix the inconsistency?

I can think of one :
First, stop the kea DHCPv4 server(s).
This will flush the full "DHCP leases pool state and whatever" to the /var/lib/kea/dhcp4.leases file.
Now you can edit this file, and you have to adhere to one : rule : syntax errors are not allowed ^^
After all, thuis file is created and maintained by 'kea'. Everyone else can read this file for whatever reason, but if you want to write it, it has to be 'exact' as kea probably doesn't do much error checking as it knows who created the file : itself.

@cybis said in Unbound lease include is missing or inconsistent:

he fact that static mapping can't be inside the DHCP pool yet (see https://redmine.pfsense.org/issues/15659),

That issue also mentions a temporary solution or workaround. That didn't work out for you ?

@cybis said in Unbound lease include is missing or inconsistent:

Renaming/removing the 2 files mentioned above and restarting the services doesn't help either.

Stopping the server, and removing the file /var/lib/kea/dhcp4.leases, will remove very knowledge of the 'past' for kea.
Removing the file means removing the inconsistency.
When you start it again, it will treat every incoming lease request as new, and will act upon it as a new lease.
If the file /var/lib/kea/dhcp4.leases becomes 'inconsistent' again, what / who triggers this situation ?

Btw : /var/unbound/leases/leases4.conf is derived from /var/lib/kea/dhcp4.leases.

@cybis said in Unbound lease include is missing or inconsistent:

I know, those files shouldn't be touched,

You are the boss. It's your data, you decide.
These files are created and maintained by a process. The 'real' data set remains in memory, in RAM, and is flushed out to disk ones in a while. When the process (kea) restarts, reading the file upon start give kea the possibility to know what it did before it was restarted. The file is it's persistent memory.

cybis

@Gertjan said in Unbound lease include is missing or inconsistent:

I can think of one :
First, stop the kea DHCPv4 server(s).
This will flush the full "DHCP leases pool state and whatever" to the /var/lib/kea/dhcp4.leases file.
Now you can edit this file, and you have to adhere to one : rule : syntax errors are not allowed ^^
After all, thuis file is created and maintained by 'kea'. Everyone else can read this file for whatever reason, but if you want to write it, it has to be 'exact' as kea probably doesn't do much error checking as it knows who created the file : itself.

Thanks a lot for your inputs. Yes, in /var/lib/kea/dhcp4.leases I only edited the IP adresses of the hosts I wanted to change. Everything else was untouched, including the lease duration info. No new entries were added manually.

That issue also mentions a temporary solution or workaround. That didn't work out for you ?

I don't see a temporary solution / workaround in that issue As far as I understand, pfSense's GUI input validation prevents adding static IPs which are inside the DCHP pool (although Kea would support it with its current config). This is exactly what I observed when I tried on my firewall

Stopping the server, and removing the file /var/lib/kea/dhcp4.leases, will remove very knowledge of the 'past' for kea.
Removing the file means removing the inconsistency.
When you start it again, it will treat every incoming lease request as new, and will act upon it as a new lease.
If the file /var/lib/kea/dhcp4.leases becomes 'inconsistent' again, what / who triggers this situation ?

Btw : /var/unbound/leases/leases4.conf is derived from /var/lib/kea/dhcp4.leases.

Just for clarification: the reported inconsistency is in unbound's /var/lib/kea/dhcp4.leases, the DHCP lease file is fine. Similarly to what you suggested, I did the following steps:

Disable DNS registration and early DNS registration under Kea's general settings
Stop Kea and unbound
Rename/move /var/unbound/dhcpleases_entries.conf, /var/unbound/host_entries.conf and /var/unbound/leases/leases4.conf
Start unbound and Kea
Enable DNS registration and early DNS registration (restarts Kea)

According to the DHCP log, there's no inconsistency message anymore with the last DHCP server restart:

You are the boss. It's your data, you decide.
These files are created and maintained by a process. The 'real' data set remains in memory, in RAM, and is flushed out to disk ones in a while. When the process (kea) restarts, reading the file upon start give kea the possibility to know what it did before it was restarted. The file is it's persistent memory.

Makes sense. I stopped unbound to check what exaclty is flushed to the files. Based on the file timestamp, both /var/unbound/hoset_entries.conf and /var/unbound/dhcpleases_entries.conf are updated. This triggers the inconsistency message in the DHCP logs again though. After restarting the DHCP server, the message is gone (similar to the log screenshot above). I'm just puzzled that my /var/unbound/dhcpleases_entries.conf file is completely empty now and /var/unbound/leases/leases4.conf doesn't contain all the hosts from my DHCP pools. DNS resolution works fine for both static and dynamic clients though

giuliafw70

If DNS still resolves for the clients, I would be careful not to read too much into dhcpleases_entries.conf being empty. In pfSense those unbound include files are generated pieces, so static mappings/host overrides and dynamic DHCP registrations do not necessarily end up in the same file. The order I would use is: make the lease changes with Kea stopped, start Kea, then restart/apply DNS Resolver after Kea is stable, and avoid editing anything under /var/unbound directly except as a temporary cleanup. If the warning only comes back while stopping Unbound, it may be a stale generated include during the service transition rather than the active DNS state.

cybis

@giuliafw70 Yes, the host names of the static mappings are stored in /var/unbound/host_entries.conf. I agree, as long as DNS resolution works, there's no need to further look into the content of those files.