Kea stops handing out DHCP leases while active; kea2unbound generated even with Unbound disabled

empbilly

Hi everyone,

We are running pfSense CE 2.8.1 with Kea DHCPv4 in HA hot-standby mode.

Environment:

pfSense CE 2.8.1-RELEASE
DHCP backend: Kea DHCPv4
HA enabled: pfmaster / pfbackup, hot-standby mode
DNS service in use: DNS Forwarder / dnsmasq
DNS Resolver / Unbound: disabled
Kea DNS Registration: disabled
Kea Early DNS Registration: disabled
Many VLANs and DHCP scopes, including static mappings for access points

We are seeing recurring issues where Kea appears as running in the WebUI, but clients stop receiving DHCP addresses. Restarting Kea from the WebUI usually restores service temporarily.

Kea status looks OK when checked:

DHCPv4 server: active
DHCPv6 server: inactive
Control Agent: inactive

The HA status from the Kea control socket also looks healthy:

"ha-mode": "hot-standby",
"local": {
  "role": "primary",
  "server-name": "pfmaster",
  "state": "hot-standby"
},
"remote": {
  "communication-interrupted": false,
  "in-touch": true,
  "last-state": "hot-standby",
  "role": "standby",
  "server-name": "pfbackup"
},
"globally-disabled": false,
"sockets": { "status": "ready" }

However, we found that pfSense keeps generating and executing:

/conf/kea4_scripts.d/kea2unbound.sh

even though Unbound is disabled and DNS Registration is disabled.

The generated script contains:

#!/bin/sh
# Automatically generated! DO NOT EDIT!

case "$1" in
	hook_load)
		;&
	lease4_add)
		;&
	lease4_delete)
		;&
	lease4_update)
		/usr/local/bin/kea2unbound --kea-conf /usr/local/etc/kea/kea-dhcp4.conf --unbound-conf /var/unbound/unbound.conf --include-file /var/unbound/leases/leases4.conf sync 
		;;
esac

Since Unbound is not running, we see repeated errors like:

unbound-control error: connect: Connection refused for 127.0.0.1 port 953

Unable to query Unbound Control: /var/unbound/unbound.conf

/conf/kea4_scripts.d/kea2unbound.sh: /usr/local/bin/kea2unbound: Text file busy

Unable to obtain lock after 5 seconds: /usr/local/bin/kea2unbound

We also confirmed that only dnsmasq is listening on port 53:

nobody   dnsmasq    udp4   *:53
nobody   dnsmasq    tcp4   *:53

No Unbound process is listening on port 53 or 953.

If we manually move kea2unbound.sh out of /conf/kea4_scripts.d/ and kill the kea2unbound processes, the error storm stops:

mv /conf/kea4_scripts.d/kea2unbound.sh /conf/kea4_scripts.d.disabled/
pkill -f '/usr/local/bin/kea2unbound'
pkill -f '/conf/kea4_scripts.d/kea2unbound.sh'

But this is only temporary, because pfSense recreates the script after Kea/WebUI configuration reloads.

We also observed:

ALLOC_ENGINE_V4_LEASES_RECLAMATION_SLOW expired leases still exist after 5 reclamations

COMMAND_WATCH_SOCKET_MARK_READY_ERROR watch socket failed to mark ready: WatchSocket markReady failed: select_fd was closed!

COMMAND_SOCKET_WRITE_FAIL Error while writing to command socket -1 : Bad file descriptor

HA_LEASE_UPDATE_COMMUNICATIONS_FAILED pfmaster: failed to send lease update to pfbackup: End of file

We already ran:

leases-reclaim remove=true

through the Kea control socket, and it completed successfully:

{ "result": 0, "text": "Reclamation of expired leases is complete." }

We also tuned expired lease processing in custom Kea config.

Another issue we noticed: when replacing an access point and changing only the MAC address in an existing DHCP static mapping, Kea may keep the old lease associated with the reserved IP, causing address conflict messages until the old lease is cleared.

Example:

ALLOC_ENGINE_V4_DISCOVER_ADDRESS_CONFLICT
conflicting reservation for address 172.19.0.148 with existing lease
new device MAC: ac:8b:a9:2b:b9:09
old lease MAC: e4:38:83:72:aa:7d

Has anyone else seen Kea stop handing out DHCP leases while still showing as active in the WebUI?

Also, is it expected that pfSense generates and runs kea2unbound.sh when Unbound is disabled and Kea DNS Registration is disabled?

At this point we are considering rolling back to ISC DHCP temporarily, because this is becoming unstable for production use.

I also opened a Redmine bug report with the collected details, logs, and configuration evidence:

https://redmine.pfsense.org/issues/16865

Any feedback from other users running Kea DHCP on pfSense CE 2.8.1, especially with HA and DNS Forwarder/dnsmasq instead of Unbound, would be very helpful.

SteveITS

@empbilly re MAC address, see https://forum.netgate.com/topic/198115/changing-the-mac-address-on-a-kea-static-lease-does-not-work/

Kea2unbound I think runs even if not using Kea (?) but I’ve not tried it without unbound.