Intermittent Kea DHCP failures, high CPU and command socket timeouts on pfSense CE 2.8.1

empbilly

Hello,

We are experiencing recurrent, intermittent DHCP failures on multiple VLANs, usually during periods of increased client activity around 13:00, although one confirmed incident occurred in the morning.

The issue and supporting evidence were also reported in pfSense Redmine #16994.

Environment

pfSense CE 2.8.1 / FreeBSD 15.0-CURRENT
Kea 2.6.2, memfile backend
HA hot-standby: one primary and one standby
27 subnets/interfaces
Multi-threading enabled: 8 threads, queue size 64

What we found

Similar events were recorded on several dates between June and August. During the incidents, some laboratory and Wi-Fi clients could not obtain an address, while other VLANs continued receiving DHCP replies. Kea never stopped completely.

The clearest incident occurred on August 11, between approximately 08:00 and 08:25:

frequent status-get timeouts on the Kea control socket;
84 COMMAND_SOCKET_WRITE_FAIL ... Broken pipe errors;
kea-dhcp4 reached approximately 92% CPU and had 29 threads;
multiple terminated child processes had Kea as their parent;
packet capture showed 118 Discover, 45 Offer, 101 Request and 63 ACK packets in 30 seconds;
pools were not exhausted, interfaces had no errors/drops, and memory/mbufs were normal;
the standby process remained running and HA was normally in-touch.

At 08:25 we issued config-reload. The Kea PID did not change, but CPU usage immediately returned to normal, control socket errors stopped, and an affected client obtained an address.

Current hypothesis

Kea loads libdhcp_run_script.so with /usr/local/bin/kea_run4 and sync=false. Although /conf/kea4_scripts.d/ is empty, every matching lease event can still create a shell process that scans the empty directory and exits. We directly observed children such as:

/bin/sh /usr/local/bin/kea_run4 lease4_renew
/bin/sh /usr/local/bin/kea_run4 leases4_committed

Kea previously fixed an issue where this hook left defunct processes (#1878). Kea 2.6.2 already contains that fix, so we are not claiming it is the same bug. However, the process creation, terminated children, high CPU and recovery after reload suggest a possible regression, FreeBSD-specific behavior or high-load edge case. An A/B test with the hook disabled is still required.

An earlier, separate pfSense issue involving kea2unbound.sh was already mitigated with the official Redmine #16865 patch. The current failures continue without kea2unbound or unbound-control processes.

Has anyone observed similar behavior with Kea 2.6.2 on pfSense 2.8.1, especially involving libdhcp_run_script.so and kea_run4 under DHCP load?

empbilly

Update: root cause confirmed with DTrace

The problem occurred again on August 25 between approximately 08:00 and 08:24. The details are also being tracked in pfSense Redmine #16994.

During the incident, kea-dhcp4 reached approximately 94% CPU, the control socket repeatedly timed out, and packet capture showed 18,528 Discover packets but only 3,308 Offer packets. The standby remained healthy.

DTrace confirmed that CPU time was being spent creating processes from the Run Script hook:

_malloc_prefork / _malloc_postfork
isc::asiolink::ProcessSpawnImpl::spawn(bool)
libdhcp_run_script.so: lease4_renew
libdhcp_run_script.so: leases4_committed

In a 10-second sample, Kea performed 38 fork() calls, 38 sh executions and 76 wait4() calls. Both /conf/kea4_scripts.d/ and /cf/conf/kea4_scripts.d/ were empty, so these processes performed no useful work. A configuration reload with the hook still enabled did not resolve the CPU usage.

We then performed an A/B test on both HA nodes. We changed the pfSense generator to load libdhcp_run_script.so only when an executable script exists and removed the hook from the active Kea configuration.

The mitigation applied to /etc/inc/services.inc checks the scripts directory before adding the hook:

$has_run_script = false;
foreach (glob("{$run_scriptd_base}/*") ?: [] as $candidate) {
    if (is_file($candidate) && is_executable($candidate)) {
        $has_run_script = true;
        break;
    }
}

if (file_exists($run_script_main) && $has_run_script) {
    // Add libdhcp_run_script.so to hooks-libraries.
}

We created backups on both nodes with the suffix .before-run-script-mitigation-20260825, validated /etc/inc/services.inc with php -l, validated the generated Kea configuration with kea-dhcp4 -t, and then used config-reload. The Kea PIDs did not change. This survives reboot, but the patch may need to be reapplied after a pfSense upgrade replaces /etc/inc/services.inc.

After removing the hook:

CPU dropped from approximately 94% to 0.4%;
a 15-second DTrace sample showed zero forks and zero shell executions;
the packet queue returned to approximately 1,1,1.2;
status-get returned in approximately 0.11-0.12 seconds;
HA remained hot-standby and in-touch;
no new command socket errors were observed.

This confirms that, in our environment, the failure is caused by unnecessary asynchronous process creation by libdhcp_run_script.so and the pfSense kea_run4 wrapper when no scripts are configured. pfSense should only load this hook when the scripts directory contains at least one executable file.