DHCPv4 HA intermittently fails to start: CmdHttpListener bind: Address already in use + command socket errors

[empbilly]

hello,

Environment

• pfSense CE 2.8.0-RELEASE on two appliances (HA pair)
• Kea DHCPv4 2.6.2 (hot-standby HA over a dedicated sync network)
• Config file: /usr/local/etc/kea/kea-dhcp4.conf

Latest errors

2025-08-22 10:33:46.986484-03:00 kea-dhcp4 93981 ERROR [kea-dhcp4.commands.0x2c836c012000] COMMAND_SOCKET_WRITE_FAIL Error while writing to command socket -1 : Bad file descriptor
2025-08-22 10:33:46.986156-03:00 kea-dhcp4 93981 ERROR [kea-dhcp4.commands.0x2c836c012000] COMMAND_WATCH_SOCKET_MARK_READY_ERROR watch socket failed to mark ready: WatchSocket markReady failed: select_fd was closed!
2025-08-22 10:31:34.223555-03:00 kea-dhcp4 9031  ERROR [kea-dhcp4.dhcp4.0x3414f3412000] DHCP4_INIT_FAIL failed to initialize Kea server: configuration error using file '/usr/local/etc/kea/kea-dhcp4.conf': Error initializing hooks: CmdHttpListener::run failed: unable to setup TCP acceptor for listening to the incoming HTTP requests: bind: Address already in use [system:48 at /usr/local/include/boost/asio/detail/reactive_socket_service.hpp:161:33 in function 'bind']
2025-08-22 10:31:34.220073-03:00 kea-dhcp4 9031  ERROR [kea-dhcp4.dhcp4.0x3414f3412000] DHCP4_CONFIG_LOAD_FAIL configuration error using file: /usr/local/etc/kea/kea-dhcp4.conf, reason: Error initializing hooks: CmdHttpListener::run failed: unable to setup TCP acceptor for listening to the incoming HTTP requests: bind: Address already in use [system:48 at /usr/local/include/boost/asio/detail/reactive_socket_service.hpp:161:33 in function 'bind']

What I’ve already tried (essentials)

• Changed the HA port on both nodes from the default 8765 to a high, unused port (same on both sides).

  • Verified connectivity both ways (nc -vz <peer> <NEWPORT>) and that only kea-dhcp4 is listening on the new port (sockstat -4 -l -p <NEWPORT>).
  • Ensured 8765 is free (no listeners).

• Performed clean restarts to avoid dup instances:

  pkill -f kea-dhcp4
  rm -f /var/run/kea/kea-dhcp4.kea-dhcp4.pid
  <start via webgui>
  

• Config test passes consistently:

  kea-dhcp4 -t /usr/local/etc/kea/kea-dhcp4.conf
  

• Earlier (before port change) I occasionally saw short-lived php/sh listeners on 8765, likely GUI/diagnostics. After moving to a new port and closing GUI tabs, the issue still occurs intermittently on service start/boot.

Questions / guidance requested

1. Is there a known issue after 2.8.0 where the Kea HA HTTP acceptor (hooks) intermittently fails to bind even when only kea-dhcp4 is supposed to own the port?
2. Any insight into the COMMAND_SOCKET_WRITE_FAIL and WatchSocket ... select_fd was closed errors—race during restart, stale control socket, or something else?
3. Recommended best practices to avoid races on HA restarts (ordering, delays, disabling Service Watchdog for Kea)? Any lock files beyond the PID to check?
4. Any additional debug flags/logging I should enable to pinpoint why the listener thinks the address is in use at that moment?

Thanks!!