Kea-dhcp dns registration high load
-
Latest plus 24.08 dev image.
Has been working for some time
Suddenly web console became very slow/can't login and eventually ened up to the crash screen.
Nothing to be reported though either on php or freebsdOn console I would see extremely high load running top
Lots of lockf processes and unblound-checkconfDisabling dns registration and early dns registration solves the issue without having to kill processes
ct 8 00:46:18 pf php-fpm[63489]: /services_dhcp_settings.php: Configuration Change: admin@192.168.31.87 (Local Database): DHCP Server - Settings changed Oct 8 00:46:18 pf check_reload_status[671]: Syncing firewall Oct 8 00:46:27 pf check_reload_status[671]: Reloading filter Oct 8 00:46:27 pf kea2unbound[1752]: Unable to find a Kea control socket: /var/run/kea4-ctrl-socket Oct 8 00:46:29 pf php-fpm[64928]: /rc.filter_configure_sync: The gateway: Internet_DHCP6 is invalid or unknown, not using it. Oct 8 00:46:58 pf kea2unbound[29857]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:46:58 pf kea2unbound[15352]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:02 pf kea2unbound[96017]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:02 pf kea2unbound[96331]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:02 pf kea2unbound[95854]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:05 pf kea2unbound[17505]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:05 pf kea2unbound[17725]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:05 pf kea2unbound[16701]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:08 pf kea2unbound[36985]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:08 pf kea2unbound[35977]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:08 pf kea2unbound[35884]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:12 pf kea2unbound[59418]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:12 pf kea2unbound[57895]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:15 pf kea2unbound[77634]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:15 pf kea2unbound[79047]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:19 pf kea2unbound[99856]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:19 pf kea2unbound[23]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:19 pf kea2unbound[99288]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:19 pf kea2unbound[98177]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:21 pf kea2unbound[2931]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:21 pf kea2unbound[3115]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:21 pf kea2unbound[932]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:21 pf kea2unbound[1746]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:22 pf kea2unbound[10748]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:22 pf kea2unbound[12925]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:22 pf kea2unbound[13267]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:22 pf kea2unbound[13650]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:26 pf kea2unbound[25727]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:26 pf kea2unbound[26411]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:26 pf kea2unbound[26858]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:26 pf kea2unbound[26454]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:26 pf kea2unbound[28887]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:26 pf kea2unbound[27988]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:29 pf kea2unbound[31374]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock Oct 8 00:47:29 pf php-fpm[11020]: /services_dhcp_settings.php: Configuration Change: admin@192.168.31.87 (Local Database): DHCP Server - Settings changed Oct 8 00:47:29 pf check_reload_status[671]: Syncing firewall
Have tried restarting, no change.
Has been working fine for some time, only change was enabling openvpn client registration.
I have switched that back to disabled, but the issue remains.dhcpd.log also complaints
und[97019]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock
Oct 8 00:54:06 pf kea2unbound[98339]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock
Oct 8 00:54:06 pf kea2unbound[98557]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lock
Oct 8 00:54:06 pf kea2unbound[98754]: Failed to obtain lock after 30s: /tmp/kea2unbound4.lockls -l *.lock
-rw-rw-rw- 1 root wheel 0 Oct 8 00:53 config.lock
-rw-rw-rw- 1 root wheel 0 Oct 8 00:54 dhcpdconfigure.lock
-rw-rw-rw- 1 root wheel 0 Oct 8 00:54 filter.lock
-rw-rw-rw- 1 root wheel 0 Oct 7 18:40 haproxy.lock
-rw-r--r-- 1 root wheel 0 Oct 8 00:54 kea2unbound4.lock
-rw-rw-rw- 1 root wheel 0 Oct 7 18:41 notifyqueue.lock
-rw-rw-rw- 1 root wheel 0 Oct 7 18:39 notifyqueue_running.lock
-rw-rw-rw- 1 root wheel 0 Oct 7 18:38 openvpnserviceclient3.lock
-rw-rw-rw- 1 root wheel 0 Oct 7 18:38 openvpnserviceserver2.lockWhat else should I look at?
-
This has been resolved in internal snapshots
-
@cmcdonald Good to hear.
Any chances this to be an interim external release? -
@netblues an interim public snapshot would slow us down towards a GA release.
Soon.