Kea DHCP stops working
-
Hi,
I am writing this topic to report on an issue that has been happening sporadically without any obvious cause.
I've been using the KEA DHCP since it came out. Since then, this issue probably occurred around 5 times give it or take it.
Basically lose connection to some machines, and when I go to pfSense I see the services as Stopped.
Trying to start them does not work.
Happens for both DHCP v4 and v6On the error log I got this:
May 22 12:16:03 kea-dhcp4 20512 ERROR [kea-dhcp4.dhcp4.0xdd682612000] DHCP4_INIT_FAIL failed to initialize Kea server: configuration error using file '/usr/local/etc/kea/kea-dhcp4.conf': cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable May 22 12:16:03 kea-dhcp4 20512 ERROR [kea-dhcp4.dhcp4.0xdd682612000] DHCP4_CONFIG_LOAD_FAIL configuration error using file: /usr/local/etc/kea/kea-dhcp4.conf, reason: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable May 22 12:16:03 kea-dhcp6 20782 ERROR [kea-dhcp6.dhcp6.0x3731b6012000] DHCP6_INIT_FAIL failed to initialize Kea server: configuration error using file '/usr/local/etc/kea/kea-dhcp6.conf': cannot lock socket lockfile, /tmp/kea6-ctrl-socket.lock, : Resource temporarily unavailable May 22 12:16:03 kea-dhcp4 20512 ERROR [kea-dhcp4.dhcp4.0xdd682612000] DHCP4_PARSER_COMMIT_FAIL parser failed to commit changes: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable May 22 12:16:03 kea-dhcp6 20782 ERROR [kea-dhcp6.dhcp6.0x3731b6012000] DHCP6_CONFIG_LOAD_FAIL configuration error using file: /usr/local/etc/kea/kea-dhcp6.conf, reason: cannot lock socket lockfile, /tmp/kea6-ctrl-socket.lock, : Resource temporarily unavailable May 22 12:16:03 kea-dhcp4 20512 INFO [kea-dhcp4.dhcpsrv.0xdd682612000] DHCPSRV_CFGMGR_SOCKET_TYPE_DEFAULT "dhcp-socket-type" not specified , using default socket type raw May 22 12:16:03 kea-dhcp6 20782 ERROR [kea-dhcp6.dhcp6.0x3731b6012000] DHCP6_PARSER_COMMIT_FAIL parser failed to commit changes: cannot lock socket lockfile, /tmp/kea6-ctrl-socket.lock, : Resource temporarily unavailable
-
@maverickws Probably https://redmine.pfsense.org/issues/14977
-
Maybe this is a situation where the Service Watchdog would be of use.
-
@provels AFAIK that just tries to start the service, and it can't start because of the stale lock file.
Now, why there is a lock file is another question...presumably either Kea is crashing, or it's not cleaning up after itself when stopped.
For now I would just use ISC until Kea is out of "feature preview."
Edit: Service Watchdog may send an email...? Been quite a while since I've looked at it.
-
@provels said in Kea DHCP stops working:
Maybe this is a situation where the Service Watchdog would be of use.
Nah. Just one more situation where the "watchdog package would make thing worse".
The error is mentioned.
Remove (delete) the lock file the old fashioned way, and the issue is solved. The "watchdog package" would just :
Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - .... etc.
[ crash ]and somewhere at that moment the system will fail ....
Because pfSense is probably going to be rebooted, the /tmp/ folder will get wiped, and you won't find the issue. -
@SteveITS it does seem like the same issue. I've added a comment to the redmine report. Thanks
-
Is this fixed? I have the issue with 24.03-RELEASE:
ERROR [kea-dhcp4.dhcp4.0x220050812000] DHCP4_PARSER_COMMIT_FAIL parser failed to commit changes: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable ERROR [kea-dhcp4.dhcp4.0x220050812000] DHCP4_CONFIG_LOAD_FAIL configuration error using file: /usr/local/etc/kea/kea-dhcp4.conf, reason: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable ERROR [kea-dhcp4.dhcp4.0x220050812000] DHCP4_INIT_FAIL failed to initialize Kea server: configuration error using file '/usr/local/etc/kea/kea-dhcp4.conf': cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable
For some reason (I don't know) the DHCP stopped working and after that it's not able to restart
-
Know issue.
The kea process 'stopped without cleaning up', this event is also known as a 'crash'.
pfSense restart the process ... but there is an issue : the previous pid file, containing the pid of the previous now defunct kea process, is still there ... the startup fails.What I did : (as I wanted it to start anyway, and the issue is a bit silly) : I modded kea's shell startup scripts so it unconditionally deletes the PID file if one exists during startup.
or
Say to yourself : No way, I use ISC.
or
Delete the pid file manually and retry.I've been using kea for several weeks, despite the fact I've read the blog post about the user conditions, caveats, warnings and other fine print. It worked pretty well, and I haven't been confronted with crashes. My LANs are small, 50 devices or so, and a captive portal with a boatload of guest users. Don't recall having any issues.
I had to abandon kea as another test project obliged me to use "DHCP options" and these are also on the kea-not-yet list. -
This issue has impacted me on numerous occasions, only when the service is killed in an unclean fashion.
I also can't replicate the behavior by creating dummy lock files -- the service will still start.
Due to this unexplained behavior, I implemented a hacky workaround: creating a cron job which executes the following script:
[ `keactrl status | grep 'DHCPv4 server:' | awk '{print $3}'` = "inactive" ] && [ -f /tmp/kea4-ctrl-socket.lock ] && rm -f /tmp/kea4-ctrl-socket.lock && keactrl start -s dhcp4