Kea DHCP stops working

[maverickws]

Hi,

I am writing this topic to report on an issue that has been happening sporadically without any obvious cause.
I've been using the KEA DHCP since it came out. Since then, this issue probably occurred around 5 times give it or take it.
Basically lose connection to some machines, and when I go to pfSense I see the services as Stopped.
Trying to start them does not work.
Happens for both DHCP v4 and v6

On the error log I got this:

May 22 12:16:03	kea-dhcp4	20512	ERROR [kea-dhcp4.dhcp4.0xdd682612000] DHCP4_INIT_FAIL failed to initialize Kea server: configuration error using file '/usr/local/etc/kea/kea-dhcp4.conf': cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable
May 22 12:16:03	kea-dhcp4	20512	ERROR [kea-dhcp4.dhcp4.0xdd682612000] DHCP4_CONFIG_LOAD_FAIL configuration error using file: /usr/local/etc/kea/kea-dhcp4.conf, reason: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable
May 22 12:16:03	kea-dhcp6	20782	ERROR [kea-dhcp6.dhcp6.0x3731b6012000] DHCP6_INIT_FAIL failed to initialize Kea server: configuration error using file '/usr/local/etc/kea/kea-dhcp6.conf': cannot lock socket lockfile, /tmp/kea6-ctrl-socket.lock, : Resource temporarily unavailable
May 22 12:16:03	kea-dhcp4	20512	ERROR [kea-dhcp4.dhcp4.0xdd682612000] DHCP4_PARSER_COMMIT_FAIL parser failed to commit changes: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable
May 22 12:16:03	kea-dhcp6	20782	ERROR [kea-dhcp6.dhcp6.0x3731b6012000] DHCP6_CONFIG_LOAD_FAIL configuration error using file: /usr/local/etc/kea/kea-dhcp6.conf, reason: cannot lock socket lockfile, /tmp/kea6-ctrl-socket.lock, : Resource temporarily unavailable
May 22 12:16:03	kea-dhcp4	20512	INFO [kea-dhcp4.dhcpsrv.0xdd682612000] DHCPSRV_CFGMGR_SOCKET_TYPE_DEFAULT "dhcp-socket-type" not specified , using default socket type raw
May 22 12:16:03	kea-dhcp6	20782	ERROR [kea-dhcp6.dhcp6.0x3731b6012000] DHCP6_PARSER_COMMIT_FAIL parser failed to commit changes: cannot lock socket lockfile, /tmp/kea6-ctrl-socket.lock, : Resource temporarily unavailable

[SteveITS]

@maverickws Probably https://redmine.pfsense.org/issues/14977

[provels]

Maybe this is a situation where the Service Watchdog would be of use.

[SteveITS]

@provels AFAIK that just tries to start the service, and it can't start because of the stale lock file.

Now, why there is a lock file is another question...presumably either Kea is crashing, or it's not cleaning up after itself when stopped.

For now I would just use ISC until Kea is out of "feature preview."

Edit: Service Watchdog may send an email...? Been quite a while since I've looked at it.

[Gertjan]

@provels said in Kea DHCP stops working:

Maybe this is a situation where the Service Watchdog would be of use.

Nah. Just one more situation where the "watchdog package would make thing worse".
The error is mentioned.
Remove (delete) the lock file the old fashioned way, and the issue is solved. The "watchdog package" would just :
Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - Kea not running => let's start it - wait 60 seconds - .... etc.
[ crash ]

and somewhere at that moment the system will fail ....
Because pfSense is probably going to be rebooted, the /tmp/ folder will get wiped, and you won't find the issue.

[maverickws]

@SteveITS it does seem like the same issue. I've added a comment to the redmine report. Thanks

[Overlord]

Is this fixed? I have the issue with 24.03-RELEASE:

ERROR [kea-dhcp4.dhcp4.0x220050812000] DHCP4_PARSER_COMMIT_FAIL parser failed to commit changes: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable

ERROR [kea-dhcp4.dhcp4.0x220050812000] DHCP4_CONFIG_LOAD_FAIL configuration error using file: /usr/local/etc/kea/kea-dhcp4.conf, reason: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable

ERROR [kea-dhcp4.dhcp4.0x220050812000] DHCP4_INIT_FAIL failed to initialize Kea server: configuration error using file '/usr/local/etc/kea/kea-dhcp4.conf': cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable

For some reason (I don't know) the DHCP stopped working and after that it's not able to restart

[Gertjan]

@Overlord

Know issue.
The kea process 'stopped without cleaning up', this event is also known as a 'crash'.
pfSense restart the process ... but there is an issue : the previous pid file, containing the pid of the previous now defunct kea process, is still there ... the startup fails.

What I did : (as I wanted it to start anyway, and the issue is a bit silly) : I modded kea's shell startup scripts so it unconditionally deletes the PID file if one exists during startup.
or
Say to yourself : No way, I use ISC.
or
Delete the pid file manually and retry.

I've been using kea for several weeks, despite the fact I've read the blog post about the user conditions, caveats, warnings and other fine print. It worked pretty well, and I haven't been confronted with crashes. My LANs are small, 50 devices or so, and a captive portal with a boatload of guest users. Don't recall having any issues.
I had to abandon kea as another test project obliged me to use "DHCP options" and these are also on the kea-not-yet list.

[zkhcohen]

This issue has impacted me on numerous occasions, only when the service is killed in an unclean fashion.

I also can't replicate the behavior by creating dummy lock files -- the service will still start.

Due to this unexplained behavior, I implemented a hacky workaround: creating a cron job which executes the following script:

[ `keactrl status | grep 'DHCPv4 server:' | awk '{print $3}'` = "inactive" ] && [ -f /tmp/kea4-ctrl-socket.lock ] && rm -f /tmp/kea4-ctrl-socket.lock && keactrl start -s dhcp4

[noisyjohn]

@maverickws
The same here ,,, But I found the solution:
login with an sftp client

directory /root/tmp chmod 777 (allow all)
delete /root/tmp/kea4-ctrl-socket
delete /root/tmp/kea4-ctrl-socket.lock
restart Kea DHCP server (the sockets are automaticaly created)
done!
however this issue may hapen again after some days ...

[maverickws]

@noisyjohn Hi there and thank you for your input.

However, I have been forced out of KEA.
The reasons are described in this topic:
Ater Kea for 1 year, reverted back to ISC

[cmcdonald]

I believe we now have a fix for this. Keep an eye out for a public beta of 24.11 very soon.

[zkhcohen]

@cmcdonald Is there any intent to release this in the Community Edition? It's now been over a year since the last major version release, and the issues are stacking up. It's starting to feel like the project is stagnating.

[Gertjan]

@zkhcohen said in Kea DHCP stops working:

and the issues are stacking up ...

This thread, this issue, only exists in 24.03.
It concerns KEA, the 'new' DHCP server/client that is going to replace ISC DHCP in the future. The latter is used in 2.7.2 and is working very well - I'm using it also in 24.03.

Right now, no one really wants 'new versions' as most of us try to deal with 24H2 and its 'new things' that no one was asking for, but we still got (to debug) it ...

[zkhcohen]

@Gertjan I can personally confirm that this issue is present in CE 2.7.2 with the same symptoms.

I'm not sure how you can claim that no one "wants a new version" when there are numerous threads off and on this forum asking for exactly that.

[Gertjan]

@zkhcohen said in Kea DHCP stops working:

issue is present in CE 2.7.2 with the same symptoms.

Aha, so 2.7.2 has also the KEA !? I stand corrected.
I really had the impression that only pfSense Plus (24.03) had this test-drive KEA addition.

Easy way out then : keep the 30+ years tested an proved "DHCP ISC" and the issue will be gone.

[datpif]

I see this is still ongoing. I found a workaround via watchdog service. The script does a cleanup before attempting to restart the sevice

• list itemmake sure you install the watchdog service in pfsense.

• list Add the kea dhcp 4 service.

• Shell into pfsense and change to /usr/local/etc/rc.d

• list Create a backup of kea service script. cp kea kea.old

• list edit the kea file and replace contents with script below

#!/bin/sh

# PROVIDE: kea
# REQUIRE: NETWORK netif routing    
# KEYWORD: shutdown

. /etc/rc.subr

name=kea
desc="Kea DHCP Server"
rcvar=kea_enable

load_rc_config $name

kea_enable=${kea_enable:-"NO"}

command="/usr/local/sbin/keactrl"
required_files="/usr/local/etc/${name}/keactrl.conf"

# Add cleanup function
cleanup_kea() {
    # Clean up stale lock files
    rm -f /tmp/kea4-ctrl-socket.lock
    # Kill any zombie processes
    pkill -9 kea-dhcp4
    # Wait for processes to die
    sleep 2
}

# Modify start command to include cleanup
start_cmd() {
    cleanup_kea
    ${command} start
    logger -t kea-watchdog "Kea DHCP4 started with cleanup"
}

# Modify stop command to include cleanup
stop_cmd() {
    ${command} stop
    cleanup_kea
    logger -t kea-watchdog "Kea DHCP4 stopped with cleanup"
}

status_cmd="$command status"
reload_cmd="$command reload"
extra_commands="reload"

run_rc_command "$1"

Watchdog should auto restart the service