KEA service stopping through the day
-
My KEA service (in an HA setup with two pfSense instances, both 2.7.2) kept behaving oddly, so I reverted to the ISC DHCP server which so far has been 100% stable. I don't think KEA is ready for prime time yet. Here's what I've observed:
- Once, I entered 10-15 static IP mappings, and the next day I noticed they were all gone.
- KEA server sometimes ran OK on both HA nodes, sometimes it stopped on one node but not on the other
- While adding static IP mappings today, I went back to edit a handful to enable Static ARP on the MASTER HA NODE. Reproduction steps:
** Click EDIT on a static DHCP rule.
** Click on static Arp checkbox to enable
** Hit Enter
** Go to Status -> DHCP Leases
** It's blank. The KEA Service on MASTER NODE had stopped itself with the error: "failed to initialize Kea server: configuration error using file '/usr/local/etc/kea/kea-dhcp4.conf': cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable"
** This was very reproducible - Also, when doing CARP failover, sometimes KEA would stop on one node and not run on both after I disabled the forced-failover
Again, wayyyyy too many issues.
-
@Markito said in KEA service stopping through the day:
sometimes ran OK on both HA nodes
https://docs.netgate.com/pfsense/en/latest/releases/23-09.html#kea-dhcp-server-feature-preview-now-available
"Currently the Kea implementation lacks the following DHCP server features:
... High Availability Failover" -
@SteveITS Thanks :) I had not realized that.
-
@w0w
This can happen if you have switched from dhcpd to kea but have not changed the service watchdog. -
@jimp Today see TS state
unexpected state: NoState
and removing/tmp/kea4-ctrl-socket.lock
does no helpsomething new?
-
This issue should be handled with the 24.11-RC. Feedback on it would be helpful if you were hitting this previously.
-
@marcosm This issue is happening to me
a few nights prior i woke up to some "IOT" things flashing as they couldnt connect to their wifi.
and found i didnt have internet, however when i got up at 6 it was working again without user intervention so i am not sure..this morning i woke up to no "internet"
(some statically set things over ethernet were working, obv) but everything wifi was offline.on the router, kea ipv4 was offline i had to click the start button, for now i have installed the watch dog server to auto restart. id send logs if i knew where and which ones you wanted to help diag this? or if this is even related? (running 24.11)
best regards
-Rolder -
@darkrolder said in KEA service stopping through the day:
on the router, kea ipv4 was offline i had to click the start button, for now i have installed the watch dog server to auto restart. id send logs if i knew where and which ones you wanted to help diag this? or if this is even related? (running 24.11)
I have also had this experience on my Netgate 3100. Only details I could find in the logs was:
Dec 5 20:58:54 kernel pid 67465 (kea-dhcp4), jid 0, uid 0: exited on signal 6 (core dumped)
For some reason the kia-dhcp4 process does not seem to be generating any log entries on my device so really hard to work out if it's connected.
I have after reading a few posts increased the size of my DHCP pool in case some IoT devices are doing something odd (I saw this as a possibility in another thread.I am now in the monitoring phase but I would not have expected a DHCP service to fail so spectacularly if it ran out of addresses in it's pool to give out.
-
@DavidIr I don't think i have that many IOT devices, maybe 10-15? mostly light switches and my
cell phones, tv, etc. i have them on their own vlan with the DHCP pool size at about 150 IPs, my brother (IT admin) has suggested changing the DHCP lease from the default 2 hours to 24 hours. i have done this, so far so good. but its only been 1 day. s: only time will tell if this helps-Rolder
-
@DavidIr Did this happen on 24.11? The core dump file should be in /root - sharing that would help determine what happened.
-
@marcosm said in KEA service stopping through the day:
@DavidIr Did this happen on 24.11? The core dump file should be in /root - sharing that would help determine what happened.
Yes it did.
I assume the kea-dhcp4.core file?
I'm not familiar with these files - is it safe to just attach to the forum, or should I send in some other way? -
-
@marcosm I have uploaded the file I downloaded the other day, but will also upload another from today which definitely aligns to when the service unexpectedly stopped this morning - I resolved by simply restarting the service. I was surprised that the watchdog did not restart the service for me.
-
-
@DavidIr It would help to have some additional info about the system. You can get that by going to /status.php.
-
@marcosm status_output.tgz uploaded to the same link provided above.
Since the previous messages I have installed and configured the Service Watchdog plugin
-
In case you need any additional info I am now on holiday until Jan 5th so will not see or be able to respond to any posts or requests for info until I return.
-
Hi
this weekend the core dump happened exactly the same in my Netgate 3100.I wonder if there is a solution for this problem?
Regards,
-
"Good news" is that the reason of the core dump was a signal 6, which means the process itself has chosen to 'pull the brakes', most probably because resources were missing, like not enough RAM to name one.
-
@Gertjan said in KEA service stopping through the day:
"Good news" is that the reason of the core dump was a signal 6, which means the process itself has chosen to 'pull the brakes', most probably because resources were missing, like not enough RAM to name one.
Yes, heap corruption in this case. This is turning into quite the rabbit hole. Unfortunately, this looks like an issue deeper than Kea, like failure in libcxxrt or jemalloc. We've got some test hardware setup with some additional logging and tuning to jemalloc to try to get a better view of the state of the world before the abort. But the core dump is gnarly, the heap is trashed. The effort required to fix this might be out of scope for an EOL platform, both for us and for upstream. Will know more soon.