Seeing Kea DHCP Issues after upgrade to 24.11

3aandl

Decided to give Kea another shot after the upgrade to 24.11 on our 1541 but after doing so while things seemed ok last night, today I am seeing the DHCP logs flooded with the following and I am unable to pull a DHCP address on this network (VLAN). If I switch back to ISC everything works just fine without issue. This looks similar to the following bug in Redmine: https://redmine.pfsense.org/issues/15735 and https://redmine.pfsense.org/issues/15328 which states it was fixed in 24.11. Has anyone seen this or have any ideas? We literally just switched back to ISC with no other changes and everything works without issue again so definitely seems like a KEA issue.

Nov 26 18:28:02 kea-dhcp4 18175 WARN [kea-dhcp4.alloc-engine.0x1aefa3e15f00] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1 2c:aa:8e:63:dd:a3], cid=[01:2c:aa:8e:63:dd:a3], tid=0x18106178: Failed to allocate an IPv4 address for client with classes: ALL, VENDOR_CLASS_udhcp 1.33.1, pool_lan_0, pool_opt3_0, pool_opt2_0, pool_opt4_0, UNKNOWN
Nov 26 18:28:02 kea-dhcp4 18175 WARN [kea-dhcp4.alloc-engine.0x1aefa3e15f00] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1 2c:aa:8e:63:dd:a3], cid=[01:2c:aa:8e:63:dd:a3], tid=0x18106178: failed to allocate an IPv4 address after 101 attempt(s)
Nov 26 18:28:02 kea-dhcp4 18175 WARN [kea-dhcp4.alloc-engine.0x1aefa3e15f00] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1 2c:aa:8e:63:dd:a3], cid=[01:2c:aa:8e:63:dd:a3], tid=0x18106178: failed to allocate an IPv4 lease in the subnet 192.168.3.0/24, subnet-id 2, shared network (none)
Nov 26 18:28:02 kea-dhcp4 18175 WARN [kea-dhcp4.alloc-engine.0x1aefa3e15f00] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1 ec:81:50:aa:a4:cb], cid=[01:ec:81:50:aa:a4:cb], tid=0x866909c8: Failed to allocate an IPv4 address for client with classes: ALL, pool_lan_0, pool_opt3_0, pool_opt2_0, pool_opt4_0, UNKNOWN
Nov 26 18:28:02 kea-dhcp4 18175 WARN [kea-dhcp4.alloc-engine.0x1aefa3e15f00] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1 ec:81:50:aa:a4:cb], cid=[01:ec:81:50:aa:a4:cb], tid=0x866909c8: failed to allocate an IPv4 address after 101 attempt(s)
Nov 26 18:28:02 kea-dhcp4 18175 WARN [kea-dhcp4.alloc-engine.0x1aefa3e15f00] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1 ec:81:50:aa:a4:cb], cid=[01:ec:81:50:aa:a4:cb], tid=0x866909c8: failed to allocate an IPv4 lease in the subnet 192.168.3.0/24, subnet-id 2, shared network (none)
Nov 26 18:28:00 kea-dhcp4 18175 WARN [kea-dhcp4.alloc-engine.0x1aefa3e15f00] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1 e8:68:e7:1f:9c:e0], cid=[no info], tid=0x207bdfb4: Failed to allocate an IPv4 address for client with classes: ALL, pool_lan_0, pool_opt3_0, pool_opt2_0, pool_opt4_0, UNKNOWN
Nov 26 18:28:00 kea-dhcp4 18175 WARN [kea-dhcp4.alloc-engine.0x1aefa3e15f00] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1 e8:68:e7:1f:9c:e0], cid=[no info], tid=0x207bdfb4: failed to allocate an IPv4 address after 101 attempt(s)
Nov 26 18:28:00 kea-dhcp4 18175 WARN [kea-dhcp4.alloc-engine.0x1aefa3e15f00] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1 e8:68:e7:1f:9c:e0], cid=[no info], tid=0x207bdfb4: failed to allocate an IPv4 lease in the subnet 192.168.3.0/24, subnet-id 2, shared network (none)
Nov 26 18:28:00 kea-dhcp4 18175 WARN [kea-dhcp4.alloc-engine.0x1aefa3e15f00] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1 a4:e5:7c:c5:57:70], cid=[no info], tid=0x961dc84e: Failed to allocate an IPv4 address for client with classes: ALL, pool_lan_0, pool_opt3_0, pool_opt2_0, pool_opt4_0, UNKNOWN
Nov 26 18:28:00 kea-dhcp4 18175 WARN [kea-dhcp4.alloc-engine.0x1aefa3e15f00] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1 a4:e5:7c:c5:57:70], cid=[no info], tid=0x961dc84e: failed to allocate an IPv4 address after 101 attempt(s)
Nov 26 18:28:00 kea-dhcp4 18175 WARN [kea-dhcp4.alloc-engine.0x1aefa3e15f00] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1 a4:e5:7c:c5:57:70], cid=[no info], tid=0x961dc84e: failed to allocate an IPv4 lease in the subnet 192.168.3.0/24, subnet-id 2, shared network (none)
Nov 26 18:28:00 kea-dhcp4 18175 WARN [kea-dhcp4.alloc-engine.0x1aefa3e15f00] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1 70:31:7f:c3:6c:6b], cid=[01:70:31:7f:c3:6c:6b], tid=0x80049ece: Failed to allocate an IPv4 address for client with classes: ALL, pool_lan_0, pool_opt3_0, pool_opt2_0, pool_opt4_0, UNKNOWN
Nov 26 18:28:00 kea-dhcp4 18175 WARN [kea-dhcp4.alloc-engine.0x1aefa3e15f00] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1 70:31:7f:c3:6c:6b], cid=[01:70:31:7f:c3:6c:6b], tid=0x80049ece: failed to allocate an IPv4 address after 101 attempt(s)
Nov 26 18:28:00 kea-dhcp4 18175 WARN [kea-dhcp4.alloc-engine.0x1aefa3e15f00] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1 70:31:7f:c3:6c:6b], cid=[01:70:31:7f:c3:6c:6b], tid=0x80049ece: failed to allocate an IPv4 lease in the subnet 192.168.3.0/24, subnet-id 2, shared network (none)
Nov 26 18:27:59 kea-dhcp4 18175 WARN [kea-dhcp4.alloc-engine.0x1aefa3e15f00] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1 2c:aa:8e:63:dd:a3], cid=[01:2c:aa:8e:63:dd:a3], tid=0x84d1813e: Failed to allocate an IPv4 address for client with classes: ALL, VENDOR_CLASS_udhcp 1.33.1, pool_lan_0, pool_opt3_0, pool_opt2_0, pool_opt4_0, UNKNOWN
Nov 26 18:27:59 kea-dhcp4 18175 WARN [kea-dhcp4.alloc-engine.0x1aefa3e15f00] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1 2c:aa:8e:63:dd:a3], cid=[01:2c:aa:8e:63:dd:a3], tid=0x84d1813e: failed to allocate an IPv4 address after 101 attempt(s)

cmcdonald

Can you try deleting all leases and trying again?

3aandl

@cmcdonald OK, went in a deleted everything and it seemed ok for a very short amount of time and then right back to flooding the logs again with the same errors and unable to pull an IP address. I also checked and there are plenty of IP's available in the pool.

cmcdonald

@3aandl can you run at Diagnostics > Command Prompt

echo '{"command":"config-get"}' | nc -U /var/run/kea4-ctrl-socket | jq

and

echo '{"command":"lease4-get-all"}' | nc -U /var/run/kea4-ctrl-socket | jq

cmcdonald

@3aandl This sort of seems like there is a misbehaving client on the network that is exhausting the pool, resulting in an unintentional (or intentional....) DoS attack on DHCP services. When Kea receives a DISCOVER it will soft-allocate a lease and wait for the client to send a REQUEST before transitioning to a hard allocation. However, during this window those addresses are unavailable. There is likely some tuning that we can do to mitigate that, but it would also be helpful to confirm that this is indeed was is happening.

3aandl

@cmcdonald I will reenable and run the command as soon as I’m back onsite. Do you feel it is just a client that doesn’t like the way Kea hands the addresses out? I tested with laptops and two iPhones (8 & 15 ProMax) and they would not pull an address but literally just going in and ticking ISC and applying and everything works as normal?

3aandl

@3aandl Reenabled and ran the command. I have the output but did lnow if I should attach the entire thing as they are both quite long.....here is a new snippet that shows it runs ok for a bit after restarting but then starts throwing errors and can't pull an IP. In this case testing with an Ipad. Can pull from one network but move over on wireless to another and no go. Again, turn off KEA and turn on ISC and all is good again with no issues. I also tested this with multiple client etc. and there are still at least 25+ IP's left in the pool to hand out.

marcosm

@3aandl Yes, please include the output of the commands. You may upload a text file as well.

3aandl

@marcosm pfkea1.txt pfkea2.txt

ysam

Hello,

We have the exact same issue. After upgrading to 24.11 dhcp on a specific vlan refuses to allocate an IP. Even though we specifically assigned an IP to a mac address, still same issue.

Dec 3 15:10:43	kea-dhcp4	69034	WARN [kea-dhcp4.alloc-engine.0x289b25d8700] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1 06:91:00:80:e0:85], cid=[01:06:91:00:80:e0:85], tid=0x61e91a5e: Failed to allocate an IPv4 address for client with classes: ALL, VENDOR_CLASS_MSFT 5.0, pool_opt4_0, pool_opt9_0, pool_opt3_0, pool_opt1_0, pool_opt6_0, pool_opt8_0, pool_lan_0, pool_opt5_0, pool_opt2_0, pool_opt7_0, pool_opt11_0, pool_opt10_0, UNKNOWN
Dec 3 15:10:43	kea-dhcp4	69034	WARN [kea-dhcp4.alloc-engine.0x289b25d8700] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1 06:91:00:80:e0:85], cid=[01:06:91:00:80:e0:85], tid=0x61e91a5e: failed to allocate an IPv4 address after 2 attempt(s)
Dec 3 15:10:43	kea-dhcp4	69034	WARN [kea-dhcp4.alloc-engine.0x289b25d8700] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1 06:91:00:80:e0:85], cid=[01:06:91:00:80:e0:85], tid=0x61e91a5e: failed to allocate an IPv4 lease in the subnet 192.168.0.0/24, subnet-id 11, shared network (none)
Dec 3 15:10:43	kea-dhcp4	69034	WARN [kea-dhcp4.alloc-engine.0x289b2418200] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1 06:91:00:80:e0:85], cid=[01:06:91:00:80:e0:85], tid=0x61e91a5e: Failed to allocate an IPv4 address for client with classes: ALL, VENDOR_CLASS_MSFT 5.0, pool_opt4_0, pool_opt9_0, pool_opt3_0, pool_opt1_0, pool_opt6_0, pool_opt8_0, pool_lan_0, pool_opt5_0, pool_opt2_0, pool_opt7_0, pool_opt11_0, pool_opt10_0, UNKNOWN
Dec 3 15:10:43	kea-dhcp4	69034	WARN [kea-dhcp4.alloc-engine.0x289b2418200] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1 06:91:00:80:e0:85], cid=[01:06:91:00:80:e0:85], tid=0x61e91a5e: failed to allocate an IPv4 address after 2 attempt(s)
Dec 3 15:10:43	kea-dhcp4	69034	WARN [kea-dhcp4.alloc-engine.0x289b2418200] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1 06:91:00:80:e0:85], cid=[01:06:91:00:80:e0:85], tid=0x61e91a5e: failed to allocate an IPv4 lease in the subnet 192.168.0.0/24, subnet-id 11, shared network (none)
Dec 3 15:10:27	kea-dhcp4	69034	WARN [kea-dhcp4.alloc-engine.0x289b2417b00] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1 06:91:00:80:e0:85], cid=[01:06:91:00:80:e0:85], tid=0x61e91a5e: Failed to allocate an IPv4 address for client with classes: ALL, VENDOR_CLASS_MSFT 5.0, pool_opt4_0, pool_opt9_0, pool_opt3_0, pool_opt1_0, pool_opt6_0, pool_opt8_0, pool_lan_0, pool_opt5_0, pool_opt2_0, pool_opt7_0, pool_opt11_0, pool_opt10_0, UNKNOWN
Dec 3 15:10:27	kea-dhcp4	69034	WARN [kea-dhcp4.alloc-engine.0x289b2417b00] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1 06:91:00:80:e0:85], cid=[01:06:91:00:80:e0:85], tid=0x61e91a5e: failed to allocate an IPv4 address after 2 attempt(s)
Dec 3 15:10:27	kea-dhcp4	69034	WARN [kea-dhcp4.alloc-engine.0x289b2417b00] ALLOC_ENGINE_V4_ALLOC_FAIL_SUBNET [hwtype=1 06:91:00:80:e0:85], cid=[01:06:91:00:80:e0:85], tid=0x61e91a5e: failed to allocate an IPv4 lease in the subnet 192.168.0.0/24, subnet-id 11, shared network (none)
Dec 3 15:10:27	kea-dhcp4	69034	WARN [kea-dhcp4.alloc-engine.0x289b2417b00] ALLOC_ENGINE_V4_ALLOC_FAIL_CLASSES [hwtype=1 06:91:00:80:e0:85], cid=[01:06:91:00:80:e0:85], tid=0x61e91a5e: Failed to allocate an IPv4 address for client with classes: ALL, VENDOR_CLASS_MSFT 5.0, pool_opt4_0, pool_opt9_0, pool_opt3_0, pool_opt1_0, pool_opt6_0, pool_opt8_0, pool_lan_0, pool_opt5_0, pool_opt2_0, pool_opt7_0, pool_opt11_0, pool_opt10_0, UNKNOWN
Dec 3 15:10:27	kea-dhcp4	69034	WARN [kea-dhcp4.alloc-engine.0x289b2417b00] ALLOC_ENGINE_V4_ALLOC_FAIL [hwtype=1 06:91:00:80:e0:85], cid=[01:06:91:00:80:e0:85], tid=0x61e91a5e: failed to allocate an IPv4 address after 2 attempt(s)

kea-config.txt

3aandl

@ysam Yeah, we switched back to ISC and everything runs just fine with no issues. We have messed with this on and off on several firewalls over the past week and we consistently see the same issues. Clear out the leases, things run ok for a bit and then the logs are flooded with what you are seeing and a number of devices quit pulling DHCP. We merely switch back to ISC and everything works right away and continues working with no errors and/or issues. Seems like there are still some issues on the implementation of KEA

Cylosoft

I came here to post a Kea issue. Seems to be along these lines. I don't have good info and can't get good info because it caused problems and we had to get back to ISC asap.

DHCP ran great for a couple of days. Had a VM with a static IP assigned by MAC that rebooted and when it did it was assigned a new IP from the pool. In the DHCP Leases Status page it showed the same MAC on the assigned IP and the new IP from the pool. We rebooted the VM several more times and it kept assigning the next IP on each reboot. So this MAC had 4 IPs. We went back to ISC and the issue went away.

Cylosoft

@3aandl Yeah we had planned to switch all devices to Kea. Got 6 in and ended up rolling 4 of them back to ISC so we stopped the project.

Biggest complaint I got was that it was re-IPing the entire network. It doesn't respect leases given out by ISC already. Then we ran into the static IP not being respected and we had to hit the brakes.

ysam

Problem is we cannot switch to ISC without huge pain!
We have 11 Vlans all with /24 dhcp space... so we will suffer a lot to do that.

Any luck for a fix?

Gertjan

@Cylosoft said in Seeing Kea DHCP Issues after upgrade to 24.11:

Then we ran into the static IP not being respected

I have about 50 of these :

and all my device still get the same IP.

@Cylosoft said in Seeing Kea DHCP Issues after upgrade to 24.11:

Biggest complaint I got was that it was re-IPing the entire network. It doesn't respect leases given out by ISC already.

AFAIK, kea doesn't use the dhcpleases storage file ISC created.
But, as the IP pool is probably (right ?) the same, and say a device using a lease like IP 192.168.1.10 right now and wants to renew 192.168.1.10,- knowing that that at that moment, 192.168.1.10 hasn't been given away to some one (as kea just started), it will give 192.168.1.10 to 192.168.1.10

cmcdonald

@Gertjan said in Seeing Kea DHCP Issues after upgrade to 24.11:

AFAIK, kea doesn't use the dhcpleases storage file ISC created.

dhcpleases will die along with ISC DHCPD. It is not used with Kea.

Cylosoft

@Gertjan Yeah static mappings work. Except when they don't. This particular network has 91 static mappings. 90 of them had no issue. The VM with an issue rebooted once and got it's correct static IP back, then the next reboot it got one from the pool. Then I forced several more reboots and it kept pulling the next pool IP. I actually deleted the static mapping and recreated, rebooted and it pulled the static, then I rebooted again and it got one from the pool. I should have taken a screenshot with the same MAC having a bunch of IPs consumed, but it was causing issues so no time for that.

After switching from ISC to Kea I fully expected Kea to give out the same lease when the client did a renewal request. On the 6 networks we did it was always that every renewal request started from the first available on the pool and went up. So on every network we had issues with duplicate IPs. Then we had to cycle switches and APs to get that quickly cleaned up. Then you get a few people doing things by IP that shouldn't be "but the IP never has changed in years". So complaints about the entire network being re-IP'd.

I agree it should be if we had 192.168.1.10 to 192.168.1.100 as a pool and the client requests 192.168.1.80 as a renewal it gets 192.168.1.80 if that's not already used. But Kea forces it down to 192.168.1.10. So then you get a duplicate IP issue for a bit.

It happened twice and I actually told my guy he must be wrong because I was sure ISC would have given out the requested IP again and no way would Kea be setup to not do that. I switched 4 of the networks myself and saw it every time.

3aandl

@Cylosoft Same here, as we dug in, we saw several times where leases that were assigned elsewhere were given to another device etc. The crazy part is there were still IP's available in the pool when it started flooding the logs and no longer handing out addresses. In most cases KEA seemed to hand out the next IP in the pool to a device and if it is rebooted etc. it would pull the next one so it seemed like one MAC may have 4-5 IP's etc. tied to it.

3aandl

@cmcdonald Don't they use "Host Reservations" in KEA which from what I understand are basically the same thing or am I not understanding this correctly?

https://kb.isc.org/docs/what-are-host-reservations-how-to-use-them

Gertjan

@3aandl said in Seeing Kea DHCP Issues after upgrade to 24.11:

https://kb.isc.org/docs/what-are-host-reservations-how-to-use-them

That's what my /usr/local/etc/kea/kea-dhcp4.conf shows :

                "reservations": [
                    {
                        "hw-address": "00:4e:01:ca:ca:9c",
                        "ip-address": "192.168.1.2",
                        "hostname": "bureau2"
                    },
                    {
                        "hw-address": "ac:15:a2:42:b0:0b",
                        "ip-address": "192.168.1.3",
                        "hostname": "TL-SG108E"
                    },
                    {
                        "hw-address": "00:15:71:f6:ce:77",
                        "ip-address": "192.168.1.4",
                        "hostname": "poweredget310"
                    },
.......