Kea DHCP4 lease file cleanup failed and crashed pfSense

Terho

Network connectivity was lost. pfSense LAN (igb1) gateway did not respond to ping from LAN (igb1) host using IPv4 /24 subnet. DHCP renewal time was 600 secs. Baremetal hardware running only pfSense using 1GB interfaces (igb0, igb1, igb1.20, igb1.30).

Full pfSense recovery and normal network availability achieved after cold boot by switching pfSense power OFF and back ON. Three dhcp clients were connected when fault occurred.

Two last system log messages just before network was lost:

2025-05-18 18:37:31.034495+03:00, kea-dhcp4, 37620,INFO [kea-dhcp4.dhcpsrv.0x615d6412000] DHCPSRV_MEMFILE_LFC_EXECUTE executing Lease File Cleanup using: /usr/local/sbin/kea-lfc -4 -x /var/lib/kea/dhcp4.leases.2 -i /var/lib/kea/dhcp4.leases.1 -o /var/lib/kea/dhcp4.leases.output -f /var/lib/kea/dhcp4.leases.completed -p /var/lib/kea/dhcp4.leases.pid -c ignored-path

2025-05-18 18:37:31.032762+03:00, kea-dhcp4, 37620, INFO [kea-dhcp4.dhcpsrv.0x615d6412000] DHCPSRV_MEMFILE_LFC_START starting Lease File Cleanup

Gertjan

@Terho said in Kea DHCP4 lease file cleanup failed and crashed pfSense:

Two last system log messages just before network was lost:

These are ordinary 'INFO' messages signaling that it was about to clean its less file.
Nothing special, happens all the time.

@Terho said in Kea DHCP4 lease file cleanup failed and crashed pfSense:

DHCP renewal time was 600 secs

Are you sure about that 600 seconds ?
This means renewal happens after 300 seconds.
Why so low ? 7200 seconds or more, ok. 600 is waaaaay to low.

Btw : have a look at the var/lib/kea/ folder, check (read, look at them, the files) where the leases files are stored.
Nothing special ?

Mine are a couple of Kb in size :

[25.03-BETA][root@pfSense.bhf.tld]/var/lib/kea: ls -al
total 31
drwxr-xr-x  2 root wheel      6 May 19 11:29 .
drwxr-xr-x  4 root wheel      4 Nov 19  2023 ..
-rw-r--r--  1 root wheel  17078 May 19 12:22 dhcp4.leases
-rw-r--r--  1 root wheel   5422 May 19 11:29 dhcp4.leases.2
-rw-r--r--  1 root wheel 169244 May 19 12:22 dhcp6.leases
-rw-r--r--  1 root wheel   4635 May 19 11:29 dhcp6.leases.2

Terho

The 600 sec came from an ISP DHCP server.

After I studied logs and SIEM events it turned out that the crash was caused by some sort of DDoS. Just before the crash occurred, there were plenty of blocked ingress WAN packets coming from multiple malicious or suspective IP-adddresses as tagged by Virustotal.

So, eventually this might have been a "normal DDoS" and not a pfSense software problem.