Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    KEA service stopping through the day

    Scheduled Pinned Locked Moved
    DHCP and DNS
    13
    30
    4.9k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • GertjanG
      Gertjan @jimp
      last edited by Gertjan

      @jimp said in KEA service stopping through the day:

      If it happens again look back farther in the log for other errors. Something had to have made it fail and leave that lock file behind.

      kea pocess ID 66699 runs fine :

      <134>1 2023-11-13T12:27:46.196757+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.leases.0xfd5a1a16d00] DHCP4_LEASE_ALLOC [hwtype=1 f0:03:8c:bd:82:51], cid=[01:f0:03:8c:bd:82:51], tid=0xe6ad31c9: lease 192.168.1.26 has been allocated for 86400 seconds
<134>1 2023-11-13T12:27:53.094095+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_lan_0 evaluated to 1
<134>1 2023-11-13T12:27:53.094150+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_opt2_0 evaluated to 1
<134>1 2023-11-13T12:27:53.094186+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_opt1_0 evaluated to 1
<134>1 2023-11-13T12:27:53.094393+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.leases.0xfd5a1a16d00] DHCP4_INIT_REBOOT [hwtype=1 a4:bb:6d:ba:0a:cc], cid=[01:a4:bb:6d:ba:0a:cc], tid=0x8b206146: client is in INIT-REBOOT state and requests address 192.168.1.7
<134>1 2023-11-13T12:27:53.094659+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.leases.0xfd5a1a16d00] DHCP4_LEASE_ALLOC [hwtype=1 a4:bb:6d:ba:0a:cc], cid=[01:a4:bb:6d:ba:0a:cc], tid=0x8b206146: lease 192.168.1.7 has been allocated for 86400 seconds
<134>1 2023-11-13T12:27:53.182388+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_lan_0 evaluated to 1
<134>1 2023-11-13T12:27:53.182435+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_opt2_0 evaluated to 1
<134>1 2023-11-13T12:27:53.182470+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_opt1_0 evaluated to 1
<134>1 2023-11-13T12:27:53.182640+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.leases.0xfd5a1a16d00] DHCP4_INIT_REBOOT [hwtype=1 a4:bb:6d:ba:0a:cc], cid=[01:a4:bb:6d:ba:0a:cc], tid=0x191ea2: client is in INIT-REBOOT state and requests address 192.168.1.7
<134>1 2023-11-13T12:27:53.182865+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.leases.0xfd5a1a16d00] DHCP4_LEASE_ALLOC [hwtype=1 a4:bb:6d:ba:0a:cc], cid=[01:a4:bb:6d:ba:0a:cc], tid=0x191ea2: lease 192.168.1.7 has been allocated for 86400 seconds
<134>1 2023-11-13T12:28:12.056238+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_lan_0 evaluated to 1
<134>1 2023-11-13T12:28:12.056365+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_opt2_0 evaluated to 1
<134>1 2023-11-13T12:28:12.056494+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_opt1_0 evaluated to 1
<134>1 2023-11-13T12:28:12.057010+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.leases.0xfd5a1a16d00] DHCP4_INIT_REBOOT [hwtype=1 f0:03:8c:bd:82:51], cid=[01:f0:03:8c:bd:82:51], tid=0x9fed9b30: client is in INIT-REBOOT state and requests address 192.168.1.26
<134>1 2023-11-13T12:28:12.057586+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.leases.0xfd5a1a16d00] DHCP4_LEASE_ALLOC [hwtype=1 f0:03:8c:bd:82:51], cid=[01:f0:03:8c:bd:82:51], tid=0x9fed9b30: lease 192.168.1.26 has been allocated for 86400 seconds

      At this moment, I edited some Unboud (resolver) settings, as a result DHCP (kea) got restarted ( ? ):
      A new process is created : 36914:

      

<134>1 2023-11-13T12:29:48.269908+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - INFO  [kea-dhcp4.hosts.0x2aa006212000] HOSTS_BACKENDS_REGISTERED the following host backend types are available: 
<132>1 2023-11-13T12:29:48.273600+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - WARN  [kea-dhcp4.dhcpsrv.0x2aa006212000] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled.
<132>1 2023-11-13T12:29:48.275977+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - WARN  [kea-dhcp4.dhcp4.0x2aa006212000] DHCP4_RESERVATIONS_LOOKUP_FIRST_ENABLED Multi-threading is enabled and host reservations lookup is always performed first.
<134>1 2023-11-13T12:29:48.276515+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - INFO  [kea-dhcp4.dhcpsrv.0x2aa006212000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc0
<134>1 2023-11-13T12:29:48.313747+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - INFO  [kea-dhcp4.dhcpsrv.0x2aa006212000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc1
<134>1 2023-11-13T12:29:48.313806+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - INFO  [kea-dhcp4.dhcpsrv.0x2aa006212000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc2
<134>1 2023-11-13T12:29:48.313844+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - INFO  [kea-dhcp4.dhcpsrv.0x2aa006212000] DHCPSRV_CFGMGR_SOCKET_TYPE_DEFAULT "dhcp-socket-type" not specified , using default socket type raw
<134>1 2023-11-13T12:29:48.316396+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - INFO  [kea-dhcp4.hooks.0x2aa006212000] HOOKS_LIBRARY_CLOSED hooks library /usr/local/lib/kea/hooks/libdhcp_lease_cmds.so successfully closed
<134>1 2023-11-13T12:29:48.317096+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - INFO  [kea-dhcp4.dhcpsrv.0x2aa006212000] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 192.168.1.0/24 with params: valid-lifetime=86400
<134>1 2023-11-13T12:29:48.320114+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - INFO  [kea-dhcp4.dhcpsrv.0x2aa006212000] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 192.168.2.0/24 with params: valid-lifetime=86400
<134>1 2023-11-13T12:29:48.320724+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - INFO  [kea-dhcp4.dhcpsrv.0x2aa006212000] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 192.168.100.0/24 with params: valid-lifetime=7200
<134>1 2023-11-13T12:29:48.322962+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - INFO  [kea-dhcp4.dhcpsrv.0x2aa006212000] DHCPSRV_CFGMGR_SOCKET_TYPE_SELECT using socket type raw
<134>1 2023-11-13T12:29:48.323307+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - INFO  [kea-dhcp4.dhcpsrv.0x2aa006212000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc0
<134>1 2023-11-13T12:29:48.323345+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - INFO  [kea-dhcp4.dhcpsrv.0x2aa006212000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc1
<134>1 2023-11-13T12:29:48.323390+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - INFO  [kea-dhcp4.dhcpsrv.0x2aa006212000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc2
<134>1 2023-11-13T12:29:48.323427+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - INFO  [kea-dhcp4.dhcpsrv.0x2aa006212000] DHCPSRV_CFGMGR_SOCKET_TYPE_DEFAULT "dhcp-socket-type" not specified , using default socket type raw
<131>1 2023-11-13T12:29:48.323711+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - ERROR [kea-dhcp4.dhcp4.0x2aa006212000] DHCP4_PARSER_COMMIT_FAIL parser failed to commit changes: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable
<131>1 2023-11-13T12:29:48.324342+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - ERROR [kea-dhcp4.dhcp4.0x2aa006212000] DHCP4_CONFIG_LOAD_FAIL configuration error using file: /usr/local/etc/kea/kea-dhcp4.conf, reason: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable
<131>1 2023-11-13T12:29:48.324808+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 36914 - - ERROR [kea-dhcp4.dhcp4.0x2aa006212000] DHCP4_INIT_FAIL failed to initialize Kea server: configuration error using file '/usr/local/etc/kea/kea-dhcp4.conf': cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable

      Take note at the end of 36914 : it failed , as 66699 is still running and doing its job.

      
<134>1 2023-11-13T12:29:48.683307+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - INFO  [kea-dhcp4.hosts.0x31db2a812000] HOSTS_BACKENDS_REGISTERED the following host backend types are available: 
<132>1 2023-11-13T12:29:48.685062+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - WARN  [kea-dhcp4.dhcpsrv.0x31db2a812000] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled.
<132>1 2023-11-13T12:29:48.685672+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - WARN  [kea-dhcp4.dhcp4.0x31db2a812000] DHCP4_RESERVATIONS_LOOKUP_FIRST_ENABLED Multi-threading is enabled and host reservations lookup is always performed first.
<134>1 2023-11-13T12:29:48.689584+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - INFO  [kea-dhcp4.dhcpsrv.0x31db2a812000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc0
<134>1 2023-11-13T12:29:48.689821+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - INFO  [kea-dhcp4.dhcpsrv.0x31db2a812000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc1
<134>1 2023-11-13T12:29:48.689924+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - INFO  [kea-dhcp4.dhcpsrv.0x31db2a812000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc2
<134>1 2023-11-13T12:29:48.690030+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - INFO  [kea-dhcp4.dhcpsrv.0x31db2a812000] DHCPSRV_CFGMGR_SOCKET_TYPE_DEFAULT "dhcp-socket-type" not specified , using default socket type raw
<134>1 2023-11-13T12:29:48.692835+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - INFO  [kea-dhcp4.hooks.0x31db2a812000] HOOKS_LIBRARY_CLOSED hooks library /usr/local/lib/kea/hooks/libdhcp_lease_cmds.so successfully closed
<134>1 2023-11-13T12:29:48.693681+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - INFO  [kea-dhcp4.dhcpsrv.0x31db2a812000] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 192.168.1.0/24 with params: valid-lifetime=86400
<134>1 2023-11-13T12:29:48.696870+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - INFO  [kea-dhcp4.dhcpsrv.0x31db2a812000] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 192.168.2.0/24 with params: valid-lifetime=86400
<134>1 2023-11-13T12:29:48.697584+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - INFO  [kea-dhcp4.dhcpsrv.0x31db2a812000] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 192.168.100.0/24 with params: valid-lifetime=7200
<134>1 2023-11-13T12:29:48.699934+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - INFO  [kea-dhcp4.dhcpsrv.0x31db2a812000] DHCPSRV_CFGMGR_SOCKET_TYPE_SELECT using socket type raw
<134>1 2023-11-13T12:29:48.700783+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - INFO  [kea-dhcp4.dhcpsrv.0x31db2a812000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc0
<134>1 2023-11-13T12:29:48.700910+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - INFO  [kea-dhcp4.dhcpsrv.0x31db2a812000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc1
<134>1 2023-11-13T12:29:48.701014+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - INFO  [kea-dhcp4.dhcpsrv.0x31db2a812000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc2
<134>1 2023-11-13T12:29:48.701113+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - INFO  [kea-dhcp4.dhcpsrv.0x31db2a812000] DHCPSRV_CFGMGR_SOCKET_TYPE_DEFAULT "dhcp-socket-type" not specified , using default socket type raw
<131>1 2023-11-13T12:29:48.701479+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - ERROR [kea-dhcp4.dhcp4.0x31db2a812000] DHCP4_PARSER_COMMIT_FAIL parser failed to commit changes: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable
<131>1 2023-11-13T12:29:48.702220+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - ERROR [kea-dhcp4.dhcp4.0x31db2a812000] DHCP4_CONFIG_LOAD_FAIL configuration error using file: /usr/local/etc/kea/kea-dhcp4.conf, reason: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable
<131>1 2023-11-13T12:29:48.702769+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 57958 - - ERROR [kea-dhcp4.dhcp4.0x31db2a812000] DHCP4_INIT_FAIL failed to initialize Kea server: configuration error using file '/usr/local/etc/kea/kea-dhcp4.conf': cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable

      I've restarted kea on the dash board GUI, as it was signalled 'stopped'.
      Take note at the end of 57958 : it failed , as 66699 is still running and doing its job.

      
<134>1 2023-11-13T12:29:57.883187+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_lan_0 evaluated to 1
<134>1 2023-11-13T12:29:57.883236+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_opt2_0 evaluated to 1
<134>1 2023-11-13T12:29:57.883270+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_opt1_0 evaluated to 1

<134>1 2023-11-13T12:30:52.944681+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - INFO  [kea-dhcp4.hosts.0x2c2191012000] HOSTS_BACKENDS_REGISTERED the following host backend types are available: 
<132>1 2023-11-13T12:30:52.946233+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - WARN  [kea-dhcp4.dhcpsrv.0x2c2191012000] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled.
<132>1 2023-11-13T12:30:52.946413+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - WARN  [kea-dhcp4.dhcp4.0x2c2191012000] DHCP4_RESERVATIONS_LOOKUP_FIRST_ENABLED Multi-threading is enabled and host reservations lookup is always performed first.
<134>1 2023-11-13T12:30:52.946581+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - INFO  [kea-dhcp4.dhcpsrv.0x2c2191012000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc0
<134>1 2023-11-13T12:30:52.946701+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - INFO  [kea-dhcp4.dhcpsrv.0x2c2191012000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc1
<134>1 2023-11-13T12:30:52.946808+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - INFO  [kea-dhcp4.dhcpsrv.0x2c2191012000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc2
<134>1 2023-11-13T12:30:52.946913+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - INFO  [kea-dhcp4.dhcpsrv.0x2c2191012000] DHCPSRV_CFGMGR_SOCKET_TYPE_DEFAULT "dhcp-socket-type" not specified , using default socket type raw
<134>1 2023-11-13T12:30:52.949597+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - INFO  [kea-dhcp4.hooks.0x2c2191012000] HOOKS_LIBRARY_CLOSED hooks library /usr/local/lib/kea/hooks/libdhcp_lease_cmds.so successfully closed
<134>1 2023-11-13T12:30:52.950462+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - INFO  [kea-dhcp4.dhcpsrv.0x2c2191012000] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 192.168.1.0/24 with params: valid-lifetime=86400
<134>1 2023-11-13T12:30:52.953709+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - INFO  [kea-dhcp4.dhcpsrv.0x2c2191012000] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 192.168.2.0/24 with params: valid-lifetime=86400
<134>1 2023-11-13T12:30:52.954451+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - INFO  [kea-dhcp4.dhcpsrv.0x2c2191012000] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 192.168.100.0/24 with params: valid-lifetime=7200
<134>1 2023-11-13T12:30:52.956837+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - INFO  [kea-dhcp4.dhcpsrv.0x2c2191012000] DHCPSRV_CFGMGR_SOCKET_TYPE_SELECT using socket type raw
<134>1 2023-11-13T12:30:52.957271+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - INFO  [kea-dhcp4.dhcpsrv.0x2c2191012000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc0
<134>1 2023-11-13T12:30:52.957398+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - INFO  [kea-dhcp4.dhcpsrv.0x2c2191012000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc1
<134>1 2023-11-13T12:30:52.957509+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - INFO  [kea-dhcp4.dhcpsrv.0x2c2191012000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc2
<134>1 2023-11-13T12:30:52.957615+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - INFO  [kea-dhcp4.dhcpsrv.0x2c2191012000] DHCPSRV_CFGMGR_SOCKET_TYPE_DEFAULT "dhcp-socket-type" not specified , using default socket type raw
<131>1 2023-11-13T12:30:52.957957+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - ERROR [kea-dhcp4.dhcp4.0x2c2191012000] DHCP4_PARSER_COMMIT_FAIL parser failed to commit changes: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable
<131>1 2023-11-13T12:30:52.958709+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - ERROR [kea-dhcp4.dhcp4.0x2c2191012000] DHCP4_CONFIG_LOAD_FAIL configuration error using file: /usr/local/etc/kea/kea-dhcp4.conf, reason: cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable
<131>1 2023-11-13T12:30:52.959259+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 18912 - - ERROR [kea-dhcp4.dhcp4.0x2c2191012000] DHCP4_INIT_FAIL failed to initialize Kea server: configuration error using file '/usr/local/etc/kea/kea-dhcp4.conf': cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable

      Take note at the end of 18912 : it failed to staet, as 66699 is still running and doing its job.

      
<134>1 2023-11-13T12:33:22.241513+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_lan_0 evaluated to 1
<134>1 2023-11-13T12:33:22.241642+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_opt2_0 evaluated to 1
<134>1 2023-11-13T12:33:22.241738+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_opt1_0 evaluated to 1
<134>1 2023-11-13T12:33:22.242195+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.leases.0xfd5a1a16d00] DHCP4_INIT_REBOOT [hwtype=1 f0:03:8c:bd:82:51], cid=[01:f0:03:8c:bd:82:51], tid=0xf86e2fe4: client is in INIT-REBOOT state and requests address 192.168.1.26
<134>1 2023-11-13T12:33:22.242848+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.leases.0xfd5a1a16d00] DHCP4_LEASE_ALLOC [hwtype=1 f0:03:8c:bd:82:51], cid=[01:f0:03:8c:bd:82:51], tid=0xf86e2fe4: lease 192.168.1.26 has been allocated for 86400 seconds
<134>1 2023-11-13T12:33:24.233442+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_lan_0 evaluated to 1
<134>1 2023-11-13T12:33:24.233572+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_opt2_0 evaluated to 1
<134>1 2023-11-13T12:33:24.233667+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_opt1_0 evaluated to 1
<134>1 2023-11-13T12:33:24.234106+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.leases.0xfd5a1a16d00] DHCP4_INIT_REBOOT [hwtype=1 f0:03:8c:bd:82:51], cid=[01:f0:03:8c:bd:82:51], tid=0xf86e2fe4: client is in INIT-REBOOT state and requests address 192.168.1.26
<134>1 2023-11-13T12:33:24.234730+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.leases.0xfd5a1a16d00] DHCP4_LEASE_ALLOC [hwtype=1 f0:03:8c:bd:82:51], cid=[01:f0:03:8c:bd:82:51], tid=0xf86e2fe4: lease 192.168.1.26 has been allocated for 86400 seconds
<134>1 2023-11-13T12:33:40.932704+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_lan_0 evaluated to 1
<134>1 2023-11-13T12:33:40.932818+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_opt2_0 evaluated to 1
<134>1 2023-11-13T12:33:40.932903+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcpsrv.0xfd5a1a16d00] EVAL_RESULT Expression pool_opt1_0 evaluated to 1

<134>1 2023-11-13T12:34:12.983864+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.dhcp4.0xfd5a1a12000] DHCP4_SHUTDOWN server shutdown
<134>1 2023-11-13T12:34:12.984898+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.lease-cmds-hooks.0xfd5a1a12000] LEASE_CMDS_DEINIT_OK unloading Lease Commands hooks library successful
<134>1 2023-11-13T12:34:12.985316+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 66699 - - INFO  [kea-dhcp4.hooks.0xfd5a1a12000] HOOKS_LIBRARY_CLOSED hooks library /usr/local/lib/kea/hooks/libdhcp_lease_cmds.so successfully closed

      Now, I've killed 66699 manually.
      Delete the lock file.
      Started kea from the dashboard, kea process ID 81504 was created - and running since (already yesterday).

      
<134>1 2023-11-13T12:34:35.689415+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 81504 - - INFO  [kea-dhcp4.hosts.0x248197412000] HOSTS_BACKENDS_REGISTERED the following host backend types are available: 
<132>1 2023-11-13T12:34:35.691663+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 81504 - - WARN  [kea-dhcp4.dhcpsrv.0x248197412000] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled.
<132>1 2023-11-13T12:34:35.694022+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 81504 - - WARN  [kea-dhcp4.dhcp4.0x248197412000] DHCP4_RESERVATIONS_LOOKUP_FIRST_ENABLED Multi-threading is enabled and host reservations lookup is always performed first.
<134>1 2023-11-13T12:34:35.694348+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 81504 - - INFO  [kea-dhcp4.dhcpsrv.0x248197412000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc0
<134>1 2023-11-13T12:34:35.694646+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 81504 - - INFO  [kea-dhcp4.dhcpsrv.0x248197412000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc1
<134>1 2023-11-13T12:34:35.696831+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 81504 - - INFO  [kea-dhcp4.dhcpsrv.0x248197412000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc2
<134>1 2023-11-13T12:34:35.697090+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 81504 - - INFO  [kea-dhcp4.dhcpsrv.0x248197412000] DHCPSRV_CFGMGR_SOCKET_TYPE_DEFAULT "dhcp-socket-type" not specified , using default socket type raw
<134>1 2023-11-13T12:34:35.699837+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 81504 - - INFO  [kea-dhcp4.hooks.0x248197412000] HOOKS_LIBRARY_CLOSED hooks library /usr/local/lib/kea/hooks/libdhcp_lease_cmds.so successfully closed
<134>1 2023-11-13T12:34:35.700531+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 81504 - - INFO  [kea-dhcp4.dhcpsrv.0x248197412000] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 192.168.1.0/24 with params: valid-lifetime=86400
<134>1 2023-11-13T12:34:35.703503+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 81504 - - INFO  [kea-dhcp4.dhcpsrv.0x248197412000] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 192.168.2.0/24 with params: valid-lifetime=86400
<134>1 2023-11-13T12:34:35.704083+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 81504 - - INFO  [kea-dhcp4.dhcpsrv.0x248197412000] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 192.168.100.0/24 with params: valid-lifetime=7200
<134>1 2023-11-13T12:34:35.706428+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 81504 - - INFO  [kea-dhcp4.dhcpsrv.0x248197412000] DHCPSRV_CFGMGR_SOCKET_TYPE_SELECT using socket type raw
<134>1 2023-11-13T12:34:35.707055+01:00 pfSense.brit-hotel-fumel.net kea-dhcp4 81504 - - INFO  [kea-dhcp4.dhcpsrv.0x248197412000] DHCPSRV_CFGMGR_ADD_IFACE listening on interface igc0

      66699 never died or someting like that.
      There is no indication in the log file that the initial kea process 66699 actually received a 'stop' or 'kill' signal.

      No "help me" PM's please. Use the forum, the community will thank you.
      Edit : and where are the logs ??

      1 Reply Last reply Reply Quote 0
      • jimpJ
        jimp Rebel Alliance Developer Netgate
        last edited by

        Curious. I wonder what the contents of /var/run/kea/kea-dhcp4.kea-dhcp4.pid were during that time.

        I could maybe see something like that happening if the PID file in /var/run/kea/kea-dhcp4.kea-dhcp4.pid contained a PID that didn't match the daemon that was running.

        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

        Need help fast? Netgate Global Support!

        Do not Chat/PM for help!

        GertjanG chudakC 2 Replies Last reply Reply Quote 0
        • GertjanG
          Gertjan @jimp
          last edited by

          @jimp

          That's the one I'll be looking at - the content - the next time.

          No "help me" PM's please. Use the forum, the community will thank you.
          Edit : and where are the logs ??

          1 Reply Last reply Reply Quote 1
          • GertjanG Gertjan referenced this topic on
          • C
            Cobrax2
            last edited by

            I switched to kea too, and it can't restart if it somehow gets hung or killed by unbound because of that lock file. Is there a way to add the rm command in the startup file of kea? What is the startup command for it, where is it located? I'd hate to have to run a cron job every minute just to remove it...
            Thanks

            1 Reply Last reply Reply Quote 0
            • M
              Markito
              last edited by

              My KEA service (in an HA setup with two pfSense instances, both 2.7.2) kept behaving oddly, so I reverted to the ISC DHCP server which so far has been 100% stable. I don't think KEA is ready for prime time yet. Here's what I've observed:

              • Once, I entered 10-15 static IP mappings, and the next day I noticed they were all gone.
              • KEA server sometimes ran OK on both HA nodes, sometimes it stopped on one node but not on the other
              • While adding static IP mappings today, I went back to edit a handful to enable Static ARP on the MASTER HA NODE. Reproduction steps:
                ** Click EDIT on a static DHCP rule.
                ** Click on static Arp checkbox to enable
                ** Hit Enter
                ** Go to Status -> DHCP Leases
                ** It's blank. The KEA Service on MASTER NODE had stopped itself with the error: "failed to initialize Kea server: configuration error using file '/usr/local/etc/kea/kea-dhcp4.conf': cannot lock socket lockfile, /tmp/kea4-ctrl-socket.lock, : Resource temporarily unavailable"
                ** This was very reproducible
              • Also, when doing CARP failover, sometimes KEA would stop on one node and not run on both after I disabled the forced-failover

              Again, wayyyyy too many issues.

              S 1 Reply Last reply Reply Quote 0
              • S
                SteveITS Rebel Alliance @Markito
                last edited by

                @Markito said in KEA service stopping through the day:

                sometimes ran OK on both HA nodes

                https://docs.netgate.com/pfsense/en/latest/releases/23-09.html#kea-dhcp-server-feature-preview-now-available

                "Currently the Kea implementation lacks the following DHCP server features:
                ... High Availability Failover"

                Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                Upvote 👍 helpful posts!

                M 1 Reply Last reply Reply Quote 1
                • M
                  Markito @SteveITS
                  last edited by

                  @SteveITS Thanks :) I had not realized that.

                  1 Reply Last reply Reply Quote 0
                  • T
                    ThomasDr @w0w
                    last edited by

                    @w0w
                    This can happen if you have switched from dhcpd to kea but have not changed the service watchdog.

                    1 Reply Last reply Reply Quote 1
                    • chudakC
                      chudak @jimp
                      last edited by

                      @jimp Today see TS state unexpected state: NoState and removing /tmp/kea4-ctrl-socket.lock does no help

                      something new?

                      1 Reply Last reply Reply Quote 0
                      • M
                        marcosm Netgate
                        last edited by

                        This issue should be handled with the 24.11-RC. Feedback on it would be helpful if you were hitting this previously.

                        D 1 Reply Last reply Reply Quote 0
                        • D
                          darkrolder @marcosm
                          last edited by

                          @marcosm This issue is happening to me
                          a few nights prior i woke up to some "IOT" things flashing as they couldnt connect to their wifi.
                          and found i didnt have internet, however when i got up at 6 it was working again without user intervention so i am not sure..

                          this morning i woke up to no "internet"
                          (some statically set things over ethernet were working, obv) but everything wifi was offline.

                          on the router, kea ipv4 was offline i had to click the start button, for now i have installed the watch dog server to auto restart. id send logs if i knew where and which ones you wanted to help diag this? or if this is even related? (running 24.11)

                          best regards
                          -Rolder

                          D 1 Reply Last reply Reply Quote 0
                          • D
                            DavidIr @darkrolder
                            last edited by

                            @darkrolder said in KEA service stopping through the day:

                            on the router, kea ipv4 was offline i had to click the start button, for now i have installed the watch dog server to auto restart. id send logs if i knew where and which ones you wanted to help diag this? or if this is even related? (running 24.11)

                            I have also had this experience on my Netgate 3100. Only details I could find in the logs was:

                            Dec 5 20:58:54	kernel		pid 67465 (kea-dhcp4), jid 0, uid 0: exited on signal 6 (core dumped)

                            For some reason the kia-dhcp4 process does not seem to be generating any log entries on my device so really hard to work out if it's connected.
                            I have after reading a few posts increased the size of my DHCP pool in case some IoT devices are doing something odd (I saw this as a possibility in another thread.

                            I am now in the monitoring phase but I would not have expected a DHCP service to fail so spectacularly if it ran out of addresses in it's pool to give out.

                            D M 2 Replies Last reply Reply Quote 1
                            • D
                              darkrolder @DavidIr
                              last edited by

                              @DavidIr I don't think i have that many IOT devices, maybe 10-15? mostly light switches and my
                              cell phones, tv, etc. i have them on their own vlan with the DHCP pool size at about 150 IPs, my brother (IT admin) has suggested changing the DHCP lease from the default 2 hours to 24 hours. i have done this, so far so good. but its only been 1 day. s: only time will tell if this helps

                              -Rolder

                              1 Reply Last reply Reply Quote 0
                              • M
                                marcosm Netgate @DavidIr
                                last edited by

                                @DavidIr Did this happen on 24.11? The core dump file should be in /root - sharing that would help determine what happened.

                                D 1 Reply Last reply Reply Quote 0
                                • D
                                  DavidIr @marcosm
                                  last edited by

                                  @marcosm

                                  @marcosm said in KEA service stopping through the day:

                                  @DavidIr Did this happen on 24.11? The core dump file should be in /root - sharing that would help determine what happened.

                                  Yes it did.
                                  cefcd600-2ba6-4aa8-97fe-f820e1c96dc1-image.png
                                  I assume the kea-dhcp4.core file?
                                  I'm not familiar with these files - is it safe to just attach to the forum, or should I send in some other way?

                                  M 1 Reply Last reply Reply Quote 0
                                  • M
                                    marcosm Netgate @DavidIr
                                    last edited by

                                    @DavidIr Yes. Check the timestamp of the core files, e.g. with ls -lha /root/*.core and if they are smilar (i.e. potentially related), upload them here.

                                    D 1 Reply Last reply Reply Quote 0
                                    • D
                                      DavidIr @marcosm
                                      last edited by DavidIr

                                      @marcosm I have uploaded the file I downloaded the other day, but will also upload another from today which definitely aligns to when the service unexpectedly stopped this morning - I resolved by simply restarting the service. I was surprised that the watchdog did not restart the service for me.

                                      9a05935d-f121-4067-b75c-b70a99c6ecc5-image.png

                                      D 1 Reply Last reply Reply Quote 0
                                      • D
                                        DavidIr @DavidIr
                                        last edited by

                                        @DavidIr Hi @marcosm any joy looking at the dump files?

                                        I have implemented the changes in https://forum.netgate.com/post/1199521 for now in the hope this will restart the DHCP service when it fails, but would love to understand what's going on and help solve the potentially wider issue.

                                        Thank you

                                        1 Reply Last reply Reply Quote 0
                                        • First post
                                          Last post