LAN switches to crashing Kiadhcp
-
After configuring an OPT interface as an additional LAN2 and configuring two LAN switches.
When I use LAN2, Kiadhcp crashes periodically. If I use LAN1, it will be ok, I think.
I do not have an error log from the dhcp, only warm and from the watchdog and also visually from the status of the services.A has used the following manuals:
Switches https://docs.netgate.com/pfsense/en/latest/solutions/xg-7100/configuring-the-switch-ports.html
Additional LAN https://docs.netgate.com/pfsense/en/latest/solutions/xg-7100/opt-lan.htmlSystem logs
Dec 31 16:33:01 php-cgi 27491 servicewatchdog_cron.php: Service Watchdog detected service kea-dhcp6 stopped. Restarting kea-dhcp6 (Kea DHCPv6 Server) Dec 31 16:31:01 php-cgi 25565 servicewatchdog_cron.php: Service Watchdog detected service kea-dhcp6 stopped. Restarting kea-dhcp6 (Kea DHCPv6 Server) Dec 31 16:29:29 php-cgi 21151 servicewatchdog_cron.php: Service Watchdog detected service kea-dhcp6 stopped. Restarting kea-dhcp6 (Kea DHCPv6 Server) Dec 31 16:29:01 php-cgi 13317 servicewatchdog_cron.php: Service Watchdog detected service kea-dhcp6 stopped. Restarting kea-dhcp6 (Kea DHCPv6 Server) Dec 31 16:28:01 php-cgi 65799 servicewatchdog_cron.php: Service Watchdog detected service kea-dhcp4 stopped. Restarting kea-dhcp4 (Kea DHCP Server) Dec 31 16:27:02 php-cgi 21151 servicewatchdog_cron.php: Service Watchdog detected service kea-dhcp4 stopped. Restarting kea-dhcp4 (Kea DHCP Server) Dec 31 16:26:41 php-cgi 20805 servicewatchdog_cron.php: Service Watchdog detected service kea-dhcp6 stopped. Restarting kea-dhcp6 (Kea DHCPv6 Server) Dec 31 16:26:02 php-cgi 72636 servicewatchdog_cron.php: Service Watchdog detected service kea-dhcp4 stopped. Restarting kea-dhcp4 (Kea DHCP Server) Dec 31 16:25:02 php-cgi 20805 servicewatchdog_cron.php: Service Watchdog detected service kea-dhcp4 stopped. Restarting kea-dhcp4 (Kea DHCP Server)DHCP logs
Dec 31 16:33:48 kea-dhcp4 56502 WARN [kea-dhcp4.dhcp4.0x3448b0612000] DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 2, queue size: 64 Dec 31 16:33:48 kea-dhcp4 56502 WARN [kea-dhcp4.dhcp4.0x3448b0612000] DHCP4_RESERVATIONS_LOOKUP_FIRST_ENABLED Multi-threading is enabled and host reservations lookup is always performed first. Dec 31 16:33:48 kea-dhcp4 56502 WARN [kea-dhcp4.dhcpsrv.0x3448b0612000] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled. Dec 31 16:33:20 kea-dhcp6 39042 WARN [kea-dhcp6.dhcp6.0x2f9238c12000] DHCP6_MULTI_THREADING_INFO enabled: yes, number of threads: 2, queue size: 64 Dec 31 16:33:20 kea-dhcp6 39042 WARN [kea-dhcp6.dhcpsrv.0x2f9238c12000] DHCPSRV_NO_SOCKETS_OPEN no interface configured to listen to DHCP traffic Dec 31 16:33:20 kea-dhcp6 39042 WARN [kea-dhcp6.dhcp6.0x2f9238c12000] DHCP6_RESERVATIONS_LOOKUP_FIRST_ENABLED Multi-threading is enabled and host reservations lookup is always performed first. Dec 31 16:33:20 kea-dhcp6 39042 WARN [kea-dhcp6.dhcpsrv.0x2f9238c12000] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled. Dec 31 16:32:53 kea-dhcp4 25981 WARN [kea-dhcp4.dhcp4.0x556a1a412000] DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 2, queue size: 64 Dec 31 16:32:53 kea-dhcp4 25981 WARN [kea-dhcp4.dhcp4.0x556a1a412000] DHCP4_RESERVATIONS_LOOKUP_FIRST_ENABLED Multi-threading is enabled and host reservations lookup is always performed first. Dec 31 16:32:53 kea-dhcp4 25981 WARN [kea-dhcp4.dhcpsrv.0x556a1a412000] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled. Dec 31 16:31:54 kea-dhcp4 71839 WARN [kea-dhcp4.dhcp4.0x567ed1c12000] DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 2, queue size: 64 Dec 31 16:31:54 kea-dhcp4 71839 WARN [kea-dhcp4.dhcp4.0x567ed1c12000] DHCP4_RESERVATIONS_LOOKUP_FIRST_ENABLED Multi-threading is enabled and host reservations lookup is always performed first. Dec 31 16:31:54 kea-dhcp4 71839 WARN [kea-dhcp4.dhcpsrv.0x567ed1c12000] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled. Dec 31 16:31:25 kea-dhcp6 38178 WARN [kea-dhcp6.dhcp6.0x2ed040a12000] DHCP6_MULTI_THREADING_INFO enabled: yes, number of threads: 2, queue size: 64 Dec 31 16:31:25 kea-dhcp6 38178 WARN [kea-dhcp6.dhcpsrv.0x2ed040a12000] DHCPSRV_NO_SOCKETS_OPEN no interface configured to listen to DHCP traffic Dec 31 16:31:25 kea-dhcp6 38178 WARN [kea-dhcp6.dhcp6.0x2ed040a12000] DHCP6_RESERVATIONS_LOOKUP_FIRST_ENABLED Multi-threading is enabled and host reservations lookup is always performed first. Dec 31 16:31:25 kea-dhcp6 38178 WARN [kea-dhcp6.dhcpsrv.0x2ed040a12000] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled. Dec 31 16:30:57 kea-dhcp4 23763 WARN [kea-dhcp4.dhcp4.0x2e494ce12000] DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 2, queue size: 64 Dec 31 16:30:57 kea-dhcp4 23763 WARN [kea-dhcp4.dhcp4.0x2e494ce12000] DHCP4_RESERVATIONS_LOOKUP_FIRST_ENABLED Multi-threading is enabled and host reservations lookup is always performed first. Dec 31 16:30:57 kea-dhcp4 23763 WARN [kea-dhcp4.dhcpsrv.0x2e494ce12000] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled. Dec 31 16:29:57 kea-dhcp4 63415 WARN [kea-dhcp4.dhcp4.0x2dd3b6012000] DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 2, queue size: 64 Dec 31 16:29:57 kea-dhcp4 63415 WARN [kea-dhcp4.dhcp4.0x2dd3b6012000] DHCP4_RESERVATIONS_LOOKUP_FIRST_ENABLED Multi-threading is enabled and host reservations lookup is always performed first. Dec 31 16:29:57 kea-dhcp4 63415 WARN [kea-dhcp4.dhcpsrv.0x2dd3b6012000] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled. Dec 31 16:29:29 kea-dhcp6 46047 WARN [kea-dhcp6.dhcp6.0x4c5512612000] DHCP6_MULTI_THREADING_INFO enabled: yes, number of threads: 2, queue size: 64 Dec 31 16:29:29 kea-dhcp6 46047 WARN [kea-dhcp6.dhcpsrv.0x4c5512612000] DHCPSRV_NO_SOCKETS_OPEN no interface configured to listen to DHCP traffic Dec 31 16:29:29 kea-dhcp6 46047 WARN [kea-dhcp6.dhcp6.0x4c5512612000] DHCP6_RESERVATIONS_LOOKUP_FIRST_ENABLED Multi-threading is enabled and host reservations lookup is always performed first. Dec 31 16:29:29 kea-dhcp6 46047 WARN [kea-dhcp6.dhcpsrv.0x4c5512612000] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled. Dec 31 16:28:59 kea-dhcp4 12111 WARN [kea-dhcp4.dhcp4.0x574984612000] DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 2, queue size: 64 Dec 31 16:28:59 kea-dhcp4 12111 WARN [kea-dhcp4.dhcp4.0x574984612000] DHCP4_RESERVATIONS_LOOKUP_FIRST_ENABLED Multi-threading is enabled and host reservations lookup is always performed first. Dec 31 16:28:59 kea-dhcp4 12111 WARN [kea-dhcp4.dhcpsrv.0x574984612000] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled.Switches VLANs
Switches ports
Thanks for help and Happy New Year.
-
@WhoAmI68 said in LAN switches to crashing Kiadhcp:
https://docs.netgate.com/pfsense/en/latest/solutions/xg-7100/configuring-the-switch-ports.html
The watchdog package is "for developers only". Believe me, you don't need it.
Most, if not all log messages you've shown are normal startup messages.
The system log servicewatchdog_cron messages are probably you as the admin (example) editing some interface related options, like switch options, and after the Save, the interface related services - like the DHCP server(s), get restarted. That's normal.
But while it was restarting, the watchdog package detected that is wasn't running (isn't that genius ?), so it will also (re) started the same service...
You got the issue here ? At beast, you'll get a nasty race condition.
At worst .... well, I never went there (as I'm not a developer ...) but I'm not going in that rabbit hole.
( told you, the watchdog is that dumb - it often breaks more then that it solves )I'm using 24.11 on a 4100. Been using kea server IPv4 and IPv6 on several LAN interfaces. No issue what so ever, and I'm not surprised. True, Every kea aspect isn't tested for a full 100 % (99,9 is probably reached ^^), but Netgate did test 24.11 on their own equipment before releasing it.
-
@Gertjan said in LAN switches to crashing Kiadhcp:
The watchdog package is "for developers only". Believe me, you don't need it.
Maybe, but I see that Surigata is down at some point in the morning. After this issue, I was used to Watchdog.
@Gertjan said in LAN switches to crashing Kiadhcp:
Most, if not all log messages you've shown are normal startup messages.
The system log servicewatchdog_cron messages are probably you as the admin (example) editing some interface related options, like switch options, and after the Save, the interface related services - like the DHCP server(s), get restarted. That's normal.
But while it was restarting, the watchdog package detected that is wasn't running (isn't that genius ?), so it will also (re) started the same service...
You got the issue here ? At beast, you'll get a nasty race condition.
At worst .... well, I never went there (as I'm not a developer ...) but I'm not going in that rabbit hole.
( told you, the watchdog is that dumb - it often breaks more then that it solves )Maybe you are right, but I do not understand why it was working well before I configured switches :). It is good that the watchdog restarts the service, but if you have 50 notifications a night that the service was down........
In any case, my opinion is that it is not a good thing.
-
@WhoAmI68 said in LAN switches to crashing Kiadhcp:
It is good that the watchdog restarts the service, but if you have 50 notifications a night that the service was down........
for that of course.
And during the morning, a good cup of coffee, and your repair the issue and call it a day.
If kea was needing the "watchdog" ti stay active, that would have been known here on the forum.
Guess what - it isn't ?So, what about undoing what you've done, to reach a stable situation again ?
D you have by any chance the log snippet where it shows that kea crashes ?
And what did the other (system, gateway alert etc) logs mention at that time ?Btw : are you using an (arm based ) 2100 or the (intel based) 7100 ?
-
@Gertjan said in LAN switches to crashing Kiadhcp:
2100
@Gertjan said in LAN switches to crashing Kiadhcp:
And during the morning, a good cup of coffee, and your repair the issue and call it a day.
If kea was needing the "watchdog" ti stay active, that would have been known here on the forum.
Guess what - it isn't ?So, what about undoing what you've done, to reach a stable situation again ?
Ok, I remove kiadhcp from watchdog. I'll test it a few days and let you know if it's down in the morning.
@Gertjan said in LAN switches to crashing Kiadhcp:
Btw : are you using an (arm based ) 2100 or the (intel based) 7100 ?
I using 2100 so correct manual was https://docs.netgate.com/pfsense/en/latest/solutions/netgate-2100/configuring-the-switch-ports.html .
Thanks for help and Happy New Year.
-
@WhoAmI68 said in LAN switches to crashing Kiadhcp:
Maybe, but I see that Surigata is down at some point in the morning. After this issue, I was used to Watchdog.
Using Service Watchdog with Suricata or Snort is another recipe for disaster. I'm the developer/maintainer for both of those packages, so trust me
, I know of what I am speaking.Service Watchdog does not know how to properly monitor the running state of the IDS/IPS packages, especially if deployed on multiple interfaces. It also does not understand that the packages periodically restart themselves for things like rules updates. The Service Watchdog package will blindly issue a restart command of its own in that circumstance which can lead to multiple copies of the IDS/IPS binary running on the same interface.
-
@Gertjan Surprisingly, it even works without a watchdog and does not go down.
Thanks for help
-
You can find my config of switch VLANs bolow.
For example, when using LAN1, I have no warm logs etc. When I am using LAN2 or LAN3, i have the log below.
WARN [kea-dhcp4.dhcpsrv.0x5f459be12000] DHCPSRV_OPEN_SOCKET_FAIL failed to open socket: Failed to open socket on interface mvneta1, reason: failed to bind fallback socket to address 192.168.103.1, port 67, reason: Address already in use - is another DHCP server running?Also, when I log in to the admin console, dhcp is down, but after reloading it is OK.
Please, someone explain it to me.
-
Go to the console or SSH.
Use option 8.ps ax | grep 'kea'Kill all the running instances.
Then, in the GUI, start kea.
-
@Gertjan Unfortunately, there is a problem with this set-up. If it is set up as shown in the pictures so it's work fine.
A solution was found
.Pics
