kea-dhcp6 crashes

patient0

I'm on the latest snapshot and switched from ISC DHCP to KEA DHCP after upgrading to lastest DEV.

On my Macbook Air I have three 'Location' (= networks) profiles, 'Automatic' (for IPv4 & IPv6), 'IPv4 Only' and 'IPv6 Only').

When I change from either 'Automatic' or 'IPv6 Only' to 'IPv4 Only' or from 'IPv6 Only' to 'Automatic', kea-dhcp6 crashes.

For the DHCPv6 server 'DNS registration' and 'Early DNS Registration' is checked and for the LAN interface (where the error occurs) 'Tracking Server' is set for both.

Addition: Turns out it's always crashing when changing from any location to another, not only to IPv4 locations.

In the below logs entries 1774 shows the kernel message about kea-dhcp6 dumped.

   1765 Sep 30 11:28:16 pfSense kea2unbound[67758]: Add record: "16.169.168.192.in-addr.arpa. 100 IN PTR nextcloudpivm.hsh.patient0.xyz."
   1766 Sep 30 11:28:16 pfSense kea2unbound[67758]: Add record: "5.169.168.192.in-addr.arpa. 100 IN PTR nat64dns64.hsh.patient0.xyz."
   1767 Sep 30 11:28:16 pfSense kea2unbound[67758]: Add record: "7.169.168.192.in-addr.arpa. 100 IN PTR gogs.hsh.patient0.xyz."
   1768 Sep 30 11:28:16 pfSense kea2unbound[67758]: Add record: "79.169.168.192.in-addr.arpa. 100 IN PTR xmg1915-10e.hsh.patient0.xyz."
   1769 Sep 30 11:28:16 pfSense kea2unbound[67758]: Add record: "gogs.hsh.patient0.xyz. 100 IN A 192.168.169.7"
   1770 Sep 30 11:28:16 pfSense kea2unbound[67758]: Add record: "nat64dns64.hsh.patient0.xyz. 100 IN A 192.168.169.5"
   1771 Sep 30 11:28:16 pfSense kea2unbound[67758]: Add record: "nextcloudpivm.hsh.patient0.xyz. 100 IN A 192.168.169.16"
   1772 Sep 30 11:28:16 pfSense kea2unbound[67758]: Add record: "xmg1915-10e.hsh.patient0.xyz. 100 IN A 192.168.169.79"
   1773 Sep 30 11:28:16 pfSense kea2unbound[67758]: Write include: /var/unbound/leases/leases4.conf (994f3b844bc970bcd617a92b655dc9be2003e4b9d871f7375bf7022bf180dddd)
   1774 Sep 30 11:28:22 pfSense kernel: pid 90830 (kea-dhcp6), jid 0, uid 0: exited on signal 6 (core dumped)
   1775 Sep 30 11:28:24 pfSense kea2unbound[62605]: Add record: "16.169.168.192.in-addr.arpa. 100 IN PTR nextcloudpivm.hsh.patient0.xyz."
   1776 Sep 30 11:28:24 pfSense kea2unbound[62605]: Add record: "5.169.168.192.in-addr.arpa. 100 IN PTR nat64dns64.hsh.patient0.xyz."
   1777 Sep 30 11:28:24 pfSense kea2unbound[62605]: Add record: "52.169.168.192.in-addr.arpa. 100 IN PTR thomas-air.hsh.patient0.xyz."
   1778 Sep 30 11:28:24 pfSense kea2unbound[62605]: Add record: "7.169.168.192.in-addr.arpa. 100 IN PTR gogs.hsh.patient0.xyz."
   1779 Sep 30 11:28:24 pfSense kea2unbound[62605]: Add record: "79.169.168.192.in-addr.arpa. 100 IN PTR xmg1915-10e.hsh.patient0.xyz."
   1780 Sep 30 11:28:24 pfSense kea2unbound[62605]: Add record: "gogs.hsh.patient0.xyz. 100 IN A 192.168.169.7"
   1781 Sep 30 11:28:24 pfSense kea2unbound[62605]: Add record: "nat64dns64.hsh.patient0.xyz. 100 IN A 192.168.169.5"
   1782 Sep 30 11:28:24 pfSense kea2unbound[62605]: Add record: "nextcloudpivm.hsh.patient0.xyz. 100 IN A 192.168.169.16"
   1783 Sep 30 11:28:24 pfSense kea2unbound[62605]: Add record: "thomas-air.hsh.patient0.xyz. 100 IN A 192.168.169.52"
   1784 Sep 30 11:28:24 pfSense kea2unbound[62605]: Add record: "xmg1915-10e.hsh.patient0.xyz. 100 IN A 192.168.169.79"
   1785 Sep 30 11:28:24 pfSense kea2unbound[62605]: Write include: /var/unbound/leases/leases4.conf (d0f9de272b3a1ff2c6f5aea6f5c4d2ee7580e424de38a4bc63f7843ea2fe5e64)
   1786 Sep 30 11:28:25 pfSense kea2unbound[84196]: Add record: "16.169.168.192.in-addr.arpa. 100 IN PTR nextcloudpivm.hsh.patient0.xyz."
   1787 Sep 30 11:28:25 pfSense kea2unbound[84196]: Add record: "5.169.168.192.in-addr.arpa. 100 IN PTR nat64dns64.hsh.patient0.xyz."
   1788 Sep 30 11:28:25 pfSense kea2unbound[84196]: Add record: "52.169.168.192.in-addr.arpa. 100 IN PTR thomas-air.hsh.patient0.xyz."
   1789 Sep 30 11:28:25 pfSense kea2unbound[84196]: Add record: "7.169.168.192.in-addr.arpa. 100 IN PTR gogs.hsh.patient0.xyz."
   1790 Sep 30 11:28:25 pfSense kea2unbound[84196]: Add record: "79.169.168.192.in-addr.arpa. 100 IN PTR xmg1915-10e.hsh.patient0.xyz."
   1791 Sep 30 11:28:25 pfSense kea2unbound[84196]: Add record: "gogs.hsh.patient0.xyz. 100 IN A 192.168.169.7"
   1792 Sep 30 11:28:25 pfSense kea2unbound[84196]: Add record: "nat64dns64.hsh.patient0.xyz. 100 IN A 192.168.169.5"
   1793 Sep 30 11:28:25 pfSense kea2unbound[84196]: Add record: "nextcloudpivm.hsh.patient0.xyz. 100 IN A 192.168.169.16"
   1794 Sep 30 11:28:25 pfSense kea2unbound[84196]: Add record: "thomas-air.hsh.patient0.xyz. 100 IN A 192.168.169.52"
   1795 Sep 30 11:28:25 pfSense kea2unbound[84196]: Add record: "xmg1915-10e.hsh.patient0.xyz. 100 IN A 192.168.169.79"
   1796 Sep 30 11:28:25 pfSense kea2unbound[84196]: Write include: /var/unbound/leases/leases4.conf (06e810e759549379accbaa5596bae1f220008ad9a1d364798aef85936e484f0d)
   1797 Sep 30 11:28:26 pfSense kea2unbound[63496]: Add record: "16.169.168.192.in-addr.arpa. 100 IN PTR nextcloudpivm.hsh.patient0.xyz."
   1798 Sep 30 11:28:26 pfSense kea2unbound[63496]: Add record: "5.169.168.192.in-addr.arpa. 100 IN PTR nat64dns64.hsh.patient0.xyz."
   1799 Sep 30 11:28:26 pfSense kea2unbound[63496]: Add record: "52.169.168.192.in-addr.arpa. 100 IN PTR thomas-air.hsh.patient0.xyz."
   1800 Sep 30 11:28:26 pfSense kea2unbound[63496]: Add record: "7.169.168.192.in-addr.arpa. 100 IN PTR gogs.hsh.patient0.xyz."
   1801 Sep 30 11:28:26 pfSense kea2unbound[63496]: Add record: "79.169.168.192.in-addr.arpa. 100 IN PTR xmg1915-10e.hsh.patient0.xyz."
   1802 Sep 30 11:28:26 pfSense kea2unbound[63496]: Add record: "gogs.hsh.patient0.xyz. 100 IN A 192.168.169.7"
   1803 Sep 30 11:28:26 pfSense kea2unbound[63496]: Add record: "nat64dns64.hsh.patient0.xyz. 100 IN A 192.168.169.5"
   1804 Sep 30 11:28:26 pfSense kea2unbound[63496]: Add record: "nextcloudpivm.hsh.patient0.xyz. 100 IN A 192.168.169.16"
   1805 Sep 30 11:28:26 pfSense kea2unbound[63496]: Add record: "thomas-air.hsh.patient0.xyz. 100 IN A 192.168.169.52"
   1806 Sep 30 11:28:26 pfSense kea2unbound[63496]: Add record: "xmg1915-10e.hsh.patient0.xyz. 100 IN A 192.168.169.79"
   1807 Sep 30 11:28:26 pfSense kea2unbound[63496]: Write include: /var/unbound/leases/leases4.conf (4f0adee5da9bcf645205717b0fd57ad4df664e4af0f0901744320666873754f3)
   1808 Sep 30 11:28:27 pfSense kea2unbound[83938]: Add record: "16.169.168.192.in-addr.arpa. 100 IN PTR nextcloudpivm.hsh.patient0.xyz."
   1809 Sep 30 11:28:27 pfSense kea2unbound[83938]: Add record: "5.169.168.192.in-addr.arpa. 100 IN PTR nat64dns64.hsh.patient0.xyz."
   1810 Sep 30 11:28:27 pfSense kea2unbound[83938]: Add record: "52.169.168.192.in-addr.arpa. 100 IN PTR thomas-air.hsh.patient0.xyz."
   1811 Sep 30 11:28:27 pfSense kea2unbound[83938]: Add record: "7.169.168.192.in-addr.arpa. 100 IN PTR gogs.hsh.patient0.xyz."
   1812 Sep 30 11:28:27 pfSense kea2unbound[83938]: Add record: "79.169.168.192.in-addr.arpa. 100 IN PTR xmg1915-10e.hsh.patient0.xyz."
   1813 Sep 30 11:28:27 pfSense kea2unbound[83938]: Add record: "gogs.hsh.patient0.xyz. 100 IN A 192.168.169.7"
   1814 Sep 30 11:28:27 pfSense kea2unbound[83938]: Add record: "nat64dns64.hsh.patient0.xyz. 100 IN A 192.168.169.5"
   1815 Sep 30 11:28:27 pfSense kea2unbound[83938]: Add record: "nextcloudpivm.hsh.patient0.xyz. 100 IN A 192.168.169.16"
   1816 Sep 30 11:28:27 pfSense kea2unbound[83938]: Add record: "thomas-air.hsh.patient0.xyz. 100 IN A 192.168.169.52"
   1817 Sep 30 11:28:27 pfSense kea2unbound[83938]: Add record: "xmg1915-10e.hsh.patient0.xyz. 100 IN A 192.168.169.79"
   1818 Sep 30 11:28:27 pfSense kea2unbound[56686]: Remove record: "1.0.0.1.d.0.0.0.0.0.0.0.0.0.0.0.9.6.9.1.2.e.1.3.9.6.1.0.2.0.a.2.ip6.arpa. 1200 IN PTR rt-ac88u.hsh.patient0.xyz."
   1819 Sep 30 11:28:27 pfSense kea2unbound[56686]: Remove record: "rt-ac88u.hsh.patient0.xyz. 1200 IN AAAA 2a02:169:31e2:1969::d:1001"
   1820 Sep 30 11:28:27 pfSense kea2unbound[83938]: Write include: /var/unbound/leases/leases4.conf (1b1d4a0cfd4f7686da65e593c25fb378e8e82dcd94e53bec458622978c9a5543)
   1821 Sep 30 11:28:27 pfSense kea2unbound[62799]: Add record: "16.169.168.192.in-addr.arpa. 100 IN PTR nextcloudpivm.hsh.patient0.xyz."
   1822 Sep 30 11:28:27 pfSense kea2unbound[62799]: Add record: "5.169.168.192.in-addr.arpa. 100 IN PTR nat64dns64.hsh.patient0.xyz."
   1823 Sep 30 11:28:27 pfSense kea2unbound[62799]: Add record: "52.169.168.192.in-addr.arpa. 100 IN PTR thomas-air.hsh.patient0.xyz."
   1824 Sep 30 11:28:27 pfSense kea2unbound[62799]: Add record: "7.169.168.192.in-addr.arpa. 100 IN PTR gogs.hsh.patient0.xyz."
   1825 Sep 30 11:28:27 pfSense kea2unbound[62799]: Add record: "79.169.168.192.in-addr.arpa. 100 IN PTR xmg1915-10e.hsh.patient0.xyz."
   1826 Sep 30 11:28:27 pfSense kea2unbound[62799]: Add record: "gogs.hsh.patient0.xyz. 100 IN A 192.168.169.7"
   1827 Sep 30 11:28:27 pfSense kea2unbound[62799]: Add record: "nat64dns64.hsh.patient0.xyz. 100 IN A 192.168.169.5"
   1828 Sep 30 11:28:27 pfSense kea2unbound[62799]: Add record: "nextcloudpivm.hsh.patient0.xyz. 100 IN A 192.168.169.16"
   1829 Sep 30 11:28:27 pfSense kea2unbound[62799]: Add record: "thomas-air.hsh.patient0.xyz. 100 IN A 192.168.169.52"
   1830 Sep 30 11:28:27 pfSense kea2unbound[62799]: Add record: "xmg1915-10e.hsh.patient0.xyz. 100 IN A 192.168.169.79"
   1831 Sep 30 11:28:27 pfSense kea2unbound[62799]: Write include: /var/unbound/leases/leases4.conf (1b1d4a0cfd4f7686da65e593c25fb378e8e82dcd94e53bec458622978c9a5543)
   1832 Sep 30 11:28:28 pfSense kea2unbound[83522]: Add record: "16.169.168.192.in-addr.arpa. 100 IN PTR nextcloudpivm.hsh.patient0.xyz."
   1833 Sep 30 11:28:28 pfSense kea2unbound[83522]: Add record: "5.169.168.192.in-addr.arpa. 100 IN PTR nat64dns64.hsh.patient0.xyz."
   1834 Sep 30 11:28:28 pfSense kea2unbound[83522]: Add record: "52.169.168.192.in-addr.arpa. 100 IN PTR thomas-air.hsh.patient0.xyz."
   1835 Sep 30 11:28:28 pfSense kea2unbound[83522]: Add record: "7.169.168.192.in-addr.arpa. 100 IN PTR gogs.hsh.patient0.xyz."
   1836 Sep 30 11:28:28 pfSense kea2unbound[83522]: Add record: "79.169.168.192.in-addr.arpa. 100 IN PTR xmg1915-10e.hsh.patient0.xyz."
   1837 Sep 30 11:28:28 pfSense kea2unbound[83522]: Add record: "gogs.hsh.patient0.xyz. 100 IN A 192.168.169.7"
   1838 Sep 30 11:28:28 pfSense kea2unbound[83522]: Add record: "nat64dns64.hsh.patient0.xyz. 100 IN A 192.168.169.5"
   1839 Sep 30 11:28:28 pfSense kea2unbound[83522]: Add record: "nextcloudpivm.hsh.patient0.xyz. 100 IN A 192.168.169.16"
   1840 Sep 30 11:28:28 pfSense kea2unbound[83522]: Add record: "thomas-air.hsh.patient0.xyz. 100 IN A 192.168.169.52"
   1841 Sep 30 11:28:28 pfSense kea2unbound[83522]: Add record: "xmg1915-10e.hsh.patient0.xyz. 100 IN A 192.168.169.79"
   1842 Sep 30 11:28:28 pfSense kea2unbound[83522]: Write include: /var/unbound/leases/leases4.conf (12b2a378a448f1b19dca5cc8cd7c5c223c160cf369204eb58a597eaa7db2444c)
   1843 Sep 30 11:28:28 pfSense kea2unbound[56686]: Write include: /var/unbound/leases/leases6.conf (89ccc764bb0ab7a7bda1bcc7bb73c036c2d2f14d92132b116dd3ac960d4496e9)

Is that an issue others stumble upon too?

patient0

Update: disabling all the DNS registration features doesn't help.

stephenw10

Hmm, I would have sworn that was the kea2unbound function.

You see the same error with no dhcp lease registration set?

Is it only that client that triggers it?

Can you upload that core file for us to check? It should be in /root. https://nc.netgate.com/nextcloud/s/4FNpZJZfHQtoHia

Steve

patient0

Good Morning @stephenw10, thanks for getting back to me.

I disabled DNS Registration and Early DNS Registration, yes.

Screenshot 2024-10-02 at 09.14.55.png Screenshot 2024-10-02 at 09.15.09.png

And just a few minutes ago switched on my Macbook Air M2 from the Automatic location to IPv6 Only. And it crashed again.

The kea-dhcp6.core dump is uploaded to the nextcloud.

It also happens with my older Macbook Pro 2018 model which does have the same locations. And it is connected to the same Wifi AP (ASUS RT-AC88U).

On one of the wired Linux boxes I couldn't replicate it yet, but there I have to disable/enable the interface to disable or enable IPv4 or IPv6.

stephenw10

Great I see that. Let me see what we can get from it...

stephenw10

If you can easily replicate this can you get a packet capture of the request that triggers it?

patient0

@stephenw10 it happens every time I change to a location including a IPv4 address, so yes it's easily replicated.

Is a package capture for UPD and port 67 sufficent?

stephenw10

dhcpv6 uses 546/547 so I'd capture on 67,68,546 and 547. Or maybe just filter by the client MAC. Since this appears to require switching between v4 and v6 we probably need to see both to be sure.

patient0

Forgot about DHCPv6, you are right of course.

I uploaded pfSense-Plus_pcap_kea-dhcp6-crash__20241002_001.tar.gz containing the core file, pcap captured for udp and client mac and one for udp and ports 67 68 546 547.

The relevant network infos are:

client MAC is 34:b1:eb:f3:ba:76
client IPv4 192.168.169.52 (static DHCP)
client IPv6 2a02:169:31e2:1969::f:52 (static DHCPv6)
server IPv4 192.168.169.1
server IPv6 2a02:169:31e2:1969::1

In general:

LANv4: 192.168.169.0/24
LANv6: 2a02:169:31e2:1969::/64
IPv6 addresses ending in ::c:xxxx are static IPs assigned on the node itself
IPv6 addresses ending in ::f:xxxx are static DHCPv6 mappings
IPv6 addresses ending in ::d:xxxx are from the DHCPv6 address pool

stephenw10

That's great thanks!

cmcdonald

I fixed a segfault in kea-dhcp6 a few days ago. Likely is the same issue.

patient0

@cmcdonald Any chance of getting a binary to test?

stephenw10

New image should be available 'real soon now'. Like actually soon!

behemyth

@stephenw10 said in kea-dhcp6 crashes:

New image should be available 'real soon now'. Like actually soon!

We sha’ll see, lol…

stephenw10

I mean soon is a relative term. (but it really should be, new bugs aside)

juanzelli

Well, the Redmine roadmap page changed. It's no longer showing 24.08 release. It seems something really is happening

https://redmine.pfsense.org/projects/pfsense-plus/roadmap

netblues

@juanzelli said in kea-dhcp6 crashes:

https://redmine.pfsense.org/projects/pfsense-plus/roadmap

Ah yes. Well, by looking at the open bugs (which are the same as they were in the 24.8 release) obviously the 24.08 has been merged into 24.11 which is due in ~20 days from now
So an rc version can be expected really soon, followed by the official 24.11 within November.

behemyth

Like I said above, we’ll see.

24.08 was being tested back in frickin May, and now its not even being released and now its 24.11.

Heck, this could get pushed and become 25.03.

Until I see a release, I’m not holding my breath. I dont know why you would just completely drop a release and roll it into another release.

Just release something, and then work on the next one.

stephenw10

Well releasing something broken would be the worst thing we could do IMO!

By new image I mean a new public snapshot for testing.

netblues

@behemyth said in kea-dhcp6 crashes:

Until I see a release, I’m not holding my breath. I dont know why you would just completely drop a release and roll it into another release.

Just release something, and then work on the next one.

Releases don't really mean a lot, especially when they are not bound to a specific feature set.
So as long as the product is secure, and released features work as expected, date bound releases aren't that critical.
I could say that working towards a 3 or 4 per year releases as a target is nice, but setting dates becomes unrealistic, especially when new great features come into the pipeline (in this case, cloud management of multiple devices).

Perhaps naming the release as "next", and renaming it into the month it is finally released is better.