ipv6 broken: radvd: can't join ipv6-allrouters on <interface>

strangegopher · Apr 8, 2019, 7:06 PM

I keep getting log spam radvd: can't join ipv6-allrouters on cxgbe0.22
Only solution seems to be stopping radvd and starting it again. Restarting does not work.
But after a while the messages appears again.
I don't know why, couldn't find a conclusive answer on google.

🔒 Log in to view

my settings:
🔒 Log in to view
🔒 Log in to view
🔒 Log in to view
🔒 Log in to view

~~edit: note that i am able to visit ipv6 sites with no issues~~ ipv6 breaks when this spam starts

strangegopher · Apr 10, 2019, 7:06 PM

I was able to trace back to beginning of the message. I looked at other logs like firewall, dhcp, general, dns but nothing significant happened at the time that this error started.

Apr 10 10:42:37	radvd	15739	can't join ipv6-allrouters on cxgbe0.7
Apr 10 10:42:32	radvd	15739	can't join ipv6-allrouters on cxgbe0.42
Apr 10 10:42:31	radvd	15739	can't join ipv6-allrouters on cxgbe0.7
Apr 10 10:42:23	radvd	15739	can't join ipv6-allrouters on cxgbe0.42
Apr 10 01:13:14	radvd	15451	version 2.17 started
Apr 10 01:06:08	radvd	8151	resuming normal operation
Apr 10 01:06:08	radvd	8151	attempting to reread config file
Apr 10 01:06:06	radvd	8151	resuming normal operation
Apr 10 01:06:06	radvd	8151	attempting to reread config file
Apr 10 01:04:53	radvd	8151	resuming normal operation
Apr 10 01:04:53	radvd	8151	attempting to reread config file
Apr 10 01:04:50	radvd	8151	resuming normal operation
Apr 10 01:04:50	radvd	8151	attempting to reread config file
Apr 10 01:02:51	radvd	8151	resuming normal operation
Apr 10 01:02:51	radvd	8151	attempting to reread config file
Apr 10 01:02:49	radvd	8151	resuming normal operation
Apr 10 01:02:49	radvd	8151	attempting to reread config file
Apr 10 01:00:39	radvd	8088	version 2.17 started
Apr 10 00:55:32	radvd	22375	resuming normal operation
Apr 10 00:55:32	radvd	22375	attempting to reread config file
Apr 10 00:55:30	radvd	22375	resuming normal operation
Apr 10 00:55:30	radvd	22375	attempting to reread config file
Apr 10 00:54:19	radvd	22375	resuming normal operation
Apr 10 00:54:19	radvd	22375	attempting to reread config file
Apr 10 00:52:25	radvd	22206	version 2.17 started
Apr 9 17:26:42	radvd	38270	resuming normal operation
Apr 9 17:26:42	radvd	38270	attempting to reread config file
Apr 9 17:26:40	radvd	38270	resuming normal operation
Apr 9 17:26:40	radvd	38270	attempting to reread config file
Apr 9 17:02:23	radvd	38270	resuming normal operation
Apr 9 17:02:23	radvd	38270	attempting to reread config file
Apr 9 17:02:21	radvd	38270	resuming normal operation
Apr 9 17:02:21	radvd	38270	attempting to reread config file
Apr 9 17:00:57	radvd	38142	version 2.17 started
Apr 9 16:52:06	radvd	59526	resuming normal operation
Apr 9 16:52:06	radvd	59526	attempting to reread config file
Apr 9 16:52:02	radvd	59526	resuming normal operation
Apr 9 16:52:02	radvd	59526	attempting to reread config file
Apr 9 16:51:59	radvd	59526	resuming normal operation
Apr 9 16:51:59	radvd	59526	attempting to reread config file
Apr 9 16:51:56	radvd	59526	resuming normal operation
Apr 9 16:51:56	radvd	59526	attempting to reread config file
Apr 9 16:50:54	radvd	59267	invalid all-zeros prefix in /var/etc/radvd.conf, line 51
Apr 9 16:50:54	radvd	59267	invalid all-zeros prefix in /var/etc/radvd.conf, line 37
Apr 9 16:50:54	radvd	59267	invalid all-zeros prefix in /var/etc/radvd.conf, line 23
Apr 9 16:50:54	radvd	59267	invalid all-zeros prefix in /var/etc/radvd.conf, line 9
Apr 9 16:50:54	radvd	59267	version 2.17 started
Apr 9 15:25:00	radvd	38338	resuming normal operation
Apr 9 15:25:00	radvd	38338	attempting to reread config file
Apr 9 15:06:11	radvd	38338	resuming normal operation
Apr 9 15:06:11	radvd	38338	attempting to reread config file
Apr 9 12:00:40	radvd	38338	resuming normal operation
Apr 9 12:00:40	radvd	38338	attempting to reread config file
Apr 9 12:00:38	radvd	38338	resuming normal operation
Apr 9 12:00:38	radvd	38338	attempting to reread config file
Apr 9 11:58:14	radvd	38120	version 2.17 started

strangegopher · Apr 13, 2019, 8:49 PM

bump

strangegopher · Apr 14, 2019, 9:14 PM

edit: I have disabled ipv6 for now

strangegopher · May 10, 2019, 1:41 PM

Just to note, when this spam starts, ipv6 stops working.

Irata · May 12, 2019, 9:16 AM

I can confirm this must be an issue.

I have exactly the same situation, after around 6-7 hours of running, radvd spams the logs with "can't join ipv6-allrouters". IPv6 stops works as all clients release their IPv6 address since no more router advertisements are ever sent.
As an interim fix, I stop and start radvd via cron every 6 hours, a click of restart does not fix it. It has to be a stop, sleep 1 second, and a start. Is there some memory leak in radvd? Or binding and not releasing to the interface? There is radvd 2.18 available now with numerous fixes.

This issue does unfortunately break IPv6 when using un-managed router advertisements.

I have turned up logging levels, but it does not add anything more related to radvd.

strangegopher · May 12, 2019, 4:59 PM

@Irata thank you, good to know that I am not the only one.
Can someone from pfsense team let us know if a bug report needs to be opened?

Irata · May 12, 2019, 5:52 PM

@strangegopher thanks, glad I found your post as I also thought it was just me!

It happens on all interfaces at the same time, so I took a packet capture on a quiet subnet - but nothing of interest again, can just see when the router advertisements stop.

Sadly my restart workaround using cron has side effects, any active IPv6 connections are dropped each time it runs. And one Android device even disconnects from WiFi each time radvd is restarted, it treats a loss of IPv6 as a network problem, bizarre but true.

Be interested to hear if anyone using radvd and IPv6 does not have this issue? Check Routing logs in System Logs.

I'm at a loss what else to do. It's interesting that a periodic restart masks the problem, does that point to radvd being the problem?

maverick_slo · Sep 1, 2019, 7:37 PM

I don't have this issue.
I'm on 2.5 with uptime 3 days.

strangegopher · May 14, 2019, 2:50 PM

@maverick_slo do u have ipv6?

maverick_slo · May 14, 2019, 3:21 PM

Since 2013

arafey · May 15, 2019, 9:45 PM

I've been having this problem as well and I also noticed that it's about 6 hours before IPv6 suddenly breaks. This should definitely be fixed before 2.5.0 is released.

crimper · May 21, 2019, 5:37 AM

I am having the same issue. IPv6 would stop working within 24 hours. I backup the config.xml from the 2.5 and restored the file to a 2.4.4_p2 and it does not have the problem.

Irata · May 21, 2019, 5:43 PM

I'm not sure how to find what's common.

I'm running pfsense within VMware ESXi and have this issue. Are others with this issue the same?

I can also confirm the latest 2.4 branch does not have this issue with the same config.

strangegopher · May 21, 2019, 5:45 PM

I am running it on baremetal dell box but I am using chelsio 10G sfp+ nic.

strangegopher · May 30, 2019, 9:14 PM

@jimp sorry for the ping but do u have any idea why ipv6 stops working?

motific · Jun 6, 2019, 7:28 PM

How is it on a fresh 2.5 install with the old config?

I’ve just had some bad behaviour with radvd not responding at all (not quite the same as your issue) and that has cleared it.

kiokoman · Jun 6, 2019, 9:54 PM

i have similar behavior on my log

Jun 6 21:32:29 	radvd 	66014 	resuming normal operation
Jun 6 21:32:29 	radvd 	66014 	attempting to reread config file
Jun 6 18:57:34 	radvd 	66014 	resuming normal operation
Jun 6 18:57:34 	radvd 	66014 	attempting to reread config file
Jun 6 14:01:49 	radvd 	65728 	version 2.17 started 
Jun 6 14:00:39 	radvd 	67952 	can't join ipv6-allrouters on igb2
Jun 6 14:00:39 	radvd 	67952 	can't join ipv6-allrouters on ath0_wlan0
Jun 6 14:00:33 	radvd 	67952 	can't join ipv6-allrouters on igb2
Jun 6 14:00:22 	radvd 	67952 	can't join ipv6-allrouters on igb2
Jun 6 14:00:20 	radvd 	67952 	can't join ipv6-allrouters on ath0_wlan0
Jun 6 14:00:08 	radvd 	67952 	can't join ipv6-allrouters on igb2
Jun 6 14:00:01 	radvd 	67952 	can't join ipv6-allrouters on ath0_wlan0
Jun 6 13:59:53 	radvd 	67952 	can't join ipv6-allrouters on ath0_wlan0
Jun 6 13:59:53 	radvd 	67952 	can't join ipv6-allrouters on igb2
Jun 6 13:59:40 	radvd 	67952 	can't join ipv6-allrouters on ath0_wlan0
Jun 6 13:59:36 	radvd 	67952 	can't join ipv6-allrouters on igb2
Jun 6 13:59:30 	radvd 	67952 	can't join ipv6-allrouters on igb2
Jun 6 13:59:25 	radvd 	67952 	can't join ipv6-allrouters on ath0_wlan0

igb0 is wan
igb1 is lan
on igb2 i have only 1 pc, a server connected with static ipv6
on ath0_wlan0 i have a couple of notebook with windows 10 and a couple of android phone

Irata · Jun 6, 2019, 10:04 PM

A fresh install causes the same error for me, once stateless IPv6 is setup. I can't find a stable working system whatever I do, other than back to 2.4.

Because I'm using an ESXi testbed, it's easy to spin up new installs.

I do think there is something fundamental here, given the reports from wide ranging setups.

kiokoman · Jun 7, 2019, 8:18 AM

the error is generated from device-bsd44.c, a patch it's inside the port of radvd for freebsd
tracking back there was similar problem. related Bug #2878 https://redmine.pfsense.org/issues/2878 from 2013

/* XXX: See pfSense ticket #2878 */
+	setsockopt(sock, IPPROTO_IPV6, IPV6_LEAVE_GROUP, &mreq, sizeof(mreq));
+
+	if (setsockopt(sock, IPPROTO_IPV6, IPV6_JOIN_GROUP,
+			&mreq, sizeof(mreq)) < 0) {
+		flog(LOG_ERR, "can't join ipv6-allrouters on %s", iface->props.name);
+		return (-1);
+ }

maybe it's time for a new ticket
https://redmine.pfsense.org/issues/9577