ipv6 broken: radvd: can't join ipv6-allrouters on <interface>



  • I keep getting log spam radvd: can't join ipv6-allrouters on cxgbe0.22
    Only solution seems to be stopping radvd and starting it again. Restarting does not work.
    But after a while the messages appears again.
    I don't know why, couldn't find a conclusive answer on google.

    926b914a-6c46-4918-9f62-721e8813c2cf-image.png

    my settings:
    16112b7f-4157-42c0-800b-3708a07202fb-image.png
    01626846-52f7-415a-9d1a-ae525d7e5eba-image.png
    22be4866-42c0-414b-844c-d50eaccef264-image.png
    7b957cd6-5684-4034-a242-7adcd2550fcb-image.png

    edit: note that i am able to visit ipv6 sites with no issues ipv6 breaks when this spam starts



  • I was able to trace back to beginning of the message. I looked at other logs like firewall, dhcp, general, dns but nothing significant happened at the time that this error started.

    Apr 10 10:42:37	radvd	15739	can't join ipv6-allrouters on cxgbe0.7
    Apr 10 10:42:32	radvd	15739	can't join ipv6-allrouters on cxgbe0.42
    Apr 10 10:42:31	radvd	15739	can't join ipv6-allrouters on cxgbe0.7
    Apr 10 10:42:23	radvd	15739	can't join ipv6-allrouters on cxgbe0.42
    Apr 10 01:13:14	radvd	15451	version 2.17 started
    Apr 10 01:06:08	radvd	8151	resuming normal operation
    Apr 10 01:06:08	radvd	8151	attempting to reread config file
    Apr 10 01:06:06	radvd	8151	resuming normal operation
    Apr 10 01:06:06	radvd	8151	attempting to reread config file
    Apr 10 01:04:53	radvd	8151	resuming normal operation
    Apr 10 01:04:53	radvd	8151	attempting to reread config file
    Apr 10 01:04:50	radvd	8151	resuming normal operation
    Apr 10 01:04:50	radvd	8151	attempting to reread config file
    Apr 10 01:02:51	radvd	8151	resuming normal operation
    Apr 10 01:02:51	radvd	8151	attempting to reread config file
    Apr 10 01:02:49	radvd	8151	resuming normal operation
    Apr 10 01:02:49	radvd	8151	attempting to reread config file
    Apr 10 01:00:39	radvd	8088	version 2.17 started
    Apr 10 00:55:32	radvd	22375	resuming normal operation
    Apr 10 00:55:32	radvd	22375	attempting to reread config file
    Apr 10 00:55:30	radvd	22375	resuming normal operation
    Apr 10 00:55:30	radvd	22375	attempting to reread config file
    Apr 10 00:54:19	radvd	22375	resuming normal operation
    Apr 10 00:54:19	radvd	22375	attempting to reread config file
    Apr 10 00:52:25	radvd	22206	version 2.17 started
    Apr 9 17:26:42	radvd	38270	resuming normal operation
    Apr 9 17:26:42	radvd	38270	attempting to reread config file
    Apr 9 17:26:40	radvd	38270	resuming normal operation
    Apr 9 17:26:40	radvd	38270	attempting to reread config file
    Apr 9 17:02:23	radvd	38270	resuming normal operation
    Apr 9 17:02:23	radvd	38270	attempting to reread config file
    Apr 9 17:02:21	radvd	38270	resuming normal operation
    Apr 9 17:02:21	radvd	38270	attempting to reread config file
    Apr 9 17:00:57	radvd	38142	version 2.17 started
    Apr 9 16:52:06	radvd	59526	resuming normal operation
    Apr 9 16:52:06	radvd	59526	attempting to reread config file
    Apr 9 16:52:02	radvd	59526	resuming normal operation
    Apr 9 16:52:02	radvd	59526	attempting to reread config file
    Apr 9 16:51:59	radvd	59526	resuming normal operation
    Apr 9 16:51:59	radvd	59526	attempting to reread config file
    Apr 9 16:51:56	radvd	59526	resuming normal operation
    Apr 9 16:51:56	radvd	59526	attempting to reread config file
    Apr 9 16:50:54	radvd	59267	invalid all-zeros prefix in /var/etc/radvd.conf, line 51
    Apr 9 16:50:54	radvd	59267	invalid all-zeros prefix in /var/etc/radvd.conf, line 37
    Apr 9 16:50:54	radvd	59267	invalid all-zeros prefix in /var/etc/radvd.conf, line 23
    Apr 9 16:50:54	radvd	59267	invalid all-zeros prefix in /var/etc/radvd.conf, line 9
    Apr 9 16:50:54	radvd	59267	version 2.17 started
    Apr 9 15:25:00	radvd	38338	resuming normal operation
    Apr 9 15:25:00	radvd	38338	attempting to reread config file
    Apr 9 15:06:11	radvd	38338	resuming normal operation
    Apr 9 15:06:11	radvd	38338	attempting to reread config file
    Apr 9 12:00:40	radvd	38338	resuming normal operation
    Apr 9 12:00:40	radvd	38338	attempting to reread config file
    Apr 9 12:00:38	radvd	38338	resuming normal operation
    Apr 9 12:00:38	radvd	38338	attempting to reread config file
    Apr 9 11:58:14	radvd	38120	version 2.17 started
    


  • bump



  • edit: I have disabled ipv6 for now



  • Just to note, when this spam starts, ipv6 stops working.



  • I can confirm this must be an issue.

    I have exactly the same situation, after around 6-7 hours of running, radvd spams the logs with "can't join ipv6-allrouters". IPv6 stops works as all clients release their IPv6 address since no more router advertisements are ever sent.
    As an interim fix, I stop and start radvd via cron every 6 hours, a click of restart does not fix it. It has to be a stop, sleep 1 second, and a start. Is there some memory leak in radvd? Or binding and not releasing to the interface? There is radvd 2.18 available now with numerous fixes.

    This issue does unfortunately break IPv6 when using un-managed router advertisements.

    I have turned up logging levels, but it does not add anything more related to radvd.



  • @Irata thank you, good to know that I am not the only one.
    Can someone from pfsense team let us know if a bug report needs to be opened?



  • @strangegopher thanks, glad I found your post as I also thought it was just me!

    It happens on all interfaces at the same time, so I took a packet capture on a quiet subnet - but nothing of interest again, can just see when the router advertisements stop.

    Sadly my restart workaround using cron has side effects, any active IPv6 connections are dropped each time it runs. And one Android device even disconnects from WiFi each time radvd is restarted, it treats a loss of IPv6 as a network problem, bizarre but true.

    Be interested to hear if anyone using radvd and IPv6 does not have this issue? Check Routing logs in System Logs.

    I'm at a loss what else to do. It's interesting that a periodic restart masks the problem, does that point to radvd being the problem?



  • I dont have this issue. Im on 2.5 with uptime 3 days.



  • @maverick_slo do u have ipv6?



  • Since 2013 😁



  • I've been having this problem as well and I also noticed that it's about 6 hours before IPv6 suddenly breaks. This should definitely be fixed before 2.5.0 is released.



  • I am having the same issue. IPv6 would stop working within 24 hours. I backup the config.xml from the 2.5 and restored the file to a 2.4.4_p2 and it does not have the problem.



  • I'm not sure how to find what's common.

    I'm running pfsense within VMware ESXi and have this issue. Are others with this issue the same?

    I can also confirm the latest 2.4 branch does not have this issue with the same config.



  • I am running it on baremetal dell box but I am using chelsio 10G sfp+ nic.



  • @jimp sorry for the ping but do u have any idea why ipv6 stops working?



  • How is it on a fresh 2.5 install with the old config?

    I’ve just had some bad behaviour with radvd not responding at all (not quite the same as your issue) and that has cleared it.



  • i have similar behavior on my log

    Jun 6 21:32:29 	radvd 	66014 	resuming normal operation
    Jun 6 21:32:29 	radvd 	66014 	attempting to reread config file
    Jun 6 18:57:34 	radvd 	66014 	resuming normal operation
    Jun 6 18:57:34 	radvd 	66014 	attempting to reread config file
    Jun 6 14:01:49 	radvd 	65728 	version 2.17 started 
    Jun 6 14:00:39 	radvd 	67952 	can't join ipv6-allrouters on igb2
    Jun 6 14:00:39 	radvd 	67952 	can't join ipv6-allrouters on ath0_wlan0
    Jun 6 14:00:33 	radvd 	67952 	can't join ipv6-allrouters on igb2
    Jun 6 14:00:22 	radvd 	67952 	can't join ipv6-allrouters on igb2
    Jun 6 14:00:20 	radvd 	67952 	can't join ipv6-allrouters on ath0_wlan0
    Jun 6 14:00:08 	radvd 	67952 	can't join ipv6-allrouters on igb2
    Jun 6 14:00:01 	radvd 	67952 	can't join ipv6-allrouters on ath0_wlan0
    Jun 6 13:59:53 	radvd 	67952 	can't join ipv6-allrouters on ath0_wlan0
    Jun 6 13:59:53 	radvd 	67952 	can't join ipv6-allrouters on igb2
    Jun 6 13:59:40 	radvd 	67952 	can't join ipv6-allrouters on ath0_wlan0
    Jun 6 13:59:36 	radvd 	67952 	can't join ipv6-allrouters on igb2
    Jun 6 13:59:30 	radvd 	67952 	can't join ipv6-allrouters on igb2
    Jun 6 13:59:25 	radvd 	67952 	can't join ipv6-allrouters on ath0_wlan0
    

    igb0 is wan
    igb1 is lan
    on igb2 i have only 1 pc, a server connected with static ipv6
    on ath0_wlan0 i have a couple of notebook with windows 10 and a couple of android phone



  • A fresh install causes the same error for me, once stateless IPv6 is setup. I can't find a stable working system whatever I do, other than back to 2.4.

    Because I'm using an ESXi testbed, it's easy to spin up new installs.

    I do think there is something fundamental here, given the reports from wide ranging setups.



  • the error is generated from device-bsd44.c, a patch it's inside the port of radvd for freebsd
    tracking back there was similar problem. related Bug #2878 https://redmine.pfsense.org/issues/2878 from 2013

    /* XXX: See pfSense ticket #2878 */
    +	setsockopt(sock, IPPROTO_IPV6, IPV6_LEAVE_GROUP, &mreq, sizeof(mreq));
    +
    +	if (setsockopt(sock, IPPROTO_IPV6, IPV6_JOIN_GROUP,
    +			&mreq, sizeof(mreq)) < 0) {
    +		flog(LOG_ERR, "can't join ipv6-allrouters on %s", iface->props.name);
    +		return (-1);
    + }
    

    maybe it's time for a new ticket
    https://redmine.pfsense.org/issues/9577



  • idk if we open a ticket devs might just ignore it, first we should get devs on the forums to tell us if we need to open a bug.



  • @strangegopher - they're aware of it, one of the devs referred to the ticket in a thread with an issue I'm having.



  • Now it hit me too:

    Jun 29 07:17:29 radvd 62926 can't join ipv6-allrouters on hn0.10
    Jun 29 07:15:22 radvd 62926 can't join ipv6-allrouters on hn0.10
    Jun 29 07:15:00 radvd 62926 can't join ipv6-allrouters on hn0.9
    Jun 29 07:13:07 radvd 62926 can't join ipv6-allrouters on hn0.7
    Jun 29 07:12:47 radvd 62926 can't join ipv6-allrouters on hn0.10
    Jun 29 07:11:25 radvd 62926 can't join ipv6-allrouters on hn0.8
    Jun 29 07:11:23 radvd 62926 can't join ipv6-allrouters on hn0.9
    Jun 29 07:10:22 radvd 62926 can't join ipv6-allrouters on hn0.10
    Jun 29 07:08:10 radvd 62926 can't join ipv6-allrouters on hn0.10



  • welcome to the party !



  • @maverick_slo made any changes recently, or added any new IPv6 devices?



  • No changes no new devices.
    I just came home from vacation...



  • i put here a workaround that seems to work for me

    [2.5.0-DEVELOPMENT][root@pfSense.localdomain]/root: crontab -l
    @hourly killall radvd && sleep 5 && /usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog
    


  • On the recent snapshot (2.5.0.a.20190728.0536) I now also get the following repeated in the routing system log:

    
    Jul 30 16:50:49	radvd	72128	IPv6 forwarding on interface seems to be disabled, but continuing anyway
    Jul 30 16:50:42	radvd	72128	IPv6 forwarding on interface seems to be disabled, but continuing anyway
    Jul 30 16:50:41	radvd	72128	IPv6 forwarding on interface seems to be disabled, but continuing anyway
    Jul 30 16:50:34	radvd	72128	IPv6 forwarding on interface seems to be disabled, but continuing anyway
    Jul 30 16:50:29	radvd	72128	IPv6 forwarding on interface seems to be disabled, but continuing anyway
    Jul 30 16:50:24	radvd	72128	IPv6 forwarding on interface seems to be disabled, but continuing anyway
    Jul 30 16:50:21	radvd	72128	IPv6 forwarding on interface seems to be disabled, but continuing anyway
    


  • Yup same here, 2.5.0 is getting worse by every snapshot 😁
    Ah well probably Ill go back to 2.4.4...



  • Looks like it is failing within this code in device-linux.c in radvd:

    int check_ip6_forwarding(void)
    {
    	int value;
    	FILE *fp = NULL;
    
    	fp = fopen(PROC_SYS_IP6_FORWARDING, "r");
    .
    .
    .
    

    Which I think is checking for /proc/sys./... but there is no /proc/sys/... directory. There is also alternative code called in device-bsd44.c. So I'm wondering if radvd in 2.5.0 is being built for the wrong platform, or it's expecting a file that is now removed. But I admit this is a wild guess.



  • Netgate can't possibly ship 2.5.0 with this bug that essentially disables ipv6 support in pfsense. Still no dev replies or acknowledgment of Bug #9577. Hope people that are paying for netgate products or support ask for this to be fixed.



  • Actually it is working for me, it's only a message log spamming out, they have time to correct it before the official release



  • It's "working" here too with a radvd restart every 6 hours. I do a radvd service restart via Cron instead of a kill, which seemed cleaner but my approach causes IPv6 sessions to be dropped when the prefix is unadvertised briefly.

    Looking at the source code, that spam logging should not even be executing on a freebsd platform. So maybe it's a hint of what's broken.



  • After 6 days my IPv6 died as well.
    Static assigned IPv6 works just fine, slaac and RA doesn`t. Client does not get IPv6 address at all.

    LOG:
    Aug 6 09:48:50 radvd 35714 can't join ipv6-allrouters on hn0.9
    Aug 6 09:48:50 radvd 35714 IPv6 forwarding on interface seems to be disabled, but continuing anyway
    Aug 6 09:48:10 radvd 35714 can't join ipv6-allrouters on hn0.7
    Aug 6 09:48:10 radvd 35714 IPv6 forwarding on interface seems to be disabled, but continuing anyway
    Aug 6 09:47:15 radvd 35714 can't join ipv6-allrouters on hn0.10
    Aug 6 09:47:15 radvd 35714 IPv6 forwarding on interface seems to be disabled, but continuing anyway
    Aug 6 09:44:37 radvd 35714 can't join ipv6-allrouters on hn0.10
    Aug 6 09:44:37 radvd 35714 IPv6 forwarding on interface seems to be disabled, but continuing anyway
    Aug 6 09:43:06 radvd 35714 can't join ipv6-allrouters on hn0.7
    Aug 6 09:43:06 radvd 35714 IPv6 forwarding on interface seems to be disabled, but continuing anyway
    Aug 6 09:42:05 radvd 35714 can't join ipv6-allrouters on hn0.8
    Aug 6 09:42:05 radvd 35714 IPv6 forwarding on interface seems to be disabled, but continuing anyway
    Aug 6 09:41:55 radvd 35714 can't join ipv6-allrouters on hn0.9
    Aug 6 09:41:55 radvd 35714 IPv6 forwarding on interface seems to be disabled, but continuing anyway
    Aug 6 09:41:46 radvd 35714 can't join ipv6-allrouters on hn0.10

    Ah well....



  • me too . how fix it?

    alt text



  • for the moment afaik only this -> https://forum.netgate.com/post/851662



  • @kiokoman said in ipv6 broken: radvd: can't join ipv6-allrouters on <interface>:

    i put here a workaround that seems to work for me

    [2.5.0-DEVELOPMENT][root@pfSense.localdomain]/root: crontab -l
    @hourly killall radvd && sleep 5 && /usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog
    

    What is this principle?



  • restart Router Advertisement Daemon every hour, to me the problem arise every some hours so restarting it mitigate the problem



  • @kiokoman

    I tried to restart the service yesterday and did not solve the problem. Restart the system to work.I have problems after about 20 hours of running.



  • well i dont know how to solve the problem, you have 2 options, wait for the staff to solve it or go back to 2.4.4


Log in to reply