New latency every 30 seconds with 2.4.2 caused by radvd 2.17_3 *****HAVE A TEMPORARY FIX*****

Flole

Having the same issue here, did this ever get fixed?

chpalmer

I missed the response 8 mos. ago.. Is this happening with 2.4.3 for you? Are you on a bonded connection or just load balancing?

Flole

I'm using lacp with 4 interfaces and vlans on the bond, running pfsense 2.4.3. I started radvd in debug mode but nothing that indicates what might cause the problem. It results in pings getting lost or go up all the way to 750ms whenever that spike happens. I also have some messages in dmesg about a "listen queue overflow". I am using snort, but not on the interfaces that I am having problems with, so I don't think it is related (just wanted to mention it as its an installed package).

Flole

Unfortunately this still exists in 2.4.4. I also noticed some latency spikes when PPPOE reconnects, so maybe this is related to IPv6 in General? dpinger and dhcp are using lots of CPU on the PPPOE reconnection event.

Only thing in dmesg that could be related is

sa6_recoverscope: embedded scope mismatch: xxxxxx sin6_scope_id was overridden

a few times.

Flole

I just removed the LACP and that fixed this issue (or at least made it better).
What I have noticed then is, that I'm having ping spikes on every second block of logs (so the first one is fine, no ping spike, but the second one causes a ping spike):

[Nov 04 04:10:33] radvd (46168): ioctl(SIOCGIFINDEX) succeeded on em2.1
[Nov 04 04:10:33] radvd (46168): ioctl(SIOCGIFFLAGS) succeeded on em2.1
[Nov 04 04:10:33] radvd (46168): em2.1 is up
[Nov 04 04:10:33] radvd (46168): em2.1 is running
[Nov 04 04:10:33] radvd (46168): em2.1 supports multicast
[Nov 04 04:10:33] radvd (46168): sysctl ifdata succeeded on em2.1
[Nov 04 04:10:33] radvd (46168): ioctl(SIOCGIFMEDIA) succeeded on em2.1
[Nov 04 04:10:33] radvd (46168): em2.1 is active


[Nov 04 04:10:36] radvd (46168): ioctl(SIOCGIFINDEX) succeeded on em2.1
[Nov 04 04:10:36] radvd (46168): ioctl(SIOCGIFFLAGS) succeeded on em2.1
[Nov 04 04:10:36] radvd (46168): em2.1 is up
[Nov 04 04:10:36] radvd (46168): em2.1 is running
[Nov 04 04:10:36] radvd (46168): em2.1 supports multicast
[Nov 04 04:10:36] radvd (46168): sysctl ifdata succeeded on em2.1
[Nov 04 04:10:36] radvd (46168): ioctl(SIOCGIFMEDIA) succeeded on em2.1
[Nov 04 04:10:36] radvd (46168): em2.1 is active

However, to me that's more like a workaround instead of a fix. Does that help finding the issue @chpalmer ?

aharrison

I was seeing the exact same thing and was driving me crazy.

ServicesDHCPv6 Server & RALANDHCPv6 Server

Disabled the above - the issue went away instantly like magic

Flole

@aharrison What NIC are you using? I think I have found the issue.

aharrison

It's the Netgate RCC-VE 2440 System

Flole

Well I guess someone has to look into this now.... Most likely it's some kind of driver issue.... Are you using LACP?

Flole

Also it would be interesting to know if you've touched the tunables. I'm trying to figure out what all these 3 machines have in common, obviously it's not the NIC that caused the issue here.

aharrison

sorry for the delay. LACP is not in use - its a rather simple one WAN, two LAN out configuration with a couple VLANs in play. Minimal changes have been made to the defaults.

tcsac

@Flole - my fix was to disable radvd per aharrison's suggestion (apologize just seeing this now, it never notified me). I am also doing LACP from chelsio NICs to a pair of cisco nexus switches. Still have LACP enabled, but disabled RADVD and the lag spikes are gone. I stumbled upon this again because when I upgraded to 2.4.4 it re-enabled radvd again and I couldn't recall what I had done the last time.

**I should note that disabling radvd isn't a fix as it means no dhcpv6 on the LAN which isn't really ideal. This is definitely a bug in radvd 2.17 and it's unfortunate it hasn't been addressed. It's not the CPU or hyperthreading or anything related to the NICs - it's 100% caused by radvd, you can see the process spike (even with priority set to low) and a corresponding lag spike goes with it. Disable radvd and the spikes disappear.

tcsac

@aharrison @Flole @chpalmer

I believe I have a fix - I've been running this for about 20 minutes with no lag spikes. I won't call it ideal, or even great, but it proves without a doubt that the issue is with the 2.x builds of radvd and not a network card, or vlan or lacp or insert whatever excuse issue, it's radvd.

I installed an older 1.x binary I was able to find on the freebsd packages mirror to replace the 2.17 binary. It seems to work perfectly fine (it's advertising as expected) and no more lag issues. Steps below (1.15 was the newest version I could find):

First stop radvd (disable advertisements from the GUI)
next you need to ssh into the system and go to the console
cd /usr/local/sbin
mv radvd radvd.bak
mv radvdump radvdump.bak
cd /tmp
fetch http://pkg.freebsd.org/FreeBSD:10:amd64/release_3/All/radvd-1.15.txz
tar xf radvd-1.15.txz
cd /tmp/usr/local/sbin
cp radvd* /usr/local/sbin/

restart radvd from the GUI and you should be good to go.

Hopefully someone at netgate will address this more formally. As far as I can tell at this point it breaks nothing. If you do run into issues you should have no problem backing out, just delete the radvd and move radvd.bak to radvd.