Verizon Fios and IPV6, Which Settings Work?

JKnott

Router advertisements should happen frequently. Take a packet capture of the DHCPv6 sequence.

To do that:

Shutdown pfsense and unplug WAN cable.
Reboot pfsense and start Packet Capture, filtering on DHCPv6.
Reconnect WAN cable.
After a couple of minutes, download packet capture and examine with Wireshark.

jeremy.duncan

@jknott way ahead of you i am doing a tcpdump on the external interface... bupkiss. just my DHCPv6 solicits

tcpdump -ni lagg0_vlan2 ip6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lagg0_vlan2, link-type EN10MB (Ethernet), capture size 262144 bytes

16:11:02.856512 IP6 fe80::eef4:bbff:fed3:a3e8.546 > ff02::1:2.547: dhcp6 solicit
16:11:59.768654 IP6 fe80::eef4:bbff:fec1:d2e8.546 > ff02::1:2.547: dhcp6 solicit
16:13:14.636935 IP6 fe80::eef4:bbff:fed3:a3e8.546 > ff02::1:2.547: dhcp6 solicit
16:14:02.094810 IP6 fe80::eef4:bbff:fec1:d2e8.546 > ff02::1:2.547: dhcp6 solicit

jeremy.duncan

@jknott i also force router solicitations..

16:17:57.834172 IP6 fe80::eef4:bbff:fed3:a3e8 > ff02::2: ICMP6, router solicitation, length 16
16:18:01.846522 IP6 fe80::eef4:bbff:fed3:a3e8 > ff02::2: ICMP6, router solicitation, length 16
16:18:05.848481 IP6 fe80::eef4:bbff:fed3:a3e8 > ff02::2: ICMP6, router solicitation, length 16

MikeV7896

@jeremy-duncan Guessing it's just not on in Chesapeake yet then... the only reports from other DSLR users have been Newport News and Yorktown so far... nothing from Norfolk/Chesapeake/VA Beach.

jeremy.duncan

@mikev7896 understood thanks

SirSilentBob

@MikeV7896 Thank you! I'd greatly appreciate it. From what you are saying, it sounds like you are maybe using an on-board NIC for one of your connections and you have four LANs, maybe with a four-port NIC? I am using a 4-port intel gigabit card, and disabled my on-board NIC so I've just got 1 WAN and three LANs. But I'm looking to do a similar setup, a primary lan, a "spare/experimentation" lan and a Atlas Probe lan.

@jeremy-duncan Sorry it isn't working yet for you. Maybe once Verizon finishes up with rolling it out in Newport News & Hampton then they'll migrate over to the southside and start deploying it there...

MikeV7896

@sirsilentbob

My pfSense box has a SuperMicro motherboard with five onboard NICs... one is OS-accessible but is also shared with the built-in out-of-band management firmware (which I don't use and have disabled in the BIOS), the other four are all dedicated Intel Gigabit NICs. One WAN, 3 LAN. Main LAN connects to a switch for everything else, LAN2 connects to the mesh router, and ATLAS connects directly to the device my Atlas probe is running on.

Our WAN settings vary a little, but I don't think the variances are significant to where your LANs would be affected.

LAN settings, well... there's not much there. I will admit that I'm not using prefix 0 though... LAN = 10, LAN2 = 20, ATLAS = 50...

I did look at the advanced DHCP6 config on my WAN interface (since I didn't see any advanced options on my LANs) and I did see this...

Though I don't have advanced configuration checked normally, and none of the other options in that section are enabled or filled out. WAN was selected by default.

SirSilentBob

@mikev7896 Hmm, thanks for sharing. Your drop-down prefix interface being set to WAN is quite interesting to me, I'll try duplicating that! I'll try changing some things when everyone is asleep and won't miss the internet. Also I'll try changing the LAN prefixes from single digit (0, 1 & 2) to double digit ones too.

Could you share how you have your Router Advertisements / Router Mode set, if you get a chance, please? Just wondering if that is a hang-up.

Thanks again!

MikeV7896

@sirsilentbob My RA router mode setting is Unmanaged. I let SLAAC and RDNSS do its thing, and I've had no issues with any device.

JKnott

@jeremy-duncan

Do they even support IPv6?

JKnott

@mikev7896

It doesn't matter which prefix ID you use, so long as they're unique for each interface. I don't use advanced IPv6 settings.

MikeV7896

@jknott they’re beginning to roll it out on a larger scale. Over the past couple of weeks, reports of IPv6 now being available have come in from five areas near Baltimore MD and three in VA. A business account in NY (don’t remember where in the state) received an email that it should be rolling out up there in June.

SirSilentBob

@MikeV7896 Thanks for your gracious help and sharing of config info.

Unfortunately I guess at this time I think all I can do is go back to my "wrong" configuration that is only allowing IPv6 on a single LAN. I've pretty much duplicated your setup exactly, but it isn't working, WAN_DHCP6 just stays in a Pending state. I checked the box to start DHCP6 client in debug mode, the only "hint" I have is this below log entry, and a search on that missing dhcp6cctlkey file has been fruitless, and even found posts saying that error is unimportant. It must have some sort of importance though, because my config that gives a single LAN IPv6 (when the prefix interface is set to that LAN and not WAN) does not generate that error. I'm just not sure why I can't get it to work on all LAN interfaces.

May 14 13:30:57	dhcp6c	29013	skip opening control port
May 14 13:30:57	dhcp6c	29013	failed initialize control message authentication
May 14 13:30:57	dhcp6c	29013	failed to open /usr/local/etc/dhcp6cctlkey: No such file or directory

If anyone has any thoughts or things to try, I am open to suggestions...

Edit: Something else I am seeing, the Service Status for RADVD disappears from the dashboard too but according to the command line it is still running? I have a backup I can roll back to though that should get me back to a single LAN with IPv6. Just changing the settings back doesn't restore connectivity, so something is sticking somewhere... I might save another config file to compare differences between files and then restore to the working single-lan config.

SirSilentBob

Well after a few resets, and messing around with configurations, IPv6 just "started working" on a second LAN... I can't claim to understand what change allowed it, but it works now, so hopefully it will continue to do so.

Hopefully over time my knowledge of IPv6 and the process of setting up routers will improve and this will be easier for those who come to use it after us!

Thanks @MikeV7896 !

MikeV7896

Something of note that has been discovered...

It appears that the Alcatel-Lucent ONTs that Verizon provides have firmware that is mangling IPv6 packets by adding additional data after the packet checksum, and Intel wired NICs with hardware checksum offloading are being negatively affected by this (only Intel NICs have been identified as being affected by this issue so far). While the issue has been discovered with other routers (especially Verizon's own G1100 router) and Intel NICs on Windows PCs, it sounds like this could affect pfSense routers where an Intel NIC is used for your WAN connection AND you have hardware checksum offloading enabled.

I personally have always had it disabled... when I started using pfSense, I had seen some articles or posts about problems with it being enabled for some NICs, so I just never chanced it and kept it turned off (I don't remember exactly, but it might've been that I started using pfSense on a PC with Realtek NICs, and just left it disabled after moving my hardware over to Intel NICs). I have not tried enabling the setting to see if things break.

So if you have an Intel NIC for your WAN AND you're experiencing problems with IPv6 connectivity, you might want to try disabling the Hardware Checksum Offload setting in System > Advanced > Networking. My understanding is that it's only the checksum offloading that needs to be disabled... the other hardware offload settings should be fine.

As I've mentioned before also... IPv6 is still very much being rolled out throughout the Fios service areas (still currently only in DC/MD/VA). So you might want to leave your hardware checksum offloading enabled until you know IPv6 is available in your area, then see if it affects your ability to connect via IPv6 or not.

mattlach

@mikev7896 said in Verizon Fios and IPV6, Which Settings Work?:

Something of note that has been discovered...

It appears that the Alcatel-Lucent ONTs that Verizon provides have firmware that is mangling IPv6 packets by adding additional data after the packet checksum, and Intel wired NICs with hardware checksum offloading are being negatively affected by this (only Intel NICs have been identified as being affected by this issue so far). While the issue has been discovered with other routers (especially Verizon's own G1100 router) and Intel NICs on Windows PCs, it sounds like this could affect pfSense routers where an Intel NIC is used for your WAN connection AND you have hardware checksum offloading enabled.

I personally have always had it disabled... when I started using pfSense, I had seen some articles or posts about problems with it being enabled for some NICs, so I just never chanced it and kept it turned off (I don't remember exactly, but it might've been that I started using pfSense on a PC with Realtek NICs, and just left it disabled after moving my hardware over to Intel NICs). I have not tried enabling the setting to see if things break.

So if you have an Intel NIC for your WAN AND you're experiencing problems with IPv6 connectivity, you might want to try disabling the Hardware Checksum Offload setting in System > Advanced > Networking. My understanding is that it's only the checksum offloading that needs to be disabled... the other hardware offload settings should be fine.

As I've mentioned before also... IPv6 is still very much being rolled out throughout the Fios service areas (still currently only in DC/MD/VA). So you might want to leave your hardware checksum offloading enabled until you know IPv6 is available in your area, then see if it affects your ability to connect via IPv6 or not.

Sounds like Verizon is up to its evil non-removable fingerprinting of users again in order to data mine them.

At some point collection use of user data HAS TO be made illegal. It's an outright assault on peoples right to privacy.

MikeV7896

@mattlach It's not only a Verizon issue... The first item I read about the issue was in the Intel community and was from a user of a fiber service in Canada... no Verizon there. They have an Alcatel-Lucent ONT though.

SirSilentBob

@mikev7896 said in Verizon Fios and IPV6, Which Settings Work?:

@mattlach It's not only a Verizon issue... The first item I read about the issue was in the Intel community and was from a user of a fiber service in Canada... no Verizon there. They have an Alcatel-Lucent ONT though.

Mike, I'm curious, how are your other offloading settings configured?

I checked mine, and apparently I have hardware checksum offloading enabled. I checked what ONT I have, and I believe it was installed in mid-late 2011. Going by the ONT S/N, which the first 4 characters are T0211, I'm assuming that means it was made in February of 2011. The ONT I have is a Motorola DBBU-1238 Firmware rev. C (This might just be the model number of the in-door unit with battery and power supply though.) Assuming the outside guts are also Motorola, then I guess this old Motorola unit doesn't have the IPv6 bug. (Frankly I'm amazed that Motorola seems to have passed on an opportunity for a bug/deficiency!)

So I guess this can be a confirmation that this particular Motorola ONT doesn't have the same issue.

Here's my settings, just curious how they compare to what you are running:

SirSilentBob

I have sent a local friend of mine a message to check what ONT he has, he's on the same street and CO as me, so he should have IPv6 but it's just not working. If he's got an Alcatel/Lucent unit then he'll just have to figure out how to disable hardware checksum offloading in BSD and try again, as I'm 99% sure he's got a quad intel gigabit card like I'm running. If disabling that makes it work, I'll see if I can get him to provide the info on his ONT so it can be confirmed as a "bugged" one if anyone is keeping track.

MikeV7896

@sirsilentbob said in Verizon Fios and IPV6, Which Settings Work?:

Here's my settings, just curious how they compare to what you are running:

I have all of the hardware offloading settings disabled. I'm guessing my CPU is powerful enough to handle everything, because with gigabit service I can still get full 940 Mbps results on speed tests.

As far as the ONT tracking, I think that's a bit outside of the scope of this community. The original issue has only been mentioned as happening with the Alcatel-Lucent ONTs, and I don't believe there have been any reports of other ONTs having a similar issue.