Verizon Fios and IPV6, Which Settings Work?

SirSilentBob

@MikeV7896 Thanks for your gracious help and sharing of config info.

Unfortunately I guess at this time I think all I can do is go back to my "wrong" configuration that is only allowing IPv6 on a single LAN. I've pretty much duplicated your setup exactly, but it isn't working, WAN_DHCP6 just stays in a Pending state. I checked the box to start DHCP6 client in debug mode, the only "hint" I have is this below log entry, and a search on that missing dhcp6cctlkey file has been fruitless, and even found posts saying that error is unimportant. It must have some sort of importance though, because my config that gives a single LAN IPv6 (when the prefix interface is set to that LAN and not WAN) does not generate that error. I'm just not sure why I can't get it to work on all LAN interfaces.

May 14 13:30:57	dhcp6c	29013	skip opening control port
May 14 13:30:57	dhcp6c	29013	failed initialize control message authentication
May 14 13:30:57	dhcp6c	29013	failed to open /usr/local/etc/dhcp6cctlkey: No such file or directory

If anyone has any thoughts or things to try, I am open to suggestions...

Edit: Something else I am seeing, the Service Status for RADVD disappears from the dashboard too but according to the command line it is still running? I have a backup I can roll back to though that should get me back to a single LAN with IPv6. Just changing the settings back doesn't restore connectivity, so something is sticking somewhere... I might save another config file to compare differences between files and then restore to the working single-lan config.

SirSilentBob

Well after a few resets, and messing around with configurations, IPv6 just "started working" on a second LAN... I can't claim to understand what change allowed it, but it works now, so hopefully it will continue to do so.

Hopefully over time my knowledge of IPv6 and the process of setting up routers will improve and this will be easier for those who come to use it after us!

Thanks @MikeV7896 !

MikeV7896

Something of note that has been discovered...

It appears that the Alcatel-Lucent ONTs that Verizon provides have firmware that is mangling IPv6 packets by adding additional data after the packet checksum, and Intel wired NICs with hardware checksum offloading are being negatively affected by this (only Intel NICs have been identified as being affected by this issue so far). While the issue has been discovered with other routers (especially Verizon's own G1100 router) and Intel NICs on Windows PCs, it sounds like this could affect pfSense routers where an Intel NIC is used for your WAN connection AND you have hardware checksum offloading enabled.

I personally have always had it disabled... when I started using pfSense, I had seen some articles or posts about problems with it being enabled for some NICs, so I just never chanced it and kept it turned off (I don't remember exactly, but it might've been that I started using pfSense on a PC with Realtek NICs, and just left it disabled after moving my hardware over to Intel NICs). I have not tried enabling the setting to see if things break.

So if you have an Intel NIC for your WAN AND you're experiencing problems with IPv6 connectivity, you might want to try disabling the Hardware Checksum Offload setting in System > Advanced > Networking. My understanding is that it's only the checksum offloading that needs to be disabled... the other hardware offload settings should be fine.

As I've mentioned before also... IPv6 is still very much being rolled out throughout the Fios service areas (still currently only in DC/MD/VA). So you might want to leave your hardware checksum offloading enabled until you know IPv6 is available in your area, then see if it affects your ability to connect via IPv6 or not.

mattlach

@mikev7896 said in Verizon Fios and IPV6, Which Settings Work?:

Something of note that has been discovered...

It appears that the Alcatel-Lucent ONTs that Verizon provides have firmware that is mangling IPv6 packets by adding additional data after the packet checksum, and Intel wired NICs with hardware checksum offloading are being negatively affected by this (only Intel NICs have been identified as being affected by this issue so far). While the issue has been discovered with other routers (especially Verizon's own G1100 router) and Intel NICs on Windows PCs, it sounds like this could affect pfSense routers where an Intel NIC is used for your WAN connection AND you have hardware checksum offloading enabled.

I personally have always had it disabled... when I started using pfSense, I had seen some articles or posts about problems with it being enabled for some NICs, so I just never chanced it and kept it turned off (I don't remember exactly, but it might've been that I started using pfSense on a PC with Realtek NICs, and just left it disabled after moving my hardware over to Intel NICs). I have not tried enabling the setting to see if things break.

So if you have an Intel NIC for your WAN AND you're experiencing problems with IPv6 connectivity, you might want to try disabling the Hardware Checksum Offload setting in System > Advanced > Networking. My understanding is that it's only the checksum offloading that needs to be disabled... the other hardware offload settings should be fine.

As I've mentioned before also... IPv6 is still very much being rolled out throughout the Fios service areas (still currently only in DC/MD/VA). So you might want to leave your hardware checksum offloading enabled until you know IPv6 is available in your area, then see if it affects your ability to connect via IPv6 or not.

Sounds like Verizon is up to its evil non-removable fingerprinting of users again in order to data mine them.

At some point collection use of user data HAS TO be made illegal. It's an outright assault on peoples right to privacy.

MikeV7896

@mattlach It's not only a Verizon issue... The first item I read about the issue was in the Intel community and was from a user of a fiber service in Canada... no Verizon there. They have an Alcatel-Lucent ONT though.

SirSilentBob

@mikev7896 said in Verizon Fios and IPV6, Which Settings Work?:

@mattlach It's not only a Verizon issue... The first item I read about the issue was in the Intel community and was from a user of a fiber service in Canada... no Verizon there. They have an Alcatel-Lucent ONT though.

Mike, I'm curious, how are your other offloading settings configured?

I checked mine, and apparently I have hardware checksum offloading enabled. I checked what ONT I have, and I believe it was installed in mid-late 2011. Going by the ONT S/N, which the first 4 characters are T0211, I'm assuming that means it was made in February of 2011. The ONT I have is a Motorola DBBU-1238 Firmware rev. C (This might just be the model number of the in-door unit with battery and power supply though.) Assuming the outside guts are also Motorola, then I guess this old Motorola unit doesn't have the IPv6 bug. (Frankly I'm amazed that Motorola seems to have passed on an opportunity for a bug/deficiency!)

So I guess this can be a confirmation that this particular Motorola ONT doesn't have the same issue.

Here's my settings, just curious how they compare to what you are running:

SirSilentBob

I have sent a local friend of mine a message to check what ONT he has, he's on the same street and CO as me, so he should have IPv6 but it's just not working. If he's got an Alcatel/Lucent unit then he'll just have to figure out how to disable hardware checksum offloading in BSD and try again, as I'm 99% sure he's got a quad intel gigabit card like I'm running. If disabling that makes it work, I'll see if I can get him to provide the info on his ONT so it can be confirmed as a "bugged" one if anyone is keeping track.

MikeV7896

@sirsilentbob said in Verizon Fios and IPV6, Which Settings Work?:

Here's my settings, just curious how they compare to what you are running:

I have all of the hardware offloading settings disabled. I'm guessing my CPU is powerful enough to handle everything, because with gigabit service I can still get full 940 Mbps results on speed tests.

As far as the ONT tracking, I think that's a bit outside of the scope of this community. The original issue has only been mentioned as happening with the Alcatel-Lucent ONTs, and I don't believe there have been any reports of other ONTs having a similar issue.

kohenkatz

@sirsilentbob said in Verizon Fios and IPV6, Which Settings Work?:

this old Motorola unit

Since the beginning of FiOS rollouts, Verizon has used at least 24 different Motorola ONT models, 11 Tellabs models, and 14 Alcatel models. Of those, all the Tellabs models and 10 of the Motorola models were using a technology (BPON) that Verizon no longer uses. DSLReports has a list. As far as I can tell, the Motorola units have never exhibited this IPv6 issue.

nolaquen

For the folks that have had IPv6 up and running for a while, has anyone had the /56 prefix change on them? It just went active for me this week, and curious how much effort I need to put into insulating my configuration (to the extent I can) from WAN prefix changes.

For IPv4, the only time in recent memory that it changed was actually this week when it went down and the link came back up with IPv6. Prior to that, it was consistent through power outages, equipment poweroffs, etc.

jeremy.duncan

Woo hoo! Verizon FiOS finally turned up my area for IPv6. I was able to get it working for a few zones on my firewall. I have a few subnets and other routed subnets on my network that I am keeping on my HE tunnel as it's permanent prefix - using some policy-based routing. Otherwise everytime the firewall resets or connection resets it will change the IPv6 prefix. Sure wish Verizon would just allocate a permanent prefix.

SirSilentBob

@jeremy-duncan

Nice, so the IPv6 has made it across the water to Chesapeake finally, congrats!

As for the changing prefix issue, have you considered something like using ULAs?

I have really really considered it (mostly have not yet because lazy and a lot of stuff to set up) because I want to be able to have devices, some of which are mobile and will only use SLAAC have a consistent IPv6, the same as I do on the IPv4 RFC1918 side of things. I think my solution for the mobile devices that can't take IPv6 via DHCP is if I did this, just manually set the ULA on the device and take it off "automatic". Honestly, I would probably make the tail end of the IPv6 address just be the same as it is for IPv4. Even if that meant my LAN devices had a ULA like fd48:dead:beef:50:192:168:50:10 and it's IPv4 was 192.168.50.10.

I have certain devices that I want the addresses to stay the same so I can do device specific things / exemptions for filtering.

I thought about leaving a HE tunnel up, but doing it this way I'd avoid issues with changing prefixes and the devices won't be limited to the tunnel speed, they'd all have the full gigabit line speed.

Let me know how it goes if you try this out, please! I'm staring at my access point right now, with currently 28 devices on and active on the network, and it's just me having to probably commit a whole day to this change-over that is stopping me from already having done it, and I still can think of 5 or 6 more devices that are just off right now...

jeremy.duncan

@sirsilentbob ah. No. My fios business is in chesapeake and that's still limping along on v4. My home network in fairfax county was the one that was finally enabled... as for ULA, no because that just means I'd have to do network prefix translation on the firewall which wouldn't be any better performance than a tunnel even with a lower MTU. So to keep these subnets somewhat static I'll just keep the policy routing going and do the cool IPv6 on my wifi and another DMZ.

ingenium13

Verizon rolled out IPv6 in my neighborhood. I'm able to assign /64s (in the right IP ranges) to my vlans, however pfsense won't pass traffic. Packet captures show that pfsense is sending neighbor solicitations, but not getting a response, and upstream is also sending neighbor solicitations, but pfsense isn't responding.

11:04:24.681659 IP6 fe80::2e0:67ff:fe2a:da56 > fe80::e86:10ff:fea1:7bc2: ICMP6, echo request, seq 4139, length 9
11:04:25.143801 IP6 2600:4041:170::1 > ff02::1:ff2a:da56: ICMP6, neighbor solicitation, who has fe80::2e0:67ff:fe2a:da56, length 32
11:04:25.182549 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:26.182638 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:27.182427 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:28.189624 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:28.191316 IP6 2600:4041:170::1 > ff02::1:ff2a:da56: ICMP6, neighbor solicitation, who has fe80::2e0:67ff:fe2a:da56, length 32
11:04:29.189601 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:30.189391 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:31.155646 IP6 2600:4041:170::1 > ff02::1:ff2a:da56: ICMP6, neighbor solicitation, who has fe80::2e0:67ff:fe2a:da56, length 32
11:04:31.195582 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:32.195400 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:33.195391 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:34.162249 IP6 2600:4041:170::1 > ff02::1:ff2a:da56: ICMP6, neighbor solicitation, who has fe80::2e0:67ff:fe2a:da56, length 32
11:04:34.201399 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32

I tried to ping one of my vlan assigned IPv6 addresses from my VPS, and I see the ping come in in a packet capture on WAN, but again, no response.

I had he.net working previously. I added a firewall rule at the top of WAN to allow all IPv6. "Block private networks and loopback addresses" is disabled.

Any thoughts?

jeremy.duncan

@ingenium13 yeah I had this exact problem when I had the Network Prefix Translation enabled for a few subnets that I was keeping on HE. Do you also have that enabled? If so, just disable them and restart the WAN NIC because that package doesn't seem to work correctly.

Otherwise, you need to make sure your firewall rules allow for IPv6 ICMP-IPv6 for neighbor advertisement, solicitation, and packet too big at the every least on the WAN interface.

lastly, a reboot fixes a lot of hosed up stuff with the firewall and handling multiple IPv6 gateways

ingenium13

@jeremy-duncan nope, I have nothing configured for NPT. Is the only place to confirm it under firewall, NAT?

I added firewall rule at the top of WAN explicitly allowing all IPv6 traffic of any type. I've rebooted several times too. I disabled the he.net interface (I hadn't used it in years, but I left it configured just in case I wanted to eventually use it again).

I also checked in the firewall logs and there's nothing indicating any blocked IPv6 traffic. The States table shows 4 states corresponding to these neighbor solicitations, and they all say NO_TRAFFIC as the state.

jeremy.duncan

@ingenium13 did you follow the instructions for Track interface on the LAN side interfaces for IPv6? You can't statically assign subnets from a DHCPv6-PD allocation unfortunately.

ingenium13

@jeremy-duncan Yeah, I have it set to track interface on the LAN side. The LAN router interface correctly gets a routable IPv6 address, and clients get routable IPv6 addresses as well. However the WAN interface just refuses to respond to the neighbor solicitations, so everything just fails with no route to host.

The gateway is fe80::e86:10ff:fea1:7bc2 and WAN is automatically assigned fe80::e86:10ff:fea1:7bc2%igb0. Manually trying to ping the gateway fails:

[2.6.0-RELEASE][root@pfSense]/root: ping6 -I igb0 fe80::e86:10ff:fea1:7bc2
PING6(56=40+8+8 bytes) fe80::2e0:67ff:fe2a:da56%igb0 --> fe80::e86:10ff:fea1:7bc2
^C
--- fe80::e86:10ff:fea1:7bc2 ping6 statistics ---
12 packets transmitted, 0 packets received, 100.0% packet loss

The packet capture just shows repeat attempts by pfsense to send neighbor solicitation upstream, but no response. And then 2600:4041:170::1 sends a solicitation to pfsense, which also doesn't respond.

It seems like a firewall issue, but I'm not sure what else I can do besides setting a rule to allow all IPv6 traffic, which I've already done.

I've tried pinging from the LAN interface as well with the same result.

jeremy.duncan

@ingenium13 as stupid as this sounds.. did you reboot it?

ingenium13

@jeremy-duncan Yes, many times