Verizon Fios and IPV6, Which Settings Work?
-
Something of note that has been discovered...
It appears that the Alcatel-Lucent ONTs that Verizon provides have firmware that is mangling IPv6 packets by adding additional data after the packet checksum, and Intel wired NICs with hardware checksum offloading are being negatively affected by this (only Intel NICs have been identified as being affected by this issue so far). While the issue has been discovered with other routers (especially Verizon's own G1100 router) and Intel NICs on Windows PCs, it sounds like this could affect pfSense routers where an Intel NIC is used for your WAN connection AND you have hardware checksum offloading enabled.
I personally have always had it disabled... when I started using pfSense, I had seen some articles or posts about problems with it being enabled for some NICs, so I just never chanced it and kept it turned off (I don't remember exactly, but it might've been that I started using pfSense on a PC with Realtek NICs, and just left it disabled after moving my hardware over to Intel NICs). I have not tried enabling the setting to see if things break.
So if you have an Intel NIC for your WAN AND you're experiencing problems with IPv6 connectivity, you might want to try disabling the Hardware Checksum Offload setting in System > Advanced > Networking. My understanding is that it's only the checksum offloading that needs to be disabled... the other hardware offload settings should be fine.
As I've mentioned before also... IPv6 is still very much being rolled out throughout the Fios service areas (still currently only in DC/MD/VA). So you might want to leave your hardware checksum offloading enabled until you know IPv6 is available in your area, then see if it affects your ability to connect via IPv6 or not.
-
@mikev7896 said in Verizon Fios and IPV6, Which Settings Work?:
Something of note that has been discovered...
It appears that the Alcatel-Lucent ONTs that Verizon provides have firmware that is mangling IPv6 packets by adding additional data after the packet checksum, and Intel wired NICs with hardware checksum offloading are being negatively affected by this (only Intel NICs have been identified as being affected by this issue so far). While the issue has been discovered with other routers (especially Verizon's own G1100 router) and Intel NICs on Windows PCs, it sounds like this could affect pfSense routers where an Intel NIC is used for your WAN connection AND you have hardware checksum offloading enabled.
I personally have always had it disabled... when I started using pfSense, I had seen some articles or posts about problems with it being enabled for some NICs, so I just never chanced it and kept it turned off (I don't remember exactly, but it might've been that I started using pfSense on a PC with Realtek NICs, and just left it disabled after moving my hardware over to Intel NICs). I have not tried enabling the setting to see if things break.
So if you have an Intel NIC for your WAN AND you're experiencing problems with IPv6 connectivity, you might want to try disabling the Hardware Checksum Offload setting in System > Advanced > Networking. My understanding is that it's only the checksum offloading that needs to be disabled... the other hardware offload settings should be fine.
As I've mentioned before also... IPv6 is still very much being rolled out throughout the Fios service areas (still currently only in DC/MD/VA). So you might want to leave your hardware checksum offloading enabled until you know IPv6 is available in your area, then see if it affects your ability to connect via IPv6 or not.
Sounds like Verizon is up to its evil non-removable fingerprinting of users again in order to data mine them.
At some point collection use of user data HAS TO be made illegal. It's an outright assault on peoples right to privacy.
-
@mattlach It's not only a Verizon issue... The first item I read about the issue was in the Intel community and was from a user of a fiber service in Canada... no Verizon there. They have an Alcatel-Lucent ONT though.
-
@mikev7896 said in Verizon Fios and IPV6, Which Settings Work?:
@mattlach It's not only a Verizon issue... The first item I read about the issue was in the Intel community and was from a user of a fiber service in Canada... no Verizon there. They have an Alcatel-Lucent ONT though.
Mike, I'm curious, how are your other offloading settings configured?
I checked mine, and apparently I have hardware checksum offloading enabled. I checked what ONT I have, and I believe it was installed in mid-late 2011. Going by the ONT S/N, which the first 4 characters are T0211, I'm assuming that means it was made in February of 2011. The ONT I have is a Motorola DBBU-1238 Firmware rev. C (This might just be the model number of the in-door unit with battery and power supply though.) Assuming the outside guts are also Motorola, then I guess this old Motorola unit doesn't have the IPv6 bug. (Frankly I'm amazed that Motorola seems to have passed on an opportunity for a bug/deficiency!)
So I guess this can be a confirmation that this particular Motorola ONT doesn't have the same issue.
Here's my settings, just curious how they compare to what you are running:
-
I have sent a local friend of mine a message to check what ONT he has, he's on the same street and CO as me, so he should have IPv6 but it's just not working. If he's got an Alcatel/Lucent unit then he'll just have to figure out how to disable hardware checksum offloading in BSD and try again, as I'm 99% sure he's got a quad intel gigabit card like I'm running. If disabling that makes it work, I'll see if I can get him to provide the info on his ONT so it can be confirmed as a "bugged" one if anyone is keeping track.
-
@sirsilentbob said in Verizon Fios and IPV6, Which Settings Work?:
Here's my settings, just curious how they compare to what you are running:
I have all of the hardware offloading settings disabled. I'm guessing my CPU is powerful enough to handle everything, because with gigabit service I can still get full 940 Mbps results on speed tests.
As far as the ONT tracking, I think that's a bit outside of the scope of this community. The original issue has only been mentioned as happening with the Alcatel-Lucent ONTs, and I don't believe there have been any reports of other ONTs having a similar issue.
-
@sirsilentbob said in Verizon Fios and IPV6, Which Settings Work?:
this old Motorola unit
Since the beginning of FiOS rollouts, Verizon has used at least 24 different Motorola ONT models, 11 Tellabs models, and 14 Alcatel models. Of those, all the Tellabs models and 10 of the Motorola models were using a technology (BPON) that Verizon no longer uses. DSLReports has a list. As far as I can tell, the Motorola units have never exhibited this IPv6 issue.
-
For the folks that have had IPv6 up and running for a while, has anyone had the /56 prefix change on them? It just went active for me this week, and curious how much effort I need to put into insulating my configuration (to the extent I can) from WAN prefix changes.
For IPv4, the only time in recent memory that it changed was actually this week when it went down and the link came back up with IPv6. Prior to that, it was consistent through power outages, equipment poweroffs, etc.
-
Woo hoo! Verizon FiOS finally turned up my area for IPv6. I was able to get it working for a few zones on my firewall. I have a few subnets and other routed subnets on my network that I am keeping on my HE tunnel as it's permanent prefix - using some policy-based routing. Otherwise everytime the firewall resets or connection resets it will change the IPv6 prefix. Sure wish Verizon would just allocate a permanent prefix.
-
Nice, so the IPv6 has made it across the water to Chesapeake finally, congrats!
As for the changing prefix issue, have you considered something like using ULAs?
I have really really considered it (mostly have not yet because lazy and a lot of stuff to set up) because I want to be able to have devices, some of which are mobile and will only use SLAAC have a consistent IPv6, the same as I do on the IPv4 RFC1918 side of things. I think my solution for the mobile devices that can't take IPv6 via DHCP is if I did this, just manually set the ULA on the device and take it off "automatic". Honestly, I would probably make the tail end of the IPv6 address just be the same as it is for IPv4. Even if that meant my LAN devices had a ULA like fd48:dead:beef:50:192:168:50:10 and it's IPv4 was 192.168.50.10.
I have certain devices that I want the addresses to stay the same so I can do device specific things / exemptions for filtering.
I thought about leaving a HE tunnel up, but doing it this way I'd avoid issues with changing prefixes and the devices won't be limited to the tunnel speed, they'd all have the full gigabit line speed.
Let me know how it goes if you try this out, please! I'm staring at my access point right now, with currently 28 devices on and active on the network, and it's just me having to probably commit a whole day to this change-over that is stopping me from already having done it, and I still can think of 5 or 6 more devices that are just off right now...
-
@sirsilentbob ah. No. My fios business is in chesapeake and that's still limping along on v4. My home network in fairfax county was the one that was finally enabled... as for ULA, no because that just means I'd have to do network prefix translation on the firewall which wouldn't be any better performance than a tunnel even with a lower MTU. So to keep these subnets somewhat static I'll just keep the policy routing going and do the cool IPv6 on my wifi and another DMZ.
-
Verizon rolled out IPv6 in my neighborhood. I'm able to assign /64s (in the right IP ranges) to my vlans, however pfsense won't pass traffic. Packet captures show that pfsense is sending neighbor solicitations, but not getting a response, and upstream is also sending neighbor solicitations, but pfsense isn't responding.
11:04:24.681659 IP6 fe80::2e0:67ff:fe2a:da56 > fe80::e86:10ff:fea1:7bc2: ICMP6, echo request, seq 4139, length 9 11:04:25.143801 IP6 2600:4041:170::1 > ff02::1:ff2a:da56: ICMP6, neighbor solicitation, who has fe80::2e0:67ff:fe2a:da56, length 32 11:04:25.182549 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32 11:04:26.182638 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32 11:04:27.182427 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32 11:04:28.189624 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32 11:04:28.191316 IP6 2600:4041:170::1 > ff02::1:ff2a:da56: ICMP6, neighbor solicitation, who has fe80::2e0:67ff:fe2a:da56, length 32 11:04:29.189601 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32 11:04:30.189391 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32 11:04:31.155646 IP6 2600:4041:170::1 > ff02::1:ff2a:da56: ICMP6, neighbor solicitation, who has fe80::2e0:67ff:fe2a:da56, length 32 11:04:31.195582 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32 11:04:32.195400 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32 11:04:33.195391 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32 11:04:34.162249 IP6 2600:4041:170::1 > ff02::1:ff2a:da56: ICMP6, neighbor solicitation, who has fe80::2e0:67ff:fe2a:da56, length 32 11:04:34.201399 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32I tried to ping one of my vlan assigned IPv6 addresses from my VPS, and I see the ping come in in a packet capture on WAN, but again, no response.
I had he.net working previously. I added a firewall rule at the top of WAN to allow all IPv6. "Block private networks and loopback addresses" is disabled.
Any thoughts?
-
@ingenium13 yeah I had this exact problem when I had the Network Prefix Translation enabled for a few subnets that I was keeping on HE. Do you also have that enabled? If so, just disable them and restart the WAN NIC because that package doesn't seem to work correctly.
Otherwise, you need to make sure your firewall rules allow for IPv6 ICMP-IPv6 for neighbor advertisement, solicitation, and packet too big at the every least on the WAN interface.
lastly, a reboot fixes a lot of hosed up stuff with the firewall and handling multiple IPv6 gateways
-
@jeremy-duncan nope, I have nothing configured for NPT. Is the only place to confirm it under firewall, NAT?
I added firewall rule at the top of WAN explicitly allowing all IPv6 traffic of any type. I've rebooted several times too. I disabled the he.net interface (I hadn't used it in years, but I left it configured just in case I wanted to eventually use it again).
I also checked in the firewall logs and there's nothing indicating any blocked IPv6 traffic. The States table shows 4 states corresponding to these neighbor solicitations, and they all say NO_TRAFFIC as the state.
-
@ingenium13 did you follow the instructions for Track interface on the LAN side interfaces for IPv6? You can't statically assign subnets from a DHCPv6-PD allocation unfortunately.
-
@jeremy-duncan Yeah, I have it set to track interface on the LAN side. The LAN router interface correctly gets a routable IPv6 address, and clients get routable IPv6 addresses as well. However the WAN interface just refuses to respond to the neighbor solicitations, so everything just fails with no route to host.
The gateway is fe80::e86:10ff:fea1:7bc2 and WAN is automatically assigned fe80::e86:10ff:fea1:7bc2%igb0. Manually trying to ping the gateway fails:
[2.6.0-RELEASE][root@pfSense]/root: ping6 -I igb0 fe80::e86:10ff:fea1:7bc2 PING6(56=40+8+8 bytes) fe80::2e0:67ff:fe2a:da56%igb0 --> fe80::e86:10ff:fea1:7bc2 ^C --- fe80::e86:10ff:fea1:7bc2 ping6 statistics --- 12 packets transmitted, 0 packets received, 100.0% packet lossThe packet capture just shows repeat attempts by pfsense to send neighbor solicitation upstream, but no response. And then 2600:4041:170::1 sends a solicitation to pfsense, which also doesn't respond.
It seems like a firewall issue, but I'm not sure what else I can do besides setting a rule to allow all IPv6 traffic, which I've already done.
I've tried pinging from the LAN interface as well with the same result.
-
@ingenium13 as stupid as this sounds.. did you reboot it?
-
@jeremy-duncan Yes, many times
-
@ingenium13 what other 3rd party or extraneous apps you have running? Do you have a Dynamic DNS agent running?
-
@jeremy-duncan Yeah I have Dynamic DNS running updating an IP on Cloudflare, pfblockerng, suricata (not in blocking mode, and only on some LAN interfaces, but not the one I'm testing IPv6 with), avahi (with ipv6 support turned off), haproxy, bandwidthd, and igmpproxy. And wireguard.
Firewall logs aren't showing any blocked IPv6 traffic.
