Verizon Fios and IPV6, Which Settings Work?

SirSilentBob

@mikev7896 said in Verizon Fios and IPV6, Which Settings Work?:

@mattlach It's not only a Verizon issue... The first item I read about the issue was in the Intel community and was from a user of a fiber service in Canada... no Verizon there. They have an Alcatel-Lucent ONT though.

Mike, I'm curious, how are your other offloading settings configured?

I checked mine, and apparently I have hardware checksum offloading enabled. I checked what ONT I have, and I believe it was installed in mid-late 2011. Going by the ONT S/N, which the first 4 characters are T0211, I'm assuming that means it was made in February of 2011. The ONT I have is a Motorola DBBU-1238 Firmware rev. C (This might just be the model number of the in-door unit with battery and power supply though.) Assuming the outside guts are also Motorola, then I guess this old Motorola unit doesn't have the IPv6 bug. (Frankly I'm amazed that Motorola seems to have passed on an opportunity for a bug/deficiency!)

So I guess this can be a confirmation that this particular Motorola ONT doesn't have the same issue.

Here's my settings, just curious how they compare to what you are running:

SirSilentBob

I have sent a local friend of mine a message to check what ONT he has, he's on the same street and CO as me, so he should have IPv6 but it's just not working. If he's got an Alcatel/Lucent unit then he'll just have to figure out how to disable hardware checksum offloading in BSD and try again, as I'm 99% sure he's got a quad intel gigabit card like I'm running. If disabling that makes it work, I'll see if I can get him to provide the info on his ONT so it can be confirmed as a "bugged" one if anyone is keeping track.

MikeV7896

@sirsilentbob said in Verizon Fios and IPV6, Which Settings Work?:

Here's my settings, just curious how they compare to what you are running:

I have all of the hardware offloading settings disabled. I'm guessing my CPU is powerful enough to handle everything, because with gigabit service I can still get full 940 Mbps results on speed tests.

As far as the ONT tracking, I think that's a bit outside of the scope of this community. The original issue has only been mentioned as happening with the Alcatel-Lucent ONTs, and I don't believe there have been any reports of other ONTs having a similar issue.

kohenkatz

@sirsilentbob said in Verizon Fios and IPV6, Which Settings Work?:

this old Motorola unit

Since the beginning of FiOS rollouts, Verizon has used at least 24 different Motorola ONT models, 11 Tellabs models, and 14 Alcatel models. Of those, all the Tellabs models and 10 of the Motorola models were using a technology (BPON) that Verizon no longer uses. DSLReports has a list. As far as I can tell, the Motorola units have never exhibited this IPv6 issue.

nolaquen

For the folks that have had IPv6 up and running for a while, has anyone had the /56 prefix change on them? It just went active for me this week, and curious how much effort I need to put into insulating my configuration (to the extent I can) from WAN prefix changes.

For IPv4, the only time in recent memory that it changed was actually this week when it went down and the link came back up with IPv6. Prior to that, it was consistent through power outages, equipment poweroffs, etc.

jeremy.duncan

Woo hoo! Verizon FiOS finally turned up my area for IPv6. I was able to get it working for a few zones on my firewall. I have a few subnets and other routed subnets on my network that I am keeping on my HE tunnel as it's permanent prefix - using some policy-based routing. Otherwise everytime the firewall resets or connection resets it will change the IPv6 prefix. Sure wish Verizon would just allocate a permanent prefix.

SirSilentBob

@jeremy-duncan

Nice, so the IPv6 has made it across the water to Chesapeake finally, congrats!

As for the changing prefix issue, have you considered something like using ULAs?

I have really really considered it (mostly have not yet because lazy and a lot of stuff to set up) because I want to be able to have devices, some of which are mobile and will only use SLAAC have a consistent IPv6, the same as I do on the IPv4 RFC1918 side of things. I think my solution for the mobile devices that can't take IPv6 via DHCP is if I did this, just manually set the ULA on the device and take it off "automatic". Honestly, I would probably make the tail end of the IPv6 address just be the same as it is for IPv4. Even if that meant my LAN devices had a ULA like fd48:dead:beef:50:192:168:50:10 and it's IPv4 was 192.168.50.10.

I have certain devices that I want the addresses to stay the same so I can do device specific things / exemptions for filtering.

I thought about leaving a HE tunnel up, but doing it this way I'd avoid issues with changing prefixes and the devices won't be limited to the tunnel speed, they'd all have the full gigabit line speed.

Let me know how it goes if you try this out, please! I'm staring at my access point right now, with currently 28 devices on and active on the network, and it's just me having to probably commit a whole day to this change-over that is stopping me from already having done it, and I still can think of 5 or 6 more devices that are just off right now...

jeremy.duncan

@sirsilentbob ah. No. My fios business is in chesapeake and that's still limping along on v4. My home network in fairfax county was the one that was finally enabled... as for ULA, no because that just means I'd have to do network prefix translation on the firewall which wouldn't be any better performance than a tunnel even with a lower MTU. So to keep these subnets somewhat static I'll just keep the policy routing going and do the cool IPv6 on my wifi and another DMZ.

ingenium13

Verizon rolled out IPv6 in my neighborhood. I'm able to assign /64s (in the right IP ranges) to my vlans, however pfsense won't pass traffic. Packet captures show that pfsense is sending neighbor solicitations, but not getting a response, and upstream is also sending neighbor solicitations, but pfsense isn't responding.

11:04:24.681659 IP6 fe80::2e0:67ff:fe2a:da56 > fe80::e86:10ff:fea1:7bc2: ICMP6, echo request, seq 4139, length 9
11:04:25.143801 IP6 2600:4041:170::1 > ff02::1:ff2a:da56: ICMP6, neighbor solicitation, who has fe80::2e0:67ff:fe2a:da56, length 32
11:04:25.182549 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:26.182638 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:27.182427 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:28.189624 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:28.191316 IP6 2600:4041:170::1 > ff02::1:ff2a:da56: ICMP6, neighbor solicitation, who has fe80::2e0:67ff:fe2a:da56, length 32
11:04:29.189601 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:30.189391 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:31.155646 IP6 2600:4041:170::1 > ff02::1:ff2a:da56: ICMP6, neighbor solicitation, who has fe80::2e0:67ff:fe2a:da56, length 32
11:04:31.195582 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:32.195400 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:33.195391 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32
11:04:34.162249 IP6 2600:4041:170::1 > ff02::1:ff2a:da56: ICMP6, neighbor solicitation, who has fe80::2e0:67ff:fe2a:da56, length 32
11:04:34.201399 IP6 fe80::2e0:67ff:fe2a:da56 > ff02::1:ffa1:7bc2: ICMP6, neighbor solicitation, who has fe80::e86:10ff:fea1:7bc2, length 32

I tried to ping one of my vlan assigned IPv6 addresses from my VPS, and I see the ping come in in a packet capture on WAN, but again, no response.

I had he.net working previously. I added a firewall rule at the top of WAN to allow all IPv6. "Block private networks and loopback addresses" is disabled.

Any thoughts?

jeremy.duncan

@ingenium13 yeah I had this exact problem when I had the Network Prefix Translation enabled for a few subnets that I was keeping on HE. Do you also have that enabled? If so, just disable them and restart the WAN NIC because that package doesn't seem to work correctly.

Otherwise, you need to make sure your firewall rules allow for IPv6 ICMP-IPv6 for neighbor advertisement, solicitation, and packet too big at the every least on the WAN interface.

lastly, a reboot fixes a lot of hosed up stuff with the firewall and handling multiple IPv6 gateways

ingenium13

@jeremy-duncan nope, I have nothing configured for NPT. Is the only place to confirm it under firewall, NAT?

I added firewall rule at the top of WAN explicitly allowing all IPv6 traffic of any type. I've rebooted several times too. I disabled the he.net interface (I hadn't used it in years, but I left it configured just in case I wanted to eventually use it again).

I also checked in the firewall logs and there's nothing indicating any blocked IPv6 traffic. The States table shows 4 states corresponding to these neighbor solicitations, and they all say NO_TRAFFIC as the state.

jeremy.duncan

@ingenium13 did you follow the instructions for Track interface on the LAN side interfaces for IPv6? You can't statically assign subnets from a DHCPv6-PD allocation unfortunately.

ingenium13

@jeremy-duncan Yeah, I have it set to track interface on the LAN side. The LAN router interface correctly gets a routable IPv6 address, and clients get routable IPv6 addresses as well. However the WAN interface just refuses to respond to the neighbor solicitations, so everything just fails with no route to host.

The gateway is fe80::e86:10ff:fea1:7bc2 and WAN is automatically assigned fe80::e86:10ff:fea1:7bc2%igb0. Manually trying to ping the gateway fails:

[2.6.0-RELEASE][root@pfSense]/root: ping6 -I igb0 fe80::e86:10ff:fea1:7bc2
PING6(56=40+8+8 bytes) fe80::2e0:67ff:fe2a:da56%igb0 --> fe80::e86:10ff:fea1:7bc2
^C
--- fe80::e86:10ff:fea1:7bc2 ping6 statistics ---
12 packets transmitted, 0 packets received, 100.0% packet loss

The packet capture just shows repeat attempts by pfsense to send neighbor solicitation upstream, but no response. And then 2600:4041:170::1 sends a solicitation to pfsense, which also doesn't respond.

It seems like a firewall issue, but I'm not sure what else I can do besides setting a rule to allow all IPv6 traffic, which I've already done.

I've tried pinging from the LAN interface as well with the same result.

jeremy.duncan

@ingenium13 as stupid as this sounds.. did you reboot it?

ingenium13

@jeremy-duncan Yes, many times

jeremy.duncan

@ingenium13 what other 3rd party or extraneous apps you have running? Do you have a Dynamic DNS agent running?

ingenium13

@jeremy-duncan Yeah I have Dynamic DNS running updating an IP on Cloudflare, pfblockerng, suricata (not in blocking mode, and only on some LAN interfaces, but not the one I'm testing IPv6 with), avahi (with ipv6 support turned off), haproxy, bandwidthd, and igmpproxy. And wireguard.

Firewall logs aren't showing any blocked IPv6 traffic.

jeremy.duncan

@ingenium13 I've seen some jacked up stuff with dyn dns trying to update the DNS with the IPv6 address instead of the IPv4. Try disabling those and rebooting and see what happens

ingenium13

@jeremy-duncan Didn't make a difference. I even deleted my he.net config just in case that was somehow causing a problem.

jeremy.duncan

@ingenium13 odd... I'm all outta ideas