Comcast IPv6 woes

wwwdrich

We just added a Comcast Business line at home and I'm tearing my hair out trying to get the IPv6 setup to behave. I am running multi-wan, which I'm sure it part of the issue.

I have been running IPv6 via. a tunnel with my first provider (Sonic) for some time. That was a straight-forward setup. However, since Comcast uses DHCPv6-PD and not static IPs, it's more of a challenge.

The good news -- I have the Comcast WAN interface successfully configured for DHCPv6 and the LAN interface configured for tracking, and I am getting IP addresses on my clients. The problem is that I cannot get pfsense to route reliably.

Routing from the firewall itself works fine:

: mtr --report-wide --report-cycles=5 -6 ipv6.google.com
Start: 2019-04-15T15:25:46-0700
HOST: fw01.lapseofthought.com                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ???                                            100.0     5    0.0   0.0   0.0   0.0   0.0
  2.|-- be-10008-rur01.sanjose.ca.sfba.comcast.net      0.0%     5    9.8  10.3   8.1  15.5   3.0
  3.|-- be-231-rar01.santaclara.ca.sfba.comcast.net     0.0%     5    7.9   9.8   7.9  11.1   1.3
  4.|-- hu-0-2-0-3-ar01.santaclara.ca.sfba.comcast.net 60.0%     5    8.8   8.9   8.8   9.1   0.2
  5.|-- 2001:558:fe0d:7::16                             0.0%     5   12.2  10.9   9.2  12.2   1.2
  6.|-- 2607:f8b0:80f7::1                               0.0%     5   15.7  11.2   9.9  15.7   2.5
  7.|-- sfo03s07-in-x0e.1e100.net                       0.0%     5    9.9  15.2   9.9  34.5  10.8

But routing from my home server tries to go out the old Sonic interface and falls on the floor somewhere in Sonic's net:

: mtr --report-wide --report-cycles=5 -6 ipv6.google.com
Start: Mon Apr 15 15:26:54 2019
HOST: morpheus                                                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 0000-0000-0000-0000-04f7-2002-3420-2062.6rd.ip6.sonic.net  0.0%     5   35.0  31.3  17.2  35.4   7.9
  2.|-- lo0.cgn-gw.equinix-sj.sonic.net                            0.0%     5   17.5  19.7  16.9  23.5   2.6
  3.|-- 201.ge-6-1-1.gw.equinix-sj.sonic.net                       0.0%     5   17.7  69.2  17.6 259.4 106.5
  4.|-- ???                                                       100.0     5    0.0   0.0   0.0   0.0   0.0

I have tried just about every combination I can think of with the gateway settings. Both gateways are showing up in the gateway status and all of the interfaces have IPs (both v4 and v6). I realize that I can't set up load balancing or CARP with DHCPv6-PD, but even setting the IPv6 default gateway to point directly to the Comcast interface doesn't help (the settings below were used to create the above mtr output).

pfSense Interface IPs
pfSense Gateway Settings

I'm sure I'm just missing something obvious, but I'm stumped!

(NOTE: FWIW I'm also being bitten by https://redmine.pfsense.org/issues/5999, but I removed the IP alias as a workaround for now)

wwwdrich

Curiouser and curiouser... With the routing settings above I am now occasionally seeing traffic go out through both WAN interfaces! Which actually sort-of works, I just had a successful test to https://test-ipv6.com/. However my in-house Sensu is really unhappy as it frequently failing to connect to all sorts of resources on the firewalls. And anything connecting to the outside world over IPv6 works intermittently (I assume depending on how it is routing).

: mtr -6 ipv6.google.com
                            My traceroute  [v0.86]
morpheus (::)                                          Mon Apr 15 17:59:17 2019
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                       Packets               Pings
 Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 0000-0000-0000-0000-04f7-2002-34  0.0%    61   17.9  16.2   0.2  47.2  12.9
    fw01.lapseofthought.com
 2. lo0.cgn-gw.equinix-sj.sonic.net   0.0%    61   16.8  16.2   8.2  30.9   4.7
    2001:558:4000:9b::1
 3. 201.ge-6-1-1.gw.equinix-sj.sonic  0.0%    61   17.3  52.6   8.3 301.1  73.1
    be-20008-rur02.sanjose.ca.sfba.comcast.net
 4. be-331-rar01.hayward.ca.sfba.com 71.7%    61   15.5  14.9  10.2  28.9   5.6
 5. hu-0-2-0-2-ar01.santaclara.ca.sf 68.3%    61   10.7  11.3   9.6  17.2   1.8
 6. 2001:558:fe0d:7::a               68.3%    61   11.4  12.2  10.5  17.8   1.6
 7. ???
 8. 2001:4860:0:1::790               66.7%    61   12.4  12.9  10.9  16.6   1.4
 9. 2001:4860:0:1004::f              66.7%    61   11.4  14.9  10.7  66.0  12.1
10. 2001:4860::c:4000:f2d3           66.7%    61   12.6  14.7  11.3  44.9   7.2
11. 2001:4860::9:4000:f387           80.0%    61   19.9  14.0  11.1  29.5   5.4
12. 2001:4860:0:1::965               76.7%    61   11.8  13.6  11.3  20.7   2.4
13. ???
14. sfo03s18-in-x0e.1e100.net        71.2%    60   12.0  14.0  10.3  53.5  10.2

Derelict

Multi-WAN GUA IPv6 is going to be a trick. Especially with one of them DHCP6.

Are you replacing the sonic tunnel with comcast or do you want to run them both simultaneously?

What do you expect of all of the clients? They need to choose what IPv6 address to use as a source address for connections they make. How are you anticipating they will be making those decisions?

Some thoughts here:

https://docs.netgate.com/pfsense/en/latest/book/multiwan/multi-wan-for-ipv6.html

wwwdrich

@Derelict said in Comcast IPv6 woes:

https://docs.netgate.com/pfsense/en/latest/book/multiwan/multi-wan-for-ipv6.html

I read that page when I first started setting this up, and because of it I'm not going to try and do multi-wan routing with IPv6. I would be perfectly happy running all of the IPv6 traffic over Comcast (well, as happy as anyone ever is with Comcast).

The clients are all getting their IPv6 addresses from the Comcast block, so that appears to be working as expected. I had hopes of setting them up with IP aliases for the Sonic tunnel addresses at some point, but if that isn't going to work I can live with it -- I have already pulled the IPv6 service IPs out of my DNS. I have a though for scraping the logs down the road and updating DNS via DDNS from a script, but I'm not there yet.

About the only thing I haven't tried is deleting all of the IPv6 gateway groups and the IPv6 gateway for my Sonic WAN connection. I assumed pfSense would just ignore them if they weren't configured as a default gateway.

Derelict

All you should have to do is remove any traces from the sonic tunnel from your DHCP6 and RAs settings. If the clients don't receive one of those addresses, they should only use the Comcast addresses.

Once the router stops advertising the prefix, the clients should drop the related addresses in short order.

Look at Diagnostics > Routes under IPv6. You should not see any Sonic tunnel traces - particularly the default route.

You can leave the tunnel up and policy route to it if desired - just don't make it the default gateway. That's how I run here with a Cox PD and a hurricane tunnel.

chrcoluk

yeah I initially had a hurricane electric for my ipv6, and I noticed when I left it enabled after adding native ipv6 from my isp (also via dhcp), I would get weird behaviour like certain traffic randomly going out over HE.

I didnt really want to delete the HE config from my device tho so what I ended up doing was adding a command at startup to disable its interface via shellcmd.

There may well have been a more elegant solution tho, I just did what I found the quickest to do at the time other than of course to delete HE.

wwwdrich

Well, I got it working last night... oddly enough the problem was actually due to the bug I mentioned in my original post - https://redmine.pfsense.org/issues/5999. Because IP aliases break tracking I had removed my IPv6 interface from CARP. That meant that my secondary was picking up some percentage of the IPv6 traffic on the LAN. It still had routes to my old ISP over the v6 tunnel, so it was sending some of the traffic out that way.

The solution for the moment was to turn the secondary off, I'm going to do some more config tweaking tonight and see if I can get to a state (probably with no IPv6 on the secondary) where I can leave it on-line so I have at least some level of failover.