Comcast dropping prefix delegation?
-
Same here. Except I always get a new prefix with each renew. I traced it backward and forward and it traces everything except the last hop. It's the one step beyond pfsense that's the problem.
Out of curiosity, what modem do you use JC Denton?
-
Out of curiosity, what modem do you use JC Denton?
I have a Motorola SB6141.
cmb, the traceroute ends one hop before pfSense (i.e., my IPv6 upstream gateway) when tracing externally. Do a renew and it's back to normal.
-
And it barfed again shortly within an hour or two. Renewed and got a different /60. Very frustrating.
I think my current workaround will be to have cron HUP dhcp6c every few hours until I can find a Comcast rep that can speak v6.
-
"until I can find a Comcast rep that can speak v6"
I gave up on that ;) I just run a tunnel with HE until comcast is ready for primetime on their ipv6.. Rock solid… One thing for sure I always have the same /48 never have to worry about that changing..
-
I'm pretty sure that Comcast's IPv6 is done and ready, considering they gave a talk last month in France at a Cisco symposeum about moving to IPv6-only (don't worry, there's still at least a few years before that's a reality; though "IPv4 as a service" could be sooner).
I can honestly say that I've had very minimal issues with pfSense and Comcast's IPv6. Usually they appear when the modem reboots, then comes back online (like during maintenance)… sometimes IPv6 comes back, sometimes it doesn't, sometimes it does but not completely. But I've had pfSense uptime stretches of over a month with no IPv6 issues.
-
Maybe in your area it is.. But it for sure not primetime in all their areas… Shoot try calling and asking for help with, good luck talking to anyone with a clue.. So great that they tell the world they are ready, and not saying they are not ahead of the game compared to other isps.
So their ipv6 is 100% and ready, just like they tell the world their customer service is great... <rofl>I wouldn't trust everything that comes out of the corp PR mouth ;)
So they plan on going ipv6 only internally... Which is interesting, but they sure an the hell can not just give their customers ipv6 only..</rofl>
-
Maybe in your area it is.. But it for sure not primetime in all their areas… Shoot try calling and asking for help with, good luck talking to anyone with a clue.. So great that they tell the world they are ready, and not saying they are not ahead of the game compared to other isps.
So their ipv6 is 100% and ready, just like they tell the world their customer service is great... <rofl>I wouldn't trust everything that comes out of the corp PR mouth ;)
So they plan on going ipv6 only internally... Which is interesting, but they sure an the hell can not just give their customers ipv6 only..</rofl>
Maybe in my area… which is still running an old CMTS and only has 8 downstream channels, while there are other areas that are upgraded and ready for DOCSIS 3.1 and have 24 or more channels. My modem supports 16, so that's how I know I only have 8 available.
As for phone support... I don't expect them to know a THING about IPv6. I will only ever call as a last resort, and so far in over a year and a half I've only needed to call twice... once was to confirm an outage with their automatic system (the outage was fixed less than an hour later), the second was to activate a modem that wouldn't activate through their "walled garden".
-
@JC:
And it barfed again shortly within an hour or two. Renewed and got a different /60. Very frustrating.
Ugly. Definitely sounds like they have something screwed up there. The one thing beyond what you've tried that I'd do is run a ping to your IPv6 LAN IP from the Internet, and packet capture on WAN filtered on ICMPv6. The traceroute makes it sound like it's definitely not getting to you, but that'll 100% confirm or deny that. Assuming you likely won't see the ICMPv6 on WAN, that means they did drop your PD route. That's probably the likely scenario since everything else you've noted matches the reports of about a handful of other people who are having that problem.
-
@cmb:
@JC:
And it barfed again shortly within an hour or two. Renewed and got a different /60. Very frustrating.
Ugly. Definitely sounds like they have something screwed up there. The one thing beyond what you've tried that I'd do is run a ping to your IPv6 LAN IP from the Internet, and packet capture on WAN filtered on ICMPv6. The traceroute makes it sound like it's definitely not getting to you, but that'll 100% confirm or deny that. Assuming you likely won't see the ICMPv6 on WAN, that means they did drop your PD route. That's probably the likely scenario since everything else you've noted matches the reports of about a handful of other people who are having that problem.
Yep, no packets. I don't suppose anyone here would have an intelligent contact at Comcast? FWIW, I tried again and they offered to roll a truck to connect my "version 6 server" :-X
In the meantime, I've brought my Tunnelbroker tunnel back up for reliable v6.
-
@JC:
Yep, no packets. I don't suppose anyone here would have an intelligent contact at Comcast? FWIW, I tried again and they offered to roll a truck to connect my "version 6 server" :-X
Man, they really need to train their support people on IPv6…
But your mention of an intelligent contact at Comcast made me remember, John Brzozowski himself has an account on this site. You'd be hard-pressed to find a more intelligent IPv6 contact anywhere in the world.
I don't want to inundate him with people's support issues, but this is definitely a problem that's impacting multiple Comcast customers and it's pretty clearly something in their network. It's not that many relative to how many people use pfSense with Comcast IPv6, maybe a handful that I've heard from recently. But it might be something they're unaware of, or are working on and could use a technically competent customer to help. I emailed John with a link to this thread, maybe he'll be able to chip in.
-
Hello all,
I have posted here before in the past. I run the IPv6 program at Comcast. Quick clarifying question, at some point those who had a /60 are now getting a /64 for PD? Or no PD at all?
John
-
Quick clarifying question, at some point those who had a /60 are now getting a /64 for PD? Or no PD at all?
Hi John,
I get a /60 again if I force a renew. There's about a 20-25% chance I get the /60 I was using back.
I know there's no guarantee I'll have the same PD for any length of time. What's getting me though, is the PD's route seems to drop silently (and prematurely) on the Comcast side. So IPv6 traffic on my side grinds to a halt until I manually intervene or dhcp6c goes out and attempts a renew when its timer expires. I've been able to replicate this using two different routers in two vastly different Comcast service locations (two different states).
Happy to help troubleshoot this any way I can. Just let me know what you need :)
Thanks!
-
OK this helps. When we perform moves and splits for capacity addresses (for IPv4 and IPv6) and prefixes for IPv6 change unless of course you are a commercial static customer. Commercial static customer IP blocks (IPv4 and IPv6) are static across moves and splits. NOTE - commercial static has NOT yet launched, targeting this summer.
One thing that we did find is that after a mode or a split RENEWs for /60s are responded to with a /64. I think this is what folks are seeing. This is a bug in the DHCP server that I believe has been fixed. I need to check on the rollout of the same.
If the router in use does not deactivate the /60 or parts of the same this will properly result in traffic for the same not being routed. Remember the router was just moved from one CMTS to another. The new /64 should be properly routed. I understand this does not to the crux of the issue a /60 is needed/wanted.
Once you have the /64, the best way to get a /60 back is to change the DUID of your DHCPv6 client. This will result in a new DHCPv6 from your router which will in turn be responded to with a /60 (make sure you request a /60 in the DHCPv6 SOLICIT). This is the best self service approach I can offer, otherwise, the only way for me to help is to manually request the deactivation of the target /64 which in turn prevent the same from being RENEWed. The latter has some scaling issues, i.e. you need me to be involved with each and every request.
Let me know if the above does or does not make sense. And more importantly if what I suggested addresses the issue.
John
-
So what it sounds like might be happening is that since pfSense is requesting renewal of a /60, and Comcast is responding with a /64, pfSense is ignoring the /64 (not necessarily a bad practice if you have set up multiple networks using prefix IDs of a /60). Eventually, the /60 lease expires, but pfSense isn't showing that the /60 isn't valid anymore.
So pfSense might need to take action if the IPv6 prefix renewal doesn't complete as expected and/or the lease received previously expires… though it does sound like the root issue here is on Comcast's side, with the server responding with incorrect prefix size.
-
I have not tested this personally, so I am offering my best guess here. I think what you suggest sounds plausible. pfSense should accept what is offered back as part of the renewal even if it is not the same length as what it currently has, this would at least ensure that IPv6 does not entirely disappear. Note the RENEW message that is sent back in this case contains two IA_PDs, one with zero lifetimes (i.e. expiring the same) and one with non-zero lifetimes, the new prefix which is most likely a /64.
The root of the issue is the change of the prefix size here, however, RFC3315 and RFC3633 indicate that this is legal from a DHCPv6 point of view. Trust me the prefix length change was not intentional.
The mechanism, modulo the prefix length change, is how prefix renumbering is triggered, as such we should look into fixing the same.
Let me know what I can do to help,
John
-
So I'm not seeing a /64 in my logs (just the split(s) of the /60 for my sub-interfaces). Digging a bit, I am seeing two different DHCP6 server DUIDs responding. Not sure if that's relevant, though?
John, I'll share my dhcp6c log with you in a PM so you have a bit more context…
-
I might be in a good position to be a test subject for any changes, at least for the next week… Comcast is getting ready for a CMTS upgrade in my area, so there has been maintenance the past few nights. Last night, it looks like my LAN prefix was lost, as my remote smokeping shows it dropping along with WAN addresses at the start of the work, but it never recovered while the WAN side did.
I'll be happy to recover any logs or anything this afternoon when I get home... just let me know what's needed.
-
@JC:
So I'm not seeing a /64 in my logs (just the split(s) of the /60 for my sub-interfaces). Digging a bit, I am seeing two different DHCP6 server DUIDs responding. Not sure if that's relevant, though?
John, I'll share my dhcp6c log with you in a PM so you have a bit more context…
Seeing two ADVERTISE messages one each from two separate DHCPv6 server is correct. The full four message exchange will only complete with one of the DHCP servers. You should not be seeing REPLIES from two DHCP servers.
John
-
@virgiliomi:
I might be in a good position to be a test subject for any changes, at least for the next week… Comcast is getting ready for a CMTS upgrade in my area, so there has been maintenance the past few nights. Last night, it looks like my LAN prefix was lost, as my remote smokeping shows it dropping along with WAN addresses at the start of the work, but it never recovered while the WAN side did.
I'll be happy to recover any logs or anything this afternoon when I get home... just let me know what's needed.
Just to confirm, you have no delegated prefix now? Or you are fully back online? Either way ping me by messenger or email.
John
-
Thanks all for continuing to work on this -I've been in the field a bunch and not able to keep up here but I did want to chime in/confirm with others.
@JC:
I get a /60 again if I force a renew. There's about a 20-25% chance I get the /60 I was using back.
I know there's no guarantee I'll have the same PD for any length of time. What's getting me though, is the PD's route seems to drop silently (and prematurely) on the Comcast side. So IPv6 traffic on my side grinds to a halt until I manually intervene …
The same here for me except always a new /60.
If the router in use does not deactivate the /60 or parts of the same this will properly result in traffic for the same not being routed.
I've also thought this may be an issue for a while. pfSense doesn't seem to know and/or indicate the prefix is dead and the clients still receive the dead prefix. It would be great if -for whatever reason and whatever ISP- the prefix expires/dies etc. that pfSense would indicate that somehow or there would be some way to handle that. It is easy to see and set what prefix information is being requested, but I haven't found a way to easily view the status, that is, to see when or what has been received etc. My old cheap router would show the prefix received and that was handy -if pfSense had some status information regarding the prefix that would be nice. So far all I've done is look at the internal interface addresses -please let me know if there is an easier way to view the prefix info.
Thanks!
