IPv6 Route Troubleshooting
-
I had a similar problem with my ISP a few months ago. One question, when you ping have you tried from pfSense, as well as computers behind it? When I had that problem pinging from pfSense would work, but not from anything on my LAN. This indicated a routing problem was likely. Also, that address assigned to pfSense is probably not used for routing. If it has a /128 mask, then it's only used to identify that computer, and be used as the destination for accessing your firewall/router. It's the address you'd use for VPNs, etc.. Routing on IPv6 is very often done over the link local addresses. As part of my testing, I used a managed switch, configured for port mirroring, between my cable modem and pfSense, to capture what was happening when pfSense booted up. In that capture, I found a packet that described an error on the CMTS, about no prefixes available and even identified the CMTS by name. I also ran into the same traceroute problem when trying addresses on my LAN, but to pfSense worked fine.
So, try testing to both LAN and WAN addresses and see if there's a difference. Also, use Wireshark, on a computer connected to that managed switch, to examine the packets during boot up, as Packet Capture is not usable at that time.
It took a lot of effort, but I was finally able to get the problem resolved.
You might also mention the ISP, so that other users might be able to help.
-
The ISP is Greenlight Networks. It's a small ISP based in Upstate New York (US). It's fiber and competitive with Spectrum so I can't complain there but sometimes they leave a little bit to be desired on the support side.
Anyway, I've had the same issues via the pfSense web-based command prompt. Nothing responds. I'm not sure which address you mean for the one assigned to pfSense but the /128 on the LAN interface was what I had been using for remote access type things. However, the WAN interface is also a /128 but I have pfSense set to request an IP not just a prefix (or whatever the wording is) on the WAN interface.
Good idea on the port mirroring and Wireshark. It's just weird to me that IPv6 doesn't seem to route out and I can't get anything to route back in from a different (work) ISP.
The other thing with the "maintenance" is that they've promoted it as new/improved hardware. So I'm still leaning towards something on their end and their announced IPv6 prefixes have changed a bit lately. I was hoping things would work themselves out but that was a week ago and my config hasn't changed.
-
Are you sure you have a /128 on the LAN interface? It should be a /64.
By address assigned to pfSense, I'm referring to the one that comes from the ISP and is not part of your LAN.
When you request a prefix, it's for the LAN, not WAN. Without it, you won't have much of a network.
A few weeks ago, I posted an article here about creating a "data tap", with a managed switch. One thing you'll want to do is ensure the ports used between the modem and firewall are on a separate VLAN, as I described, to keep packets from the computer from interfering with what you're testing.
However, given the ISP made some changes, the problem is very likely their fault. If they also can't ping the pfSense assigned address, then they should fix that.
-
Yeah, I mean my LAN is a /64 from the ISP but pfSense has a single IP in that /64 that is assigned to it's LAN interface and that is the public IP I use for remote access to services on that machine.
To clarify on the WAN setup. For the WAN DHCP6 setting, I have "Request only an IPv6 Prefix" unchecked so that I get both a Prefix for the LAN and an IP for the WAN interface that uses a different prefix (if that makes sense).
Thanks for the link.
-
I'm not sure if the ISP has changed anything but they have someone looking into it... In the meantime I tried again to ping/traceroute from the web-based pfSense command prompt and that all works now.
I also tried ping and traceroute from work (different ISP) and I can get a response from the WAN interface IP but not the LAN interface IP. Also, LAN clients are still not able to route successfully (traceroute hits my pfSense LAN IP then fails beyond that). Now it sounds like the problem is on my end? Any more ideas?
Thanks for all the help.
-
@digitalgeek said in IPv6 Route Troubleshooting:
I also tried ping and traceroute from work (different ISP) and I can get a response from the WAN interface IP but not the LAN interface IP. Also, LAN clients are still not able to route successfully (traceroute hits my pfSense LAN IP then fails beyond that). Now it sounds like the problem is on my end? Any more ideas?
This is exactly the situation I had and it was caused by the ISP. This is where Wireshark comes in handy to see what's happening when pfSense boots up. In my case, I found an error message from the CMTS. When it comes to IPv6, there are 2 routes involved. One to the /128 address and 1 to your LAN. They should both go through your WAN interface. If they don't it's something causing routing issues before it gets to you. One of the things I did in my testing and also had the tier 2 support try is pinging a LAN address, while I watched for the pings. If you don't see them, it's the ISP's problem. There is nothing you can do to fix the problem, if the pings don't even reach your router.
-
I've attached the packet where I found the error message. It's in frame 29 in DHCPv6 > Identity Association for Prefix Delegation > Status code > Status Message. It says no prefix available and names the CMTS I'm connected to.
-
I told my ISP to try pinging my router's LAN IP address. It has been setup for many months now to respond to ICMP requests (use it for external-based uptime monitoring) and I had no issues with it until their "maintenance" so they should be able to ping it if/when everything is working correctly on their end.
Thanks for the packet capture. I'll see what I can figure out from my end but based on what you've said it sounds like an issue only they can fix...even if I'm able to point them in the right direction.
-
@digitalgeek said in IPv6 Route Troubleshooting:
even if I'm able to point them in the right direction.
I was able to point them in the right direction as soon as I made my first call. It still took about 3 months to fix and even then it only happened after a senior tech took his modem to the head end and tried connecting to 4 different CMTS and it only failed on mine.
-
@JKnott said in IPv6 Route Troubleshooting:
A few weeks ago, I posted an article here about creating a "data tap", with a managed switch.
Would there be anything gained using the managed switch versus just letting pfSense boot up with the WAN port unplugged, start a WAN interface packet capture, plug the WAN interface in, and see what gets logged?
-
@digitalgeek said in IPv6 Route Troubleshooting:
@JKnott said in IPv6 Route Troubleshooting:
A few weeks ago, I posted an article here about creating a "data tap", with a managed switch.
Would there be anything gained using the managed switch versus just letting pfSense boot up with the WAN port unplugged, start a WAN interface packet capture, plug the WAN interface in, and see what gets logged?
You could probably do that. However, you'd still want to download the capture into Wireshark. I had the managed switch, configured as a data tap, so I used it. I have other occasions when I want to capture packets when simply disconnecting/reconnecting won't help.
-
Just wanted to give an update for anyone who stumbles across this:
My ISP was able to verify the problem was on their end and they have implemented a fix. Everything is now working correctly.
-
@digitalgeek said in IPv6 Route Troubleshooting:
My ISP was able to verify the problem was on their end and they have implemented a fix. Everything is now working correctly.
You were able to do that a lot faster than I was. A big part of the problem was the people responsible to fix the problem refused to even look at it, because I had my own firewall/router. This was even after both tier 2 support and a senior tech said the problem wasn't at my end. It even affected my neighbour. I wonder how pfSense was able to affect him!
It was only after the senior tech took his modem to the head end and tested with 4 different CMTS, proving the problem was on mine, that those guys finally fixed the problem.
-
Yeah, It took a week to actually get a response from tier 1 email support (I think they were swamped with the maintenance...but still) and then another couple days for them to realize what I was talking about and that it was over their heads. After that it went to the network engineering team (props to them) and the guy that handled my ticket was able to verify the date I told him the issue started was the same day that he had tracked the issue to. He told me the maintenance was a large migration and they were still working out some of the bugs. A few days later and he told me everything should be fixed...and it was! I'm not sure how much of it was on their schedule to fix versus me complaining but regardless I'm happy to have IPv6 remote connectivity back. Thanks @JKnott for all the help and support!
-
@digitalgeek said in IPv6 Route Troubleshooting:
and then another couple days for them to realize what I was talking about and that it was over their heads
I had the same experience with both the tier 2 and senior tech guys. I had to teach both of them about IPv6. While they had the basics, they didn't understand DHCPv6-PD, nor how the WAN address is not used for routing and more. I still have a problem with those guys who refused to look at the problem, even though it was proven to be elsewhere, just because I have my own router. Hopefully, someone straightened them out about how to do their job. Even the office of the president was getting push back from them on this. The tier 2 & tech guys at least put some effort in and were prepared to learn, so that they could do their job.
BTW, I didn't even bother with tier 1, as I knew they'd be clueless about the problem.
-
Wow, it sounds like you had quite the nightmare. I always hate when "tech support" knows less about the technology than I do but I guess that is tier 1 for you. I've already learned a lot more about IPv6 from you @JKnott just from this thread!
-
@digitalgeek said in IPv6 Route Troubleshooting:
I've already learned a lot more about IPv6 from you @JKnott just from this thread!
I've learned much about it from this book. Also, I learned a lot by using Wireshark to look at the packets.
Tier 1 is fine for basic problems, but they don't have much in the way of problem determination skills. I just feel sorry for the customers who don't have my background, when they have a difficult problem.
As an example, several years ago, I had intermittent failures on my Internet and phone connections, but not TV. A tech came out, wanting to replace the cable from my living room to my computer room, claiming it was old. He couldn't explain why the problem might not be in the cable from the utility room, which was a few years older, might not be the problem. Of course, he wanted to staple the cable along the baseboards, around the doors, etc. to replace a cable, nicely installed by my ISPs tech, which was hidden behind walls, ceiling, etc.. I refused. In my unit I have 2 feeds from the utility room. I did some testing, including running a script that pinged the ISPs gateway, and logging the failures. I then moved my modem to the other feed and testing showed the failure still happening. This was without any of the original coax in the path. This proved the problem was not in my unit. Further testing by the cable company determined the problem was out in the front of my condo, where the cable comes in from the street. If it hadn't been for my testing and determination, that problem would not have been resolved. Most customers would not have been capable of what I did.
BTW, I have many years experience in telecommunications, networks and computers.
-
Thanks for the book link. I'll have to take a look. Yeah, it seems just having a willingness to learn will get you a lot of the way there. I've done similar troubleshooting type things for friends who otherwise would've just accepted unstable internet service as normal. Their ISP usually tells them everything is fine so the ISP doesn't have to spend the resources to figure it out and some of them won't really do much until you beat them over the head with hard data that they can act on after you have basically done their job for them. And even then it can be like pulling teeth as you know. I'm basically tech support for all my friends and family. I'm sure you know the feeling. My experience has come from a lot of tinkering, breaking things, and attempting to fix them before someone yells at me for it.
-
@digitalgeek - Did Greenlight Networks ever tell you what the problem was that they fixed? I’m also using them, and after their Maintenance in Q4 of 2020, ipv6 has not worked. It worked before then, but not after. They also refuse to look into the problem. They even finally stopped responding to repeated inquiries. I’ve been taken all routers here out of the config just for testing (eg directly connecting multiple different systems running different OSs directly to the ONT and they don’t work. Even with that GLN won’t even do anything.
-
@gary201 The issue from July 2019 was resolved without them really going into detail about what was happening during their large maintenance/migration. When I got in touch with them they were still in the "putting out fires" mode. They made a note of my issue, emailed me a few days later when they had a fix in place for me to verify, and all was good.
Around December 2nd of 2020 I did have an IPv6 outage after a maintenance window. No IPv6 traffic was routing. I also tried different machines directly wired to the ONT at that time to verify it wasn't something on my end (not that I had changed anything). I reached out to them and they were able to in their words, "remove a filter" and it fixed my issue. I'm not sure how helpful that is, but it's all they told me.
