Trouble configuring IPv6

JKnott

You're saying you pinged from a computer on the LAN side and the WAN addresses were used? That doesn't make sense, unless you're using NAT.
What do you see on the LAN side?

wineguy

@jknott -- Thank you very much for your time!

The LAN side shows 100% packet loss.

I have a single ipv4 port forward active. No ipv6 NAT.

Digression...
The ping behavior is typical of some *nix flavors. If I have a server with multiple interfaces and cripple one, so that it is UP, but has no route to a destination, then

"ping -I CRIPPLED_IFACE UNREACHABLE_DEST"

I'll get a warning from ping that it may use a different interface and then it does exactly that - and the ping results reflect the ip of the interface it used, rather than the ip of the interface I requested. Quite like what's happening here.

I'm not familiar with how pfSense is written (which is why I'm here), so I have no idea what's really happening, but it looks like the OS doesn't see a route to google from LAN.

JKnott

@wineguy

Where are you pinging from? Yes, I know you can force an interface in some routers.

Ping from a computer on the LAN. Use packet capture to see what's on the LAN interface or run Wireshark on the computer you're pinging from. Also, do a packet capture on the WAN side. Are you saying the packets on the LAN side have different addresses than they have on the WAN side?

BTW, pfsense runs on FreeBSD, which is a *nix flavour.

wineguy

@jknott
Sorry I didn't make this clear. The results I just showed were from the following test:

Start Packet Capture
pfSense:Diagnostics-Packetcapture (WAN,IPv6,any, 2607:f8b0:4009:80b::200e)

Start Ping
pfSense:Diagnostics-Ping (2607:f8b0:4009:80b::200e,IPv6,LAN,10)

Stop Packet Capture

I shall now run the following test:
Ping google from server somewhere on LAN
+Packet capture on LAN
+Packet capture on WAN

![0_1619546676974_packetcaptureWAN.cap](Uploading 100%)

![0_1619546797070_packetcaptureLAN.cap](Uploading 100%)

BTW, this is the error I see when attaching packet captures...

wineguy

@wineguy

Here are the packet captures...

packetcapture.zip

JKnott

@wineguy

OK, both those show no response. So, when you got a response, it was from pfsense and not a computer behind it. When you get the successful pings, it goes out from the WAN address? Is that correct? The prefix on the one that worked is different from the one that failed, though from the same ISP. This tells me that the WAN address works, but not a LAN address. I ran into this same situation a couple of years ago, when there was a problem with my ISP. I did some more testing with a 2nd connection to my ISP (they provide 2 IPv4 addresses and support 2 devices on the cable modem) and was able to show that traffic for the WAN address arrived, but that for a LAN address didn't. This indicated a problem elsewhere than my network. In my testing, I determined there was a problem with my ISP and even identified the failing system by host name. The way I did that was I used Wireshark and a managed switch configured as a data tap to capture the full DHCPv6-PD sequence. By examining that, I could see that the CMTS at my ISP's head end was failing. If you don't have a spare managed switch and computer to run Wireshark on, you can still use Packet Capture. To use Packet Capture, shut down pfsense and disconnect from your modem. Then power up pfsense and start Packet Capture on the WAN port, filtering on DHCPv6 (port 546 or 547). Then reconnect the modem. This should capture the full DHCPv6-PD sequence. Post the capture here.

JKnott

@jknott

BTW, I just tried pinging your WAN and LAN addresses.

Here's what I get when I ping the LAN address:

ping 2603:300a:164f:10e0::167
PING 2603:300a:164f:10e0::167(2603:300a:164f:10e0::167) 56 data bytes
From 2001:558:4040:189:3a17:e1ff:fef8:ef5c: icmp_seq=1 Destination unreachable: Address unreachable
From 2001:558:4040:189:3a17:e1ff:fef8:ef5c: icmp_seq=2 Destination unreachable: Address unreachable
From 2001:558:4040:189:3a17:e1ff:fef8:ef5c: icmp_seq=3 Destination unreachable: Address unreachable

But I get nothing when I try your WAN address. Is it blocked by your firewall?

JKnott

@jknott

I also tried traceroute:

WAN

traceroute 2603:300a:164f:1000:208:a2ff:fe11:a354
traceroute to 2603:300a:164f:1000:208:a2ff:fe11:a354 (2603:300a:164f:1000:208:a2ff:fe11:a354), 30 hops max, 80 byte packets
1 firewall.jknott.net (2607:fea8:4c82:5900:4262:31ff:fe12:b66c) 0.258 ms 0.192 ms 0.174 ms
2 * * *
3 2607:f798:10:10d3:0:241:5615:221 (2607:f798:10:10d3:0:241:5615:221) 20.620 ms 20.790 ms 20.772 ms
4 2607:f798:10:10e1:0:690:6324:9086 (2607:f798:10:10e1:0:690:6324:9086) 21.165 ms 2607:f798:10:31f:0:2091:4823:3185 (2607:f798:10:31f:0:2091:4823:3185) 20.947 ms 2607:f798:10:2f0:0:2091:4823:2041 (2607:f798:10:2f0:0:2091:4823:2041) 20.934 ms
5 2607:f798:10:8b6:0:2091:4823:5230 (2607:f798:10:8b6:0:2091:4823:5230) 23.449 ms 23.010 ms 23.181 ms
6 2607:f798:10:378:0:2091:4823:7021 (2607:f798:10:378:0:2091:4823:7021) 59.283 ms 44.252 ms 63.357 ms
7 * * *
8 * * *
9 be-1213-cr13.350ecermak.il.ibone.comcast.net (2001:558:3:c9::2) 41.470 ms 26.243 ms be-1113-cr13.350ecermak.il.ibone.comcast.net (2001:558:3:c8::2) 25.578 ms
10 * * *
11 be-1312-cs03.chicago.il.ibone.comcast.net (2001:558:3:10a::1) 34.865 ms be-1212-cs02.chicago.il.ibone.comcast.net (2001:558:3:109::1) 38.610 ms be-1112-cs01.chicago.il.ibone.comcast.net (2001:558:3:108::1) 54.106 ms
12 2001:558:3:209::2 (2001:558:3:209::2) 31.555 ms * *
13 2001:558:300:202f::2 (2001:558:300:202f::2) 51.430 ms 37.341 ms 52.052 ms
14 2001:558:320:bd::2 (2001:558:320:bd::2) 38.598 ms 36.015 ms 40.934 ms
15 2001:558:302:840d::2 (2001:558:302:840d::2) 39.504 ms 37.290 ms 35.159 ms
16 2001:558:4040:189:3a17:e1ff:fef8:ef5c (2001:558:4040:189:3a17:e1ff:fef8:ef5c) 63.104 ms 46.823 ms 48.533 ms
17 * * *
18 * * *
19 * * *
20 * * *

This one fails after 16 hops.

LAN

traceroute 2603:300a:164f:10e0::167
traceroute to 2603:300a:164f:10e0::167 (2603:300a:164f:10e0::167), 30 hops max, 80 byte packets
1 firewall.jknott.net (2607:fea8:4c82:5900:4262:31ff:fe12:b66c) 0.255 ms 0.208 ms 0.192 ms
2 * * *
3 2607:f798:10:10d3:0:241:5615:221 (2607:f798:10:10d3:0:241:5615:221) 21.805 ms 21.968 ms 23.550 ms
4 2607:f798:10:31d:0:2091:4823:3177 (2607:f798:10:31d:0:2091:4823:3177) 23.742 ms 2607:f798:10:34c:0:2091:4823:5121 (2607:f798:10:34c:0:2091:4823:5121) 23.321 ms 2607:f798:10:31d:0:2091:4823:3177 (2607:f798:10:31d:0:2091:4823:3177) 23.918 ms
5 2607:f798:10:8b6:0:2091:4823:5230 (2607:f798:10:8b6:0:2091:4823:5230) 23.496 ms 28.617 ms 28.385 ms
6 2607:f798:10:378:0:2091:4823:7021 (2607:f798:10:378:0:2091:4823:7021) 50.861 ms 51.807 ms 51.402 ms
7 * * *
8 * * be-2101-cs01.350ecermak.il.ibone.comcast.net (2001:558:3:140::1) 55.113 ms
9 be-1313-cr13.350ecermak.il.ibone.comcast.net (2001:558:3:ca::2) 40.466 ms be-1213-cr13.350ecermak.il.ibone.comcast.net (2001:558:3:c9::2) 31.590 ms *
10 be-301-cr12.chicago.il.ibone.comcast.net (2001:558:3:1b4::2) 26.862 ms * be-302-cr12.chicago.il.ibone.comcast.net (2001:558:3:1b5::2) 38.452 ms
11 * be-1412-cs04.chicago.il.ibone.comcast.net (2001:558:3:10b::1) 39.337 ms be-1112-cs01.chicago.il.ibone.comcast.net (2001:558:3:108::1) 39.259 ms
12 2001:558:3:209::2 (2001:558:3:209::2) 23.795 ms 40.070 ms 2001:558:3:20a::2 (2001:558:3:20a::2) 41.728 ms
13 2001:558:300:202f::2 (2001:558:300:202f::2) 52.476 ms 39.521 ms 52.740 ms
14 2001:558:320:bd::2 (2001:558:320:bd::2) 35.428 ms 33.893 ms 35.697 ms
15 2001:558:302:840d::2 (2001:558:302:840d::2) 49.037 ms 49.158 ms 49.005 ms
16 2001:558:4040:189:3a17:e1ff:fef8:ef5c (2001:558:4040:189:3a17:e1ff:fef8:ef5c) 50.663 ms 49.072 ms 53.657 ms
17 2001:558:4040:189:3a17:e1ff:fef8:ef5c (2001:558:4040:189:3a17:e1ff:fef8:ef5c) 3041.129 ms !H 3033.028 ms !H 3027.778 ms !H

This one seems to have a successful completion at hop 17.

wineguy

@jknott I've been off-site. I will do this in a few hours. Thank you!

wineguy

@jknott
@jknott Re: your traceroute... I saw this in my logs and thought I was going nuts. Who the *** has been pinging me? And was successful?!? LOL.

When I do some voodoo (disconnect WAN from pfSense, reboot ISP modem, reboot pfSense, reconnect WAN to pfSense after both reboots complete), I can get some flaky, intermittent IPv6 connectivity, sometimes lasting for days.

This is starting to sound like an ISP issue, even though they claim the issue is in my firewall config. (If I connect the server directly to the ISP modem, the IPv6 connection is rock solid. In fact I can multi-home the server and get perfect IPv6 from its direct connection to the ISP modem, while getting no IPv6 through the firewall. But I really want the server behind the firewall...)

I'll run your reboot capture as soon as I get in. Thank you!

JKnott

@wineguy

Yep, it likely is an ISP issue or beyond. If it was your issue, you should see the pings for LAN addresses hitting the WAN interface, but you're not. This is exactly what I saw when I had that problem 2 years ago.

wineguy

@jknott
Here's the capture of 546|547 on the WAN interface...

packetcapture-dhcp6.zip

JKnott

@wineguy

There should be more to it than that. There should be at least 4 packets.

Here's one that works:

DHCPv6 Solicit and Renew.pcapng

Did you follow the instructions to shut down pfsense, disconnect the modem, restart pfsense and reconnect the modem? That should provide the full DHCPv6-PD sequence.

wineguy

@jknott
I followed the instructions precisely. The first time I got two packets.

I tried again - the second time I reset the ISP modem also. Waited for both to finish booting, started the packet capture, and then reconnected the WAN to the ISP modem.

What I sent was the result of the second try (which, to my untrained eyes, seemed much like the first attempt).

JKnott

@wineguy

I don't see any response from your ISP. If that's all you get, then that could be the problem. Take a look at mine. You need at least 4 packets, ending with the reply. In yours, I don't see the advertise or reply. Is IPv6 up on pfsense? I'd be surprised if it is.

wineguy

@jknott
Hmmmm.... It is NOW, but that doesn't mean that it was THEN...

Let me run the test again and WAIT until I see an IPv6 address has been assigned to the WAN before I stop the packet capture.

wineguy

@jknott
Here's where I am now. I tried a few times to get the capture and got zero->2 packets. The zero was the most frustrating, because I saw that the WAN interface acquired the address, but I captured NO packets.

I concluded that I needed to install a sniffer between the WAN interface and the ISP modem, but that involves moving a cable that will break something - my next window will be Tuesday morning.

In the interim though, I was toying with a few environment modifications and changed my RA from 'unmanaged' to 'stateless DHCP' and my IPv6 started working.

I don't believe it's actually fixed, but it's not easy to debug something that isn't obviously broken. If history is a guide, it'll break again within a few days and I'll make my physical changes on Tuesday and have captures that show the full sequence.

Until then...

Thank you, again, for your assistance!

JKnott

@wineguy

Yeah, using a sniffer is a good idea. When I had the problem with my ISP, I made a data tap with a managed switch.