ISP offering and testing an IPv6 BETA program, but it's not working and need some ideas

bmeeks

My fiber ISP recently started to offer an IPv6 BETA program. They intend to provide native IPv6 service. I think at this point I am their first and only tester. It's a small ISP in a small rural town. My experience with IPv6 is limited, but I'm learning.

The setup is supposed to offer me a /60 prefix using DHCPv6 prefix delegation. I have pfSense set up for DHCPv6 on the WAN with prefix delegation, and then "Track WAN Interface" configured on my LAN. It's not working.

I think the issue is on my ISP's side, but perhaps I am misunderstanding how things should be handled. I've captured the DHCPv6 exchange on my pfSense WAN port. There are two packet exchanges that really indicate to me there is a misconfiguration on the ISP's side. Confirmation from experts here would be helpful.

Below is the packet capture of the entire exchange. Look at the final two lines of the capture. The one thing that sticks out to me is the UDP reply from the ISP containing the prefix delegation info is being sent to UDP port 547 on my WAN's link-local address. Should not that be coming to port 546 instead? Notice that my WAN side replies back to that with an icmpv6 message saying "destination unreachable, unreachable port".

I'm thinking that because the DHCPv6 information is being sent back to the wrong destination port, my firewall's client is not seeing it and thus I am getting no IPv6 configuration.

Assuming I am correct in my analysis, what type of misconfiguration could they conceivably have on their end? Any hints I could offer them might be helpful. The technology is XGS-PON.

DHCPv6 Packet Exchange Capture

fe80::290:bff:fe7d:1ae5 below is my WAN port address
fe80::480b:bcff:fe54:cfa below is my ISP's gateway

09:38:23.671126 IP6 (hlim 1, next-header UDP (17) payload length: 105) fe80::290:bff:fe7d:1ae5.546 > ff02::1:2.547: [udp sum ok] dhcp6 solicit (xid=a8cd83 (client-ID hwaddr/time type 1 time 730928221 00900b7d1ae5) (IA_NA IAID:0 T1:0 T2:0) (elapsed-time 65535) (option-request DNS-server DNS-search-list) (IA_PD IAID:0 T1:0 T2:0 (IA_PD-prefix ::/60 pltime:4294967295 vltime:4294967295)))
09:39:40.064721 IP6 (hlim 1, next-header UDP (17) payload length: 89) fe80::290:bff:fe7d:1ae5.546 > ff02::1:2.547: [udp sum ok] dhcp6 solicit (xid=a5dd12 (client-ID hwaddr/time type 1 time 730928221 00900b7d1ae5) (elapsed-time 0) (option-request DNS-server DNS-search-list) (IA_PD IAID:0 T1:0 T2:0 (IA_PD-prefix ::/60 pltime:4294967295 vltime:4294967295)))
09:39:41.171079 IP6 (hlim 1, next-header UDP (17) payload length: 89) fe80::290:bff:fe7d:1ae5.546 > ff02::1:2.547: [udp sum ok] dhcp6 solicit (xid=a5dd12 (client-ID hwaddr/time type 1 time 730928221 00900b7d1ae5) (elapsed-time 110) (option-request DNS-server DNS-search-list) (IA_PD IAID:0 T1:0 T2:0 (IA_PD-prefix ::/60 pltime:4294967295 vltime:4294967295)))
09:39:41.853901 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::290:bff:fe7d:1ae5 > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 16
	  source link-address option (1), length 8 (1): 00:90:0b:7d:1a:e5
	    0x0000:  0090 0b7d 1ae5
09:39:41.854982 IP6 (hlim 64, next-header UDP (17) payload length: 148) fe80::480b:bcff:fe54:cfa.547 > fe80::290:bff:fe7d:1ae5.547: [udp sum ok] dhcp6 relay-reply (linkaddr=:: peeraddr=fe80::290:bff:fe7d:1ae5 (interface-ID 415a5253303030313134...) (relay-message (dhcp6 advertise (xid=a5dd12 (client-ID hwaddr/time type 1 time 730928221 00900b7d1ae5) (server-ID hwaddr type 1 64d1543b1b92) (preference 255) (IA_PD IAID:0 T1:129600 T2:207360 (IA_PD-prefix 2602:fcea:1::/60 pltime:233280 vltime:259200)))))
09:39:41.855025 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 196) fe80::290:bff:fe7d:1ae5 > fe80::480b:bcff:fe54:cfa: [icmp6 sum ok] ICMP6, destination unreachable, unreachable port, fe80::290:bff:fe7d:1ae5 udp port 547
09:39:41.855026 IP6 (hlim 64, next-header UDP (17) payload length: 148) fe80::480b:bcff:fe54:cfa.547 > fe80::290:bff:fe7d:1ae5.547: [udp sum ok] dhcp6 relay-reply (linkaddr=:: peeraddr=fe80::290:bff:fe7d:1ae5 (interface-ID 415a5253303030313134...) (relay-message (dhcp6 advertise (xid=a5dd12 (client-ID hwaddr/time type 1 time 730928221 00900b7d1ae5) (server-ID hwaddr type 1 64d1543b1b92) (preference 255) (IA_PD IAID:0 T1:129600 T2:207360 (IA_PD-prefix 2602:fcea:1::/60 pltime:233280 vltime:259200)))))
09:39:41.855036 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 196) fe80::290:bff:fe7d:1ae5 > fe80::480b:bcff:fe54:cfa: [icmp6 sum ok] ICMP6, destination unreachable, unreachable port, fe80::290:bff:fe7d:1ae5 udp port 547
09:39:43.291952 IP6 (hlim 1, next-header UDP (17) payload length: 89) fe80::290:bff:fe7d:1ae5.546 > ff02::1:2.547: [udp sum ok] dhcp6 solicit (xid=a5dd12 (client-ID hwaddr/time type 1 time 730928221 00900b7d1ae5) (elapsed-time 322) (option-request DNS-server DNS-search-list) (IA_PD IAID:0 T1:0 T2:0 (IA_PD-prefix ::/60 pltime:4294967295 vltime:4294967295)))
09:39:43.306537 IP6 (hlim 64, next-header UDP (17) payload length: 148) fe80::480b:bcff:fe54:cfa.547 > fe80::290:bff:fe7d:1ae5.547: [udp sum ok] dhcp6 relay-reply (linkaddr=:: peeraddr=fe80::290:bff:fe7d:1ae5 (interface-ID 415a5253303030313134...) (relay-message (dhcp6 advertise (xid=a5dd12 (client-ID hwaddr/time type 1 time 730928221 00900b7d1ae5) (server-ID hwaddr type 1 64d1543b1b92) (preference 255) (IA_PD IAID:0 T1:129600 T2:207360 (IA_PD-prefix 2602:fcea:1::/60 pltime:233280 vltime:259200)))))
09:39:43.306591 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 196) fe80::290:bff:fe7d:1ae5 > fe80::480b:bcff:fe54:cfa: [icmp6 sum ok] ICMP6, destination unreachable, unreachable port, fe80::290:bff:fe7d:1ae5 udp port 547
09:39:45.855620 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::290:bff:fe7d:1ae5 > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 16
	  source link-address option (1), length 8 (1): 00:90:0b:7d:1a:e5
	    0x0000:  0090 0b7d 1ae5
09:39:46.655190 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::290:bff:fe7d:1ae5 > fe80::480b:bcff:fe54:cfa: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::480b:bcff:fe54:cfa
	  source link-address option (1), length 8 (1): 00:90:0b:7d:1a:e5
	    0x0000:  0090 0b7d 1ae5
09:39:46.656387 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 24) fe80::480b:bcff:fe54:cfa > fe80::290:bff:fe7d:1ae5: [icmp6 sum ok] ICMP6, neighbor advertisement, length 24, tgt is fe80::480b:bcff:fe54:cfa, Flags [router, solicited]
09:39:46.853087 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::480b:bcff:fe54:cfa > fe80::290:bff:fe7d:1ae5: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::290:bff:fe7d:1ae5
	  source link-address option (1), length 8 (1): 4a:0b:bc:54:0c:fa
	    0x0000:  4a0b bc54 0cfa
09:39:46.853123 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 24) fe80::290:bff:fe7d:1ae5 > fe80::480b:bcff:fe54:cfa: [icmp6 sum ok] ICMP6, neighbor advertisement, length 24, tgt is fe80::290:bff:fe7d:1ae5, Flags [router, solicited]
09:39:47.276246 IP6 (hlim 1, next-header UDP (17) payload length: 89) fe80::290:bff:fe7d:1ae5.546 > ff02::1:2.547: [udp sum ok] dhcp6 solicit (xid=a5dd12 (client-ID hwaddr/time type 1 time 730928221 00900b7d1ae5) (elapsed-time 721) (option-request DNS-server DNS-search-list) (IA_PD IAID:0 T1:0 T2:0 (IA_PD-prefix ::/60 pltime:4294967295 vltime:4294967295)))
09:39:47.289787 IP6 (hlim 255, next-header UDP (17) payload length: 148) fe80::480b:bcff:fe54:cfa.547 > fe80::290:bff:fe7d:1ae5.547: [udp sum ok] dhcp6 relay-reply (linkaddr=:: peeraddr=fe80::290:bff:fe7d:1ae5 (interface-ID 415a5253303030313134...) (relay-message (dhcp6 advertise (xid=a5dd12 (client-ID hwaddr/time type 1 time 730928221 00900b7d1ae5) (server-ID hwaddr type 1 64d1543b1b92) (preference 255) (IA_PD IAID:0 T1:129600 T2:207360 (IA_PD-prefix 2602:fcea:1::/60 pltime:233280 vltime:259200)))))
09:39:47.289813 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 196) fe80::290:bff:fe7d:1ae5 > fe80::480b:bcff:fe54:cfa: [icmp6 sum ok] ICMP6, destination unreachable, unreachable port, fe80::290:bff:fe7d:1ae5 udp port 547
09:39:49.857674 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::290:bff:fe7d:1ae5 > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 16
	  source link-address option (1), length 8 (1): 00:90:0b:7d:1a:e5
	    0x0000:  0090 0b7d 1ae5
09:39:55.362707 IP6 (hlim 1, next-header UDP (17) payload length: 89) fe80::290:bff:fe7d:1ae5.546 > ff02::1:2.547: [udp sum ok] dhcp6 solicit (xid=a5dd12 (client-ID hwaddr/time type 1 time 730928221 00900b7d1ae5) (elapsed-time 1529) (option-request DNS-server DNS-search-list) (IA_PD IAID:0 T1:0 T2:0 (IA_PD-prefix ::/60 pltime:4294967295 vltime:4294967295)))
09:39:55.378278 IP6 (hlim 255, next-header UDP (17) payload length: 148) fe80::480b:bcff:fe54:cfa.547 > fe80::290:bff:fe7d:1ae5.547: [udp sum ok] dhcp6 relay-reply (linkaddr=:: peeraddr=fe80::290:bff:fe7d:1ae5 (interface-ID 415a5253303030313134...) (relay-message (dhcp6 advertise (xid=a5dd12 (client-ID hwaddr/time type 1 time 730928221 00900b7d1ae5) (server-ID hwaddr type 1 64d1543b1b92) (preference 255) (IA_PD IAID:0 T1:129600 T2:207360 (IA_PD-prefix 2602:fcea:1::/60 pltime:233280 vltime:259200)))))
09:39:55.378307 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 196) fe80::290:bff:fe7d:1ae5 > fe80::480b:bcff:fe54:cfa: [icmp6 sum ok] ICMP6, destination unreachable, unreachable port, fe80::290:bff:fe7d:1ae5 udp port 547
09:40:11.768195 IP6 (hlim 1, next-header UDP (17) payload length: 89) fe80::290:bff:fe7d:1ae5.546 > ff02::1:2.547: [udp sum ok] dhcp6 solicit (xid=a5dd12 (client-ID hwaddr/time type 1 time 730928221 00900b7d1ae5) (elapsed-time 3170) (option-request DNS-server DNS-search-list) (IA_PD IAID:0 T1:0 T2:0 (IA_PD-prefix ::/60 pltime:4294967295 vltime:4294967295)))
09:40:11.781788 IP6 (hlim 255, next-header UDP (17) payload length: 148) fe80::480b:bcff:fe54:cfa.547 > fe80::290:bff:fe7d:1ae5.547: [udp sum ok] dhcp6 relay-reply (linkaddr=:: peeraddr=fe80::290:bff:fe7d:1ae5 (interface-ID 415a5253303030313134...) (relay-message (dhcp6 advertise (xid=a5dd12 (client-ID hwaddr/time type 1 time 730928221 00900b7d1ae5) (server-ID hwaddr type 1 64d1543b1b92) (preference 255) (IA_PD IAID:0 T1:129600 T2:207360 (IA_PD-prefix 2602:fcea:1::/60 pltime:233280 vltime:259200)))))
09:40:11.781816 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 196) fe80::290:bff:fe7d:1ae5 > fe80::480b:bcff:fe54:cfa: [icmp6 sum ok] ICMP6, destination unreachable, unreachable port, fe80::290:bff:fe7d:1ae5 udp port 547
09:40:16.942043 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::290:bff:fe7d:1ae5 > fe80::480b:bcff:fe54:cfa: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::480b:bcff:fe54:cfa
	  source link-address option (1), length 8 (1): 00:90:0b:7d:1a:e5
	    0x0000:  0090 0b7d 1ae5
09:40:16.944030 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 24) fe80::480b:bcff:fe54:cfa > fe80::290:bff:fe7d:1ae5: [icmp6 sum ok] ICMP6, neighbor advertisement, length 24, tgt is fe80::480b:bcff:fe54:cfa, Flags [router, solicited]
09:40:21.943274 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::480b:bcff:fe54:cfa > fe80::290:bff:fe7d:1ae5: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::290:bff:fe7d:1ae5
	  source link-address option (1), length 8 (1): 4a:0b:bc:54:0c:fa
	    0x0000:  4a0b bc54 0cfa
09:40:21.943302 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 24) fe80::290:bff:fe7d:1ae5 > fe80::480b:bcff:fe54:cfa: [icmp6 sum ok] ICMP6, neighbor advertisement, length 24, tgt is fe80::290:bff:fe7d:1ae5, Flags [router, solicited]
09:40:43.720207 IP6 (hlim 1, next-header UDP (17) payload length: 89) fe80::290:bff:fe7d:1ae5.546 > ff02::1:2.547: [udp sum ok] dhcp6 solicit (xid=a5dd12 (client-ID hwaddr/time type 1 time 730928221 00900b7d1ae5) (elapsed-time 6365) (option-request DNS-server DNS-search-list) (IA_PD IAID:0 T1:0 T2:0 (IA_PD-prefix ::/60 pltime:4294967295 vltime:4294967295)))
09:40:43.742694 IP6 (hlim 255, next-header UDP (17) payload length: 148) fe80::480b:bcff:fe54:cfa.547 > fe80::290:bff:fe7d:1ae5.547: [udp sum ok] dhcp6 relay-reply (linkaddr=:: peeraddr=fe80::290:bff:fe7d:1ae5 (interface-ID 415a5253303030313134...) (relay-message (dhcp6 advertise (xid=a5dd12 (client-ID hwaddr/time type 1 time 730928221 00900b7d1ae5) (server-ID hwaddr type 1 64d1543b1b92) (preference 255) (IA_PD IAID:0 T1:129600 T2:207360 (IA_PD-prefix 2602:fcea:1::/60 pltime:233280 vltime:259200)))))
09:40:43.742716 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 196) fe80::290:bff:fe7d:1ae5 > fe80::480b:bcff:fe54:cfa: [icmp6 sum ok] ICMP6, destination unreachable, unreachable port, fe80::290:bff:fe7d:1ae5 udp port 547

JKnott

@bmeeks said in ISP offering and testing an IPv6 BETA program, but it's not working and need some ideas:

Below is the packet capture of the entire exchange.

Please download and post the capture file, so it can be examined with Wireshark.

bmeeks

@JKnott: here you go

packetcapture-igb0-20230822093750.pcap

JKnott

@bmeeks

Here's what it shows:

There is no response to multiple solicits.

Here's what it should look like:

This indicates a problem at the ISP. They're not responding to the solicits.

bmeeks

@JKnott:
Thanks for the reply. Can you turn on the SRC and DST ports display in your Wireshark captures of "how it should be"?

Also, if you look at the next to the last packet in the posted file you can see a UDP packet containing all the DHCPv6 prefix info coming from my ISP. But that is destined for port 547. Should it not instead be destined to port 546? What exactly is that exchange from the ISP where source and destination ports are both 547. And it is saying dhcp-relay if I recall.

bmeeks

The ISP has been trying some things on their side for the last couple of days. They reboot their router at 2:00 AM, so it's the next day before I can test the result.

Initially, before their latest "update" Monday night (or 2:00 AM Tuesday morning, actually), I never got anything containing any sort of prefix reply. After the latest change, I started getting that dhcp-relay reply packet containing a prefix delegation string. But that is being sent to the wrong port (547 instead of 546), and I believe it is actually traffic for a DHCPv6 Relay Agent that I should possibly not be seeing at all on my end.

Do you mind sharing a PCAP file from a successful prefix delegation DHCPv6 exchange from your ISP? If you had rather do so privately, send me a PM here on the forum and I will respond with an email address. I would like to see what things should look like to help me understand what may be wrong on my ISP's side.

JKnott

@bmeeks said in ISP offering and testing an IPv6 BETA program, but it's not working and need some ideas:

Thanks for the reply. Can you turn on the SRC and DST ports display in your Wireshark captures of "how it should be"?

Also, if you look at the next to the last packet in the posted file you can see a UDP packet containing all the DHCPv6 prefix info coming from my ISP. But that is destined for port 547. Should it not instead be destined to port 546? What exactly is that exchange from the ISP where source and destination ports are both 547. And it is saying dhcp-relay if I recall.

There is no port columns in Wireshark. You have to look in the expanded packets to see what the port numbers are. However, the protocol is listed in these examples.

As for 546 vs 547, that depends on which end is sending. 546 is the source port for the client, but 547 for the server.

The last packet in your capture is a neighbor advertisement, which could have come from any device. There is nothing back from your ISP for DHCPv6.

Correction, the last packet is destination unreachable.

JKnott

@bmeeks said in ISP offering and testing an IPv6 BETA program, but it's not working and need some ideas:

Do you mind sharing a PCAP file from a successful prefix delegation DHCPv6 exchange from your ISP? If you had rather do so privately, send me a PM here on the forum and I will respond with an email address. I would like to see what things should look like to help me understand what may be wrong on my ISP's side.

I've attached the file. Don't worry about revealing my addresses, as this capture was made before I got my current prefix.

DHCPv6-PD.cap

bmeeks

@JKnott said in ISP offering and testing an IPv6 BETA program, but it's not working and need some ideas:

There is no port columns in Wireshark. You have to look in the expanded packets to see what the port numbers are. However, the protocol is listed in these examples.

As for 546 vs 547, that depends on which end is sending. 546 is the source port for the client, but 547 for the server.

The last packet in your capture is a neighbor advertisement, which could have come from any device. There is nothing back from your ISP for DHCPv6.

Correction, the last packet is destination unreachable.

I found the port columns. You can turn it on by right clicking down in the details display and choosing "Apply as column". Just makes it easier to see who is talking from and to where in the top window.

I'm talking about lines #31 and #32 in the top window of the Wireshark display at times 140.049081 and 140.071568. Notice that reply from my ISP side that is sourced from port 547 (which is correct) and destined for port 547 (which is incorrect as I think it should be 546). Note also this says it is a Relay-reply message type.

About to load and examine your capture. I found another on the web of a DHCPv6 session and it clearly shows the ISP's server replies should be sent to port 546 "from" port 547.

bmeeks

@JKnott said in ISP offering and testing an IPv6 BETA program, but it's not working and need some ideas:

I've attached the file. Don't worry about revealing my addresses, as this capture was made before I got my current prefix.

DHCPv6-PD.cap

Thanks for the file. It confirms what I suspected. My ISP has something not configured correctly with regards to their DHCPv6 Relay Agents.

It is clear in your capture that your ISP replied back with the prefix delegation info using a UDP packet source from port 547 and sent to port 546. That matches what the RFC says because DHCPv6 clients listen only on port 546 for incoming replies.

My ISP has a misconfiguration in their DHCPv6 setup as their Relay Agent appears to be sending the info back with port 547 as the destination instead of 546. Thus my firewall DHCPv6 client is failing to see the returned data (as it's the wrong port). Consequently, my firewall is responding back in that last packet with an ICMPv6 "port unreachable" message to the ISP's Relay Agent or DHCPv6 server.

JKnott

@bmeeks said in ISP offering and testing an IPv6 BETA program, but it's not working and need some ideas:

I'm talking about lines #31 and #32 in the top window of the Wireshark display at times 140.049081 and 140.071568. Notice that reply from my ISP side that is sourced from port 547 (which is correct) and destined for port 547 (which is incorrect as I think it should be 546). Note also this says it is a Relay-reply message type.

I see that replay reply, which I have never seen before. I have no idea what it's about.

bmeeks

@JKnott said in ISP offering and testing an IPv6 BETA program, but it's not working and need some ideas:

@bmeeks said in ISP offering and testing an IPv6 BETA program, but it's not working and need some ideas:

I'm talking about lines #31 and #32 in the top window of the Wireshark display at times 140.049081 and 140.071568. Notice that reply from my ISP side that is sourced from port 547 (which is correct) and destined for port 547 (which is incorrect as I think it should be 546). Note also this says it is a Relay-reply message type.

I see that replay reply, which I have never seen before. I have no idea what it's about.

Yeah, me neither. I've sent your capture and mine to the consulting engineer for my ISP. I think perhaps setting all this up is new for him as well. Hence the BETA program. So, likely a learning curve for the both of us .

Thank you for your input. You validated what I thought I understood. Just wanted another more experienced IPv6 user's view.