pfSense does not reply to NS sent by ISP router, ISP does not respond to DHCPv6 request as a result

cnrd

Hi

I have problems getting IPv6 setup with my ISP and have been doing a package capture of the traffic on WAN.

When pfSense sends out a DHCPv6 request, my ISP sends out an NS, which pfSense never replies to <-- This seems to be what causes the problem.

The pattern keeps repeating, pfSense sends a DHCPv6 request, the ISP "replies" with an NS and both seem to be waiting for the other to respond.

I tried booting into Linux on the machine that runs pfSense, here I get an IP and when looking at the package capture the OS responds to the NS, which in turn causes the ISP to respond to the DHCPv6 request. I also used dhclient to get both an
IA_NA and IA_PD, which worked as expected.

Does anyone have an idea on what I could try to get pfSense to respond to the NS? Or is the problem somewhere else?

Already tried disabling filtering of private/bogon addresses on my WAN connection.

Thanks in advance.

JKnott

@cnrd

I just watched my WAN connection, though not through DHCPv6. I see the NS coming from pfSense, not my ISP and the ISP replying. With DHCP, both v4 & v6, the process starts with the client using the "unknown" address and the server then providing it. Later, in the initial DHCP sequence and in renewals, the client provides it's assigned address. Can you do a packet capture of the DHCPv6 sequence? To get the full sequence, which will require a reboot, needs to be done with a data tap.

cnrd

@JKnott

Thanks

Here is a pcap of all IPv6 traffic while pfsense is booting all the way through complete bootup.

It was captured using port mirroring on my switch.

As a trivia my ISP has told me they are using Juniper hardware on their end.

JKnott

@cnrd

Is it possible to capture only DHCPv6 and ICMP6. I can certainly do that with Wireshark. For example, I could create a rule port 546 or icmp6, which would capture those two, but nothing else. Otherwise I have to sort through a lot of noise.

cnrd

@JKnott If you open the pcap, you can see that it's all that's in the file ;-) It has 20 entries, all either ICMPv6 or DHCPv6.

EDIT: The ISP is Gigabit.dk

JKnott

@cnrd

Sorry, I was looking at the wrong file. Is fe80::ae16:2dff:fe94:bbd3 your firewall or the ISP? I see requests for that address coming from the unknown address source.

cnrd

@JKnott said in pfSense does not reply to NS sent by ISP router, ISP does not respond to DHCPv6 request as a result:

fe80::ae16:2dff:fe94:bbd3

That is my firewall. (ac:16:2d:94:bb:d3)

Not really sure why pfSense is sending NS for it self.

Those first 4 packages with unknown source seems to be origin from the firewall's NIC (src MAC matches).

JKnott

@cnrd

Then that 1st line would be duplicate address detection (DAD) and when there's no response it assumes the address is free. Later you see the multicast listener reports initially have the unknown source and then 1 with your address. Then there are DHCPv6 solicits, but no response. There should be advertisements from the DHCPv6 server, they're not there.

cnrd

This post is deleted!

cnrd

@JKnott said in pfSense does not reply to NS sent by ISP router, ISP does not respond to DHCPv6 request as a result:

@cnrd

Then there are DHCPv6 solicits, but no response. There should be advertisements from the DHCPv6 server, they're not there.

@JKnott I know it's been some time, but I got my ISP to fix their problem with sending out RA, so now it makes sense to try to fix this.

The problem here is that the DHCPv6 server have no idea about where to send the response, as pfSense is not responding to the NS: "Neighbor Solicitation for fe80::ae16:2dff:fe94:bbd3 from fe:33:42:10:c8:6b", so it has no idea about where fe80::ae16:2dff:fe94:bbd3 is.

The response should have been: "Neighbor Advertisement fe80::ae16:2dff:fe94:bbd3 (sol, ovr) is at ac:16:2d:94:bb:d3"

If pfSense had send that response, then the DHCPv6 server would have replied with a lease.

I'm pretty sure that the DHCPv6 server is not required to multicast the reply, so for unicast it needs to know the mac of the local-link IP requesting a lease.

JKnott

@cnrd said in pfSense does not reply to NS sent by ISP router, ISP does not respond to DHCPv6 request as a result:

The problem here is that the DHCPv6 server have no idea about where to send the response, as pfSense is not responding to the NS: "Neighbor Solicitation for fe80::ae16:2dff:fe94:bbd3 from fe:33:42:10:c8:6b", so it has no idea about where fe80::ae16:2dff:fe94:bbd3 is.

Is that NS being sent to the correct address? Also, is it possible to do a packet capture? You can use Packet Capture in pfSense and download the capture file to post here.

cnrd

@JKnott said in pfSense does not reply to NS sent by ISP router, ISP does not respond to DHCPv6 request as a result:

Is that NS being sent to the correct address?

Yes it's being sent to ff02::1:ff94:bbd3, pfSense' local-link address is fe80::ae16:2dff:fe94:bbd3

Also, is it possible to do a packet capture? You can use Packet Capture in pfSense and download the capture file to post here.

Here is a cap from pfSense, that is captured while reloading the WAN interface, to force it to send a DHCP request.

packetcapture.cap

It is captured by pfSense with Interface set to WAN and Address Family set to IPv6

As before:
WAN: ac:16:2d:94:bb:d3
ISP DHCP servers: fe:33:42:10:c8:6b, e6:5d:37:84:53:17

Also I'm not seeing any NS being blocked in the firewall log.

JKnott

@cnrd said in pfSense does not reply to NS sent by ISP router, ISP does not respond to DHCPv6 request as a result:

Yes it's being sent to ff02::1:ff94:bbd3, pfSense' local-link address is fe80::ae16:2dff:fe94:bbd3

That ff02 address is a solicited node multicast, which is the other end trying to determine your MAC address. However, I don't know why it's doing that, as it should be able to find that in the solicit XID. Regardless, your system should be responding.

cnrd

@JKnott Yeah, I'm not sure why they need that, but I don't think it's out of spec.

Regardless, your system should be responding.

Yeah this it the whole problem, it seems like the NS arrives at pfSense WAN interface, and running pfctl -vvsr indicates that it is even allowed through the firewall?

@11(1000000107) pass quick inet6 proto ipv6-icmp all icmp6-type neighbrsol keep state
  [ Evaluations: 3509019   Packets: 554001    Bytes: 53687917    States: 0     ]
  [ Inserted: pid 72478 State Creations: 683   ]

Do you have any idea about what I could try to figure this out?

Also thanks for the help you have already provided :-)

EDIT: Could it be related to the fact that the NS sent by the ISP DHCP server is sent from 2a00:7660::249, which is not a private address, therefor pfSense refuses to reply?

28	9.708786	2a00:7660::249	ff02::1:ff94:bbd3	ICMPv6	86	Neighbor Solicitation for fe80::ae16:2dff:fe94:bbd3 from fe:33:42:10:c8:6b

As you can see here the source is 2a00:7660::249 and the dest is ff02::1:ff94:bbd3 at this point in time pfSense only knows the address fe80::ae16:2dff:fe94:bbd3 which is not in the same space as 2a00:7660::249

JKnott

@cnrd said in pfSense does not reply to NS sent by ISP router, ISP does not respond to DHCPv6 request as a result:

EDIT: Could it be related to the fact that the NS sent by the ISP DHCP server is sent from 2a00:7660::249, which is not a private address, therefor pfSense refuses to reply?

I wouldn't think so. While you can enable Unique Local Addresses (ULA) they're not needed. Normally, you have public addresses, at least 18.4 billion, billion of 'em.
ULA == the IPv6 equivalent of RFC 1918 addresses.

cnrd

@JKnott I'm simply lost at what to try, to get this to work at this point. I know what the root cause is, namely that there is no reply from pfSense for the NS, I know that the package arrives at the pfSense firewall, I know that it is passed through the firewall, but something behind the firewall simply isn't sending out a reply.

I even tried running with pfctl -d to verify that it was not the firewall that caused the problem.

JKnott

@cnrd

Perhaps you can try an experiment. Connect another computer to the WAN port and try pinging the link local address and see what happens. Do a packet capture of it.

cnrd

@JKnott I just connected my desktop directly to pfSense WAN port, pinging the WAN local-link address worked just fine, and the package-capture showed an NS sent from my desktop and an NA reply from pfSense.

Both machines here used local-link addresses, so I still suspect that the problem lies somewhere with the public address used by my ISP's dhcp servers.

Here is the capture (I only saved the relevant parts): NS-NA.pcapng

cnrd

I also just tested it out on a totally clean install of both pfSense 2.4.5-RELEASE-p1 and a clean install of the latest 2.5-SNAPSHOT, no difference, pfSense is simply not replying.

JKnott

Ping obviously works.

I just did a packet capture with Wireshark and a managed switch, configured as a data tap.

Here's what I see
neighbour solicitation for duplicate address detection
router solicitation
router advertisement
solicit XID
advertise XID
request XID
reply XID

I have attached the packet capture. How does yours compare? I've been using pfsense for almost 4 years and haven't seen it fail. It just works. In fact IPv6 was the reason I moved to pfsense, as the Linux firewall I was running didn't support DHCPv6-PD.

pfsense startup IPv6.pcapng