pfSense does not reply to NS sent by ISP router, ISP does not respond to DHCPv6 request as a result


  • Hi

    I have problems getting IPv6 setup with my ISP and have been doing a package capture of the traffic on WAN.

    When pfSense sends out a DHCPv6 request, my ISP sends out an NS, which pfSense never replies to <-- This seems to be what causes the problem.

    The pattern keeps repeating, pfSense sends a DHCPv6 request, the ISP "replies" with an NS and both seem to be waiting for the other to respond.

    I tried booting into Linux on the machine that runs pfSense, here I get an IP and when looking at the package capture the OS responds to the NS, which in turn causes the ISP to respond to the DHCPv6 request. I also used dhclient to get both an
    IA_NA and IA_PD, which worked as expected.

    Does anyone have an idea on what I could try to get pfSense to respond to the NS? Or is the problem somewhere else?

    Already tried disabling filtering of private/bogon addresses on my WAN connection.

    Thanks in advance.


  • @cnrd

    I just watched my WAN connection, though not through DHCPv6. I see the NS coming from pfSense, not my ISP and the ISP replying. With DHCP, both v4 & v6, the process starts with the client using the "unknown" address and the server then providing it. Later, in the initial DHCP sequence and in renewals, the client provides it's assigned address. Can you do a packet capture of the DHCPv6 sequence? To get the full sequence, which will require a reboot, needs to be done with a data tap.


  • @JKnott

    Thanks

    Here is a pcap of all IPv6 traffic while pfsense is booting all the way through complete bootup.

    It was captured using port mirroring on my switch.

    As a trivia my ISP has told me they are using Juniper hardware on their end.


  • @cnrd

    Is it possible to capture only DHCPv6 and ICMP6. I can certainly do that with Wireshark. For example, I could create a rule port 546 or icmp6, which would capture those two, but nothing else. Otherwise I have to sort through a lot of noise.


  • @JKnott If you open the pcap, you can see that it's all that's in the file ;-) It has 20 entries, all either ICMPv6 or DHCPv6.

    EDIT: The ISP is Gigabit.dk


  • @cnrd

    Sorry, I was looking at the wrong file. Is fe80::ae16:2dff:fe94:bbd3 your firewall or the ISP? I see requests for that address coming from the unknown address source.


  • @JKnott said in pfSense does not reply to NS sent by ISP router, ISP does not respond to DHCPv6 request as a result:

    fe80::ae16:2dff:fe94:bbd3

    That is my firewall. (ac:16:2d:94:bb:d3)

    Not really sure why pfSense is sending NS for it self.

    Those first 4 packages with unknown source seems to be origin from the firewall's NIC (src MAC matches).


  • @cnrd

    Then that 1st line would be duplicate address detection (DAD) and when there's no response it assumes the address is free. Later you see the multicast listener reports initially have the unknown source and then 1 with your address. Then there are DHCPv6 solicits, but no response. There should be advertisements from the DHCPv6 server, they're not there.


  • This post is deleted!

  • @JKnott said in pfSense does not reply to NS sent by ISP router, ISP does not respond to DHCPv6 request as a result:

    @cnrd

    Then there are DHCPv6 solicits, but no response. There should be advertisements from the DHCPv6 server, they're not there.

    @JKnott I know it's been some time, but I got my ISP to fix their problem with sending out RA, so now it makes sense to try to fix this.

    The problem here is that the DHCPv6 server have no idea about where to send the response, as pfSense is not responding to the NS: "Neighbor Solicitation for fe80::ae16:2dff:fe94:bbd3 from fe:33:42:10:c8:6b", so it has no idea about where fe80::ae16:2dff:fe94:bbd3 is.

    The response should have been: "Neighbor Advertisement fe80::ae16:2dff:fe94:bbd3 (sol, ovr) is at ac:16:2d:94:bb:d3"

    If pfSense had send that response, then the DHCPv6 server would have replied with a lease.

    I'm pretty sure that the DHCPv6 server is not required to multicast the reply, so for unicast it needs to know the mac of the local-link IP requesting a lease.


  • @cnrd said in pfSense does not reply to NS sent by ISP router, ISP does not respond to DHCPv6 request as a result:

    The problem here is that the DHCPv6 server have no idea about where to send the response, as pfSense is not responding to the NS: "Neighbor Solicitation for fe80::ae16:2dff:fe94:bbd3 from fe:33:42:10:c8:6b", so it has no idea about where fe80::ae16:2dff:fe94:bbd3 is.

    Is that NS being sent to the correct address? Also, is it possible to do a packet capture? You can use Packet Capture in pfSense and download the capture file to post here.


  • @JKnott said in pfSense does not reply to NS sent by ISP router, ISP does not respond to DHCPv6 request as a result:

    Is that NS being sent to the correct address?

    Yes it's being sent to ff02::1:ff94:bbd3, pfSense' local-link address is fe80::ae16:2dff:fe94:bbd3

    Also, is it possible to do a packet capture? You can use Packet Capture in pfSense and download the capture file to post here.

    Here is a cap from pfSense, that is captured while reloading the WAN interface, to force it to send a DHCP request.

    packetcapture.cap

    It is captured by pfSense with Interface set to WAN and Address Family set to IPv6

    As before:
    WAN: ac:16:2d:94:bb:d3
    ISP DHCP servers: fe:33:42:10:c8:6b, e6:5d:37:84:53:17

    Also I'm not seeing any NS being blocked in the firewall log.


  • @cnrd said in pfSense does not reply to NS sent by ISP router, ISP does not respond to DHCPv6 request as a result:

    Yes it's being sent to ff02::1:ff94:bbd3, pfSense' local-link address is fe80::ae16:2dff:fe94:bbd3

    That ff02 address is a solicited node multicast, which is the other end trying to determine your MAC address. However, I don't know why it's doing that, as it should be able to find that in the solicit XID. Regardless, your system should be responding.


  • @JKnott Yeah, I'm not sure why they need that, but I don't think it's out of spec.

    Regardless, your system should be responding.

    Yeah this it the whole problem, it seems like the NS arrives at pfSense WAN interface, and running pfctl -vvsr indicates that it is even allowed through the firewall?

    @11(1000000107) pass quick inet6 proto ipv6-icmp all icmp6-type neighbrsol keep state
      [ Evaluations: 3509019   Packets: 554001    Bytes: 53687917    States: 0     ]
      [ Inserted: pid 72478 State Creations: 683   ]
    

    Do you have any idea about what I could try to figure this out?

    Also thanks for the help you have already provided :-)

    EDIT: Could it be related to the fact that the NS sent by the ISP DHCP server is sent from 2a00:7660::249, which is not a private address, therefor pfSense refuses to reply?

    28	9.708786	2a00:7660::249	ff02::1:ff94:bbd3	ICMPv6	86	Neighbor Solicitation for fe80::ae16:2dff:fe94:bbd3 from fe:33:42:10:c8:6b
    

    As you can see here the source is 2a00:7660::249 and the dest is ff02::1:ff94:bbd3 at this point in time pfSense only knows the address fe80::ae16:2dff:fe94:bbd3 which is not in the same space as 2a00:7660::249


  • @cnrd said in pfSense does not reply to NS sent by ISP router, ISP does not respond to DHCPv6 request as a result:

    EDIT: Could it be related to the fact that the NS sent by the ISP DHCP server is sent from 2a00:7660::249, which is not a private address, therefor pfSense refuses to reply?

    I wouldn't think so. While you can enable Unique Local Addresses (ULA) they're not needed. Normally, you have public addresses, at least 18.4 billion, billion of 'em.
    ULA == the IPv6 equivalent of RFC 1918 addresses.


  • @JKnott I'm simply lost at what to try, to get this to work at this point. I know what the root cause is, namely that there is no reply from pfSense for the NS, I know that the package arrives at the pfSense firewall, I know that it is passed through the firewall, but something behind the firewall simply isn't sending out a reply.

    I even tried running with pfctl -d to verify that it was not the firewall that caused the problem.


  • @cnrd

    Perhaps you can try an experiment. Connect another computer to the WAN port and try pinging the link local address and see what happens. Do a packet capture of it.


  • @JKnott I just connected my desktop directly to pfSense WAN port, pinging the WAN local-link address worked just fine, and the package-capture showed an NS sent from my desktop and an NA reply from pfSense.

    Both machines here used local-link addresses, so I still suspect that the problem lies somewhere with the public address used by my ISP's dhcp servers.

    Here is the capture (I only saved the relevant parts): NS-NA.pcapng


  • I also just tested it out on a totally clean install of both pfSense 2.4.5-RELEASE-p1 and a clean install of the latest 2.5-SNAPSHOT, no difference, pfSense is simply not replying.


  • Ping obviously works.

    I just did a packet capture with Wireshark and a managed switch, configured as a data tap.

    Here's what I see
    neighbour solicitation for duplicate address detection
    router solicitation
    router advertisement
    solicit XID
    advertise XID
    request XID
    reply XID

    I have attached the packet capture. How does yours compare? I've been using pfsense for almost 4 years and haven't seen it fail. It just works. In fact IPv6 was the reason I moved to pfsense, as the Linux firewall I was running didn't support DHCPv6-PD.

    pfsense startup IPv6.pcapng


  • @JKnott Everything here is captured in the same way, Wireshark with a port mirror.

    Here is the order:

    duplicate address detection
    solicit XID (x5)
    neighbor solicit from 2a00:7660::248 and 2a00:7660::249

    This pattern repeats, as pfSense is waiting for a reply to the solicit XID and ISP routers are waiting for a reply to the neighbor solicits before sending a reply to the XID.

    pfsense startup.pcapng

    Interestingly I also tried setting up a pure FreeBSD 12.2 machine, just to see if that would reply to the NS, but no dice, so it seems that the problem lies somewhere below pfSense.

    Here is a capture of that NS-no-response.pcapng

    As you can see, same pattern, the ISP routers refuses to do anything before they get a reply to the NS they are sending out, while FreeBSD refuses to reply.

    I know that everything will work, if I can just get pfSense/FreeBSD to reply to that NS, as I have experimented with Linux, which replies to the NS after which everything runs smoothly:

    10	5.677136	2a00:7660::248	ff02::1:ffc9:11ee	ICMPv6	86	Neighbor Solicitation for fe80::541e:337d:38c9:11ee from e6:5d:37:84:53:17
    11	5.677141	fe80::541e:337d:38c9:11ee	2a00:7660::248	ICMPv6	86	Neighbor Advertisement fe80::541e:337d:38c9:11ee (sol, ovr) is at ac:16:2d:94:bb:d3
    12	5.677527	2a00:7660::249	ff02::1:ffc9:11ee	ICMPv6	86	Neighbor Solicitation for fe80::541e:337d:38c9:11ee from fe:33:42:10:c8:6b
    13	5.677724	fe80::541e:337d:38c9:11ee	2a00:7660::249	ICMPv6	86	Neighbor Advertisement fe80::541e:337d:38c9:11ee (sol, ovr) is at ac:16:2d:94:bb:d3
    14	5.678332	fe80::e65d:37ff:fe84:5317	fe80::541e:337d:38c9:11ee	ICMPv6	78	Router Advertisement from e6:5d:37:84:53:17
    15	5.678687	fe80::fe33:42ff:fe10:c86b	fe80::541e:337d:38c9:11ee	ICMPv6	78	Router Advertisement from fe:33:42:10:c8:6b
    

  • @cnrd said in pfSense does not reply to NS sent by ISP router, ISP does not respond to DHCPv6 request as a result:

    This pattern repeats, as pfSense is waiting for a reply to the solicit XID and ISP routers are waiting for a reply to the neighbor solicits before sending a reply to the XID.
    pfsense startup.pcapng
    Interestingly I also tried setting up a pure FreeBSD 12.2 machine, just to see if that would reply to the NS, but no dice, so it seems that the problem lies somewhere below pfSense.

    It looks to me like the problem is with the ISP, given the lack of responses. I assume your modem is in bridge mode, not gateway.


  • @JKnott I'm using a fiber modem, so there is no other routers in between.

    I don't think it's a problem on my ISP's side, as everything works as soon as the client on my side replies to the NS send by the ISP router.

    Booting into another OS that will reply to the ISP router NS and then back into pfSense results in pfSense getting both an IPv6 and a PD. The problem is that this only works for a couple of hours until the ISP router's cache run out and pfSense does not reply to the new NS send out at that time.

    I'm starting to suspect that this may actually be a bug in the ND implementation of FreeBSD.


  • @cnrd

    If whatever device is in gateway mode, pfsense will not work properly. Do you get NAT addresses on IPv4? If so, the modem is likely in gateway mode. I don't know what you have, but around here, the phone company provides fibre connections through the exact same device as they use for ADSL. It has connectors for both a phone line and Ethernet. If the customer is on fibre, then the Ethernet port connects to the fibre interface. I have a cable modem and the first thing I did when I got it, was to put it in bridge mode.


  • @JKnott there is no router function in the modem I have it is literally a fiber modem. I have a public IPv4 and in other OS'es than pfSense I get an IPv6-PD/NA.


  • @cnrd

    Well, I don't know what else to tell you. It works fine for me and many others. But when I look at your capture and see all those unanswered solicitations I don't think the problem is with pfsense. You have to find out why the ISP is not responding to them. I see 14 lines of them and not a single response. While I don't see a router solicitation, I do see the solicit XID to a multicast address. Why no response to that?

    What is your WAN config? Mine says DHCP6.

    Also, in your first post you said "When pfSense sends out a DHCPv6 request, my ISP sends out an NS, which pfSense never replies to <-- This seems to be what causes the problem.", but I don't see that in the packet capture.


  • @JKnott yeah I don't really know what else there is to try either.

    WAN is set to DHCPv6.

    The NS from the ISP comes right after the XID in the latest capture.

    All I can say is that in other OS'es it's working because they reply to that NS.

    I don't really know what to ask my ISP about, as I haven't found any documentation/RFC showing that what they are doing is out of spec.

    Anyways thanks for trying :-) as the same thing is happening upstream in FreeBSD, I'll probably try over there.


  • @cnrd

    You might also mention who your ISP is. Someone else might have experience with them.


  • @JKnott My ISP is Gigabit.dk

    As I wanted to test my theory that it would not reply to global addresses, I hand-crafted two different packages using scapy.

    working.pcap
    not-working.pcap

    The only difference between these two packages is the fact that one uses a global IP as the src, while the other uses a local-link address.

    I'm going to open a bug-report with those two minimal examples.

  • LAYER 8 Netgate

    Not sure this is a bug.

    How is an IPv6 host supposed to source a packet to a GUA unicast address from a link-local address? Link-local is link-local, not GUA. The host has no idea that the GUA address is on the next hop. If it is not, the router receiving the packet should not forward the packet if it is sourced from the link-local address. The host cannot source the packet from a GUA address because DHCP6 has not occurred yet (It doesn't have one).

    I tried to find something hard in the RFCs that states this but came up empty.

    macos to pfSense:
    
    Ping in the link-local context works:
    $ ping6 -S fe80::183d:38c9:7896:973b%vlan0 fe80::1:1%vlan0
    PING6(56=40+8+8 bytes) fe80::183d:38c9:7896:973b%vlan0 --> fe80::1:1%vlan0
    16 bytes from fe80::1:1%vlan0, icmp_seq=0 hlim=64 time=0.168 ms
    16 bytes from fe80::1:1%vlan0, icmp_seq=1 hlim=64 time=0.151 ms
    16 bytes from fe80::1:1%vlan0, icmp_seq=2 hlim=64 time=0.142 ms
    16 bytes from fe80::1:1%vlan0, icmp_seq=3 hlim=64 time=0.225 ms
    ^C
    --- fe80::1:1%vlan0 ping6 statistics ---
    4 packets transmitted, 4 packets received, 0.0% packet loss
    round-trip min/avg/max/std-dev = 0.142/0.171/0.225/0.032 ms
    
    Ping link-local to GUA fails:
    $ ping6 -S fe80::183d:38c9:7896:973b%vlan0 2001:470:beef:1::1
    PING6(56=40+8+8 bytes) fe80::183d:38c9:7896:973b%vlan0 --> 2001:470:beef:1::1
    ^C
    --- 2001:470:beef:1::1 ping6 statistics ---
    5 packets transmitted, 0 packets received, 100.0% packet loss
    
    Ping GUA to GUA works:
    $ ping6 -S 2001:470:beef:1:8444:5b18:abab:96f0 2001:470:beef:1::1
    PING6(56=40+8+8 bytes) 2001:470:beef:1:8444:5b18:abab:96f0 --> 2001:470:beef:1::1
    16 bytes from 2001:470:beef:1::1, icmp_seq=0 hlim=64 time=0.201 ms
    16 bytes from 2001:470:beef:1::1, icmp_seq=1 hlim=64 time=0.203 ms
    16 bytes from 2001:470:beef:1::1, icmp_seq=2 hlim=64 time=0.211 ms
    ^C
    --- 2001:470:beef:1::1 ping6 statistics ---
    3 packets transmitted, 3 packets received, 0.0% packet loss
    round-trip min/avg/max/std-dev = 0.201/0.205/0.211/0.004 ms
    

    Is this the same ISP that expected its customers to periodically send a Router Solicitation even though the RFCs explicitly state one MUST NOT do that except in certain instances like an interface reconfiguration, link down/up, etc?


  • @Derelict No they are sending out RA as expected, they had a problem where the RA packages was thrown away, but that was fixed by their HW vendor.

    I have been speaking to one of their internal IT guys, they have been very helpful and tried changing the config of their routers, such that it would send the NS using a link local address, that fixed this problem, but unfortunately broke the DHCP hand-out.

    I tried to find something hard in the RFCs that states this but came up empty.

    I know that it sounds wierd, but as Linux supports it and there is nothing in the RFC stating that it's wrong, I can't really see why it shouldn't be okay. I can see your argument, it does make sense, but if it's not disallowed by the RFC, then someone (in this case the HW vendor of my ISP) chooses to do it.

    How is an IPv6 host supposed to source a packet to a GUA unicast address from a link-local address?

    Here is how debian does it:
    2 0.000005 fe80::541e:337d:38c9:11ee 2a00:7660::248 ICMPv6 86 Neighbor Advertisement fe80::541e:337d:38c9:11ee (sol, ovr) is at ac:16:2d:94:bb:d3

    It is the responsibility of the receiver (router) to not forward that outside of the link local.


  • @cnrd said in pfSense does not reply to NS sent by ISP router, ISP does not respond to DHCPv6 request as a result:

    I have been speaking to one of their internal IT guys, they have been very helpful and tried changing the config of their routers, such that it would send the NS using a link local address, that fixed this problem, but unfortunately broke the DHCP hand-out.

    One thing I've noticed is the ISPs tech are not fully up to speed on IPv6. A couple of years ago, I had a problem with my ISP where I could not reach the Internet with IPv6. After testing on my own, I had determined the problem was not on my network. I called 2nd level support (I don't waste my time with 1st) and had to talk him through how DHCPv6-PD works and how the WAN address is not used for routing. He was then able to verify the problem was elsewhere. But when he tried to get the network guys to work on it, they refused because I had my own router, even though a neighbour had the same problem and he was only using the ISP's router. I had even determined the failing system, by host name, at the head end, by examining the DHCPv6-PD sequence with Wireshark. I then had a senior tech come to my home and again explained how things worked. He tried, with his own computer and modem, and it failed for him too. He then took his computer to the office and tried with 4 different systems and only the one I was connected to failed. The network guys finally accepted the problem and resolved it.

    I was able to work my way through this because I have decades of experience with telecom, computers and networks. An average customer wouldn't have a hope.

    Bottom line, you can't always count on the ISP's techs to fully understand what they're doing.


  • @JKnott can't really reveal who I talked to, but trust me, not an average tech.


  • This post is deleted!

  • @cnrd I have been following this thread with great interest.

    I use the same ISP, and have exactly the same issue, in that the NS coming in is being ignored, and the DHCP6 requests in turn keep being ignored by Gigabit.

    Funny thing is that this worked me for several years, until it stopped, around the time there was a large outage in connection with some infra upgrades being done at the time. It's possibly related with those infrastructure changes.

    Did you have any luck with support? I have a support case registered a couple of weeks ago for the same thing, and I think I will point them to this thread, unless you found a workaround on the BSD/pfSense side.


  • @abw Unfortunately not. Yeah it seems like the upgrade of hardware caused the problem.


  • @cnrd Just had a breakthrough, and now have the full /48 running with DHCPv6 on pfSense.
    My problem was due to having a user defined MAC address on the WAN interface.

    I've always had a user defined MAC address configured on the WAN interface (Interfaces > WAN > MAC Address), so as to ensure my static IPv4 address over several hardware changes.

    After reading everything you had tried, I wanted to test the theory that this user-defined address might have an impact on pfSense/FreeBSDs ability to respond to the NS.

    It seems that this is the root cause of my issue, because when removed it, and let it default to the hardware MAC, DHCPv6 and the PD came up immediately.
    Of course I no longer had an IPv4 address, but I just called up the ISP and they changed the static DHCP4 allocation to my new MAC for me on the spot.

    Now, with the hardware MAC in use, the NS works perfectly.

    So my current root cause theory is that, whatever mechanism pfSense uses to "spoof" the MAC address, does not appear to propagate to the part of FreeBSD that is responsible for NS responses. Only a theory though.

    I really hope this helps you.

    By the way, I only have the following checked in the WAN configuration now:

    Prefix Delegation Size: /48
    Send IPv6 Prefix Hint: SET

    Reboot pfSense, and that's it.


  • @abw huh wierd, any chance you can try to capture the NS/NA handshake? Maybe they have a different router setup where you are connected.

    If you cannot capture the handshake, could you try to figure out what IP the router is presenting in packages?

    Thanks!


  • @abw said in pfSense does not reply to NS sent by ISP router, ISP does not respond to DHCPv6 request as a result:

    So my current root cause theory is that, whatever mechanism pfSense uses to "spoof" the MAC address, does not appear to propagate to the part of FreeBSD that is responsible for NS responses. Only a theory though.

    There's a bit in the MAC to designate a universal or locally assigned address. Perhaps that's what's causing the problem. You should be able to check that with a packet capture.


  • @JKnott Thanks for the idea!

    My old statically coded MAC address OUI was bc:05:43 which came from a FritzBox I had for many years.
    The new OUI, which works, is 00:1f:29, which is registered to HP.

    Both have the second LSB of the first octet set to zero (UAA), so I guess that's not it.

    In my working configuration I had also set the tunable net.inet6.icmp6.nd6_onlink_ns_rfc4861 to 1, but I have not yet retested with it set to zero to determine if it's actually necessary.

    I still need to experiment a little more to isolate the fix.