IPv6 WAN Track Interface not assigning addresses to LAN/Public LAN



  • I've read through many of the threads here, and while they've been helpful unfortunately none of them have matched my situation closely enough.

    We have a Comcast Business connection and a local internet provider set up on a Multi-WAN. Comcast provides IPv6, the local does not. In IPv4 world both services are in Tier 1 in a gateway group with 66% of traffic going to Comcast, 33% going to local provider. In IPv6 world, Comcast is the default gateway and that's it. Comcast modem is a CG3000DCR providing a static IPv4 /30 (verified) and a dynamic IPv6 /60 (not verified yet). LAN and PUBLIC (LAN) VLANs are on a LAGG using igb0 and igb2. Comcast is igb1, local provider is igb3.

    The DHCPv6 process feels flaky on the WAN interface. I say this, because sometimes even changing a non-critical setting like Client Debug Mode will result in a refresh with the IPv6 gateway still having an address but listing at 100% packet loss. Sometimes the IPv6 address drops completely. Sometimes two or three superficial settings changes will fix it, sometimes a pfsense reboot will fix it. Regardless, the LAN and PUBLIC IPv6 are set to TRACK INTERFACE WAN with LAN having a prefix ID of 1, and PUBLIC having a prefix ID of f.

    The WAN interface has an IP of 2603:$:$:0:208:y:y:df69 and a link local of fe80::208:y:y:df69 (the Y's match). The gateway shows an fe80::$:$:$:ff82 link-local (which I think makes sense in IPv6 world, but I'm not sure - it matches none of the addresses listed in interfaces). The gateway currently lists as up - the monitor is pinging ipv6.google.com at 2s intervals.

    The LAN and PUBLIC interfaces don't have IPv6 addresses assigned (only fe80::1:1), which I assume means TRACK INTERFACE isn't assigning them. LAN and PUBLIC are configured with RA "assisted" and DHCPv6. I didn't notice anything was wrong until I noticed my PCs weren't getting assigned addresses, and after digging through the logs I found this repeating every 20 seconds:

    Oct 23 08:24:12 	radvd 	65798 	our AdvValidLifetime on igb1 for 2603:$:$:: doesn't agree with fe80::$:$:$:ff82
    Oct 23 08:24:12 	radvd 	65798 	our AdvPreferredLifetime on igb1 for 2603:$:$:: doesn't agree with fe80::$:$:$:ff82
    

    (The $s are to anonymize, and also to duck the character code on this BB).

    The firewall can ping IPv6, but obviously nothing past it can. I've tried all sorts of things, including changing my prefix delegation, but nothing else has resulted in a working gateway (though maybe that's partially due to the DHCPv6 flakiness I reported earlier). For anyone curious, the WAN DHCPv6 is set to "Send IPv6 prefix hint", "Debug", "Do not allow PD/Address release", and "IA Non-Temporary Address Allocation".

    Thoughts on where I go from here? Any help is much appreciated. Thanks.



  • Try checking „Request a prefix via IPv4“. And request a /60.



  • So, it's exactly like I said... after changing the config on WAN, now my IPv6 gateway lists as "pending" without an IP. It's almost like a script isn't running or completing properly on reconfiguration of the interface or on receiving a new DHCPv6 address.

    Debug DHCPv6 logs below, from immediately after the reconfig to request via v4. I notice from these logs that the "Non-Temporary Address Allocation" is being ignored...

    Oct 22 23:12:07 firewall dhcp6c[18188]: found an interface igb0 for DUID
    Oct 22 23:12:07 firewall dhcp6c[18188]: generated a new DUID: 00:01:00:01:23:61:4e:07:00:08:a2:0a:df:68
    Oct 22 23:12:07 firewall dhcp6c[18188]: saved generated DUID to /var/db/dhcp6c_duid
    Oct 22 23:12:07 firewall dhcp6c[18188]: failed to open /usr/local/etc/dhcp6cctlkey: No such file or directory
    Oct 22 23:12:07 firewall dhcp6c[18188]: failed initialize control message authentication
    Oct 22 23:12:07 firewall dhcp6c[18188]: skip opening control port
    Oct 22 23:12:07 firewall dhcp6c[18188]: <3>[interface] (9)
    Oct 22 23:12:07 firewall dhcp6c[18188]: <5>[igb1] (4)
    Oct 22 23:12:07 firewall dhcp6c[18188]: <3>begin of closure [{] (1)
    Oct 22 23:12:07 firewall dhcp6c[18188]: <3>[script] (6)
    Oct 22 23:12:07 firewall dhcp6c[18188]: <3>["/var/etc/dhcp6c_wan_script.sh"] (31)
    Oct 22 23:12:07 firewall dhcp6c[18188]: <3>end of sentence [;] (1)
    Oct 22 23:12:07 firewall dhcp6c[18188]: <3>end of closure [}] (1)
    Oct 22 23:12:07 firewall dhcp6c[18188]: <3>end of sentence [;] (1)
    Oct 22 23:12:07 firewall dhcp6c[18188]: <3>[id-assoc] (8)
    Oct 22 23:12:07 firewall dhcp6c[18188]: <13>[na] (2)
    Oct 22 23:12:07 firewall dhcp6c[18188]: <13>begin of closure [{] (1)
    Oct 22 23:12:07 firewall dhcp6c[18188]: <3>end of closure [}] (1)
    Oct 22 23:12:07 firewall dhcp6c[18188]: <3>end of sentence [;] (1)
    Oct 22 23:12:07 firewall dhcp6c[18188]: called
    Oct 22 23:12:07 firewall dhcp6c[18188]: some IA configuration defined but not used
    Oct 22 23:12:07 firewall dhcp6c[18188]: called
    Oct 22 23:12:07 firewall dhcp6c[18303]: reset a timer on igb1, state=INIT, timeo=0, retrans=891
    Oct 22 23:12:08 firewall dhcp6c[18303]: Sending Solicit
    Oct 22 23:12:08 firewall dhcp6c[18303]: a new XID (61685e) is generated
    Oct 22 23:12:08 firewall dhcp6c[18303]: set client ID (len 14)
    Oct 22 23:12:08 firewall dhcp6c[18303]: set elapsed time (len 2)
    Oct 22 23:12:08 firewall dhcp6c[18303]: send solicit to ff02::1:2%igb1
    Oct 22 23:12:08 firewall dhcp6c[18303]: reset a timer on igb1, state=SOLICIT, timeo=0, retrans=1091
    Oct 22 23:12:08 firewall dhcp6c[18303]: receive advertise from fe80::$:$:$:ff82%igb1 on igb1
    Oct 22 23:12:08 firewall dhcp6c[18303]: get DHCP option client ID, len 14
    Oct 22 23:12:08 firewall dhcp6c[18303]:   DUID: 00:01:00:01:23:61:4e:07:00:08:a2:0a:df:68
    Oct 22 23:12:08 firewall dhcp6c[18303]: get DHCP option server ID, len 10
    Oct 22 23:12:08 firewall dhcp6c[18303]:   DUID: 00:03:00:01:04:a1:51:07:ff:82
    Oct 22 23:12:08 firewall dhcp6c[18303]: server ID: 00:03:00:01:04:a1:51:07:ff:82, pref=-1
    Oct 22 23:12:08 firewall dhcp6c[18303]: reset timer for igb1 to 0.998526
    Oct 22 23:12:09 firewall dhcp6c[18303]: picked a server (ID: 00:03:00:01:04:a1:51:07:ff:82)
    Oct 22 23:12:09 firewall dhcp6c[18303]: Sending Request
    Oct 22 23:12:09 firewall dhcp6c[18303]: a new XID (623be5) is generated
    Oct 22 23:12:09 firewall dhcp6c[18303]: set client ID (len 14)
    Oct 22 23:12:09 firewall dhcp6c[18303]: set server ID (len 10)
    Oct 22 23:12:09 firewall dhcp6c[18303]: set elapsed time (len 2)
    Oct 22 23:12:09 firewall dhcp6c[18303]: send request to ff02::1:2%igb1
    Oct 22 23:12:09 firewall dhcp6c[18303]: reset a timer on igb1, state=REQUEST, timeo=0, retrans=909
    Oct 22 23:12:09 firewall dhcp6c[18303]: receive reply from fe80::$:$:$:ff82%igb1 on igb1
    Oct 22 23:12:09 firewall dhcp6c[18303]: get DHCP option client ID, len 14
    Oct 22 23:12:09 firewall dhcp6c[18303]:   DUID: 00:01:00:01:23:61:4e:07:00:08:a2:0a:df:68
    Oct 22 23:12:09 firewall dhcp6c[18303]: get DHCP option server ID, len 10
    Oct 22 23:12:09 firewall dhcp6c[18303]:   DUID: 00:03:00:01:04:a1:51:07:ff:82
    Oct 22 23:12:09 firewall dhcp6c[18303]: dhcp6c Received REQUEST
    Oct 22 23:12:09 firewall dhcp6c[18303]: executes /var/etc/dhcp6c_wan_script.sh
    Oct 22 23:12:12 firewall dhcp6c: dhcp6c REQUEST on igb1 - running rc.newwanipv6
    Oct 22 23:12:12 firewall dhcp6c[18303]: script "/var/etc/dhcp6c_wan_script.sh" terminated
    Oct 22 23:12:12 firewall dhcp6c[18303]: removing an event on igb1, state=REQUEST
    Oct 22 23:12:12 firewall dhcp6c[18303]: removing server (ID: 00:03:00:01:04:a1:51:07:ff:82)
    Oct 22 23:12:12 firewall dhcp6c[18303]: got an expected reply, sleeping.
    


  • @jsnl said in IPv6 WAN Track Interface not assigning addresses to LAN/Public LAN:

    the LAN and PUBLIC IPv6 are set to TRACK INTERFACE WAN with LAN having a prefix ID of 1, and PUBLIC having a prefix ID of f.

    So I took this part of the issue and is relating this to what I am experiencing right now. I would expect the WAN to issue the delegated prefix to the network desired.

    The WAN should have an IP from the modem with a /128. I requested a /63 (I'm using COX at my home) just to acquire two /64. You need 16 /64's, but the end result should be the same. The RA should tell the segmented network where to route the IPv6 traffic, how many addresses are available, prefix hint, etc.

    For some reason, this is where the buck stops. I have allocated IP addresses on PFSense, but my computer has nowhere to go.

    So now I am observing the radvd file...since it's supposed to advertise the correct information and route to the clients.

    The other issue to observe is the firewall. I noticed that the required localhost fe80:: is blocked and that's bad news since Track Interface is directing my clients to the LAN interface.

    My question: Why in the world is this firewall blocking the IPv6 localhost?



  • So while troubleshooting, I got IPv6 to work for 15 seconds, then it stopped working. I even did a test and the WAN IP was showing up. I turned immediate attention to the NAT.

    When I changed the option from Automatic to Hybrid +Automatic, the Router Advertisements stopped working....that shouldn't happen because the rules are still there. I wonder if there is a problem with the default Source NAT rules.



  • So I changed it to Manual NAT, created two Source NAT rules with Any, Any, for the LAN, except I changed the Translation settings; instead of using the Interface Address, I used the Subnet. Change the Pool Options to Bit Mask from the Default. The delegated prefix was assigned to my network again.

    I then deleted all of those automatic rules, because they are not allowing the localhost addresses to translate.

    I finally created the same masquerading rule for the WAN and used the Interface Address for the Translation. IPv6 worked with a test.

    0_1540646338944_9264af4d-41a9-425f-8773-c3d450e52cdd-image.png

    I highly recommend that you write down your networks, erase those default Source NAT rules and build your own into the Manual option. If you have services that you run, then you need to use 1:1 and enable the PureNAT reflection options in the System.



  • The NAT settings in PFSense should look like this to compliment the troubleshooting above.

    0_1540648660308_9646fa50-f4f9-4e1a-a136-21fee6657c56-image.png

    0_1540648593379_c27ec048-ffa8-4774-8803-fc2f35902375-image.png



  • The issue is by default, WAN interfaces have a Source with the LAN Subnet and missing localhosts that translate to the WAN Address, which confused me and the rules are wrong in general. Automatic NAT works as Source NAT per the websites information.

    The WAN interface need the WAN's network as the Source so that the next network knows that it uses the WAN Address and ports for translation, or Source NAT.

    You can't design this with the WAN as the interface and the LAN's Subnet as the Source and expect the next network to translate the WAN Address with LAN information. The next network don't know where to look.

    It will go nowhere and get blocked. This can even cause your Local Interfaces to block each other and not learn their new routes.

    Don't use the automatic NAT or Hybrid+Automatic NAT unless this area is logged as a bug. I hope this all makes sense.



  • I had to read this a bunch of times, but I think I understand what you're saying. Where I'm confused:

    • Is it your assumption that the prefix delegation is or isn't getting assigned, and that that delegation is or isn't translating over to the other two LANs with "track interface" specified?

    • What logs can I monitor to see if the prefix is successfully being assigned by the service provider? Because dhcp6c isn't showing that level of detail.

    • Why do the NAT reciprocating rules affect the IPv6 delegation? Technically IPv6 doesn't do NAT. Are you saying it's because the prefix negotiation is happening over v4?

    • How is an improper local firewall route making my ipv6 gateway fail and remain listed as "pending"? The gateway should auto-populate, as it isn't its own interface - it merely tracks the outbound routes that RA says are available after the dhcp6c handshake (Doesn't it? Or is it "built" upon WAN addressing with it's own link local address as a next-hop for LAN routes traversing the interfaces?).

    And I'm going to have to do some mental math to figure out how to adapt this for my multi-wan box, because I don't have the luxury of using "v4+v6" rules thanks to my internet setup. They have to be separate gateways.



  • That's what I was looking for earlier. I'll post a log here. The prefix delegation is being assigned. That much of it works as intended. I use a /63 which gave me two /64's to use. I guess Cox will only allow this much without asking for more...and that is enough for me anyways.

    IPv6 doesn't need to use NAT so yes you are correct. The localhost addresses need to know where to go within the machine (127.0.0.0/8, fe80::/64), especially the fe80::/64 localhost. Here is a pic of my WAN setup with a /63.

    0_1540676651288_7e9f52d7-acb5-48ce-bb9c-c128fa2a4ce8-image.png

    Where the Gateway is "Pending"; that is ok. It depends on how you configure it. When one gateway goes down, the other will come online. As long as it states that it's Online or Pending is good. You can select the option to disable gateway monitoring and then it will change the status to Online.

    @jsnl said in IPv6 WAN Track Interface not assigning addresses to LAN/Public LAN:

    Doesn't it? Or is it "built" upon WAN addressing with it's own link local address as a next-hop for LAN routes traversing the interfaces

    This is where I discovered the problem and you nailed it. When I changed the NAT options to Manual, I was able to edit each of those Automatic rules to see how it was being applied to the gateway firewall. That's where I discovered that the LAN routes were applied to translate to the WAN address, which is not how Source NAT works. I was confused there, so I erased them all and made those masquerading rules.

    You should be able to use the Internal Address for the LAN as intended. That example is my WAN in the pictures above. I chose to use the Subnet just to see if the firewall was blocking more.

    If you have delegated IP's on your client machines, then you are 99.999 percent there. :) This drove me to insanity last night. :) Again, I will post the logs. If there is a particular log you would like to see, just let me know.

    Just work your way through it by tackling one net at a time. I even rebooted the machine just to see if it went back down or had something else irregular go wrong. It's been running all day without any issues so far. The answer to your question at the end of the second bullet is the one log I'm searching for as well.



  • This log is from /var/log/dhcpd.log

    Oct 26 23:16:32 pfSense dhcpd: Sending on BPF/bce0/00:10:18:4d:6a:d4/192.168.3.0/28
    Oct 26 23:16:32 pfSense dhcpd: Listening on BPF/em0/b4:b5:2f:b9:19:68/192.168.2.0/28
    Oct 26 23:16:32 pfSense dhcpd: Sending on BPF/em0/b4:b5:2f:b9:19:68/192.168.2.0/28
    Oct 26 23:16:32 pfSense dhcpd: Sending on Socket/fallback/fallback-net
    Oct 26 23:16:32 pfSense dhcpd: Server starting service.
    Oct 26 23:16:33 pfSense dhcp6c[85204]: Sending Solicit
    Oct 26 23:16:36 pfSense dhcpd: Internet Systems Consortium DHCP Server 4.3.6-P1
    Oct 26 23:16:36 pfSense dhcpd: Copyright 2004-2018 Internet Systems Consortium.
    Oct 26 23:16:36 pfSense dhcpd: All rights reserved.
    Oct 26 23:16:36 pfSense dhcpd: For info, please visit https://www.isc.org/software/dhcp/
    Oct 26 23:16:36 pfSense dhcpd: Config file: /etc/dhcpd.conf
    Oct 26 23:16:36 pfSense dhcpd: Database file: /var/db/dhcpd.leases
    Oct 26 23:16:36 pfSense dhcpd: PID file: /var/run/dhcpd.pid
    Oct 26 23:16:36 pfSense dhcpd: Internet Systems Consortium DHCP Server 4.3.6-P1
    Oct 26 23:16:36 pfSense dhcpd: Copyright 2004-2018 Internet Systems Consortium.
    Oct 26 23:16:36 pfSense dhcpd: All rights reserved.
    Oct 26 23:16:36 pfSense dhcpd: For info, please visit https://www.isc.org/software/dhcp/
    Oct 26 23:16:36 pfSense dhcpd: Wrote 21 leases to leases file.
    Oct 26 23:16:36 pfSense dhcpd: Listening on BPF/bce0/00:10:18:4d:6a:d4/192.168.3.0/28
    Oct 26 23:16:36 pfSense dhcpd: Sending on BPF/bce0/00:10:18:4d:6a:d4/192.168.3.0/28
    Oct 26 23:16:36 pfSense dhcpd: Listening on BPF/em0/b4:b5:2f:b9:19:68/192.168.2.0/28
    Oct 26 23:16:36 pfSense dhcpd: Sending on BPF/em0/b4:b5:2f:b9:19:68/192.168.2.0/28
    Oct 26 23:16:36 pfSense dhcpd: Sending on Socket/fallback/fallback-net
    Oct 26 23:16:36 pfSense dhcpd: Server starting service.
    Oct 26 23:16:37 pfSense dhcp6c[85204]: Sending Solicit
    Oct 26 23:16:45 pfSense dhcp6c[85204]: Sending Solicit
    Oct 26 23:17:02 pfSense dhcp6c[85204]: Sending Solicit
    Oct 26 23:17:10 pfSense dhcpd: reuse_lease: lease age 2156 (secs) under 25% threshold, reply with unaltered, existing lease for 192.168.2.11
    Oct 26 23:17:10 pfSense dhcpd: DHCPREQUEST for 192.168.2.11 from a4:31:35:85:58:0d (P32) via em0
    Oct 26 23:17:10 pfSense dhcpd: DHCPACK on 192.168.2.11 to a4:31:35:85:58:0d (P32) via em0
    Oct 26 23:18:13 pfSense dhclient: PREINIT
    Oct 26 23:18:13 pfSense dhclient[5075]: DHCPREQUEST on re0 to 255.255.255.255 port 67
    Oct 26 23:18:13 pfSense dhclient[5075]: DHCPACK from 192.168.0.1
    Oct 26 23:18:13 pfSense dhclient: REBOOT
    Oct 26 23:18:13 pfSense dhclient: Starting add_new_address()
    Oct 26 23:18:13 pfSense dhclient: ifconfig re0 inet 192.168.0.3 netmask 255.255.255.240 broadcast 192.168.0.15
    Oct 26 23:18:13 pfSense dhclient: New IP Address (re0): 192.168.0.3
    Oct 26 23:18:13 pfSense dhclient: New Subnet Mask (re0): 255.255.255.240
    Oct 26 23:18:13 pfSense dhclient: New Broadcast Address (re0): 192.168.0.15
    Oct 26 23:18:13 pfSense dhclient: New Routers (re0): 192.168.0.1
    Oct 26 23:18:13 pfSense dhclient: Adding new routes to interface: re0
    Oct 26 23:18:13 pfSense dhclient: Creating resolv.conf
    Oct 26 23:18:13 pfSense dhclient[5075]: bound to 192.168.0.3 -- renewal in 1800 seconds.
    Oct 26 23:18:15 pfSense dhcp6c[16460]: failed to open /usr/local/etc/dhcp6cctlkey: No such file or directory
    Oct 26 23:18:15 pfSense dhcp6c[16460]: failed initialize control message authentication
    Oct 26 23:18:15 pfSense dhcp6c[16460]: skip opening control port
    Oct 26 23:18:16 pfSense dhcp6c[17093]: Sending Solicit
    Oct 26 23:18:17 pfSense dhcp6c[17093]: Sending Request
    Oct 26 23:18:18 pfSense dhcp6c[17093]: dhcp6c Received REQUEST
    Oct 26 23:18:18 pfSense dhcp6c[17093]: add an address 2600:8803:7800:e93e:b6b5:2fff:feb9:1968/63 on em0
    Oct 26 23:18:18 pfSense dhcp6c[17093]: status code for PD-0: success
    Oct 26 23:18:18 pfSense dhcp6c[17093]: add an address 2600:8803:7800:e930:a149:8b16:189d:3b30/128 on re0
    Oct 26 23:18:18 pfSense dhcp6c[17093]: status code for NA-0: success
    Oct 26 23:18:19 pfSense dhclient: PREINIT
    Oct 26 23:18:19 pfSense dhclient[17087]: DHCPREQUEST on bce1 to 255.255.255.255 port 67
    Oct 26 23:18:19 pfSense dhclient[17087]: DHCPACK from 192.168.0.1
    Oct 26 23:18:19 pfSense dhclient: REBOOT
    Oct 26 23:18:19 pfSense dhclient: Starting add_new_address()
    Oct 26 23:18:19 pfSense dhclient: ifconfig bce1 inet 192.168.0.2 netmask 255.255.255.240 broadcast 192.168.0.15
    Oct 26 23:18:19 pfSense dhclient: New IP Address (bce1): 192.168.0.2
    Oct 26 23:18:19 pfSense dhclient: New Subnet Mask (bce1): 255.255.255.240
    Oct 26 23:18:19 pfSense dhclient: New Broadcast Address (bce1): 192.168.0.15
    Oct 26 23:18:19 pfSense dhclient: New Routers (bce1): 192.168.0.1
    Oct 26 23:18:19 pfSense dhclient: Adding new routes to interface: bce1
    Oct 26 23:18:19 pfSense dhclient: Creating resolv.conf
    Oct 26 23:18:19 pfSense dhclient[17087]: bound to 192.168.0.2 -- renewal in 1800 seconds.



  • This is a snapshot of my NDP table:

    0_1540678354283_506a97c9-69d5-46cc-8b67-eff35759c4e8-image.png

    The interface IPV6NET has no network attached.



  • So I don't think my issue is the same. I made all of the routing changes you stated, and I have a WAN IP again, but still no addresses propagating to my LAN or PUBLIC_LAN. I also still don't have a functional IPv6 gateway (IPv6 is not multi-wan, only one service provides IPv6). I made a LAN route of 192.168.201.0/23 (LAN is x.x.201.x, VPN is x.x.202.x), a PUBLIC route of 192.168.244.0/24, and the two WAN routes plus Manual Outbound Routing.

    Screenshots attached.

    1_1540691119744_fw2.jpg

    0_1540691119743_fw1.jpg



  • Yes, I knew your issue was very different with MultiWAN. I wanted to show that assigning with the prefix and the settings I used works just to start some troubleshooting.

    I see....your Link Local is showing pending. If you have a WAN IP, it should be a /128. I would check the Interfaces of the main page. I had to manually add that WAN IP address in the Routing section. Sometimes it auto-populates.

    Are you using 5 networks? I read that you have a /60, but has not been verified yet. That gives you a prefix ID from 0-f, but f is a new network. If COMCAST has not allowed that subnet for use, then that may be why the modems DHCPv6 is not issuing the /128 address to your WAN, however....

    I see that you need at least 5 networks, so a /60 means 16 /64 networks. Using a /60 means you are on a new network. Your local WAN may not be able to see that address or entire network, but you can see the Link-Local only (fe80::), of the localhost, but that's a good thing.

    I would try /62, which would allow you a total of 7 nets or Net ID's (Prefix ID's) to use (0-7) just to troubleshoot. A /61 would give you 15 Prefix ID's.

    You should see both an IP on your WAN and a different IP for your LAN's/Additional Interfaces. Then you can add the LAN/VPN's Interface IPv6 Subnet to the RA. It will be different from the WAN.



  • So I requested a /62, and adjusted the prefixes for the two LANs to "1" and "3". Got the same /128 on the WAN, none of the other interfaces have IPv6 IPs, and the gateway now lists as "down" instead of "pending".

    I did see this in my firewall log though. I have a feeling this is part of my problem. DF69 is the newly generated gateway address.

    Oct 27 23:28:56 	WAN 	[fe80::$:$:$:ff82]:547		[2603:$:$:$:$:$:$:df69]:13626		UDP
    Default deny rule IPv6 (1000000105)


  • So you set it to /60 and the gateway was Online (required config)
    You set it to /62 and the gateway is showing Offline (troubleshooting)

    In this case, /60 is working just fine then and your WAN received the /128 address. The Gateway is Online. The Link-Local is present. Your LAN's are not receiving anything.

    Areas to check:

    • I'll assume that your NAT is set to Automatic? Your setup may be different here.
    • I'll assume that you have a default IPv6 rule on all local interfaces allowing IPv6.
    • I'll assume that you cannot determine the RA yet for the Local Interfaces until you receive the delegated IP addresses.
    • What does your LAN RA look like in the DHCPv6 settings?
    • With a /60, you can ping ipv6.google.com, so everything is working.

    @jsnl said in IPv6 WAN Track Interface not assigning addresses to LAN/Public LAN:

    2603:$:$:0:208:y:y:df69 and a link local of fe80::208:y:y:df69 (the Y's match)

    :208 Should match on all Interfaces and not necessarily the Y's. Let's say the Gateway has :2080. The LAN will have :2081, The VLAN will have :2084 and so on (different Prefix ID's)

    Correction. I was up late.....yes you are right. Both the WAN and Link-Local will match, which means that PFsense is working.

    This is where I had the headache and again turned my attention to NAT after observing the firewall traffic.

    The next step is to get the local interfaces their IP addresses and then verify that they are all /64 for advertisement to the clients. Inspect the firewall for sure.



  • There is no consistency between /60, /61, /62 as to when the gateway is up, down, or pending. It's pretty random, and often a reboot with no configuration change will fix it.

    Just so you have a status update... I'm still in manual nat, and I've added local routes and NAT rules and can now see the fe80:: packet pass my firewall rule. I had to add an fe80::/10 pass rule on my WAN, which feels odd... but I made it for only the two UDP ports to narrow it up. However I still don't have a functioning address on any other interface. So I'm guessing that 1) The script that runs to add/update the gateway isn't functioning properly or completing fully, and 2) the communication that happens between the prefix getting assigned and that delegation getting announced to the RA isn't happening - because the RADVD logs complain about "invalid ::0 routes" and other odd things.

    I won't be able to deep dive into this for most of the week... but if you have a good troubleshooting method for me to figure out where along the chain the communication is failing (or which part of the WAN update shell scripts is failing) I'm all ears.

    I'll get back to you as soon as I get time between projects. Thanks for your help.



  • No worries and glad I could help. That definitely sounds like a script issue there. I'll have to inquire about more or find a similar issue. I'm using version 2.4.4 I know that this issue exists with 2.4.3 where the track interface did not assign or advertise correctly.

    https://redmine.pfsense.org/issues/8429

    This was one bug that's listed; IPv6 didn't work properly when using a LAN bridge. I had major issues with that version and upgraded.



  • Ok, so I have some more time to poke around. I'm still noticing some of the same things... gateway goes up and down on WAN and LAN configuration randomly, is always fixed by a reboot, and isn't directly affected by any particular setting. I tried setting a MTU of 1500 (as referenced in another thread re: the arpresolve errors), that didn't help. I notice now in the logs that the WAN RA is assigning the gateway correctly (which is the cable modem link local address)... but I'm seeing a lot of errors and still can't get any of the LANs to grab addresses.

    For the purposes of these logs... "lagg0.201" is LAN, "lagg0.244" is PUBLIC_LAN.

    Nov  1 17:52:06 firewall kernel: cannot forward src fe80:$::$:$:$:b668, dst 2001:4860:4806:4::, nxt 17, rcvif lagg0.201, outif igb1
    Nov  1 17:55:02 firewall php-fpm: /rc.newwanipv6: rc.newwanipv6: Info: starting on igb1.
    Nov  1 17:55:02 firewall php-fpm: /rc.newwanipv6: rc.newwanipv6: on (IP address: 2603:$:$:$:$:$:$:df69) (interface: wan) (real interface: igb1).
    Nov  1 17:55:04 firewall dhcpleases: /etc/hosts changed size from original!
    Nov  1 17:55:04 firewall dhcpleases: Could not deliver signal HUP to process because its pidfile (/var/run/unbound.pid) does not exist, No such process.
    Nov  1 17:55:05 firewall check_reload_status: Syncing firewall
    Nov  1 17:55:09 firewall kernel: arpresolve: can't allocate llinfo for 50.$.$.58 on igb1
    Nov  1 17:55:10 firewall kernel: arpresolve: can't allocate llinfo for 50.$.$.58 on igb1
    Nov  1 17:55:10 firewall kernel: arpresolve: can't allocate llinfo for 50.$.$.58 on igb1
    Nov  1 17:55:10 firewall kernel: arpresolve: can't allocate llinfo for 50.$.$.58 on igb1
    Nov  1 17:55:10 firewall kernel: arpresolve: can't allocate llinfo for 50.$.$.58 on igb1
    Nov  1 17:55:10 firewall php-fpm[20847]: /interfaces.php: calling interface_dhcpv6_configure.
    Nov  1 17:55:10 firewall php-fpm[20847]: /interfaces.php: Accept router advertisements on interface igb1
    Nov  1 17:55:10 firewall php-fpm[20847]: /interfaces.php: Starting rtsold process
    Nov  1 17:55:12 firewall php-fpm[20847]: /interfaces.php: Gateway, switch to: ComcastGW
    Nov  1 17:55:12 firewall php-fpm[20847]: /interfaces.php: Default gateway setting Comcast Upstream Gateway as default.
    Nov  1 17:55:12 firewall check_reload_status: Restarting ipsec tunnels
    Nov  1 17:55:12 firewall rtsold: Received RA specifying route fe80::$:$:$:df69 for interface wan(igb1)
    Nov  1 17:55:12 firewall rtsold: Starting dhcp6 client for interface wan(igb1)
    Nov  1 17:55:12 firewall php-fpm: /rc.newwanipv6: Keep current gateway, its already part of the group members.
    Nov  1 17:55:13 firewall php-fpm: /rc.newwanipv6: Keep current gateway, its already part of the group members.
    Nov  1 17:55:13 firewall dhcpleases: kqueue error: unkown
    Nov  1 17:55:13 firewall dhcpleases: /etc/hosts changed size from original!
    Nov  1 17:55:15 firewall php-fpm[20847]: /interfaces.php: Keep current gateway, its already part of the group members.
    Nov  1 17:55:15 firewall php-fpm[20847]: /interfaces.php: Keep current gateway, its already part of the group members.
    Nov  1 17:55:15 firewall dhcpleases: kqueue error: unkown
    Nov  1 17:55:16 firewall dhcpleases: /etc/hosts changed size from original!
    Nov  1 17:55:16 firewall php-fpm: /rc.newwanipv6: Keep current gateway, its already part of the group members.
    Nov  1 17:55:16 firewall php-fpm: /rc.newwanipv6: Keep current gateway, its already part of the group members.
    Nov  1 17:55:16 firewall php-fpm: /rc.newwanipv6: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1541109316] unbound[93148:0] error: bind: address already in use [1541109316] unbound[93148:0] fatal error: could not open ports'
    Nov  1 17:55:16 firewall dhcpleases: kqueue error: unkown
    Nov  1 17:55:17 firewall check_reload_status: updating dyndns wan
    Nov  1 17:55:17 firewall kernel: lagg0.201: promiscuous mode disabled
    Nov  1 17:55:17 firewall kernel: vlan0: changing name to 'lagg0.201'
    Nov  1 17:55:19 firewall php-fpm: /rc.newwanipv6: Removing static route for monitor 75.75.75.75 and adding a new route through 50.$.$.58
    Nov  1 17:55:19 firewall php-fpm: /rc.newwanipv6: Removing static route for monitor 2607:f8b0:4000:812::200e and adding a new route through fe80::$:$:$:df69%igb1
    Nov  1 17:55:19 firewall php-fpm: /rc.newwanipv6: Removing static route for monitor 64.233.217.2 and adding a new route through 24.$.$.1
    Nov  1 17:55:19 firewall php-fpm: /rc.dyndns.update: Keep current gateway, its already part of the group members.
    Nov  1 17:55:20 firewall dhcpleases: /etc/hosts changed size from original!
    Nov  1 17:55:20 firewall dhcpleases: Could not deliver signal HUP to process because its pidfile (/var/run/unbound.pid) does not exist, No such process.
    Nov  1 17:55:23 firewall rc.gateway_alarm[36097]: >>> Gateway alarm: WAN_DHCP6 (Addr:2607:f8b0:4000:812::200e Alarm:1 RTT:0.000ms RTTsd:0.000ms Loss:100%)
    Nov  1 17:55:23 firewall php-fpm: /rc.newwanipv6: Keep current gateway, its already part of the group members.
    Nov  1 17:55:23 firewall check_reload_status: updating dyndns WAN_DHCP6
    Nov  1 17:55:23 firewall check_reload_status: Restarting ipsec tunnels
    Nov  1 17:55:23 firewall check_reload_status: Restarting OpenVPN tunnels/interfaces
    Nov  1 17:55:23 firewall check_reload_status: Reloading filter
    Nov  1 17:55:23 firewall check_reload_status: Reloading filter
    Nov  1 17:55:24 firewall php-fpm: /rc.openvpn: Keep current gateway, its already part of the group members.
    Nov  1 17:55:24 firewall php-fpm: /rc.dyndns.update: Keep current gateway, its already part of the group members.
    Nov  1 17:55:24 firewall php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP6.
    Nov  1 17:55:24 firewall php-fpm: /rc.filter_configure_sync: Keep current gateway, its already part of the group members.
    Nov  1 17:55:24 firewall php-fpm[20847]: /interfaces.php: Keep current gateway, its already part of the group members.
    Nov  1 17:55:24 firewall php-fpm[20847]: /interfaces.php: Keep current gateway, its already part of the group members.
    Nov  1 17:55:24 firewall dhcpleases: kqueue error: unkown
    Nov  1 17:55:25 firewall php-fpm: /rc.filter_configure_sync: Keep current gateway, its already part of the group members.
    Nov  1 17:55:25 firewall dhcpleases: /etc/hosts changed size from original!
    Nov  1 17:55:25 firewall php-fpm[20847]: /interfaces.php: Keep current gateway, its already part of the group members.
    Nov  1 17:55:25 firewall check_reload_status: Restarting ipsec tunnels
    Nov  1 17:55:27 firewall dhcpleases: /etc/hosts changed size from original!
    Nov  1 17:55:27 firewall dhcpleases: Could not deliver signal HUP to process because its pidfile (/var/run/unbound.pid) does not exist, No such process.
    Nov  1 17:55:28 firewall php-fpm[20847]: /interfaces.php: Keep current gateway, its already part of the group members.
    Nov  1 17:55:29 firewall php-fpm[20847]: /interfaces.php: Keep current gateway, its already part of the group members.
    Nov  1 17:55:29 firewall dhcpleases: kqueue error: unkown
    Nov  1 17:55:31 firewall check_reload_status: updating dyndns lan
    Nov  1 17:55:31 firewall kernel: igb0: promiscuous mode disabled
    Nov  1 17:55:31 firewall kernel: igb2: promiscuous mode disabled
    Nov  1 17:55:31 firewall kernel: lagg0: promiscuous mode disabled
    Nov  1 17:55:31 firewall kernel: lagg0.244: promiscuous mode disabled
    Nov  1 17:55:31 firewall kernel: vlan1: changing name to 'lagg0.244'
    Nov  1 17:55:32 firewall php-fpm: /rc.dyndns.update: Keep current gateway, its already part of the group members.
    Nov  1 17:55:33 firewall dhcpleases: /etc/hosts changed size from original!
    Nov  1 17:55:33 firewall dhcpleases: Could not deliver signal HUP to process because its pidfile (/var/run/unbound.pid) does not exist, No such process.
    Nov  1 17:55:34 firewall php-fpm[20847]: /interfaces.php: Keep current gateway, its already part of the group members.
    Nov  1 17:55:34 firewall php-fpm[20847]: /interfaces.php: Keep current gateway, its already part of the group members.
    Nov  1 17:55:34 firewall dhcpleases: kqueue error: unkown
    Nov  1 17:55:35 firewall php-fpm[20847]: /interfaces.php: Keep current gateway, its already part of the group members.
    Nov  1 17:55:35 firewall check_reload_status: Restarting ipsec tunnels
    Nov  1 17:55:38 firewall dhcpleases: /etc/hosts changed size from original!
    Nov  1 17:55:38 firewall dhcpleases: Could not deliver signal HUP to process because its pidfile (/var/run/unbound.pid) does not exist, No such process.
    Nov  1 17:55:39 firewall php-fpm[20847]: /interfaces.php: Keep current gateway, its already part of the group members.
    Nov  1 17:55:40 firewall php-fpm[20847]: /interfaces.php: Keep current gateway, its already part of the group members.
    Nov  1 17:55:40 firewall dhcpleases: kqueue error: unkown
    Nov  1 17:55:42 firewall check_reload_status: updating dyndns opt1
    Nov  1 17:55:43 firewall php-fpm: /rc.dyndns.update: Keep current gateway, its already part of the group members.
    Nov  1 17:55:44 firewall php-fpm[20847]: /interfaces.php: Removing static route for monitor 75.75.75.75 and adding a new route through 50.$.$.58
    Nov  1 17:55:44 firewall php-fpm[20847]: /interfaces.php: Removing static route for monitor 2607:f8b0:4000:812::200e and adding a new route through fe80::$:$:$:df69%igb1
    Nov  1 17:55:44 firewall php-fpm[20847]: /interfaces.php: Removing static route for monitor 64.233.217.2 and adding a new route through 24.$.$.1
    Nov  1 17:55:44 firewall check_reload_status: Reloading filter
    Nov  1 17:55:44 firewall php-fpm[20847]: /interfaces.php: Creating rrd update script
    Nov  1 17:55:44 firewall firewall.gardencitymi.org nginx: 2018/11/01 17:55:44 [crit] 58340#100233: *5436 SSL_write() failed (SSL:) (13: Permission denied) while processing HTTP/2 connection, client: 192.168.202.2, server: 0.0.0.0:443
    Nov  1 17:55:48 firewall php-fpm: /rc.filter_configure_sync: Keep current gateway, its already part of the group members.
    Nov  1 17:55:48 firewall rc.gateway_alarm[11005]: >>> Gateway alarm: WAN_DHCP6 (Addr:2607:f8b0:4000:812::200e Alarm:1 RTT:0.000ms RTTsd:0.000ms Loss:100%)
    Nov  1 17:55:48 firewall check_reload_status: updating dyndns WAN_DHCP6
    Nov  1 17:55:48 firewall check_reload_status: Restarting ipsec tunnels
    Nov  1 17:55:48 firewall check_reload_status: Restarting OpenVPN tunnels/interfaces
    Nov  1 17:55:48 firewall check_reload_status: Reloading filter
    Nov  1 17:55:49 firewall php-fpm: /rc.openvpn: Keep current gateway, its already part of the group members.
    Nov  1 17:55:49 firewall php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP6.
    Nov  1 17:55:49 firewall php-fpm: /rc.dyndns.update: Keep current gateway, its already part of the group members.
    Nov  1 17:55:49 firewall php-fpm[20847]: /rc.filter_configure_sync: Keep current gateway, its already part of the group members.
    

    Now I also noticed this while doing a radvdump:

    #
    # radvd configuration generated by radvdump 2.17
    # based on Router Advertisement from fe80::1:1
    # received by interface lagg0.201
    #
    
    interface lagg0.201
    {
            AdvSendAdvert on;
            # Note: {Min,Max}RtrAdvInterval cannot be obtained with radvdump
            AdvManagedFlag off;
            AdvOtherConfigFlag on;
            AdvReachableTime 0;
            AdvRetransTimer 0;
            AdvCurHopLimit 64;
            AdvDefaultLifetime 30;
            AdvHomeAgentFlag off;
            AdvDefaultPreference medium;
            AdvLinkMTU 1500;
            AdvSourceLLAddress on;
    
            DNSSL $.org
            {
                    AdvDNSSLLifetime 10;
            }; # End of DNSSL definition
    
    }; # End of interface definition
    #
    # radvd configuration generated by radvdump 2.17
    # based on Router Advertisement from fe80::1:1
    # received by interface lagg0.244
    #
    
    interface lagg0.244
    {
            AdvSendAdvert on;
            # Note: {Min,Max}RtrAdvInterval cannot be obtained with radvdump
            AdvManagedFlag off;
            AdvOtherConfigFlag on;
            AdvReachableTime 0;
            AdvRetransTimer 0;
            AdvCurHopLimit 64;
            AdvDefaultLifetime 30;
            AdvHomeAgentFlag off;
            AdvDefaultPreference medium;
            AdvLinkMTU 1500;
            AdvSourceLLAddress on;
    
            DNSSL $.org
            {
                    AdvDNSSLLifetime 10;
            }; # End of DNSSL definition
    
    }; # End of interface definition
    #
    # radvd configuration generated by radvdump 2.17
    # based on Router Advertisement from fe80::$:$:$:df69
    # received by interface igb1
    #
    
    interface igb1
    {
            AdvSendAdvert on;
            # Note: {Min,Max}RtrAdvInterval cannot be obtained with radvdump
            AdvManagedFlag off;
            AdvOtherConfigFlag off;
            AdvReachableTime 0;
            AdvRetransTimer 0;
            AdvCurHopLimit 64;
            AdvDefaultLifetime 60;
            AdvHomeAgentFlag off;
            AdvDefaultPreference high;
            AdvLinkMTU 1500;
            AdvSourceLLAddress on;
    
            prefix 2603:$:$::/64
            {
                    AdvValidLifetime 86400;
                    AdvPreferredLifetime 14400;
                    AdvOnLink on;
                    AdvAutonomous on;
                    AdvRouterAddr on;
            }; # End of prefix definition
    
    
            route ::/0
            {
                    AdvRoutePreference high;
                    AdvRouteLifetime 60;
            }; # End of route definition
    
    
            RDNSS 2001:4860:4860::8888 2001:4860:4860::8844 2001:558:feed::1
            {
                    AdvRDNSSLifetime 20;
            }; # End of RDNSS definition
    
    
            DNSSL $.org
            {
                    AdvDNSSLLifetime 20;
            }; # End of DNSSL definition
    
    }; # End of interface definition
    
    

    ...Is that telling me that I'm only getting a /64 with my /60 request on igb1?? Just as an experiment I changed LAN to prefix 0, but still got no LAN address. I read some other posts about adding static routes to make things work... but since I don't have fixed ipv6 addresses, that seems like a bad idea. I figure if I do that, I might as well do static addressing everywhere - and the first time I lose my leases, everything will go to hell.

    Thoughts?



  • @jsnl said in IPv6 WAN Track Interface not assigning addresses to LAN/Public LAN:

    2607:f8b0

    Yes. You should have a 16 /64 networks. The WAN IP should be a /64, which is what I see in both snapshots. It appears that your WAN DHCP6 is a 2607 address? It looks as if it should monitor the 2603 address from Comcast's WAN. That's what I see so far.



  • @smitheo1 said in IPv6 WAN Track Interface not assigning addresses to LAN/Public LAN:

    It appears that your WAN DHCP6 is a 2607 address?

    No, that's just the IP that it pings every 2 seconds to see if the gateway is up or not. That 2607:f8b0:4003:c00::6a address is ipv6.google.com - the address I'm pinging to make sure I actually have ipv6 connectivity to the world and not just locally with Comcast. Once I get everything up and running I might change it to something locally on Comcast's domain, but for now I want to make sure I can see the internet in IPv6.

    Is there any log I can see to know if the actual /60 prefix exchange is successful? Can I sniff it on the firewall in any way? Or do I have to wireshark it with an external laptop?



  • Ok. I'm tracking.

    Yes. You can see the exchange with Wireshark from the client side. You should also be able to sniff the firewall as well. The firewall has the Packet Capture in the Diagnostics section. That's how I learned that something was not passing through.



  • Try taking a look around in /etc/defaults/rc.conf which is where the addresses are stored to see if something is active.