Upgraded to 2.3.1-RELEASE-p5 (amd64 full-install).. Lost IPV6

  • Just upgraded from 2.2.6 to 2.3.1 and lost IPV6 connectivity on all clients.

    pfSense (the router seems fine):

    *** Welcome to pfSense 2.3.1-RELEASE-p5 (amd64 full-install) on router ***

    WAN (wan)      -> re1        -> v4/DHCP4:
                                      v6/DHCP6: 2001:558:xxxx:xxxx:19ab:43d2:6dd9:b06d/128
    LAN (lan)      -> re2        -> v4:
                                      v6/t6: 2601:cf:xxxx:xxxx:230:18ff:fec8:fdb/64
    DMZ (opt1)      -> re3        -> v4:
                                      v6/t6: 2601:cf:xxxx:xxxx:230:18ff:fec8:fdc/64

    1. Logout (SSH only)                  9) pfTop
    2. Assign Interfaces                10) Filter Logs
    3. Set interface(s) IP address      11) Restart webConfigurator
    4. Reset webConfigurator password    12) pfSense Developer Shell
    5. Reset to factory defaults        13) Update from console
    6. Reboot system                    14) Disable Secure Shell (sshd)
    7. Halt system                      15) Restore recent configuration
    8. Ping host                        16) Restart PHP-FPM
    9. Shell

    Enter an option: 8

    [2.3.1-RELEASE][root@router.localdomain]/root: ping6 google.com
    PING6(56=40+8+8 bytes) 2001:558:6011:93:19ab:43d2:6dd9:b06d –> 2607:f8b0:4002:c09::65
    16 bytes from 2607:f8b0:4002:c09::65, icmp_seq=0 hlim=55 time=12.044 ms
    16 bytes from 2607:f8b0:4002:c09::65, icmp_seq=1 hlim=55 time=10.943 ms
    --- google.com ping6 statistics ---
    3 packets transmitted, 3 packets received, 0.0% packet loss
    round-trip min/avg/max/std-dev = 10.943/11.314/12.044/0.516 ms

    However none of the clients on the LAN side can ping IPV6 addresses.

    From Windows lan side client:

    C:\Users\jcyr>ping /6 google.com

    Pinging google.com [2607:f8b0:4002:802::200e] with 32 bytes of data:
    Request timed out.
    Request timed out.

    Ping statistics for 2607:f8b0:4002:802::200e:
        Packets: Sent = 2, Received = 0, Lost = 2 (100% loss),

    From BSD lan side client:

    jcyr@linux:~$ host google.com
    google.com has address
    google.com has IPv6 address 2607:f8b0:4002:802::200e
    google.com mail is handled by 40 alt3.aspmx.l.google.com.
    google.com mail is handled by 30 alt2.aspmx.l.google.com.
    google.com mail is handled by 10 aspmx.l.google.com.
    google.com mail is handled by 20 alt1.aspmx.l.google.com.
    google.com mail is handled by 50 alt4.aspmx.l.google.com.
    jcyr@linux:~$ ping6 2607:f8b0:4002:802::200e
    PING 2607:f8b0:4002:802::200e(2607:f8b0:4002:802::200e) 56 data bytes
    –- 2607:f8b0:4002:802::200e ping statistics ---
    3 packets transmitted, 0 received, 100% packet loss, time 2015ms

    This all worked perfectly under 2.2.6, but not after upgrade to 2.3.1. No config changes were made.

    Reverting to 2.2.6 restores all functionality, and I've resorted to that in the past (this same problem occurred when I first upgraded some months ago) but I'd prefer getting to the bottom of this problem with this 2.3.1 version.

    Any suggestions on what might me wrong or how to proceed with debugging?

    PS... I X'd out the wrong portions of the V6 addresses showing that the V6 prefix delegation works fine... so here they are:

    WAN (wan)      -> re1        -> v4/DHCP4:
                                      v6/DHCP6: 2001:558:6011:93:19ab:43d2:6dd9:b06d/128
    LAN (lan)      -> re2        -> v4:
                                      v6/t6: 2601:cf:xxxx:19b0:230:18ff:fec8:fdb/64
    DMZ (opt1)      -> re3        -> v4:
                                      v6/t6: 2601:cf:xxxx:19b1:230:18ff:fec8:fdc/64

    Also, I use http://ipv6-test.com/ to determine full IPV6 functionality on the clients.

    With 2.3.1 it shows NO ipv6 support. With 2.2.6 it scores 19 out of 20, failing only the reverse IPV6 dns check, but that is expected.

  • What are your DHCPv6 and RA Settings?

    Do your IPv6 clients get an IPv6 DNS server when assigned a IPv4 and IPv6 address?

    What your DNS settings in System -> General Setup?

  • Did your delegated prefix actually change just by upgrading? Hard to tell whether that was just anonymized out or if it really did change.

    A traceroute6 out from LAN would be telling. Almost looks like you lost routing for your PD subnet.

  • Both Windows and FreeBSD clients were missing the usual V6 DNS server addresses.

    With Comcast the V6 delegated prefix changes with every pfSense reboot.

    I'm requesting a 60 bit prefix. The delegated ranges below are correct given that the masked (xxxx) parts are the same value for both interfaces.

    LAN (lan)      -> re2        -> v4:
                                      v6/t6: 2601:cf:xxxx:19b0:230:18ff:fec8:fdb/64
    DMZ (opt1)      -> re3        -> v4:
                                      v6/t6: 2601:cf:xxxx:19b1:230:18ff:fec8:fdc/64

    I think I have found the problem, and a partial solution.

    On the Resolver (unbound) config page, there is a drop-down that allows you to specify network interfaces. Out of paranoia I've always selected only the LAN and DMZ interfaces there, leaving out the WAN interface. I'm guessing that most leave this setting at the default ALL setting. On a whim, since I was having V6 DNS resolution issues, I replaced the setting with ALL. Lo and behold all started operating as expected!!!

    Reverted to the more selective setting, with the following /var/unbound/unbound.conf generated.

    Interface IP(s) to bind to

    interface:  <<=== What is this doing here. It is the WAN side V4 address which is NOT selected!
    interface: 2001:558:6011:93:3ce8:f1d4:efe4:5540 <<=== Same here for V6!
    interface: 2601:cf:8101:b550:230:18ff:fec8:fdb
    interface: 2601:cf:8101:b551:230:18ff:fec8:fdc
    interface: fe80::230:18ff:fecb:11a3%re1
    interface: fe80::1:1%re2
    interface: fe80::1:1%re3
    interface: ::1

    Unbound restarted, I ran the V6 tests, again all is well!!!

    Rebooted… No more V6 DNS resolution!!! Over to the service status page. The unbound DNS resolver is stopped!

    Manually start the resolver and V6 DNS support is restored.

    So the question is: Why does unbound fail at reboot if specific interfaces are configured for unbound.

    No error logs in the unbound logs. This in the system log:

    rc.bootup: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1468603936] unbound[35432:0] error: can't bind socket: Can't assign requested address for fe80::230:18ff:fec8:fdb [1468603936] unbound[35432:0] fatal error: could not open ports'

    OK, deleted the V6 link local interfaces from the selected unbound interfaces, rebooted, and all is well.

    Sooo… Should link local interfaces actually be considered as V6 DNS query interface candidates or not. If so, why do they cause unbound to fail on reboot. If not, should they be presented as interface candidates in the unbound interface selection drop down?