Upgraded to 2.3.1-RELEASE-p5 (amd64 full-install).. Lost IPV6

jcyr

Just upgraded from 2.2.6 to 2.3.1 and lost IPV6 connectivity on all clients.

pfSense (the router seems fine):

*** Welcome to pfSense 2.3.1-RELEASE-p5 (amd64 full-install) on router ***

WAN (wan) -> re1 -> v4/DHCP4: 73.184.240.250/23
v6/DHCP6: 2001:558:xxxx:xxxx:19ab:43d2:6dd9:b06d/128
LAN (lan) -> re2 -> v4: 192.168.1.1/24
v6/t6: 2601:cf:xxxx:xxxx:230:18ff:fec8:fdb/64
DMZ (opt1) -> re3 -> v4: 192.168.2.1/24
v6/t6: 2601:cf:xxxx:xxxx:230:18ff:fec8:fdc/64

Logout (SSH only) 9) pfTop
Assign Interfaces 10) Filter Logs
Set interface(s) IP address 11) Restart webConfigurator
Reset webConfigurator password 12) pfSense Developer Shell
Reset to factory defaults 13) Update from console
Reboot system 14) Disable Secure Shell (sshd)
Halt system 15) Restore recent configuration
Ping host 16) Restart PHP-FPM
Shell

Enter an option: 8

[2.3.1-RELEASE][root@router.localdomain]/root: ping6 google.com
PING6(56=40+8+8 bytes) 2001:558:6011:93:19ab:43d2:6dd9:b06d –> 2607:f8b0:4002:c09::65
16 bytes from 2607:f8b0:4002:c09::65, icmp_seq=0 hlim=55 time=12.044 ms
16 bytes from 2607:f8b0:4002:c09::65, icmp_seq=1 hlim=55 time=10.943 ms
^C
--- google.com ping6 statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 10.943/11.314/12.044/0.516 ms
[2.3.1-RELEASE][root@router.localdomain]/root:

However none of the clients on the LAN side can ping IPV6 addresses.

From Windows lan side client:

C:\Users\jcyr>ping /6 google.com

Pinging google.com [2607:f8b0:4002:802::200e] with 32 bytes of data:
Request timed out.
Request timed out.

Ping statistics for 2607:f8b0:4002:802::200e:
Packets: Sent = 2, Received = 0, Lost = 2 (100% loss),
Control-C
^C

From BSD lan side client:

jcyr@linux:~$ host google.com
google.com has address 216.58.193.142
google.com has IPv6 address 2607:f8b0:4002:802::200e
google.com mail is handled by 40 alt3.aspmx.l.google.com.
google.com mail is handled by 30 alt2.aspmx.l.google.com.
google.com mail is handled by 10 aspmx.l.google.com.
google.com mail is handled by 20 alt1.aspmx.l.google.com.
google.com mail is handled by 50 alt4.aspmx.l.google.com.
jcyr@linux:~$ ping6 2607:f8b0:4002:802::200e
PING 2607:f8b0:4002:802::200e(2607:f8b0:4002:802::200e) 56 data bytes
^C
–- 2607:f8b0:4002:802::200e ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2015ms

This all worked perfectly under 2.2.6, but not after upgrade to 2.3.1. No config changes were made.

Reverting to 2.2.6 restores all functionality, and I've resorted to that in the past (this same problem occurred when I first upgraded some months ago) but I'd prefer getting to the bottom of this problem with this 2.3.1 version.

Any suggestions on what might me wrong or how to proceed with debugging?

PS... I X'd out the wrong portions of the V6 addresses showing that the V6 prefix delegation works fine... so here they are:

WAN (wan) -> re1 -> v4/DHCP4: 73.184.240.250/23
v6/DHCP6: 2001:558:6011:93:19ab:43d2:6dd9:b06d/128
LAN (lan) -> re2 -> v4: 192.168.1.1/24
v6/t6: 2601:cf:xxxx:19b0:230:18ff:fec8:fdb/64
DMZ (opt1) -> re3 -> v4: 192.168.2.1/24
v6/t6: 2601:cf:xxxx:19b1:230:18ff:fec8:fdc/64

Also, I use http://ipv6-test.com/ to determine full IPV6 functionality on the clients.

With 2.3.1 it shows NO ipv6 support. With 2.2.6 it scores 19 out of 20, failing only the reverse IPV6 dns check, but that is expected.

Paint

What are your DHCPv6 and RA Settings?

Do your IPv6 clients get an IPv6 DNS server when assigned a IPv4 and IPv6 address?

What your DNS settings in System -> General Setup?

cmb

Did your delegated prefix actually change just by upgrading? Hard to tell whether that was just anonymized out or if it really did change.

A traceroute6 out from LAN would be telling. Almost looks like you lost routing for your PD subnet.

jcyr

Both Windows and FreeBSD clients were missing the usual V6 DNS server addresses.

With Comcast the V6 delegated prefix changes with every pfSense reboot.

I'm requesting a 60 bit prefix. The delegated ranges below are correct given that the masked (xxxx) parts are the same value for both interfaces.

LAN (lan) -> re2 -> v4: 192.168.1.1/24
v6/t6: 2601:cf:xxxx:19b0:230:18ff:fec8:fdb/64
DMZ (opt1) -> re3 -> v4: 192.168.2.1/24
v6/t6: 2601:cf:xxxx:19b1:230:18ff:fec8:fdc/64

I think I have found the problem, and a partial solution.

On the Resolver (unbound) config page, there is a drop-down that allows you to specify network interfaces. Out of paranoia I've always selected only the LAN and DMZ interfaces there, leaving out the WAN interface. I'm guessing that most leave this setting at the default ALL setting. On a whim, since I was having V6 DNS resolution issues, I replaced the setting with ALL. Lo and behold all started operating as expected!!!

Reverted to the more selective setting, with the following /var/unbound/unbound.conf generated.

Interface IP(s) to bind to

interface: 73.184.240.250 <<=== What is this doing here. It is the WAN side V4 address which is NOT selected!
interface: 2001:558:6011:93:3ce8:f1d4:efe4:5540 <<=== Same here for V6!
interface: 192.168.1.1
interface: 2601:cf:8101:b550:230:18ff:fec8:fdb
interface: 192.168.2.1
interface: 2601:cf:8101:b551:230:18ff:fec8:fdc
interface: fe80::230:18ff:fecb:11a3%re1
interface: fe80::1:1%re2
interface: fe80::1:1%re3
interface: 127.0.0.1
interface: ::1

Unbound restarted, I ran the V6 tests, again all is well!!!

Rebooted… No more V6 DNS resolution!!! Over to the service status page. The unbound DNS resolver is stopped!

Manually start the resolver and V6 DNS support is restored.

So the question is: Why does unbound fail at reboot if specific interfaces are configured for unbound.

No error logs in the unbound logs. This in the system log:

rc.bootup: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1468603936] unbound[35432:0] error: can't bind socket: Can't assign requested address for fe80::230:18ff:fec8:fdb [1468603936] unbound[35432:0] fatal error: could not open ports'

OK, deleted the V6 link local interfaces from the selected unbound interfaces, rebooted, and all is well.

Sooo… Should link local interfaces actually be considered as V6 DNS query interface candidates or not. If so, why do they cause unbound to fail on reboot. If not, should they be presented as interface candidates in the unbound interface selection drop down?