IPv6 IP Alias prevents Track Interface from working with DHCPv6 and RA

chewie198

I'm seeing some unexpected behavior that I wanted to run by the pfSense community before filing a bug report. I have a pfSense gateway with roughly 10 VLANS. On some of these VLANs, the firewall is set to track the WAN IPv6 interface and hand out GUAs. On other VLANs, I've assigned the firewall a ULA and distribute the ULA prefix via RAs or static assignments.

During a recent attempt to assign all of my VLANs the same DNS server IP (fd53::1), I discovered that not all of my subnets could reach the ULA address, despite simplifying my firewall rules (for testing purposes) to allow all traffic between subnets. Can anyone confirm that they're seeing the same behavior (or not) before I file a bug report? Here are some additional details:

I've assigned pfSense a ULA (fd53::1) on a VLAN interface (VLAN 53). My other VLANs all have their own interfaces and corresponding subnets, with various addressing schemes.
If the source subnet has a ULA address set on the firewall interface for that VLAN (for instance, fd00:172:28:203::1), then devices in that subnet can ping the firewall's ULA addresses from a different subnet.
If the firewall is set to track the WAN interface on a subnet (and thus, doesn't have a ULA assigned on its subnet-facing interface), then devices in that subnet cannot ping any of the firewall's ULA's in different subnets.
Devices in different subnets can ping each other's ULA's OK.
Devices in different subnets can ping GUA's on the firewall OK. For instance, using the same source and destination subnets, I can ping the firewall if I assign it an address of 53::1. If I change the firewall address to fd53::1, pings no longer succeed.
Pings run directly on the firewall itself (using the diagnostic menu) work OK between subnets regardless of ULA assignment.

To me, this suggests an issue with the firewall's routing between ULA and non-ULA interfaces, but I wanted to get some feedback before pursuing this further. I'm aware that there is already a bug report for ULA's assigned as VIP's - this isn't related because I'm not using VIP's here.

johnpoz

And what specific rules do you have on these interfaces?

Lets call it Lan1 and Lan2

So what are your rules if source is set as Lan1 Net for your rule, then that will be the prefix that is set on the interface. if you have a GUA on it then lan1 net would be that prefix.. If you have a ULA then that would be the lan net..

So lets be clear on what rules you have in place on your different interfaces.

You said you simplified the rules - so what are these exactly?

Happy to try and duplicate.. I have some vlans that are GUA, and other that have no IPv6, so should be easy to just assign a ULA on one of these.. And then your saying devices in the gua prefix can not talk to the ula pfsense IP interface?

edit: Trying to duplicate..

So I have my lan, which has gua on it from HE part of my /48
I then on my DTV vlan which had no ipv6 on it at all put

fd53::1/64 per your example

Now from the HE lan vlan I can ping pfsense ula vlan interface..
C:>ping fd53::1

Pinging fd53::1 with 32 bytes of data:
Reply from fd53::1: time<1ms
Reply from fd53::1: time<1ms
Reply from fd53::1: time<1ms

From this

If the firewall is set to track the WAN interface on a subnet (and thus, doesn't have a ULA assigned on its subnet-facing interface), then devices in that subnet cannot ping any of the firewall's ULA's in different subnets.

My setup seems to work fine, I have a gua vlan and it can ping ula vlan.. My isp does not support ipv6 so I can not easy test tracking a tracked interface... But I don't see how it would be any different than static assigned gua..

Rules on my lan which is the gua prefix are just any any ipv4+ipv6 with source be set to lan net.

chewie198

Hi @johnpoz,

Appreciate your taking the time to test this out! I'm trying to ping from a GUA to a ULA, which is what you tested, so it sounds like it's working OK for you. It would appear that we have several differences between our setups, though. My main LAN subnet is tracking the WAN IPv6 prefix, and I also advertise a ULA prefix. As you said, not sure why either of those things would make any difference, but I'm trying to narrow down what may be causing the behavior I'm seeing. Would you mind posting screenshots of your interface configurations? I'm including the same for my interfaces below, as well as a image of my firewall config. It's set to Allow All IPv4&6 from Any to Any in both VLANs.

0_1543668528081_2018-12-01 (1).png 0_1543668532684_2018-12-01 (3).png

johnpoz

sure here is my gua interface, that my pc is in - lan.. And here is the vlan I have running that my dtv interface with the ula address assigned.

And then my pc info that is in the gua lan segment that I ping the ula.. Do you have any rules in floating?

I noticed you have a /128 prefix - that is WRONG ;)

That is more than likely your issue.

chewie198

@johnpoz, are you sure that the /128 prefix is wrong? I only have one IP address in that subnet - the firewall - as incoming ping requests should be routed from a different subnet. Nevertheless, I tried changing both subnets to a /64 prefix for testing purposes, but that didn't allow me to ping the firewall.

I only have one floating rule, and that's just a Match for QoS tagging. Here are some screenshots as a sanity check. Any other ideas as to what might be causing this?
0_1543872819764_2018-12-03 (2).png
0_1543872878994_2018-12-03 (1).png

JKnott

@johnpoz said in Cannot Ping pfSense ULA From Subnet Without ULA Assigned to Firewall:

I noticed you have a /128 prefix - that is WRONG ;)

I have a /128 on my WAN interface. Works fine, including pinging. Packet Capture shows the pings coming from that /128 address.

johnpoz

Dude see the other thread your talking in - I hear you /128 is viable address for a tunnel and loopback - but its not really a global address mask.. Talk to your isp if you have problem with it, etc..

Min viable prefix in ipv6 is /64 --- your the ipv6 is better than sliced bread guy around here.. You should know this ;)

chewie198

Either way, I still can't ping the firewall address regardless of the prefix length. 53::1 works fine, fd53::1 doesn't work - and the 53::1 address works whether the prefix is /64 or /128. I've tested from Ubuntu, Windows, and Android, to rule out an OS related issue, but they're all a no-go.

Is anyone able to test this from a interface IPv6 address set to Track WAN Prefix? That's the only different variable that I can see here. If not, I'll likely spin up a clean pfSense VM to see whether this is occurring on a fresh install. Is it possible that a plugin could be causing this, and if so, is there a consistent way to disable them without complete removal?

chewie198

In case anyone encounters the same problem, I happened upon the solution by chance months later while configuring NTP on a multi-homed Ubuntu server and wanted to post it here for posterity's sake.

My main LAN subnet obtains an IPv6 address via WAN DHCPv6 prefix tracking and advertises it via RA's. Because most of my other VLAN's use ULA's and eschew dynamic GUA's, I have the firewall set up with static IPv6 addresses which get advertised to each subnet. For consistency's sake, I wanted devices on my main LAN to obtain a ULA as well as a GUA, and had configured the RA's to include a ULA prefix of fd00:172:28:198::/64. Unfortunately, since pfSense doesn't currently support multiple IPv6 prefixes per tracked interface ([https://redmine.pfsense.org/issues/5999]) this means that the client ping was going out through it's ULA, hitting it's destination, and then being routed back through the firewall. Since the firewall didn't itself have a ULA on the LAN interface, it had no route back to the client, and the traffic was dropped. Removing the ULA prefix from the RA's on that interface resolved this issue.

If anyone has a solution for allowing both static and dynamic IPv6 addresses on the same interface while avoiding the issues I encountered here, I'd appreciate the suggestions.

Derelict

Unfortunately, since pfSense doesn't currently support multiple IPv6 prefixes per tracked interface

I am doing exactly this.

DHCP6 serves the delegated prefix based on track interface settings:

0_1550862519923_Screen Shot 2019-02-22 at 11.02.50 AM.png

RA Serves BOTH the tracked interface prefix and the configured ULA:

0_1550862549999_Screen Shot 2019-02-22 at 11.04.13 AM.png

ETA: Forgot about the VIPs:

0_1550867177689_Screen Shot 2019-02-22 REDACTED at 12.23.30 PM.png

My Mac's LAN interface looks like this:

vlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=3<RXCSUM,TXCSUM>
	ether a8:60:b6:19:15:fe 
	inet6 fe80::14ea:9daa:af44:6b3d%vlan0 prefixlen 64 secured scopeid 0xb 
	inet6 2600:beef:cafe:ab01:1092:8d9e:537d:5207 prefixlen 64 autoconf secured 
	inet6 fd8c:9857:66db:1:4c9:e132:ee:396a prefixlen 64 autoconf secured 
	inet 192.168.223.6 netmask 0xffffff00 broadcast 192.168.223.255
	inet6 2600:beef:cafe:ab01::145a prefixlen 64 dynamic 
	inet6 2600:beef:cafe:ab01:edf8:e8af:4a5c:da72 prefixlen 64 deprecated autoconf temporary 
	inet6 fd8c:9857:66db:1:edf8:e8af:4a5c:da72 prefixlen 64 deprecated autoconf temporary 
	inet6 2600:beef:cafe:ab01:b95b:1aa2:a178:32c9 prefixlen 64 deprecated autoconf temporary 
	inet6 fd8c:9857:66db:1:b95b:1aa2:a178:32c9 prefixlen 64 deprecated autoconf temporary 
	inet6 2600:beef:cafe:ab01:b0f0:e7fb:d671:c1e prefixlen 64 deprecated autoconf temporary 
	inet6 fd8c:9857:66db:1:b0f0:e7fb:d671:c1e prefixlen 64 deprecated autoconf temporary 
	inet6 2600:beef:cafe:ab01:91f5:785f:2283:21fc prefixlen 64 deprecated autoconf temporary 
	inet6 fd8c:9857:66db:1:91f5:785f:2283:21fc prefixlen 64 deprecated autoconf temporary 
	inet6 2600:beef:cafe:ab01:b1d6:45a3:c20:1d15 prefixlen 64 deprecated autoconf temporary 
	inet6 fd8c:9857:66db:1:b1d6:45a3:c20:1d15 prefixlen 64 deprecated autoconf temporary 
	inet6 2600:beef:cafe:ab01:4c01:bf7b:6079:299b prefixlen 64 deprecated autoconf temporary 
	inet6 fd8c:9857:66db:1:4c01:bf7b:6079:299b prefixlen 64 deprecated autoconf temporary 
	inet6 2600:beef:cafe:ab01:347d:50ed:27e3:2de9 prefixlen 64 autoconf temporary 
	inet6 fd8c:9857:66db:1:347d:50ed:27e3:2de9 prefixlen 64 autoconf temporary 
	nd6 options=201<PERFORMNUD,DAD>
	vlan: 223 parent interface: en0
	media: autoselect (1000baseT <full-duplex>)
	status: active

The IPv6 stack in the Mac sources from a global address to global destinations and a ULA address to ULA destinations.

$ ping6 -c1 www.google.com
PING6(56=40+8+8 bytes) 2600:beef:cafe:ab01:347d:50ed:27e3:2de9 --> 2607:f8b0:4005:80b::2004
16 bytes from 2607:f8b0:4005:80b::2004, icmp_seq=0 hlim=55 time=21.027 ms

$ ping6 -c1 nc.mydomain.com
PING6(56=40+8+8 bytes) fd8c:9857:66db:1:347d:50ed:27e3:2de9 --> fd8c:9857:66db:4::c0a8:e012
16 bytes from fd8c:9857:66db:4::c0a8:e012, icmp_seq=0 hlim=63 time=0.695 ms

I use split DNS to give ULA addresses to inside queries. It's been working great. And now my inside network no longer breaks in the unlikely event Cox changes my PD.

One thing I have not tried is pinging ULA from an interface with only global addresses and pinging global from interfaces with only ULA addresses. A lot of the burden there is on the client's stack. Considering, with a /56, I don't have any problem assigning GUA and ULA to every IPv6 interface it has never come up. Let me try something real quick:

From LAN to the LAN GUA

$ ping6 -S fd8c:9857:66db:1:347d:50ed:27e3:2de9 2600:beef:cafe:ab01:208:a2ff:fe0a:593f
PING6(56=40+8+8 bytes) fd8c:9857:66db:1:347d:50ed:27e3:2de9 --> 2600:beef:cafe:ab01:208:a2ff:fe0a:593f
16 bytes from 2600:beef:cafe:ab01:208:a2ff:fe0a:593f, icmp_seq=0 hlim=64 time=0.352 ms
16 bytes from 2600:beef:cafe:ab01:208:a2ff:fe0a:593f, icmp_seq=1 hlim=64 time=0.391 ms
16 bytes from 2600:beef:cafe:ab01:208:a2ff:fe0a:593f, icmp_seq=2 hlim=64 time=0.325 ms

From LAN to another interface's GUA:

$ ping6 -S fd8c:9857:66db:1:347d:50ed:27e3:2de9 2600:beef:cafe:ab02:208:a2ff:fe0a:593f
PING6(56=40+8+8 bytes) fd8c:9857:66db:1:347d:50ed:27e3:2de9 --> 2600:beef:cafe:ab02:208:a2ff:fe0a:593f
16 bytes from 2600:beef:cafe:ab02:208:a2ff:fe0a:593f, icmp_seq=0 hlim=64 time=0.455 ms
16 bytes from 2600:beef:cafe:ab02:208:a2ff:fe0a:593f, icmp_seq=1 hlim=64 time=0.292 ms
16 bytes from 2600:beef:cafe:ab02:208:a2ff:fe0a:593f, icmp_seq=2 hlim=64 time=0.298 ms
^C
--- 2600:beef:cafe:ab02:208:a2ff:fe0a:593f ping6 statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.292/0.348/0.455/0.075 ms

From LAN to a host on DMZ GUA:

$ ping6 -S fd8c:9857:66db:1:347d:50ed:27e3:2de9 2600:beef:cafe:ab04::c0a8:e012
PING6(56=40+8+8 bytes) fd8c:9857:66db:1:347d:50ed:27e3:2de9 --> 2600:beef:cafe:ab04::c0a8:e012
16 bytes from 2600:beef:cafe:ab04::c0a8:e012, icmp_seq=0 hlim=63 time=0.775 ms
16 bytes from 2600:beef:cafe:ab04::c0a8:e012, icmp_seq=1 hlim=63 time=0.612 ms
16 bytes from 2600:beef:cafe:ab04::c0a8:e012, icmp_seq=2 hlim=63 time=0.629 ms
^C
--- 2600:beef:cafe:ab04::c0a8:e012 ping6 statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.612/0.672/0.775/0.073 ms

So it looks like it's all working as expected. There are GUA and ULA addresses on the configured interfaces.

Rebooting based on the trouble description in that redmine:

Yeah. it all just works here. GUA and ULA /64s on all interfaces configured as such. All the above pings still work. Seems fine.

For completeness, this is how I did one of the linux servers (nextcloud - accessible on GUA from the outside and ULA from the inside):

auto ens18
iface ens18 inet static
address 192.168.224.18
netmask 255.255.255.0
gateway 192.168.224.1
dns-search example.com admin.example.com
dns-nameservers 192.168.223.1

iface ens18 inet6 auto
   privext 2
   accept-ra 2
   up ip -6 addr add 2600:beef:cafe:ab04::192.168.224.18/64 dev $IFACE label $IFACE:0
   down ip -6 addr del 2600:beef:cafe:ab04::192.168.224.18/64 dev $IFACE label $IFACE:0
   up ip -6 addr add fd8c:9857:66db:4::192.168.224.18/64 dev $IFACE label $IFACE:0
   down ip -6 addr del fd8c:9857:66db:4::192.168.224.18/64 dev $IFACE label $IFACE:0

(Please do not use ULA prefix fd8c:9857:66db::/48. Please generate your own.)

chewie198

Hi @Derelict,

Thank you for your extremely detailed reply! I apologize that it took me so long to try your suggested approach, but my wife and I brought home a baby girl around the same time that I original posted this, so I've been slightly distracted :D.

That said, I was able to replicate your configuration and unfortunately, I'm still encountering the same issue. Per the bug report, "This setup works perfectly until I reboot the pfSense machine or reset the LAN interface. This causes the IP Alias / CARP address to appear as the primary interface route and the tracked interface to appear as a secondary route (only visible by running netstat -nr). Both DHCPv6 and RA give out ULA routes only."

It's hard to tell what the differences might be between our setups, and I'm sure you're much more intimately familiar with the codebase than I am as a Netgate employee, but the bug report does indeed seem to be valid for at least some firewall configurations. Any idea what I might try to troubleshoot this, or should I just accept this as a bug until it's addressed? As you can imagine, it's difficult to develop an internally stable IPv6 infrastructure with this bug plus IPv6 NPT not tracking the dynamically assigned prefix for customers on dynamic IPs. It looks like Chris Linstruth also updated that bug (#5999) with some additional info since my last reply.

Derelict

Yes. There is a problem there. I have been looking at a way to fix it but "I am not a programmer."

I only have ULA on two interfaces (LAN and DMZ where statically-addressed servers are) so when I reboot (which is rare) I:

Remove the VIPs
Edit/Save WAN
Add the VIPs back.

And it's generally good until the next reboot which is generally weeks/months.

apearson

@chewie198 Just ran into this bug as well. Let me know if you need any info from my system.

chewie198

@apearson I am not a Netgate developer, so it's not entirely up to me whether or not this gets fixed. That said, I am a software developer and I do have a vested interest in seeing this issue get resolved. If enough people are interested in seeing this fixed and other independent developers are willing to lend a hand, we may be able to submit a pull request in an attempt to resolve the issue.

I suspect it will be an uphill fight since pfSense's paying customers are largely businesses and this issue would not affect them because a business would almost certainly be using static IP ranges with either NPT or advertising their own PI address space via BGP. Nevertheless, I have added a link to this thread in the comments for https://redmine.pfsense.org/issues/5999 in case other users or developers encounter this issue and are willing to help out in some way.

I'm happy to help you troubleshoot if you need help working around the problem. Just know that you can only host internal services via IPv4 addresses for the time being so long as you have a dynamic public IP and expect your routes to persist after a reboot (which is probably the majority of individual end users).

JKnott

@chewie198

I'm trying to understand the problem here, as the thread goes back quite a while and I've forgotten the details. Is the problem that ULA addresses are not working properly? I have both ULA and GUA addresses on my network and the local DNS points to the ULA address, not GUA. This means that even if my GUA prefix should change, my local network will keep on working fine. I use tracking for my GUA prefix and configured the ULA prefix on the Router Advertisements page.

@johnpoz

Dude see the other thread your talking in - I hear you /128 is viable address for a tunnel and loopback - but its not really a global address mask.. Talk to your isp if you have problem with it, etc..

Min viable prefix in ipv6 is /64 --- your the ipv6 is better than sliced bread guy around here.. You should know this ;)

I don't see that I replied to this, so here it goes. Yes, I have a /128 for my WAN address, but the only /64 on my WAN interface is for my link local address. On IPv6, link local addresses are often used for routing.

chewie198

@JKnott The original problem was that I couldn't ping between VLANs using ULA's because I didn't have the ULA prefix in my router advertisements so there was no route back to the source. Once I added the prefix then I was able to ping successfully. That was just an overlooked mistake on my end.

However, once that was rectified the problem became that, per the bug report, "This setup works perfectly until I reboot the pfSense machine or reset the LAN interface. This causes the IP Alias / CARP address to appear as the primary interface route and the tracked interface to appear as a secondary route (only visible by running netstat -nr). Both DHCPv6 and RA give out ULA routes only."

Like you, I use tracking for my GUA prefix and configured the ULA prefix on the Router Advertisements page. As you can imagine, it's difficult to maintain global IPv6 connectivity without a globally unique prefix/address being handed out via DHCP or RA's.

Does your GUA prefix continue to be advertised after a reboot? If so, it would appear that the behavior is inconsistent. In my case, @Derelict's case, and others, we're seeing the VIP address "take over" the interface for DHCP and RA purposes.

JKnott

@chewie198 said in Cannot Ping pfSense ULA From Subnet Without ULA Assigned to Firewall:

Does your GUA prefix continue to be advertised after a reboot? If so, it would appear that the behavior is inconsistent. In my case, @Derelict's case, and others, we're seeing the VIP address "take over" the interface for DHCP and RA purposes.

Yes, both my GUA and ULA prefixes remain after rebooting. The only problem I've experienced was when I first started using pfSense was my GUA prefix would change even for something as simple as disconnecting/reconnecting the Ethernet cable. However, a setting was added, so that pfSense would retain the existing prefix.

Here's the relevant part of a recent RA:

ICMPv6 Option (Prefix information : fd48:1a37:2160::/64)
Type: Prefix information (3)
Length: 4 (32 bytes)
Prefix Length: 64
Flag: 0xe0, On-link flag(L), Autonomous address-configuration flag(A), Router address flag(R)
Valid Lifetime: 86400
Preferred Lifetime: 14400
Reserved
Prefix: fd48:1a37:2160::

And for the GUA:

ICMPv6 Option (Prefix information : 2607:fea8:abcd:dcba::/64)
Type: Prefix information (3)
Length: 4 (32 bytes)
Prefix Length: 64
Flag: 0xe0, On-link flag(L), Autonomous address-configuration flag(A), Router address flag(R)
Valid Lifetime: 86400
Preferred Lifetime: 14400
Reserved
Prefix: 2607:fea8:abcd:dcba::

(Prefix has been changed to protect the guilty)

chewie198

@JKnott As I haven't reviewed the code, I can't tell whether the behavior is merely nondeterministic or whether there's something specific to your setup that is causing it to consistently advertise the correct prefix. Given that Chris Buechler commented on the Redmine ticket that "find_interface_ipv6 returns the first IP on the interface in that case, which might not be the intended one in that situation," I would speculate that the first address in the list is random and you've just gotten lucky. Would you be willing to reboot 4-5 times in succession and confirm that ifconfig is showing the GUA first each time? If so, maybe there's something helpful we could derive from your configuration as a workaround.

I'm still concerned, however, that unless the code is specifically written to use the GUA as the basis for DHCP and RA advertisements, achieving the correct result is just going to be a lucky break based off of implementation details. The best case scenario, I think, would be to provide a dropdown for the user to select what the primary address prefix should be for DHCP solicitations. Are you seeing the correct GUA prefix for DHCP as well, or are you only using RA's? I didn't see you mention DHCPv6 in your last reply.

Derelict

@chewie198 I believe the "quick fix" is to have a checkbox somewhere that says "ignore ULA" and, somewhere in the code (ianap) honor that as the addresses on each interface are looped through.

A more-perfect solution is likely much more complicated.

chewie198

@Derelict I agree, solving this thoroughly is not easy at all. If it were, it likely would have been done already. Right now, these sections of code have been written with the assumption that there will only be one primary IP address per interface, and with IPv6, that is no longer the case. That means we now have to maintain a list of all the addresses on a given interface and modify the UI to reflect a separate list of settings for each one, or at the very least, to allow for choosing a primary. That's not a trivial amount of work.

I like your quick fix idea though because that would almost certainly be less code to write and test and it should cover a basic use case for a large percentage of end-user configurations. Then we could choose whether to allocate DHCP addresses under the GUA or ULA ranges per user preference. Given @JKnott's reply, maybe we're already in the clear regarding RA's. I don't recall seeing this behavior the last time I assigned a VIP, but I can retest if we're getting conflicting reports.