Unique Local Addresses?

JKnott

OK, I've done some testing. I've attached 3 pcap files for before the alias is added, after the alias is added but before reboot and after reboot. There are 3 local interfaces on the pfSense router
Native LAN with global address and ULA fd48:1a37:2160:0::1
VLAN 3 with ULA fd48:1a37:2160:3::1
Test network on a separate NIC with ULA fd48:1a37:2160:4::1
All interfaces have NAT IPv4 addresses. IPv4 works fine.

Desktop computer, running Linux has native LAN with global address and ULA and VLAN 3 with ULA
The ULA is always advertised and addresses appear for both native LAN and VLAN 3
Prior to enabling the alias fd48:1a37:2160:0::1 on the native LAN, internet access works fine, but I cannot ping a ULA address on a computer connected to the test network
After enabling the alias, internet still works fine and I can ping the test network computer, using the IPv6 address
After rebooting, I can still ping the computer connected to the test network, but no longer access the Internet with IPv6. DNS also fails.
When I ping ipv6.google.com, using the IPv6 address 2607:f8b0:400b:808::200e I can see the packets going out on VLAN 3, with an appropriate IPv6 address for the VLAN. Of course, this will not work over the Internet.

Through all the above, DNS lookup and IPv6 access to the Internet continue to work on the pfSense firewall.

Bottom line, for some reason, after the alias is enabled, the Linux computer decides it has to use VLAN 3 to reach the Internet. Deleting the alias and rebooting pfSense restores Internet access to the Linux computer. I expect DNS fails due to IPv6 being used to access it.

The files were captured on the Linux computer with Wireshark.

[RA without alias.pcap](/public/imported_attachments/1/RA without alias.pcap)
[RA with alias before reboot.pcap](/public/imported_attachments/1/RA with alias before reboot.pcap)
[RA with alias after reboot.pcap](/public/imported_attachments/1/RA with alias after reboot.pcap)

Derelict

What network is the linux computer supposed to be on?

JKnott

It's on the native LAN and VLAN 3. As mentioned above, the native LAN has both global and ULA addresses. VLAN 3 is ULA only.

kpa

What's the point of connecting the client to two different segments like that? You could easily have any number of different IPv6 prefixes on the main LAN, some of the globally routable, some of them ULA. IPv6 is designed with that in mind.

JKnott

@kpa:

What's the point of connecting the client to two different segments like that? You could easily have any number of different IPv6 prefixes on the main LAN, some of the globally routable, some of them ULA. IPv6 is designed with that in mind.

Experimenting. I like to try different things. But the question remains, why does adding an alias on the native LAN cause the Linux computer to start using the VLAN ULA for it's source on traffic for the Internet?

BTW, I have a /56 prefix from my ISP and use the 0 /64 on the native LAN, 4 on the other NIC and ff on the OpenVPN VPN, so I know all about lots of prefixes. I started experimenting with ULAs a while ago, as I read that IoT gear should be using them for security reasons.

johnpoz

"as I read that IoT gear should be using them for security reasons."

What scenario does your iot device even need to use IPv6 to talk to local stuff, but not be allowed global? Are you running a ipv6 only network? Why go through all the extra hassle when you could just block the iot from going outbound at the firewall.. No matter what IP it has v4 or v6, public or rfc1918, etc. etc.

ULA would make sense if you only had a /64 to play with and you have multiple segments.. And you want these segments to talk ipv6 to each other.

Clearly is this not the case since you state you have /56 to work with.

"the native LAN has both global and ULA addresses"

This comes down to running multiple layer 3 on the same layer 2 plain and simple.. Never a good idea!! You going to dual stack ipv4 and ipv6 which are completely different protocols then ok… But running either multiple v4 or v6 layer 3 on the same layer 2 is bad juju! And yeah its going to cause you grief and pain..

JKnott

As I've mentioned a few times, I like to experiment, to learn. I said quite a while back, there's no IoT here. I just decided to try ULA after reading an article about IoT. Regardless, I don't understand why a computer should start sending Internet traffic out a VLAN with only ULA on it, after an alias is placed on the pfSense native LAN interface. Everything works fine after enabling that alias, until I reboot. Also, the first time I noticed the problem was the next morning after enabling the alias and I no longer had access to the Internet. So, it appears it doesn't even need a reboot, if left sitting long enough.

So, the situation is this. at the moment I have Internet access. If I put the alias on that interface, everything will intially work as expected. But if I then reboot pfSense, my Linux computer will then try to access the Internet via VLAN 3, using a ULA. Why is that happening. Nothing has changed on the computer. I have even set the RAs on VLAN 3 to have low priortiy.

Derelict

If you look at the different packet captures you posted earlier, you will notice the RAs in a different order.

I have not had time to try to duplicate it, but it seems about equally likely that Linux is doing the wrong thing there as pfSense.

Or, as John said earlier, you are just asking for trouble and receiving it.

johnpoz

"I like to experiment, to learn"

So do I.. But your playing with shit that makes no sense… In what real world scenario would this sort of setup happen? Please give a real world use case that doesn't have a easier to manage, implement and clear design and I will be all over trying to duplicate it and make it work.

But so far its just nonsense - sorry! I can not see a valid use case for doing such a thing.. Just because you can doesn't mean you should ;)

You have created a multihomed setup. How the OS determines which IP to use has multiple variables... In windows there is the interface metric.. There is a setting to if a specific IP should be used as source, etc. I am not as familiar with how linux does it.. But have real world experience this sort of borked up configuration and mess with windows. And had to fix it.. And these were even on different interfaces.. Different binding orders, etc.. But windows like to use the wrong IP for itself, etc..

Multihoming is asking for grief, multiple layer3 on the same interface - asking for trouble.. Especially when you setup a gateway for that interface so it can get to other networks, etc.

JKnott

Linux also uses metrics, as does everything that supports routing. IPv6 also has a priority for RAs, which allows for fallback routers etc.. What I have to find out is what is causing the computer to choose the VLAN when sending to the Internet. Routing also uses longest match, when selecting the route. There's no way that a route starting with fd can be a longest match for a global address. Again, this initially works, until I reboot pfSense. I'll have to do some more testing, when I have time.

Also, as for multihomed, every router is, otherwise they wouldn't work. Linux is often used as a router and, in fact, I used it as my firewall/router for about 15 years or so, before I switched to pfSense, about 1.5 years ago. Linux is also often used in commercial gear. In fact, there were some Cisco models that ran Linux, along with many other makes. It seems the 'net runs largely on Linux or BSD.

JKnott

I've just been going through the Wireshark captures, with alias, both before and after reboot. I have notice there are differences in the RAs that may be causing the problem. The differences appear in frames 2 & 4, which are for the native LAN. For example, the first ICMPv6 Option (Prefix information…) has my global address prefix before reboot, but ULA prefix after reboot. Also, the recursive DNS Servers change from global to ULA prefix. So, the question becomes why is pfSense sending the wrong info after reboot?

Perhaps someone else can try this. I have global and ULA addresses on the native network and ULA only on a VLAN. My computer is running OpenSUSE 42.3 and has both native LAN and VLAN 3 configured. PfSense is configured the same. Perhaps someone can try with Windows too. I don't have a Windows system capable of supporting VLANs.

Here is frame 4, before reboot:

No. Time Source Destination Protocol Length Info
4 20:59:52.493110 fe80::1:1 ff02::1 ICMPv6 262 Router Advertisement from 00:16:17:a7:f2:d3

Frame 4: 262 bytes on wire (2096 bits), 262 bytes captured (2096 bits)
Ethernet II, Src: Msi_a7:f2:d3 (00:16:17:a7:f2:d3), Dst: IPv6mcast_01 (33:33:00:00:00:01)
Internet Protocol Version 6, Src: fe80::1:1, Dst: ff02::1
Internet Control Message Protocol v6
Type: Router Advertisement (134)
Code: 0
Checksum: 0x950c [correct]
[Checksum Status: Good]
Cur hop limit: 64
Flags: 0xc8
Router lifetime (s): 60
Reachable time (ms): 0
Retrans timer (ms): 0
ICMPv6 Option (Prefix information : 2607:fea8:abcd:ef00::/64)
ICMPv6 Option (Prefix information : 2607:fea8:abcd:ef00::/64)
ICMPv6 Option (Prefix information : fd48:1a37:2160::/64)
ICMPv6 Option (Route Information : Medium ::/0)
ICMPv6 Option (Recursive DNS Server 2607:fea8:abcd:ef00:216:17ff:fea7:f2d3)
ICMPv6 Option (DNS Search List Option jknott.net jknott.net)
ICMPv6 Option (MTU : 1500)
ICMPv6 Option (Source link-layer address : 00:16:17:a7:f2:d3)

And after reboot:

No. Time Source Destination Protocol Length Info
4 21:09:52.818342 fe80::1:1 ff02::1 ICMPv6 262 Router Advertisement from 00:16:17:a7:f2:d3

Frame 4: 262 bytes on wire (2096 bits), 262 bytes captured (2096 bits)
Ethernet II, Src: Msi_a7:f2:d3 (00:16:17:a7:f2:d3), Dst: IPv6mcast_01 (33:33:00:00:00:01)
Internet Protocol Version 6, Src: fe80::1:1, Dst: ff02::1
Internet Control Message Protocol v6
Type: Router Advertisement (134)
Code: 0
Checksum: 0xeffc [correct]
[Checksum Status: Good]
Cur hop limit: 64
Flags: 0xc8
Router lifetime (s): 60
Reachable time (ms): 0
Retrans timer (ms): 0
ICMPv6 Option (Prefix information : fd48:1a37:2160::/64)
ICMPv6 Option (Prefix information : 2607:fea8:abcd:ef00::/64)
ICMPv6 Option (Prefix information : fd48:1a37:2160::/64)
ICMPv6 Option (Route Information : Medium ::/0)
ICMPv6 Option (Recursive DNS Server fd48:1a37:2160::1)
ICMPv6 Option (DNS Search List Option jknott.net jknott.net)
ICMPv6 Option (MTU : 1500)
ICMPv6 Option (Source link-layer address : 00:16:17:a7:f2:d3)

Differences are in bold.

JKnott

One other thing I've noticed is that the default route entries get reversed:

Before reboot:

$ ip -6 route
2607:fea8:abcd:ef00::/64 dev eth0 proto kernel metric 256 expires 86385sec pref medium
fd48:1a37:2160::/64 dev eth0 proto kernel metric 256 expires 86385sec pref medium
fd48:1a37:2160:3::/64 dev vlan3 proto kernel metric 256 expires 86391sec pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev vlan3 proto kernel metric 256 pref medium
default via fe80::1:1 dev eth0 proto ra metric 1024 expires 45sec hoplimit 64 pref medium
default via fe80::216:17ff:fea7:f2d3 dev vlan3 proto ra metric 1024 expires 51sec hoplimit 64 pref medium

After:

$ ip -6 route
2607:fea8:abcd:ef00::/64 dev eth0 proto kernel metric 256 expires 86387sec pref medium
fd48:1a37:2160::/64 dev eth0 proto kernel metric 256 expires 86387sec pref medium
fd48:1a37:2160:3::/64 dev vlan3 proto kernel metric 256 expires 86387sec pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev vlan3 proto kernel metric 256 pref medium
default via fe80::216:17ff:fea7:f2d3 dev vlan3 proto ra metric 1024 expires 47sec hoplimit 64 pref medium
default via fe80::1:1 dev eth0 proto ra metric 1024 expires 47sec hoplimit 64 pref medium