Link-local address behavior when spoofing WAN interface MAC address

[A Former User]

My general goal was to have multiple pfSense hardware devices to be used for upgrades. Visit a site, swap out the existing unit (“old device”) with a pre-configured device (“new device”) instead of performing in-place upgrades. Limits downtime and allows bench testing of some aspects of the new device prior to installation. One thing difficult to test is the WAN behavior with IPv6 and Comcast, so the issue described below was tested from the same site with two devices.

IPv6 is configured on the WAN via DHCP6, with prefix delegation size of /60; and prefix hint. The network connection is residential Comcast/Xfinity with an Arris S33 cable modem. MAC address is overridden (spoofed) on the WAN interface of the new device to match the MAC address of the old device.

DHCP6 DUID is set in System/Advanced/Networking (DUID-LLT, with Link-layer address matching the spoofed WAN Mac).

LAN interface is set to track WAN, IPv6 Prefix ID=0.

Both the old and new devices are Netgate 5100s. The new device has been configured from a backup/restore of the old device. During troubleshooting, the new device has also been configured by hand with essentially no changes other than to the WAN/LAN interface configuration and DUID.

On the old device (running pfSense 2.4.5), IPv6 is working properly. An address for the WAN and the prefix for LAN are obtained via DHCP6.

When switching to the new device (pfSense Plus 21.05.2), IPv6 doesn’t work. Looking at a packet capture, a response to the DHCP6 solicit is never received from Xfinity.

After many iterations of testing, my conclusion is that Xfinity seems to respond to a DHCPv6 Solicit message only when the IPv6 source address is consistent with what they expect. The source address in this context is the link-local address of the WAN interface.

Because I have the WAN interface MAC spoofed, I would have expected the IPv6 link-local address to change and be derived from the spoofed MAC. Unfortunately, it seems that under most conditions, the IPv6 link-local address for the WAN doesn’t respect the spoofed MAC, and instead is derived from the hwaddr of the NIC.

For example, on the WAN interface with a spoofed MAC specified and IPv6 Configuration Type set to DHCP6, upon reboot, the Link-local address is derived from the hwaddr

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
     options=e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:90:0b:18:ac:ca
        hwaddr 00:90:0b:18:ab:cd
        inet6 fe80::290:bff:fe18:abcd%igb0 prefixlen 64 scopeid 0x1
        inet 192.168.100.22 netmask 0xffffff00 broadcast 192.168.100.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

Interestingly, if the WAN IPv6 Configuration Type set to None, upon reboot, the Link-local address is derived from the spoofed address (ether)

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:90:0b:18:ac:ca
        hwaddr 00:90:0b:18:ab:cd
        inet6 fe80::290:bff:fe18:acca%igb0 prefixlen 64 scopeid 0x1
        inet 192.168.100.22 netmask 0xffffff00 broadcast 192.168.100.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

Note: In these examples, the IPv4 address has been replaced with a placeholder value of 192.168.100.22/24, in the real world this would be a DHCP assigned internet-routable address. MAC addresses were slightly changed but still demonstrate the issue.

The inconsistent behavior with spoofed Mac addresses and Link-local addresses seems like a bug. Ideally, I would want it to always derive from the spoofed address (ether). Also, it’s not clear from the GUI that once setting a MAC address on an interface, there will be a change in behavior from the running state and a reboot that will impact the IPv6 link-local address. I’ve resorted to assuming that a reboot will be required if changing the MAC address field.

Ultimately, I am working around this issue with the shellcmd package and specifying an “earlyshellcmd” to configure the Link-local address that corresponds to the spoofed MAC, which appears to run before pfSense configures the interface with the incorrect link local address.

ifconfig igb0 inet6 fe80::290:bff:fe18:acca%igb0 prefixlen 64 alias

With this change, the DHCP6 solicit message comes from the same source address on the new device, and I receive a response from Xfinity with a prefix delegation. It is unclear how fragile this workaround will be.

It is possible that not specifying the DUID and not spoofing the WAN MAC is the expected configuration, but that didn’t really fit with my goals of device swap and preserving the ability to revert back to the old device without IPv6 addressing being impacted.

Mostly posting this in the event someone else finds themself in this situation, as it was non-obvious as to why the same IPv6 configuration was not working between devices. If there is any consensus that this behavior is a pfSense bug, I will open an issue.

[JKnott]

@oranges53

Take a close look at the spoofed address. Is it exactly the same as the original. There's one bit in the MAC to indicate locally assigned addresses. Is it the same for both NICs?

[A Former User]

@jknott I changed the MAC addresses in the output as I didn't want to post the actual MAC addresses, but I don't believe that changes the effective situation. In the real systems, both have the same first 3 octets (OUI) as they are the same vendor NIC, but the device specific last 3 octets of the MAC are unique and totally different.

Edit: I should clarify, the spoofed mac of the WAN interface on the "new" device is the physical hwaddr of the WAN NIC in the "old" device.

[JKnott]

@oranges53

When you capture a frame and examine it with Wireshark, does it show the exact same value for the U/L bit for each device? That bit is supposed to change when you override the burned in address. You might have to clone both NICs with the same value.

[A Former User]

@jknott

Not sure I'm fully understanding the significance of the U/L bit in this context. Both the old device and new device MAC addresses are the true device addresses of the respective hardware, so they would both be UAA. When setting the new device to spoof the MAC of the old device on the WAN interface, it doesn't change the nature of the address. In order for the U/L bit to reflect a locally administered address, the MAC would have to necessarily change from what I need it to be - I don't think this is related to the issue that I am encountering.

The Ethernet frame in both situations described in the original post are correctly sourced from the spoofed MAC address. So the spoofing at the Ethernet layer is working correctly.

The issue is with the auto configuration of a IPv6 link-local address. When WAN is DHCP6, it seems to be auto-configured based on the hardware address (hwaddr) instead of the spoofed address. So, when the DHCP6 solicit message does not receive a response, the DHCP6 packet has an IPv6 source address based on the hwaddr MAC.

When I influence the link-local address to be derived from the spoofed address (either by the ipconfig command at boot, or just by booting with IPv6 disabled on WAN and then reconfigured to DHCP6 without rebooting), the DHCP6 packet has an IPv6 source address that is expected (and derived not from the hwaddr MAC, but the spoofed MAC)

[JKnott]

@oranges53

The significance is if you override the original MAC, by copying another, that bit is supposed to indicate that it is a locally assigned address, not the one from the manufacturer. This is part of the Ethernet spec, so even if you think you are writing the same address, it may be different, by that bit, on the wire. The way you can see that is to do a packet capture from both NICs and compare that bit. I suspect that even if every other bit is the same, that U/L bit will show you changed the MAC on one card but not on the other. The way to get them to match would be to write the same address to both and both should then show it's a locally assigned address.

Here is an example. I just copied a frame from my network:

Ethernet II, Src: Giga-Byt_5b:f5:fa (74:d4:35:5b:f5:fa), Dst: Gifa_12:b6:6c (40:62:31:12:b6:6c)

The U/L bit is the 6th bit from the left. In the source MAC, the first byte is 74 and the 6th bit of that byte is a 0. If it was a locally assigned address, then that first byte would be 76. Likewise with the destination MAC. The first byte is 40 and it would be 42 in a locally assigned address.

However, there is some ambiguity here. IP uses Ethernet II frames, and the original DIX frame doesn't seem to define that bit, though the 802.3 frame does and Ethernet is now part of the 802.3 spec, with the Ethernet II frame.

Anyhow, do a packet capture and take a look at that bit. If it changes with a locally assigned address, then you have your answer.

[JKnott]

@oranges53

I was just thinking there may be another issue, the UUID. This is what determines the IPv6 ID to the DHCPv6 server.

[A Former User]

@jknott Thanks for your responses.

I’m certain that the U/L bit is not relevant to this situation described when spoofing a MAC address that is global (UAA). When spoofing, the U/L bit is unchanged, as by definition, changing that bit changes the MAC address entirely.

As mentioned, the old and new WAN device hardware MACs are UAA (universally administered address).

For example, given the old device WAN Mac = 00-90-0B-18-AC-CA

Representing that address as bits in network order as is used by Ethernet frames:

00000000-10010000-00001011-00011000-10101100-11001010

If the U/L bit is changed from 0 (UAA) to 1 (LAA), as in the following (7th bit from the left):

00000010-10010000-00001011-00011000-10101100-11001010

Converting that back to a MAC address (EUI-48) format results in: 02-90-0B-18-AC-CA

Obviously, 00-90-0B-18-AC-CA (old device MAC) does not equal 02-90-0B-18-AC-CA.

The Ethernet frames for the DHCPv6 traffic (and indeed, all traffic) on the old device WAN interface and the new device WAN interface when set to spoof 00-90-0B-18-AC-CA will always have a source address correctly of 00-90-0B-18-AC-CA.

The spoofing of a MAC address has no direct correlation to the U/L bit, and setting the U/L bit will have a predictable effect on the value of the MAC address, as described in this table. There is no prohibition within the OS to use a LAA when spoofing, and if spoofing a UAA, there is a responsibility not to introduce duplicate MACs on the network that would conflict with another node. So, in my case, the old device will never be on the network at the same time as the new device.

Additionally, the DHCP unique identifier (DUID) is identical between the two devices, as this was configured within the pfSense GUI as described on the original post. There are four types of DUID of which DUID-UUID is one, but that is not used in my case. The original device had a DUID-LLT set on the old device and I have preserved the same value on the new device. Packet captures from both devices confirm that the DUID used in the DHCPv6 solicit are the same.

To attempt to more succinctly describe the underlying issue, here is a more detailed description:

WAN interface hardware address of new device: 00-90-0B-18-AB-CD
WAN interface hardware address of old device: 00-90-0B-18-AC-CA

IPv6 link-local addresses are created by the OS based on the MAC, following the “Links or Nodes with IEEE 802 48-bit MACs” section of “Appendix A: Creating Modified EUI-64 Format Interface Identifiers” in RFC4291

On the new device, WAN interface MAC is spoofed to 00-90-0B-18-AC-CA and IPv6 is set to none. System is rebooted. IPv6 link local address on the wan is fe80::290:bff:fe18:acca%igb0

This is the correct, expected behavior. But, obviously, not helpful since IPv6 is not configured.

Upon changing the WAN interface IPv6 to DHCP6 with the appropriate parameters, upon reboot, the IPv6 Link local address on the WAN is fe80::290:bff:fe18:abcd%igb0

The spoofed Mac of 00-90-0B-18-AC-CA is correctly set. What has happened is that perhaps due to sequencing of when the link local address is established, the hardware Mac of the new device (00-90-0B-18-AB-CD) was used to generate the link-local address which is not correct. The spoofed MAC (00-90-0B-18-AC-CA) should have been used for the link-local address generation to be consistent.

[A Former User]

I did some additional testing on a third Netgate SG-5100. This revealed mistakes in my original post's scenario. Since I'm unable to edit the original post, the following are the updates:

In addition to how my original scenario was described, I also had an "IP Alias" type configured under Virtual IPs for WAN. Without this present, the IPv6 link-local address seems to be configured properly under both: (a) WAN IPv6 Configuration Type set to None; and (b) WAN IPv6 Configuration Type set to DHCPv6.

To correct my original post:

The WAN interface obtains IPv4 address by DHCP of 203.0.113.11/24
A Virtual IP (Firewall | Virtual IPs) of type IP alias is configured for WAN 192.168.100.24/24

In this configuration, when WAN IPv6 Configuration Type set to None, upon reboot, the Link-local address (fe80::290:bff:fe18:acca%igb0) is derived from the spoofed address (ether=00:90:0b:18:ac:ca)

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:90:0b:18:ac:ca
        hwaddr 00:90:0b:18:ab:cd
        inet6 fe80::290:bff:fe18:acca%igb0 prefixlen 64 scopeid 0x1
        inet 203.0.113.11 netmask 0xffffff00 broadcast 203.0.113.255
        inet 192.168.100.22 netmask 0xffffff00 broadcast 192.168.100.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

When WAN IPv6 Configuration Type set to DHCP6, upon reboot, the Link-local address (e80::290:bff:fe18:abcd%igb0) is derived from hwaddr(00:90:0b:18:ab:cd) incorrectly:

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
     options=e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:90:0b:18:ac:ca
        hwaddr 00:90:0b:18:ab:cd
        inet6 fe80::290:bff:fe18:abcd%igb0 prefixlen 64 scopeid 0x1
        inet 203.0.113.11 netmask 0xffffff00 broadcast 203.0.113.255
        inet 192.168.100.22 netmask 0xffffff00 broadcast 192.168.100.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

Again, these mac addresses and IPs are for demonstration purposes. The spoofed (ether) and hwaddr MACs are both from SG-5100 NICs. The IP addresses are changed, but notably different than my first post. The 203.0.113.11/24 address is the internet routable address, and the 192.168.100.22/24 address is the configured virtual IP (ip alias).

In summary, it appears that having an IP alias assigned to WAN changes the behavior of how the IPv6 link-local address is defined when the MAC address is spoofed.

[JKnott]

@oranges53 said in Link-local address behavior when spoofing WAN interface MAC address:

In summary, it appears that having an IP alias assigned to WAN changes the behavior of how the IPv6 link-local address is defined when the MAC address is spoofed.

I wonder what causes that difference. Also, no need to hide the MAC, as it's irrelevant beyond the local LAN. While the MAC is often used for the consistent address, any outgoing connection would use a privacy address, which is based on a random number.

While not the cause of the problem, I am curious about that U/L bit. As I mentioned, it is specified for 802.3, but not DIX/Ethernet II. You can have both on the same interface, as was common years ago, when IPX and NetBIOS were popular and used on the same network as IP. Does that bit change depending on whether you're using Ethernet II or 802.3? Back when I was at IBM, the first time, we had NetBIOS, IP and SNA on the same network, but since that was a Token Ring (802.5) network, it was closer to 802.3 than Ethernet II.

[A Former User]

Also, no need to hide the MAC, as it's irrelevant beyond the local LAN. While the MAC is often used for the consistent address, any outgoing connection would use a privacy address, which is based on a random number.

Given that my real MAC would be known to Comcast/Xfinity (this is a WAN interface), there is definitely a reason to hide the MAC from this forum for privacy reasons -- perhaps the threat model you have in mind is not consistent with mine. The IPv6 link local address derived from the interface MAC by pfSense does not use RFC 4941 privacy extensions, as shown in various places in this thread the IPv6 link-local address is derived from the true hardware MAC address on the NIC.

While not the cause of the problem, I am curious about that U/L bit. As I mentioned, it is specified for 802.3, but not DIX/Ethernet II. You can have both on the same interface, as was common years ago, when IPX and NetBIOS were popular and used on the same network as IP. Does that bit change depending on whether you're using Ethernet II or 802.3? Back when I was at IBM, the first time, we had NetBIOS, IP and SNA on the same network, but since that was a Token Ring (802.5) network, it was closer to 802.3 than Ethernet II.

The U/L bit is best clarified by the IEEE guidance (page 6-7), at least in the context of modern Ethernet use. In the context of a EUI-48 (MAC) address, U/L=0 means the IEEE maintains the "globally unique" nature of the address. U/L=1 means the address is locally assigned, which is now most common with mobile devices and leveraged by Extended Local Identifier (ELI) defined in the IEEE spec.

As mentioned, the U/L bit is not relevant to any situation with spoofing the MAC on pfSense (or any other platform), as the spoofed address would need to be one that sets the U/L bit. And the general idea of spoofing an address is to act as a real address, so as such U/L=1 in that context would be irregular in the context of my original post.

[A Former User]

This issue is reproduced by Netgate support and tracked as https://redmine.pfsense.org/issues/12790