IPv6 CARP Dual Master

davidredekop · Apr 29, 2023, 1:02 AM

I have been enjoying HA using pfSync and CARP interfaces for a long time. In my lab where I run one LAN segment (+ WAN) fully dual-stacked mode, however, I've never been able to resolve the dual MASTER situation with an IPv6 CARP interface.

However, the same physical interface with IPv4 HA and CARP works exactly as expected.

Running pfSense+ 23.01. Here's my issue in the GUI:

I am using IPv6 ULAs on my LAN as the ISP here offers only a /128 address on each of my CPE-attached connections.

The switch between them does not filter broadcast traffic to ff02::12

The ix1 LAN interface on my first node is:

ix1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: LAN
	options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
	ether 00:90:0b:a0:fa:9a
	inet6 fe80::290:bff:fea0:fa9a%ix1 prefixlen 64 scopeid 0x2
	inet6 fc00::10:1 prefixlen 64
	inet6 fc00::10:3 prefixlen 64 vhid 44
	inet 192.168.99.251 netmask 0xffffff00 broadcast 192.168.99.255
	inet 192.168.99.1 netmask 0xffffff00 broadcast 192.168.99.255 vhid 1
	carp: MASTER vhid 1 advbase 1 advskew 0
	carp: MASTER vhid 44 advbase 1 advskew 0
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

The vtnet1 LAN interface on my second node is:

vtnet1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=900b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,LINKSTATE>
        ether 9a:54:a6:c8:05:fc
        inet6 fe80::9854:a6ff:fec8:5fc%vtnet1 prefixlen 64 scopeid 0x2
        inet6 fc00::10:2 prefixlen 64
        inet6 fc00::10:3 prefixlen 64 vhid 44
        inet 192.168.99.252 netmask 0xffffff00 broadcast 192.168.99.255
        inet 192.168.99.1 netmask 0xffffff00 broadcast 192.168.99.255 vhid 1
        carp: BACKUP vhid 1 advbase 1 advskew 100
        carp: MASTER vhid 44 advbase 1 advskew 100
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

I don't know if this is relevant but when running a packet capture I don't see any traffic from the ULA addresses, only from link-local ones like this:

/root: tcpdump -i ix1 host ff02::12
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ix1, link-type EN10MB (Ethernet), capture size 262144 bytes
16:15:08.222211 IP6 fe80::9854:a6ff:fec8:5fc > ff02::12: ip-proto-112 36
16:15:08.304765 IP6 fe80::290:bff:fea0:fa9a > ff02::12: ip-proto-112 36

/root: tcpdump -i vtnet1 host ff02::12
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet1, link-type EN10MB (Ethernet), capture size 262144 bytes
16:15:01.104108 IP6 fe80::9854:a6ff:fec8:5fc > ff02::12: ip-proto-112 36
16:15:02.504098 IP6 fe80::9854:a6ff:fec8:5fc > ff02::12: ip-proto-112 36

Am I missing something obvious?

SteveITS · Apr 29, 2023, 1:02 AM

@davidredekop sorry to rtfm but did you find https://docs.netgate.com/pfsense/en/latest/troubleshooting/high-availability.html#both-nodes-appear-as-master ?

davidredekop · Apr 29, 2023, 4:07 PM

@steveits said in IPv6 CARP Dual Master:

https://docs.netgate.com/pfsense/en/latest/troubleshooting/high-availability.html#both-nodes-appear-as-master

haha yes I've exhausted every detail and I keep on coming back to the section that reads:

Both Nodes Appear as MASTER: This will happen if the secondary node cannot see the CARP hearbeat advertisements from the primary.

And sure enough it seems like that's at the root of it when I compare packet capture on both node1 and node2:

node1 shows this pattern:

11:55:20.623797 IP6 fe80::290:bff:fea0:fa9a > ff02::12: ip-proto-112 36
11:55:20.791227 IP6 fe80::9854:a6ff:fec8:5fc > ff02::12: ip-proto-112 36
11:55:21.624873 IP6 fe80::290:bff:fea0:fa9a > ff02::12: ip-proto-112 36
11:55:22.188119 IP6 fe80::9854:a6ff:fec8:5fc > ff02::12: ip-proto-112 36

It sees both gateway1 and gateway2 advertisements.

node2 shows this pattern:

11:53:37.422535 IP6 fe80::9854:a6ff:fec8:5fc > ff02::12: ip-proto-112 36
11:53:38.822564 IP6 fe80::9854:a6ff:fec8:5fc > ff02::12: ip-proto-112 36
11:53:40.222515 IP6 fe80::9854:a6ff:fec8:5fc > ff02::12: ip-proto-112 36
11:53:41.622534 IP6 fe80::9854:a6ff:fec8:5fc > ff02::12: ip-proto-112 36

No sign of receiving node1's advertisements.

Must be at the proxmox networking layer then, it seems.

davidredekop · Apr 29, 2023, 4:14 PM

Sometimes just articulating the problem in writing helps me solve my own issues more methodically.

On further investigation, looks like Proxmox applies Multicast snooping by default, that's why node2 could never receive the advertisement even as a linux bridge.

So the following setting turns off the Multicast snooping:

echo 0 > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping

After the reboot, I now have proper MASTER/BACKUP on the IPv6 also.
Good to know now how IPv4 differs so much from IPv6 :)

Derelict · Apr 30, 2023, 11:25 AM

@davidredekop Interesting. I have never had to change anything in proxmox for CARP.

As an aside, while fc00::/7 is the ULA network space, fc00::/8 is currently undefined. fd00::/8 is proper ULA addressing. Recommend implementing RFC 4193 and randomly selecting a /48 for ULA usage.