CARP chosen instead of VirtualIP

xeba

On my LAN interface I configured the following IPs:

10.0.3.252/16
10.1.1.2/24 (Virtual IP)

I also defined a CARP and a Virtual IP on top of it:
3) 10.0.3.101/16 (CARP)
4) 10.1.1.253/24 (Virtual IP on top of CARP)

What happens on the secondary node, is that when i try to reach the 10.1.1.0/24 subnet, address 4 is used insetad of address 2.
Outbound NAT is not involved, I suspect that the problem is the ordering of the ip addresses assigned to the interface:

em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 52:54:00:83:46:57
	hwaddr 52:54:00:83:46:57
	inet6 fe80::5054:ff:fe83:4657%em1 prefixlen 64 scopeid 0x2 
	inet 10.0.3.252 netmask 0xffff0000 broadcast 10.0.255.255 
	inet 10.1.1.253 netmask 0xffffff00 broadcast 10.1.1.255 vhid 2 
	inet 10.0.3.101 netmask 0xffff0000 broadcast 10.0.255.255 vhid 2 
	inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	carp: BACKUP vhid 2 advbase 1 advskew 100

The reason why I say this is that after a reboot, the ip order is actually different and I can reach the subnet:

em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 52:54:00:83:46:57
	hwaddr 52:54:00:83:46:57
	inet6 fe80::5054:ff:fe83:4657%em1 prefixlen 64 scopeid 0x2 
	inet 10.0.3.252 netmask 0xffff0000 broadcast 10.0.255.255 
	inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255 
	inet 10.0.3.101 netmask 0xffff0000 broadcast 10.0.255.255 vhid 2 
	inet 10.1.1.253 netmask 0xffffff00 broadcast 10.1.1.255 vhid 2 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	carp: BACKUP vhid 2 advbase 1 advskew 100

After a while the order changes (XMLRPC Sync??) and the issue appears again.

I can probably work around with some Outbound NAT or a dedicated interface, but wouldn't it be better if CARP ips were always placed last?

johnpoz

Why would you want to run multiple layer 3 on the same layer 2? Its a Borked Config right out of the gate - are you in the middle of migration from that Huge /16 that makes zero sense to the more reasonable /24?