CARP VIP at single pfSense (1.2.3) fails to BACKUP constantly

[tata_tulen]

Hi all,

We've problem with pfSense VM on VMware. We're using this scenario for a log time with no problem, until now. When I create CARP VIP, it fails to BACKUP state immediately (sometimes one or two ICMP echo replies come).

I'm really messed up and do not know where to start investigate, all settings on the others pfSense boxes (physical or in VMware) seems to be the same…

I'll appreciate any hint...
-tt-

Some config informations follow:

# ifconfig -a
le0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=8 <vlan_mtu>ether 00:50:56:8e:49:d1
	inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
	inet6 fe80::250:56ff:fe8e:49d1%le0 prefixlen 64 scopeid 0x1 
	media: Ethernet autoselect
	status: active
le1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=8 <vlan_mtu>ether 00:50:56:8e:2e:49
	inet x.y.58.61 netmask 0xffffffe0 broadcast x.y.58.63
	inet6 fe80::250:56ff:fe8e:2e49%le1 prefixlen 64 scopeid 0x2 
	media: Ethernet autoselect
	status: active
le2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=8 <vlan_mtu>ether 00:50:56:8e:1a:e7
	inet 192.168.254.1 netmask 0xffffff00 broadcast 192.168.254.255
	inet6 fe80::250:56ff:fe8e:1ae7%le2 prefixlen 64 scopeid 0x3 
	media: Ethernet autoselect
	status: active
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 
enc0: flags=41 <up,running>metric 0 mtu 1536
pfsync0: flags=41 <up,running>metric 0 mtu 1460
	pfsync: syncdev: lo0 syncpeer: 224.0.0.240 maxupd: 128
pflog0: flags=100 <promisc>metric 0 mtu 33204
carp0: flags=8 <loopback>metric 0 mtu 1500
	carp: INIT vhid 57 advbase 1 advskew 0</loopback></promisc></up,running></up,running></up,loopback,running,multicast></vlan_mtu></up,broadcast,running,simplex,multicast></vlan_mtu></up,broadcast,running,promisc,simplex,multicast></vlan_mtu></up,broadcast,running,simplex,multicast> 

(le0 is LAN iface, le1 is WAN iface, le2 is OPT1 iface, i'm trying to create VIP on WAN iface)

net.inet.ip.same_prefix_carp_only: 0
net.inet.carp.allow: 1
net.inet.carp.preempt: 1
net.inet.carp.log: 2
net.inet.carp.arpbalance: 0
net.inet.carp.drop_echoed: 0
net.inet.carp.suppress_preempt: 0

When I create the VIP, in the Log I can see this (log entries are reversed):

Mar 28 17:23:17 	kernel: carp0: link state changed to DOWN
Mar 28 17:23:17 	kernel: carp0: 2 link states coalesced
Mar 28 17:23:17 	kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
Mar 28 17:23:17 	check_reload_status: reloading filter
Mar 28 17:23:14 	kernel: carp0: link state changed to DOWN
Mar 28 17:23:14 	kernel: carp0: 2 link states coalesced
Mar 28 17:23:14 	kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
Mar 28 17:23:14 	kernel: carp0: INIT -> MASTER (preempting)
Mar 28 17:23:14 	kernel: carp0: link state changed to DOWN
Mar 28 17:23:14 	kernel: carp0: link state changed to DOWN
Mar 28 17:23:14 	kernel: carp0: 2 link states coalesced
Mar 28 17:23:14 	kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
Mar 28 17:23:14 	kernel: carp0: INIT -> MASTER (preempting)
Mar 28 17:23:14 	kernel: carp0: link state changed to DOWN
Mar 28 17:23:14 	kernel: carp0: link state changed to DOWN
Mar 28 17:23:14 	kernel: carp0: 2 link states coalesced
Mar 28 17:23:14 	kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
Mar 28 17:23:14 	kernel: carp0: INIT -> MASTER (preempting)
Mar 28 17:23:14 	kernel: le1: promiscuous mode enabled

When I disable and re-enable CARP, I can see this in the Log

Mar 28 17:50:00 	kernel: carp0: link state changed to DOWN
Mar 28 17:50:00 	kernel: carp0: 2 link states coalesced
Mar 28 17:50:00 	kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
Mar 28 17:49:57 	kernel: carp0: link state changed to DOWN
Mar 28 17:49:57 	kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
Mar 28 17:49:55 	kernel: carp0: link state changed to UP

This is the tcpdump of vrrp messages:

# tcpdump -en -i le1 'vrrp'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on le1, link-type EN10MB (Ethernet), capture size 96 bytes
17:55:22.672646 00:00:5e:00:01:39 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: x.y.58.61 > 224.0.0.18: VRRPv2, Advertisement, vrid 57, prio 0, authtype none, intvl 1s, length 36
17:55:22.672769 00:00:5e:00:01:39 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: x.y.58.61 > 224.0.0.18: VRRPv2, Advertisement, vrid 57, prio 0, authtype none, intvl 1s, length 36
17:55:25.682679 00:00:5e:00:01:39 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: x.y.58.61 > 224.0.0.18: VRRPv2, Advertisement, vrid 57, prio 0, authtype none, intvl 1s, length 36
17:55:25.682806 00:00:5e:00:01:39 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: x.y.58.61 > 224.0.0.18: VRRPv2, Advertisement, vrid 57, prio 0, authtype none, intvl 1s, length 36

# tcpdump -en -i carp0 'vrrp'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on carp0, link-type NULL (BSD loopback), capture size 96 bytes
17:55:34.714166 AF IPv4 (2), length 60: x.y.58.61 > 224.0.0.18: VRRPv2, Advertisement, vrid 57, prio 0, authtype none, intvl 1s, length 36
17:55:37.724078 AF IPv4 (2), length 60: x.y.58.61 > 224.0.0.18: VRRPv2, Advertisement, vrid 57, prio 0, authtype none, intvl 1s, length 36

[tata_tulen]

Whoa, I knew it's not problem of pfSense. My co-worker had done mistake in ESX advanced configuration - the 'Net.ReversePathFwdCheckPromisc' parameter must have the value of '1'.