It's disabled when the configuration format is different between them, as marked by the "<version>XX.Y</version>" in config.xml

If the configuration version is different, they cannot sync because it could push incorrect data.

That said, synchronizing between different versions has never been officially supported, nor recommended. It may have worked by chance before, but we never recommend running different versions for any measurable amount of time. Just long enough to make sure the updated node is functional/tested, which shouldn't involve any configuration changes.

Any blog post or diagram should tell you to get a /29. That is how it is done. Anything worth HA is worth doing right, IMHO.

And you cannot use Automatic outbound NAT with CARP/HA. It must be manual to the CARP VIP is used there.

Try these:

net.inet.igmp.default_version=2
net.inet.igmp.sendra=0
net.inet.igmp.legacysup=1

I have resolved the issue, it appears I was hitting a change in pfsync as of pfsense 2.2 as shown here

https://forum.pfsense.org/index.php?topic=93052.0
https://forum.pfsense.org/index.php?topic=93132.msg519077#msg519077

Since I was using VMXNET3 interfaces in ESXi and VirtIO interfaces in Proxmox they show up as different hardware since they have different drivers and pfsync cannot function properly.  The work around in the previous threads was to create a LAGG but the simpler solution in this case was to change Proxmox to use VMXNET3 interfaces and my states are synced perfectly now.  Changing both VMs to use E1000 interfaces likely would have worked too.

It works for me in Chrome here on 2.3.3. Make sure you clear the browser cache (ctrl+F5, or shift+reload) between tests.

I tried switching between each of the possible VIP modes and the correct fields were enabled each time.

Solved!

On the controller's firewall i had to disable "Prohibit IP Spoofing".

@Elegant
You already have Dial on Demand.
What I have read here http://sirlagz.net/2014/12/22/pfsense-carp-and-pppoe/ and here http://theartofservice.com/pfsense-carp-and-pppoe.html is that you also need to disable the Gateway monitoring. Then PPPoE will only be created on 2nd Firewall when 1st one goes down.
I have not done this myself though yet.

@KOM:

A port-forward / 1:1 NAT must be mapped to something.  In your case you would map it to the local IP of the CentOS box.  That's how it works.  And as I mentioned earlier, if you're uncomfortable forwarding traffic to a host on your LAN, then create a DMZ via new interface or VLAN and move your CentOS box there.

Ok, so I will run a physical ethernet cable between my OPT1 interface and a physical interface on my ESXi Server. I'll assign that interface to CENTOS within the ESXI Controller.

What will my configuration look like in PFsense?

Thanks viragomann  ;)

Does WAN stay MASTER/BACKUP or is it always MASTER/MASTER?

There's not a lot to happen there. Setting CARP maintenance mode hard sets the advskew to 254 and if the primary receives a more recent advert from the backup it will go into a BACKUP state on that VIP, likewise if the backup does not receive a more recent advert from the primary it will go MASTER.

Running a mismatched pair can be challenging and is not recommended. But this should work. State sync is another matter.

How are the WAN ports and the DCGW physically and virtually connected? Do both WAN ports see the CARP adv traffic like they should? (Packet Capture on CARP.)

I too am receiving this error that I believe is related to PHP-FPM

Sometimes if you restart php-fpmd service in the secondary node the SYNC completes. I sometimes need to reboot the FW in order to get the config syncronized.

My issue of the unexplained ignoring of the MAC address bypass list was solved by upgrading to a development snapshot (2.3.4 something) but in a failover to the backup router, which still had 2.3.2-Release-p1, the problem still existed, so we upgraded that to whatever snapshot was available that day. As soon as the 2.3.3 maintenance release is out we'll try that and see whether the problem comes back or not.

The problem of CARP LAN addresses not responding to workstation ARP requests has not been solved. (Summary: we change static IP address on interface to something else, create CARP virtual IP on LAN with the original IP address, at which point workstations on the network cannot see the CARP IP address. Using a packet sniffer we observe that the router's LAN interface is not responding to ARP requests by workstations for the gateway IP address. Rebooting the router and every other device on the network, including switches and workstations, did not solve the problem.)  We will try that again after the maintenance release as well.

thanks

Does it see the advertisements from the primary before you add the VIP?

Does the primary see those advertisements from the secondary?

It is not generally correct to add a CARP VIP to the secondary. You add it to the primary and it XMLRPC syncs over to the secondary with the proper advbase/advskew.

If you add it to the secondary manually and there is not a 1/0 skew VIP already on the network, of course it will assume MASTER.

Tested what you reported on a fairly-current 2.4-BETA VM pair:

Added VIP 172.25.236.65 on Secondary only:

xn0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
options=3 <rxcsum,txcsum>ether 12:77:26:96:5d:a3
inet6 fe80::1077:26ff:fe96:5da3%xn0 prefixlen 64 scopeid 0x5
inet6 2001:470:f00e:7e01::3 prefixlen 64
inet6 2001:470:f00e:7e01::1 prefixlen 64 vhid 239
inet 172.25.236.3 netmask 0xffffff00 broadcast 172.25.236.255
inet 172.25.236.1 netmask 0xffffff00 broadcast 172.25.236.255 vhid 236
inet 172.25.236.65 netmask 0xffffff00 broadcast 172.25.236.255 vhid 241
nd6 options=21 <performnud,auto_linklocal>media: Ethernet manual
status: active
carp: BACKUP vhid 236 advbase 1 advskew 100
carp: BACKUP vhid 239 advbase 1 advskew 100
carp: MASTER vhid 241 advbase 1 advskew 100

Deleted same:

xn0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
options=3 <rxcsum,txcsum>ether 12:77:26:96:5d:a3
inet6 fe80::1077:26ff:fe96:5da3%xn0 prefixlen 64 scopeid 0x5
inet6 2001:470:f00e:7e01::3 prefixlen 64
inet6 2001:470:f00e:7e01::1 prefixlen 64 vhid 239
inet 172.25.236.3 netmask 0xffffff00 broadcast 172.25.236.255
inet 172.25.236.1 netmask 0xffffff00 broadcast 172.25.236.255 vhid 236
nd6 options=21 <performnud,auto_linklocal>media: Ethernet manual
status: active
carp: BACKUP vhid 236 advbase 1 advskew 100
carp: BACKUP vhid 239 advbase 1 advskew 100</performnud,auto_linklocal></rxcsum,txcsum></up,broadcast,running,promisc,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum></up,broadcast,running,promisc,simplex,multicast>

@Smoothrunnings:

a. If what you are saying the WAN on both firewalls have their own public IPs, then how does the secondary assume the role of the primary when fail-over occurs?

Thanks

The secondary gains control of the Virtual (CARP) IPs, the LAN side and the Public side. If this isn't clear, please review the CARP man page, the HA documentation, etc. I feel like this discussion is going in circles.

Found my issue…

I had to put the VIP ip as the dns server in my dhcp server option.