@Derelict: I'd probably start here.: https://doc.pfsense.org/index.php/How_can_I_forward_ports_with_pfSense%3F Thats perfect.  thanks.

I've seen this behavior under several circumstances: when there is a carp misconfiguration; be very careful about the VHID, it must be unique for each virtual IP when there is something filtering CARP traffic between the nodes when there is leakage between the virtual IPs (eg: lan and dmz can see each other on layer 2).

@JeGr: @Rob Hate to disappoint you, while the main problem is indeed fixed (no aliases were created with 2.1.4 anymore), there still is a bug with deleting said aliases. They won't get deleted on the backup node, thus bringing chaos to the CARP stack on that interface leading to a split-brain (master/master) situation on that interface (can be resolved by rebooting the standby node or manually deleting the aliases on the VIP interface in a root shell on console). So my advice: be careful. Interesting! Do you know how long this bug has stood for, we've always had interesting behaviour with CARP + VIPs and failovers; we've always ended up rebooting the secondary for "random" problems like these.

@secgeek: Don't know what you are trying to say, but it doesn't make sense. @pedreter: we run a datacenter firewall cluster with HA and carp on multiple physical interfaces (6) with about a dozen VIPs on our second WAN link and minimum one on the remaining links (WAN, XFER, MGMT, etc.). No problem with that. Seems like a configuration issue to me rather than any problem with CARP or IPs. Greets

Good news! Everything is working. The settings were correct as you mentioned, except for one thing under Virtual IPs: The source interface was set to WAN instead of LAN. Changing it to LAN let me have both 20.0/24 and 11.0/24 subnets on the same LAN.

You will need an IP for each physical box, so you are only going to have three IPs that will fail over. Lets say .249 is the gateway, 250 could be one firewall, 251 the second, leaving you with 252, 253, and 254. You might be able to share IPs using port forwards and have enough. You can terminate multiple IPSec tunnels on one CARP VIP.

If you use CARP VIPs, you will have to add an Alias IP on each firewall or you will get an error about no matching subnet. Another option is to use OTHER VIPs, which may work depending on how the provider is routing the block to you.

You likely need to enable MAC address spoofing on the two VMs.

Welp, as it turns out, the Motorola/Arris box that they gave me doesn't require sticky statics - It's strictly optional.  So I can continue using PFSense as a filtered bridge without any worries. :)

This patch to rc.carpmaster and rc.carpbackup is still necessary on 2.1.4 to stop/start quagga on carp role change right? It looks like some logic was added to determine whether or not to start quagga based on role at startup, but the patch is still required for failover.

Do you have your interfaces assigned in the same order on both pfSense in Interfaces > assign? This is essential for syncing correctly.

For the record, I've reproduced this on the firewalls in the office here; the patch appears to solve it. However I'm still totally confused as to how it's working at all When the bug is present the IP address of the Alias (lets say .21) isn't assigned to any of the interfaces at all. Yet if I do curl https://x.x.x.21 then I can get through the NAT and the firewall to the webserver underneath. Why is PFSense responding on an IP it doesn't own!?

Yes, if you want to fail-over you need to make a CARP address on every network/vlan. With the openVPN, bind the server to the CARP address, that way it'll work when  the secondary has taken-over

yeah it is working well, this is my bad, could have check the switch port . TIA

Viragomann - I appreciate your quick response.  "…Add all other network as IP Alias with their real /24 mask..." did the trick.  I was trying to add them with a /32 mask which was not working.  Again - Thanks.

That isn't currently possible. We've had some people try this with multiple NICs and they've had mixed success. Eventually it may be possible via netgraph virtual interfaces but that's a long way off yet.

Hi ! problem is solved : it was a bonding issue, but not regarding pfSense. The pfSense CARP cluster is linked to a couple of Juniper Virtual Chassis switches without any bonding. But  a cluster of Juniper SRX routers is also linked to the same Virtual Chassis using "Chassis Cluster Redundant Ethernet Interface Link Aggregation Groups". And my mistake was I forgot to configure LAGs on the Virtual Chassis Interfaces linked to the SRX cluster reth LAG. http://www.juniper.net/documentation/en_US/junos12.1x46/topics/example/chassis-cluster-redundant-ethernet-interface-link-aggregation-group-configuring-cli.html : "For aggregation to take place, the switch used to connect the nodes in the cluster must enable IEEE 802.3ad link aggregation for the redundant Ethernet interface physical child links on each node." Romain

Argh!  As usual, as soon as I added this thread I was able to fix the problem. On the standby firewall, I edited /config/config.xml and replaced "opt1" with "opt2" for the DMZ interface.  I then removed the /tmp/config.cache and /tmp/config.lock files then rebooted the standby.  Now, CARP status shows all interfaces in Backup mode as expected. Sorry for the noise…