Adding and deleting should both be synced, unless the rule has the box ticked to stop it from being synced.

Is it, by chance, the last rule in the list that isn't being removed?

@ace:

It looks like it is just not possible to use pfsense CARPs on a LAN in vsphere 5.

If that were true, this website wouldn't work, amongst a ton of other production systems. This site and all our sites are on a CARP IP on VMs in vsphere 5.

info here on the ESX settings that will break multicast or multiple MACs on a single VM (and hence break CARP):
http://doc.pfsense.org/index.php/CARP_Configuration_Troubleshooting#VMware_ESX.2FESXi_Users

Nobody had a similar problem?

Is there something I could check? some settings, did I do something wrong?

This turned out to be a problem due to the web server having an interface on the 192.168.0.0/24 network. Taking that interface down allowed packets to flow freely, how they were meant to.

Sounds like the port forward method. So, I also take it you are using manual outbound nat. Could you post your rules? Basically it should ready something like:
Interface: WAN
Souce: 192.168.10.51/32
SPort: any 
Destination: any
DPort: 25
NAT Address: .3 VIP
NAT Port:Blank
Static Port: unchecked

Depends on how your provider gives you those IPs. Hopefully they route them to you, or can change to doing so, then you have those subnets directly assigned to internal interfaces with just a /29 between you and the provider. You could bridge otherwise but that introduces complications if you want to VPN in, need to add private subnets, etc.

Okie.. no dice.. as intended I am sure :) CARP must be on the same subnet.

I briefly toyed with the idea of super netting.. but then I realized.. couldn't I use firewall rules to block BGP to any IP except from the CARP IPs?

I'll check this out next.

stay tuned! And please chime in with observations, criticisms, or anything else :)

@podilarius:

I have done this test with pfSense and if:

If LAN or WAN fails in the master, then the slave takes over.
If the entire box dies, then the slave takes over.
It does 2 WAN checks, pinging the gateway and link status. (so far as I can tell).

I had fun doing the testing, please post your finds once you have been able to run this in your lab.

Awesome, that was the answer i was looking for :D! btw, if the switch linked to the master box fails, will the slave take the control too? (since the lan link should go down…)

@jimp:

The sync interface is its own interface, that is not a "carp" interface. Nothing for CARP happens on the sync interface; That is for pfsync (state sync) and xmlrpc (config sync). CARP heartbeats are sent on each interface that has a CARP VIP.

Just checking back in  - Using a Juniper (ex2200-48t-4g)  Switch we created a LACP group in "active" mode and set the PFSENSE LAGG interface to type "LACP" and the CARP is working perfectly.

Thanks for the quick response!  ;D

If it can't reach it's gateway then it can't get out beyond. Usual things to look for there are to make sure that there are no conflicting IPs, that the switch connecting all three devices (ISP router, carp master, carp slave) is working properly, make sure the subnet mask matches properly (is it really a /28? what's the ISP router set to?), and so on.

Things like that usually boil down to a conflict of some kind, or a layer 1/2 issue.

solved this one via commercial support, following up here for the sake of others who find it in the future. Problem was using a CARP IP with the same VHID on two separate pairs. Input validation prevents doing so on a single pair. When you have multiple pairs on the same broadcast domain, make sure you use unique VHIDs, since the VHID determines the MAC address. When you duplicate VHIDs, you create duplicate MACs, which causes the typical issues when you have duplicate MACs - significant packet loss and general network confusion.

Also a good idea to only use each VHID once at each physical location even if separate broadcast domains (VLANs), while that should work no problem as switches should keep the MACs specific to each VLAN appropriately, it can potentially confuse your switches.

@cmb:

That almost certainly indicates you have intermittent connectivity in general on the CARP IP for some reason. Could be an IP conflict, amongst other possibilities. Test connectivity in general to the CARP IP.

Yes it was a buggy carp address indeed. Thanks!

If you re-read what I posted, I covered that already. Even if the settings are synced, squid's connections in the actual squid process – NOT the state table -- are not synchronized, so true stateful failover is not possible for the squid process.

The same applies to other daemons like OpenVPN or IPsec but in those cases using the CARP VIP is needed to make sure the right box receives/sends the remote traffic properly.

In the case of squid, that doesn't matter really, unless a remote site needs to see the CARP VIP to allow access if it filters by IP.

Cross post
http://forum.pfsense.org/index.php/topic,48555.msg256532.html#msg256532

Problem solved…  After finding release notes mentioning a gateway monitoring option that disabled clearing states I found the option below.

System->Advanced->Miscellaneous
the bottom option...

Gateway Monitoring
States

By default the monitoring process will flush states for a gateway that goes down. This option overrides that behavior by not clearing states for existing connections.

That is definitely not something you want for a cluster HA solution.  I don't see anything stopping deployment now with some more testing.

Thanks for tracking that down, I added it to: http://doc.pfsense.org/index.php/CARP_Configuration_Troubleshooting

Cheers makes sense, solves my question 1.

In regards to my Question 2,

which way is the preferred option?

not with the built in load balancer, you should be able to do that with one of the add-on options in packages, like haproxy is the one that's most frequently used for more advanced load balancing scenarios.

It does require using a static IP. What you could do is use the LAN IP there instead, and add a port forward on WAN to send that traffic to the LAN IP. The port forward will automatically update when your IP changes, and the server load balancer won't have to. You may want to add an IP Alias type VIP on LAN to use for that, if you need to use the same ports the web GUI is using. Or just change its port under System>Advanced