@podilarius:

If you are trying to connect to them from the internet, the process is the same. If you are looking for inter-LAN communication, you should be able to access them via their internal IP address in the new subnet. The allow all rule you have on each LAN should pass all the communications.

If you want to restrict that, you are going to have to create a series of aliases and change the default rule.

If you are using DNS names, then I would use the DNS forwarder's override to create a split-brained DNS. This way if you are internal and you are using the pfSense firewall for DNS services, then when someone from the inside requests the DNS name they will get the internal address. But if you are on the internet, you get the VIP address.

I am sure there are more options as well depending on what exactly you are doing.

I got it sorted and I want to thank you publicly (not just by PM). I learned a lot during the process and I will make an effort to check the forum to try and help others.

Thanks again!! I appreciate the help.

When you're syncing users, it syncs the admin password on the secondary, and then you have to change the admin password in the sync settings on the primary to match. That's usually what people don't change when it breaks after the first sync. It works fine in every version, you probably had something mismatched there from that.

The MAC of a CARP VIP is determined by its VHID - it's shared in common between all CARP VIPs on all nodes. So from the router's point of view, the MAC would be identical, but it would have switched to another port.

Sometimes CPE switches can be odd with CARP - can you try plugging the master/slave into a small switch and then uplinking to the modem?

Adding and deleting should both be synced, unless the rule has the box ticked to stop it from being synced.

Is it, by chance, the last rule in the list that isn't being removed?

@ace:

It looks like it is just not possible to use pfsense CARPs on a LAN in vsphere 5.

If that were true, this website wouldn't work, amongst a ton of other production systems. This site and all our sites are on a CARP IP on VMs in vsphere 5.

info here on the ESX settings that will break multicast or multiple MACs on a single VM (and hence break CARP):
http://doc.pfsense.org/index.php/CARP_Configuration_Troubleshooting#VMware_ESX.2FESXi_Users

Nobody had a similar problem?

Is there something I could check? some settings, did I do something wrong?

This turned out to be a problem due to the web server having an interface on the 192.168.0.0/24 network. Taking that interface down allowed packets to flow freely, how they were meant to.

Sounds like the port forward method. So, I also take it you are using manual outbound nat. Could you post your rules? Basically it should ready something like:
Interface: WAN
Souce: 192.168.10.51/32
SPort: any 
Destination: any
DPort: 25
NAT Address: .3 VIP
NAT Port:Blank
Static Port: unchecked

Depends on how your provider gives you those IPs. Hopefully they route them to you, or can change to doing so, then you have those subnets directly assigned to internal interfaces with just a /29 between you and the provider. You could bridge otherwise but that introduces complications if you want to VPN in, need to add private subnets, etc.

Okie.. no dice.. as intended I am sure :) CARP must be on the same subnet.

I briefly toyed with the idea of super netting.. but then I realized.. couldn't I use firewall rules to block BGP to any IP except from the CARP IPs?

I'll check this out next.

stay tuned! And please chime in with observations, criticisms, or anything else :)

@podilarius:

I have done this test with pfSense and if:

If LAN or WAN fails in the master, then the slave takes over.
If the entire box dies, then the slave takes over.
It does 2 WAN checks, pinging the gateway and link status. (so far as I can tell).

I had fun doing the testing, please post your finds once you have been able to run this in your lab.

Awesome, that was the answer i was looking for :D! btw, if the switch linked to the master box fails, will the slave take the control too? (since the lan link should go down…)

@jimp:

The sync interface is its own interface, that is not a "carp" interface. Nothing for CARP happens on the sync interface; That is for pfsync (state sync) and xmlrpc (config sync). CARP heartbeats are sent on each interface that has a CARP VIP.

Just checking back in  - Using a Juniper (ex2200-48t-4g)  Switch we created a LACP group in "active" mode and set the PFSENSE LAGG interface to type "LACP" and the CARP is working perfectly.

Thanks for the quick response!  ;D

If it can't reach it's gateway then it can't get out beyond. Usual things to look for there are to make sure that there are no conflicting IPs, that the switch connecting all three devices (ISP router, carp master, carp slave) is working properly, make sure the subnet mask matches properly (is it really a /28? what's the ISP router set to?), and so on.

Things like that usually boil down to a conflict of some kind, or a layer 1/2 issue.

solved this one via commercial support, following up here for the sake of others who find it in the future. Problem was using a CARP IP with the same VHID on two separate pairs. Input validation prevents doing so on a single pair. When you have multiple pairs on the same broadcast domain, make sure you use unique VHIDs, since the VHID determines the MAC address. When you duplicate VHIDs, you create duplicate MACs, which causes the typical issues when you have duplicate MACs - significant packet loss and general network confusion.

Also a good idea to only use each VHID once at each physical location even if separate broadcast domains (VLANs), while that should work no problem as switches should keep the MACs specific to each VLAN appropriately, it can potentially confuse your switches.

@cmb:

That almost certainly indicates you have intermittent connectivity in general on the CARP IP for some reason. Could be an IP conflict, amongst other possibilities. Test connectivity in general to the CARP IP.

Yes it was a buggy carp address indeed. Thanks!

If you re-read what I posted, I covered that already. Even if the settings are synced, squid's connections in the actual squid process – NOT the state table -- are not synchronized, so true stateful failover is not possible for the squid process.

The same applies to other daemons like OpenVPN or IPsec but in those cases using the CARP VIP is needed to make sure the right box receives/sends the remote traffic properly.

In the case of squid, that doesn't matter really, unless a remote site needs to see the CARP VIP to allow access if it filters by IP.

Cross post
http://forum.pfsense.org/index.php/topic,48555.msg256532.html#msg256532

Problem solved…  After finding release notes mentioning a gateway monitoring option that disabled clearing states I found the option below.

System->Advanced->Miscellaneous
the bottom option...

Gateway Monitoring
States

By default the monitoring process will flush states for a gateway that goes down. This option overrides that behavior by not clearing states for existing connections.

That is definitely not something you want for a cluster HA solution.  I don't see anything stopping deployment now with some more testing.

Thanks for tracking that down, I added it to: http://doc.pfsense.org/index.php/CARP_Configuration_Troubleshooting