@cmb: @jhabers: Hmm Im using Broadcoms Smart Load Balancing with 4 NICs teamed on each server. Could that be causing confusion? yeah some load balancing/NIC teaming does that normally. Otherwise it's generally indicative of an IP conflict or ARP poisoning. thanks, is it safe to ignore those messages? anyway to get them not to log?

As long as IPsec is bound to a CARP IP, it can't come up on the secondary until it's master. If you have dual master status, there's some kind of connectivity problem between the two hosts (though that should be no diff from 1.0.x to 1.2.x).

just curious. why you need multi-wan when your WANs are on the same network segment? CARP should be good enough for fail safe. by the way, Multi-WAN + CARP should be working on 1.2.3-Release. setup the interface(s) individually from both master and slave, then setup CARP accordingly. and then go for Multi-WAN according to the book just like without CARP.

You can.  That shared IP becomes the source IP of any traffic egressing from your network and you're able to NAT traffic inbound on that interface.

what's line 4693 in your config?

Since your CPE is already a SPOF, you might consider adding a cheap unmanaged switch behind each CPE, then you could have each CPE plugged into each switch directly, instead of having them plugged into only sw1 or sw2. That way if sw1 dies, you still have both ISP1 and ISP2 active. If a cheap unmanaged switch dies, you only lose either ISP1 or ISP2. Much more desirable scenario than losing both a managed switch and access to one ISP in the process.

That is correct; If one fails they all should fail. If the settings are right, the next thing to look at would be switches and cabling. It's possible that there is an issue there. I talked to someone last week who was getting odd master/slave flips and it turned out they were using the little switch on the back of their cable/dsl modem and it was not up to the task. Merely sticking a different switch in that role fixed all of the issues.

Great; Thanks! JIMP  I love how this works so well and easy to setup! (With a little help ;) )

Yes so each PC on the network would point to the Gatway IP at the Virtual IP on the LAN port.

There are security issues with doing state sync on LAN, and it can consume a significant amount of bandwidth depending on the rate at which states are added/removed.

No, package settings are not synced between cluster nodes.

To follow up, Chris B was gracious enough to answer this for me on the mailing list. The solution was to just route the public IP's to the interfaces (as described in section 8.2 of the book).

Stupid me. I finally figured out that I was using a vhid already in use by our switches. Duh, right there in the troubleshooting carp doc…. All seems to be working great.

Thank you, that seems to do the trick!

Sounds good, hopefully that's the end of the issue :-)

This has happened to me in the past when a vSwitch is bound to multiple NICs which are attached to a physical switch that isn't configured correctly for trunking.  What ends up happening is that the multicast CARP announce goes up one leg of the trunk, the physical switch sees the request, doesn't know that the other uplinks are trunk members, and shoots the same multicast packet right back up the other pipes back to the ESX server.  ESX passes the packet back into pfsense, and it thinks oh dear someone's responding already for that CARP IP and either marks it down or both members as master. Solution is to either remove any redundant uplinks in a vSwitch (whether active or not if they are physically in the config it'll screw it up) or properly configure the physical switch to handle trunking. ESX doesn't support LACP, so this gets a little tricky if you go for the latter.  Juniper gear needs a recent JunOS and ae interfaces configured with LACP mode set to none.  You can't use beaconing.  Ciscos should be set up properly with separate EtherChannel configs for the trunks.

Thanks!  You have been a great help!

@jimp: Packages do not sync to slaves, so that is expected. You can just manually configure the squid options on the secondary to match. But exactly that would be our problem. We want to allow access to the internet based on the clients IP address, so if we use squid, we have to put the list of allowed IPs in the squid config. Without the proxy, we would have to put that list in the firewall rules. So I guess we have to choose between using a proxy server (and duplicate configuration), or no proxy server (and auto synchronisation of the firewall rules)…