Since your CPE is already a SPOF, you might consider adding a cheap unmanaged switch behind each CPE, then you could have each CPE plugged into each switch directly, instead of having them plugged into only sw1 or sw2. That way if sw1 dies, you still have both ISP1 and ISP2 active.

If a cheap unmanaged switch dies, you only lose either ISP1 or ISP2. Much more desirable scenario than losing both a managed switch and access to one ISP in the process.

That is correct; If one fails they all should fail.

If the settings are right, the next thing to look at would be switches and cabling. It's possible that there is an issue there. I talked to someone last week who was getting odd master/slave flips and it turned out they were using the little switch on the back of their cable/dsl modem and it was not up to the task. Merely sticking a different switch in that role fixed all of the issues.

Great; Thanks! JIMP  I love how this works so well and easy to setup! (With a little help ;) )

Yes so each PC on the network would point to the Gatway IP at the Virtual IP on the LAN port.

There are security issues with doing state sync on LAN, and it can consume a significant amount of bandwidth depending on the rate at which states are added/removed.

No, package settings are not synced between cluster nodes.

To follow up, Chris B was gracious enough to answer this for me on the mailing list.

The solution was to just route the public IP's to the interfaces (as described in section 8.2 of the book).

Stupid me. I finally figured out that I was using a vhid already in use by our switches. Duh, right there in the troubleshooting carp doc…. All seems to be working great.

Thank you, that seems to do the trick!

Sounds good, hopefully that's the end of the issue :-)

This has happened to me in the past when a vSwitch is bound to multiple NICs which are attached to a physical switch that isn't configured correctly for trunking.  What ends up happening is that the multicast CARP announce goes up one leg of the trunk, the physical switch sees the request, doesn't know that the other uplinks are trunk members, and shoots the same multicast packet right back up the other pipes back to the ESX server.  ESX passes the packet back into pfsense, and it thinks oh dear someone's responding already for that CARP IP and either marks it down or both members as master.

Solution is to either remove any redundant uplinks in a vSwitch (whether active or not if they are physically in the config it'll screw it up) or properly configure the physical switch to handle trunking.

ESX doesn't support LACP, so this gets a little tricky if you go for the latter.  Juniper gear needs a recent JunOS and ae interfaces configured with LACP mode set to none.  You can't use beaconing.  Ciscos should be set up properly with separate EtherChannel configs for the trunks.

Thanks!  You have been a great help!

@jimp:

Packages do not sync to slaves, so that is expected. You can just manually configure the squid options on the secondary to match.

But exactly that would be our problem. We want to allow access to the internet based on the clients IP address, so if we use squid, we have to put the list of allowed IPs in the squid config. Without the proxy, we would have to put that list in the firewall rules. So I guess we have to choose between using a proxy server (and duplicate configuration), or no proxy server (and auto synchronisation of the firewall rules)…

Hi Jimp,

I tried that, it did reduce the errors but they were still there. As a last ditch attempt I stuck in a 160gb SATA disk i had laying around and that worked perfectly. So it must have been something strange with the converter.

Strange thing is, I have the exact same setup on my primary firewall, with a 4GB CF card and converter, upgraded that to 1.2.3 and worked without any problems. So I am not sure why I had issues with the backup firewall, it would be a very strange coincidence if there was a hardware failure at the same time as upgrading the software.

Either way things are back up and running, thanks for your help, much appreciated.

Actually, this problem may be related to another problem I posted at the same time:

http://forum.pfsense.org/index.php/topic,25874.msg135322.html#msg135322

I've been concentrating on the other problem because that one made pfSense unusable for my application.

For now I'm forced to go with another solution.

JBB