To follow up, Chris B was gracious enough to answer this for me on the mailing list.

The solution was to just route the public IP's to the interfaces (as described in section 8.2 of the book).

Stupid me. I finally figured out that I was using a vhid already in use by our switches. Duh, right there in the troubleshooting carp doc…. All seems to be working great.

Thank you, that seems to do the trick!

Sounds good, hopefully that's the end of the issue :-)

This has happened to me in the past when a vSwitch is bound to multiple NICs which are attached to a physical switch that isn't configured correctly for trunking.  What ends up happening is that the multicast CARP announce goes up one leg of the trunk, the physical switch sees the request, doesn't know that the other uplinks are trunk members, and shoots the same multicast packet right back up the other pipes back to the ESX server.  ESX passes the packet back into pfsense, and it thinks oh dear someone's responding already for that CARP IP and either marks it down or both members as master.

Solution is to either remove any redundant uplinks in a vSwitch (whether active or not if they are physically in the config it'll screw it up) or properly configure the physical switch to handle trunking.

ESX doesn't support LACP, so this gets a little tricky if you go for the latter.  Juniper gear needs a recent JunOS and ae interfaces configured with LACP mode set to none.  You can't use beaconing.  Ciscos should be set up properly with separate EtherChannel configs for the trunks.

Thanks!  You have been a great help!

@jimp:

Packages do not sync to slaves, so that is expected. You can just manually configure the squid options on the secondary to match.

But exactly that would be our problem. We want to allow access to the internet based on the clients IP address, so if we use squid, we have to put the list of allowed IPs in the squid config. Without the proxy, we would have to put that list in the firewall rules. So I guess we have to choose between using a proxy server (and duplicate configuration), or no proxy server (and auto synchronisation of the firewall rules)…

Hi Jimp,

I tried that, it did reduce the errors but they were still there. As a last ditch attempt I stuck in a 160gb SATA disk i had laying around and that worked perfectly. So it must have been something strange with the converter.

Strange thing is, I have the exact same setup on my primary firewall, with a 4GB CF card and converter, upgraded that to 1.2.3 and worked without any problems. So I am not sure why I had issues with the backup firewall, it would be a very strange coincidence if there was a hardware failure at the same time as upgrading the software.

Either way things are back up and running, thanks for your help, much appreciated.

Actually, this problem may be related to another problem I posted at the same time:

http://forum.pfsense.org/index.php/topic,25874.msg135322.html#msg135322

I've been concentrating on the other problem because that one made pfSense unusable for my application.

For now I'm forced to go with another solution.

JBB

I'm not sure that what SuperMule is suggesting makes sense in this situation.

This is where I'd suggest you start.

In the VMWare VIC (virtual infrastructure client):

On the HOST:

Configuration - Networking

Get 'properties' on the switch associated with these IP addresses.  Then, clicn on the vSwitch, and click "Edit"

Under "Securty" - set all three (Promiscuous Mode, MAC Address Changes, Forged Transmits) to "Accept"

PLEASE NOTE that this has security implications!  You may want to be more specific in how you configure this, etc.

What is the recommended procedure for recovery?

Let's say FW-A fails, FW-B becomes master… if FW-A returns to service before any changes are required, GREAT - but what it you have to make changes?

Are you better to re-connect FW-A as a slave to FW-B? Or backup and restore selective parts of the FW-B config to FW-A?

Thanks!

Ya unfortunately I have to double NAT for now since we are running two (3 actually…) firewalls in parallel all off one modem.  This is all part of an overall plan to get it down to just one.