The way that I have been able to get this working is to Create a VIP on a separate subnet, then I have a Rule that allows any traffic to that Subnet to the VIP. Since the LAN Traffic is on the same private space interface the traffic can flow between the two. Just make sure you allow ANY ANY Traffic between the two DMZ's  then any Server/Client on DMZ-1 can talk to the VIP on DMZ-2 that is a Load Balancer that points to Servers in DMZ-1 Subnet… seems a bit wonky but it works.. the biggest problem with allowing traffic to flow out the same interface and then back in.. NO Load balancer will allow this.. so you have to create another subnet for it to route to... even if the servers behind the Load Balancer are on the same subnet. Good luck!

It does but you need to manually handle the VIP settings because they can't sync automatically. Also the DHCP failover setup doesn't support 2+ peers IIRC.

@craggy: How long should it take for a secondary firewall to take over the CARP VIP and start passing traffic again when the master is shutdown or rebooted? We have set up a secondary pfSense cluster in a new Data Center and the failover times are about 10 seconds with 5 to 10 ping drops and a very obvious pause during the switchover phase whereas at our primary site our pfSense cluster fails over in about 2 seconds with usually only a single ping dropped and is generally unnoticed. Both sites are identical, VMware esxi 5.1, hp bl460c blades, same HP VC networking etc. Only difference is the data centre. My boxes are instant; I've got a failing box and it needs a reboot every few days. If you're seeing 10 seconds then it might be something about being virtualized or maybe you've changed the base & skew settings.  For base & skew I use 0/1 on the primary and 0/100 on the secondary.

Nevermind…was a cabling problem - all is working

Got busy and haven't had time to tcpdump yet. Any other ideas out there about anything else that can be looked at in the interim?

I have no blade-related experience, but did you put the vswitch ports that connect to your pfsense vm's, and which are to take part in CARP, into a port group that has promiscous mode enabled? I usually create a duplicate port group (same vlan, same vswitch), which has promiscous mode enabled, and put the pfsense interfaces into that port group, and all vm's that use the pfsense as a gateway into the port group with promiscous disabled.

For me it's easier to have only one failover, the setup is so that the slave doesn't have all features (no backup wan connection) so only 1 network doesn't have the failover when there is a network fail. If all networks will switch in depended, I still can switch the master down, all networks will go down and the slave would take over all networks. I have created a stable situation again, I found out when there is an open network (both pfsense are set to init, the network becomes unstable in a couple of hours) But still I want to failover independent, I don't get why the option has been taken out.

Also see thread http://forum.pfsense.org/index.php/topic,68439.0.html, if the suggestions in this thread don't help as they may be similar issues.

So, I turn on spanning tree protocol on both sides of the bridge, to include the wifi, vlan, pfsync and bridge interface…and it all works.

@jimp: Yes. Perfecto! Thanks Jim!

ok thanks for the help.

As I see it, the only practical reason to use separate interface for sync is to avoid blocking it by firewall mistakes or overloading the NIC with traffic. So your topology seems more than fine by me :)

Changing VHID requires restart of CARP - disable, then enable in status>carp. Other than that, CARP uses shared password which should match on both ends.

Okay i have this all up and running - the issue was that BT had not setup the bloody service despite telling me several times they had! So here is how to setup BT Business infinity with 5 IPs on PFsense: WAN: Have this setup on PPPoE as usual with the correct user name and password which was provided to you. N.B. both the user name and password are case sensitive so make sure you get it right! You'll then pick up a random dynamic IP on your WAN interface for general internet access. VIPs (your 5 static IPs) All you need to do here is on the web gui go: Firewall > Virtual IPs Then depending on what kind of VIP you want just create 1 VIP for each static IP you have. my settings: Type: IP Alias Interface: WAN IP Address(es): type: Single address, Address: x.x.x.x / 29 Press save and you are done! Now you can play around and NAT things 1:1 or just port forward all you want. Enjoy!

Well, it seems this saved the day: [2.1-RELEASE][admin@master]/root(20): pkg_add -r http://files.pfsense.org/packages/amd64/8/All/libevent-1.4.14b_2.tbz Definitely this is bug.

I took a tcpdump of both interfaces does this look normal? FW1 00:00:00.000000 IP 10.1.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 0, authtype none, intvl 2s, length 36 00:00:02.001079 IP 10.1.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 0, authtype none, intvl 2s, length 36 00:00:02.001082 IP 10.1.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 0, authtype none, intvl 2s, length 36 00:00:02.001087 IP 10.1.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 0, authtype none, intvl 2s, length 36 00:00:02.001082 IP 10.1.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 0, authtype none, intvl 2s, length 36 00:00:02.001081 IP 10.1.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 0, authtype none, intvl 2s, length 36 00:00:02.001085 IP 10.1.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 0, authtype none, intvl 2s, length 36 FW2 tcpdump -i bce1_vlan101 -ttt -n proto CARP tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on bce1_vlan101, link-type EN10MB (Ethernet), capture size 96 bytes 00:00:00.000000 IP 10.1.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype none, intvl 2s, length 36 00:00:02.392089 IP 10.1.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype none, intvl 2s, length 36 00:00:02.392086 IP 10.1.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype none, intvl 2s, length 36 00:00:02.392088 IP 10.1.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype none, intvl 2s, length 36 00:00:02.392089 IP 10.1.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype none, intvl 2s, length 36 00:00:02.392089 IP 10.1.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype none, intvl 2s, length 36 00:00:02.392093 IP 10.1.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype none, intvl 2s, length 36 00:00:02.392085 IP 10.1.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype none, intvl 2s, length 36 00:00:02.392089 IP 10.1.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype none, intvl 2s, length 36

I figured this out, apparently if you make a gateway from the interfaces tab it can gain priority and cause your routing to break? Probably user error on my end but once I removed the gateway I was able to route traffic again. Thanks for your help.

It really is necessary to have the pfSense versions match up, otherwise you will have configuration sync and/or state sync issues. Otherwise, as has been stated, you don't need identical hardware or NICs, just an identical number of interfaces and to have them assigned in the exact same order on both.