Update, I finally got the ISP out there and it was an issue on their side so all is well now.  Thanks again!

Okay from my testing, it seems that this works just fine: /sbin/sysctl net.inet.carp.allow=0 or set to 1 to re-enable.

Create another vlan and assign it to a sync interface. Then assign this vlan to a sync interface and allow all traffic on it. I preffer using vlans to do not have many cables plugged on my firewall.

edit your first post subject with [SOLVED]

Yeah, no problem.  The router had already been in production for a while and had some NAT port forwards configured, and the associated firewall rules autoconfigured.  I assumed those rules would carry right over to the CARP setup because the destination was WAN.  I went to make a new rule for some reason or another and noticed that there was a new destination choice called WAN CARP (what I had named that VIP).  When I realized the firewall was discriminating between real IPs and virtual IPs, I had my answer.  I guess I just assumed that my rules were all per-interface, but they're actually more granular than that.  Changed all my regular stuff to the CARP destination and set ICMP to pass on anything and everything worked correctly. I thought I'd have to do some manual outbound rules as well, but so far that doesn't appear to be necessary.  I'll have to read more about that to know for sure, though.

@rootlurker: Hi, I thought you have 5 STATIC IP assigned by your ISP, and you also said it on dynamic IP? Also, "So how do I get the WAN IP to respond to one of my static IP's?" which WAN IP? Hi, The weirdness of BT's setup is that when you connect using PPPoE they assign your connection a dynamic IP. However, we have 5 static IP's that we pay extra for, so somewhere in BT's network, they know to route traffic for those 5 IP's to our dynamic IP, this works fine when we're dealing with another machine on the network, we just set up 1:1 mapping and the packets coming in on that IP go to the local machine and packets destined to go out over the WAN appear to originate from the static IP. It all works fine using 1:1 mapping when there's another machine on the network, but in my case the pfsense box needs to be a VPN endpoint, so I need it to act on traffic coming in from one of the static IP's itself which is what I can't figure out what to do! Thanks

Hi, pfsense can do the LoadBalancing and the Failover ability you know from your actual system. The difference between LoadBalancing in pfsense and Barracuda is, that - as you wrote - Barracuda can detect the less busy WAN and redirect traffic to this WAN. pfsense is "just" doing a round robin. There is no difference if a WAN has high load or not. To configure this in pfsense 2.0 just create the three Gateways, put them all into a Gateway Group with same Tier and chose this Gateway Group in your firewall rules as the Gateway for outgoing/outbound traffic. This is a really easy setup in pfsense 2.0

You may need to change dsl modem to router because CARP needs at least 3 ips on same subnet. I don't know if there is a feature to do not start wan auth While in backup mode.

That setup would work for failover, yes, though you might want to call that dedicated interface "SYNC" to avoid confusing people when posting about it. pfSense doesn't support active-active, so you can't do load balancing between the two boxes.

Hi,   thanks to the help of Jim, the problem has been identified. Some of the outbound NAT rules had "source=any", so also the CARP packets were natted somehow and this brought to an "inconsistent" CARP state. The problem was solved assigning to each outbound NAT rule a proper source different by "any". After this, Ermal added some code (that will be released with 2.0.1 RELEASE) to avoid this issue in any case (http://redmine.pfsense.org/issues/1954). Thanks to Jim and Ermal for supporting! Michele

K. Thank you guys. You helped me a lot. Really nice to have such a good working community here. Maybe someday I will add my part to make it even better. In my opinion we can close this thread.

You can balance firewall work when using packages. Use nat/firewall on box1 and squid on box2 for example. The active/active firewall can be done with carps, but its not desiged for it so, not supported. Take a look in this forum topic http://forum.pfsense.org/index.php/topic,40917.0.html

Wait some seconds until your switch flush mac address table. Also connect to your switch and see if there is any problem with mac table(full) or cpu usage.

I've just noticed that if i change the VIP in PfSense from CARP to IP Alias, then the problem disappears. Any thoughts?

Nevermind, I'm pretty sure this falls under the "well, duh" heading. I probably had the wrong password set on the primary as well as the "sync usernames/passwords" enabled. So when I set the admin password on the backup machine to the correct one, the XML got synced across, which reset its admin password to the wrong value which caused future syncs to fail and locked me out of the web guis. I reset the password on both, reset them to the proper values I want and then made sure syncing of usernames and passwords was not enabled in the virtual IP XML sync settings. So far, so good.

Did you use port forward or 1:1 NAT? If you are using port forward, then you will need to use advanced outbound NAT (manual mode) to transform the outgoing ip to 201.73.17.178. Remember that it is first matching rule in AON so if your LAN rule is above your custom outbound, then the custom outbound will never happen.

@zeratoun: Exactly, i want that, from the localhost of the pfsense firewall itself it uses the VIP LAN or WAN …. it's possible ? Best regards, It is possible but highly NOT recommended. I got that running in my test environment and CARP was not happy as ping stopped to the gateway on the secondary firewall. I think this will have an adverse effect on the clusters ability to fail over correctly. I didn't have a chance to test fail over, but i did notice that I could not download packages or ping the gateway. There is not reason I can think of to do this. Would you mind telling us why you would like to do that?

The CARP bad gateway has nothing to do with that. I would upgrade to the latest 2.0 release regardless, though I don't think that will fix your problem, 1.2.2 is a very dated release. Not enough info there to have much idea what's happening, exactly what pings work and don't, and where they're initiated and destined isn't clear.