This is your problem: net.inet.carp.suppress_preempt: 4 From the man page: net.inet.carp.suppress_preempt       A read only value showing the status of preemp-       tion suppression.  Preemption can be suppressed       if link on an interface is down or when       pfsync(4) interface is not synchronized.  Value       of 0 means that preemption is not suppressed,       since no problems are detected. Every problem       increments suppression counter. Carp is detecting some issue and not letting all the VIPs fail over. Not sure where to go from here- I would verify everything was good with the sync for a start.

If you have two systems setup, all your VIPs must be CARP. IP aliases are only on the primary, can't have them on two systems as they cannot be shared.

Just add Other type VIPs on WAN, not on CARP. They don't actually do anything other than filling in places in the GUI where you can pick public IPs.

First off I have to say PFsense is awesome.  Ok I figured out a bunch of things.  I hope this helps people with the same problem. 1.  you must create an IP alias for every virtual server. 2.  make sure the subnet mask is properly set on the ip alias since it defaults to 32 which won't work. 3.  Hyper-v isn't a good BSD host and you will need to create a shell script in /usr/local/etc/rd.c I like to call it something early in the alphabet like 1st.sh since I want to it execute before other shell scripts like haproxy.sh.  put this in you script: ifconfig de0 down ifconfig de1 down ifconfig de2 down ifconfig de3 down ifconfig de0 up ifconfig de1 up ifconfig de2 up ifconfig de3 up 4.  Use HAproxy-full instead of the standard load balancer.  Just install it from the packages, it's far more full featured, than the built in one. 5.  HAproxy will crash if you try to pass persistence cookies over ssl, if you see the service stopped, that's probably what you are doing.  You must use source balancing for encrypted packets, and make sure the cookie fields are blank. 6.  Stunnel will allow you to use persistence cookies with SSL.  Install it from the packages, put it in front of HAproxy so it will decrypt the packet and send the decrypted packet to haproxy, now you can use full cookie persistence with SSL. 7.  Here's a good quick and dirty tutorial for setting up HAproxy http://conheotiensinh.blogspot.com/2011/12/config-haproxy-with-pfsense-version-201.html

It can, but you wouldn't have stateful failover, and you can't use both ISPs at once in that kind of setup.

Yep, get them talking on the sync port, setup config sync and the firewall/nat rules will copy over.

Maybe I was missunderstood: pfsense is doing routing between the block of public IPs we have and which is configured on the LAN interface and the ISP address which is configured on the WAN interface. So the goal is to have our block of IPs routed and not NAT-ed. My problem is that on the WAN interface I would like packets to have as outgoing address the VIP of WAN and not the real IP of the WAN. So in case master fails and slave takes over the receiving party will always "see" the same originating IP address.

There might be a slight performance gain with jut routing, but the extra level of security, to me, out ways that performance gain. If you are talking about a filtering bridge, then there is really no performance gain. You will still have to have a firewall whether it is at the perimeter or on the server.

It's a known issue and has been that way forever. CARP and bridging aren't really things you want to mix in general. (Search the forum, posts from myself and others over the years might convince you…)

Sounds like traffic isn't getting to the CARP IP for some reason - two most common would be an IP conflict, or a stale ARP cache upstream from where that IP was previously assigned elsewhere. Packet capture on WAN would confirm or deny that.

Hi again. good news, I solved the problem… I've looked at the PFTOP from my 2nd server, and I saw in the rules list only 3 rules... deny all... I have a alias that include a URL list.  the path was http://127.0.0.1/list.txt (the list is local on the FW) I put the list on the 2nd fw, reload filter, and  voila ! Ciao !

You have to change your interface IPs to CARP IPs, which requires having the interface IP (your default gateway internally, and destination IP on WAN-side traffic) removed briefly. Generally that can be done quickly enough that it doesn't impact any hosts that are already online, or any inbound traffic as long as it's shorter than the period of the ARP cache, which it'll easily be. I've done it many times on production networks on the fly without dropping a packet, but you need to be careful to be ready to add the CARP IP immediately upon changing the interface IP or it's possible you'll create an outage until the CARP IP is added. I always have the CARP IP ready to save, change the interface IP, save the CARP IP. In that case it's just a matter of how quickly you can apply changes on one tab, and click Save and Apply on another.

Some more details/ideas here after reading the book again: Since I don't appear to have a true routed subnet, it looks like I could connect a switch to the Verizon ONT.  Off that switch I'd see my 5 IPs (I believe I did test this back at install time).  I'd use 3 of those 5 for a CARP failover set-up.  If that's all true (I think it is), then my question is how I can use the other 2 IPs.  Without the switch I use CARP/VIPs to associate those other addresses to my primary IP.  It's not clear to me how that looks with a switch in between the two now.  Seems like I could either still do the CARP/VIP trick (switch has no effect other than splitting off the two IPs I need to separate for failover), or it seems like I might have to have pfSense see the split extra IPs as multiple WAN IPs (which I'd use without failover).  The problem with that second scenario would seem to be that I can no longer pool the extras for a set of NAT rules to fan them out to the DMZ behind pfSense – that if I split them either one pfSense box would get the extra two, or each pfSense box (primary, secondary) would get one of the two extras.

Finally, they manage to route the whole block of IPs to one interface of their equipment (the one connected to pfsense), so I created IP Aliases for routing services to other IPs. Best regards Kostas

@cmb: You can't. Your interconnect with your ISP must be a /29. They should be willing to switch you over to that, it's not an uncommon request since basically every router/firewall redundancy protocol requires it. Ok, thats what I thought. I'll see about getting that changed.

link light blinking rapidly wouldn't be indicating traffic (if it actually is the link light, should have a different light for activity if it has one at all), that's more likely to indicate link cycling on the NIC. I have seen NICs not play nicely when directly connected, does it behave better if you throw a small switch between? That'd at least confirm or deny that suspicion.

To use carp you will need one real ip for each pfsense plus all others using carp. I suggest you to use one of your 4 ethernet ports to sync between boxes. A new feature on 2.x that will help on vip assigns is in this post from jimp http://forum.pfsense.org/index.php/topic,45209.msg240909.html#msg240909 After sync and carp, just create your 1:1 nat on firewall -> nat and then change your outbound nat to manual to create your specific outgouing nat translation rules. By default, all interfaces but lan has no access to anywhere. You will need to change this default rule to deny access from lan to dmz. All other rules you can create on interface that traffic starts. If you want to allow internet access from a host on dmz, the rule will be on dmz. If you want to allow that everyone can reach your web server, then rule will be assigned on wan.