Cross post
http://forum.pfsense.org/index.php/topic,48555.msg256532.html#msg256532

Problem solved…  After finding release notes mentioning a gateway monitoring option that disabled clearing states I found the option below.

System->Advanced->Miscellaneous
the bottom option...

Gateway Monitoring
States

By default the monitoring process will flush states for a gateway that goes down. This option overrides that behavior by not clearing states for existing connections.

That is definitely not something you want for a cluster HA solution.  I don't see anything stopping deployment now with some more testing.

Thanks for tracking that down, I added it to: http://doc.pfsense.org/index.php/CARP_Configuration_Troubleshooting

Cheers makes sense, solves my question 1.

In regards to my Question 2,

which way is the preferred option?

not with the built in load balancer, you should be able to do that with one of the add-on options in packages, like haproxy is the one that's most frequently used for more advanced load balancing scenarios.

It does require using a static IP. What you could do is use the LAN IP there instead, and add a port forward on WAN to send that traffic to the LAN IP. The port forward will automatically update when your IP changes, and the server load balancer won't have to. You may want to add an IP Alias type VIP on LAN to use for that, if you need to use the same ports the web GUI is using. Or just change its port under System>Advanced

Not possible with IPsec tunnel mode (some people have it there and disabled and manually go in and enable it as a solution). With OpenVPN or transport mode IPsec with GRE or gif plus a routing protocol, it is possible (generally, depends on routing in general in your network, it can get complex as any dynamic routing can).

This is your problem:
net.inet.carp.suppress_preempt: 4

From the man page:
net.inet.carp.suppress_preempt
      A read only value showing the status of preemp-
      tion suppression.  Preemption can be suppressed
      if link on an interface is down or when
      pfsync(4) interface is not synchronized.  Value
      of 0 means that preemption is not suppressed,
      since no problems are detected. Every problem
      increments suppression counter.
Carp is detecting some issue and not letting all the VIPs fail over. Not sure where to go from here- I would verify everything was good with the sync for a start.

If you have two systems setup, all your VIPs must be CARP. IP aliases are only on the primary, can't have them on two systems as they cannot be shared.

Just add Other type VIPs on WAN, not on CARP. They don't actually do anything other than filling in places in the GUI where you can pick public IPs.

First off I have to say PFsense is awesome.  Ok I figured out a bunch of things.  I hope this helps people with the same problem.
1.  you must create an IP alias for every virtual server.
2.  make sure the subnet mask is properly set on the ip alias since it defaults to 32 which won't work.
3.  Hyper-v isn't a good BSD host and you will need to create a shell script in /usr/local/etc/rd.c I like to call it something early in the alphabet like 1st.sh since I want to it execute before other shell scripts like haproxy.sh.  put this in you script:
ifconfig de0 down
ifconfig de1 down
ifconfig de2 down
ifconfig de3 down
ifconfig de0 up
ifconfig de1 up
ifconfig de2 up
ifconfig de3 up
4.  Use HAproxy-full instead of the standard load balancer.  Just install it from the packages, it's far more full featured, than the built in one.
5.  HAproxy will crash if you try to pass persistence cookies over ssl, if you see the service stopped, that's probably what you are doing.  You must use source balancing for encrypted packets, and make sure the cookie fields are blank.
6.  Stunnel will allow you to use persistence cookies with SSL.  Install it from the packages, put it in front of HAproxy so it will decrypt the packet and send the decrypted packet to haproxy, now you can use full cookie persistence with SSL.
7.  Here's a good quick and dirty tutorial for setting up HAproxy http://conheotiensinh.blogspot.com/2011/12/config-haproxy-with-pfsense-version-201.html

It can, but you wouldn't have stateful failover, and you can't use both ISPs at once in that kind of setup.

Yep, get them talking on the sync port, setup config sync and the firewall/nat rules will copy over.

Maybe I was missunderstood: pfsense is doing routing between the block of public IPs we have and which is configured on the LAN interface and the ISP address which is configured on the WAN interface. So the goal is to have our block of IPs routed and not NAT-ed. My problem is that on the WAN interface I would like packets to have as outgoing address the VIP of WAN and not the real IP of the WAN. So in case master fails and slave takes over the receiving party will always "see" the same originating IP address.

There might be a slight performance gain with jut routing, but the extra level of security, to me, out ways that performance gain. If you are talking about a filtering bridge, then there is really no performance gain. You will still have to have a firewall whether it is at the perimeter or on the server.