Aaaaand… I broke it again - same behavior.  Unclear exactly how I did that.  I was putting some Snort stuff together, but even suspecting that, and disabling it, still get no-resume behavior (testing from one of the WAN interface sides.

Interestingly, if I reverse the scenario - start downloading a file from the LAN side, pull a cable, that does resume.  So it is somehow related to the WAN side, or the number of VIPS/1:1NATs I have?  B/c WAN, DMZ, and LAN are all using CARP VIPs.  I'll do some more testing, but yes, FW2 (looking in Diag->States) does have that in there (http connection), so states are synching.

If you have CARP and Traffic shaper configured take a look here:
http://forum.pfsense.org/index.php?topic=45045

armando

Thanks. Gee it says so right there. (I did read the entire HA chapter in the book again before I asked :/ ) You're right. I'm probably over-thinking it. Creating the laggs would probably be more disruptive anyway.

I was able to correct this.

Because my testing environment is using different hardware and interfaces I needed to setup the interfaces more carefully

What I discovered is that when CARP assigns matches interfaces, it must choose them in sequential order from the assign interfaces page, matching them with the other firewall.

What I had was
Firewall 1
#1 LAN
#2 EM1
#3 EM2
#4 Bridge0

Firewall 2
#1 WAN
#2 LAN
#3 CXL0
#4 CXL1
#5 Bridge0

So, what I think happened was that CARP was matching #4 from each list, so my bridge0 (#4 on Firewall 1) was being matched with CXL1 (#4 on Firewall 2)

Once I reassigned my interfaces and lined up the interface numbers CARP matched the correct interfaces.

BGP and NAT are two different systems
on BGP you have neighbor not gateway

how many BGP sessions you have with your providers?and which ip is recognized by your provider for BGP session?

That is good to know. Thank you very much for your help. Really appreciate it.

In this scenario, both routers are advertising the 4.15.227.0/25 subnet, only the secondary is advertising with an artificially more distant path, this wouldn't be chosen unless the primary is down.
The inside,  when using OSPF, for example, the secondary advertises the default gateway with a less favorable metric than the primary, thus on your inside switches, you end up with two default routes, but only the best one would be used.
In the case that the primary pfsense goes down, then the secondary's routes become the only remaining routes on the ISP and Internally, and it keeps on working.

You could also put a link between the two boxes to route traffic over it in the event that either just the inside or the outside link goes down on the primary, then the traffic would flow through the secondary on the cross-over link, or if your switches support multi chassis LAGG you could add redundancy that way too.

Because pfSense is a stateful firewall, under certain circumstances, the session would drop, but for web traffic it wouldn't be noticeable for the most part.

I think I may have fixed this one!

It took me a lot of experimenting to get the ARP cache on the Draytek to recognise the PFSense Carp IPs….
Finaly seems to be Port forwarding ok though.
Next I will see if I can use a Carp IP as the Outgoing network IP.

You can. Check the sync box on settings tab.

I just spoke with my isp and they will route all subnets through one ip so everythig is fine. Thank you!

1. The log data wouldn't be synced – If you want to keep log data in that fashion you should be using an external syslog server to retain the logs.
2. The cron package doesn't sync, you'll have to copy those manually
3. The proxy logs don't sync, so the reports would not sync.

There is a good section in the book about bringing up a new HA member. That's what I use. It's for 2.1.5 but I've used it on 2.2.X.

A key issue is to add interfaces in the same order. There is a lot to be done "just so" esp if the active unit is in production.

If you have a third, identical unit you might restore a backup to it and get the new HA member configured on the work bench.

I don't use FreeRadius for anything other than authenticating customer CPE's to my Ubiquiti AirMax AP's. So no accounting at all. Just username/password passed from the CPE to the AP. AP checks and if the FreeRadius server says ok, the CPE connects and gets an IP address and internet. Basically all I need is to be sure that usernames/passwords are synced between the servers. Everything else doesn't matter.

Thanks , i could solve the problem by

Enable promiscuous mode on the vSwitch on my VM ware

Couple of questions.
Is the VM host using load balancing over multiple nics? If so make sure it is set to IP Hash with the switch configured accordingly.
Have you created separate port groups on the virtual switch with  promiscuous mode only enabled on the group that carries the VRRP? Port groups are probably the way forward.

Carp, VRRP etc are notoriously idiosyncratic on VMWare

So, thanks to everyone!!

Was an issue related to the specific IP.

In love with Pfsense again :)

Thanks, I really appreciate your input. It sounds like it's not a great idea. I'll raise it with our provider, but I think we're pushing the boundaries of what they offer. They're a hosting/server provider who deal in bulk rather that bespoke, so anything that deviates from their standard setup is unlikely to be possible.

We may simply add a second pfsense instance to test out these extra IPs. The bulk of them are for one very specific use case which could be diverted through a separate pfsense VM without too much effort.

Thanks again!