What are you using to balance between the servers? The built-in relayd load balancer? HAProxy?

For management purposes you'd always have to connect directly to each individual backend server to query them. You should never attempt to manage anything HA using the failover address, it doesn't matter if it's pfSense or something else. Always address the units individually for management.

If it was the exact same issue I had then the real problem ended up being my testing method.  Although I never understood why yanking a cable isn't a valid test.  When I simply unplug POWER from the primary router the failover is nearly instantaneous.  I never posted a follow up on simulating a switch failure but that also failed over as expected.

On another note, I have had to enter maintenance mode 5 times recently so that I could change out some other equipment inline with these.  Three out of five the switchover was nearly 30seconds and the WebInterface on both routers locks up completely which switching over.  Looking at top via ssh during the switchover doesn't show anything locked up nor maxing out RAM/CPU so not sure what the problem was there if it is related at all.

Is looks normal to me, MASTER on first node, Backup on second.

After shutdown of the MASTER, the second becomes MASTER
To my big surprise, in about 33% of the pings to the CARP VIP I get a reply now, the others time out…..

Hi hackce,

what do the logs on the second PFSense say? Perhaps you are using the wrong password or
the firewall prevents the synchronization.
Did you try to add an "allow all" firewall rule to your sync network for testing?

Cheers,
Jesper

The clustered hypervisor would protect against node failure by transferring the VM to another running node, but it would not protect against software failure from within the VM. If, for some reason, a CARP'd pfSense instance were to freeze/fail, in this scenario the failover CARP pfSense node should take over. Frankly, I'm not sure which is the more likely scenario: a failing hypervisor node or a failing pfsense instance.

In my particular configuration, I'm not clustering my Hyper-V nodes but instead as stand-alone nodes.

In case anyone else needs an answer to a similar problem, while searching for something entirely unrelated, I came across this link
https://forum.pfsense.org/index.php?topic=42532.0

In the final post was the solution to my problem which I have cut and pasted from there to here for ease of reference and added some notes of my own relating to the key points.

The following also pertained to my situation…
"The router had already been in production for a while and had some NAT port forwards configured"
"I assumed those rules would carry right over to the CARP setup because the destination was WAN."

The following is what got me on the right track ....
"I went to make a new rule for some reason or another and noticed that there was a new destination choice called WAN CARP (what I had named that VIP).  When I realized the firewall was discriminating between real IPs and virtual IPs, I had my answer."

This bit summed up my situation perfectly too ...
"I guess I just assumed that my rules were all per-interface, but they're actually more granular than that.  Changed all my regular stuff to the CARP destination"

When I did the above - it worked.  As the original poster sad, it is worth noting that the NAT rules are quite so granular ...

Build all the interfaces first. Make them exactly match the primary, in the same order, but with a different interface address, obviously.

Sorry, my mistake.

I missed one thing clearly written on the ufficial guide: the states syncronization MUST be enabled on the slave node too!
After enabling this everything workey, now my OpenVPN/SSH connections remain up&running even if I shutdown the primary node, pretty impressive :)

Thanks all for your help!

We use pfSense version '2.3.2_1' and pfBlockerNG '2.1.1_4', we don't experience this issue, check you're on the latest version.

If you're using a dedicated NIC for XMLRPC (recommended) you should use a cross-over cable and set the speed\duplex to 100base TX with static IPs.

Enter the following settings for pfBlockerNG -> Sync (might be an issue with it using the generic settings):
Enable Sync: Sync to hosts defined below
Protocol: HTTPS
Target IP/Hostname: backup pfSense IP
Target Port: 443
Target Password: your admin password

We had a few problems when we initially built our environment on Broadcom NICs, we now use Intel.

Hope this helps.

That ticket was referring to the keep-alive pinger process, which is what I already mentioned.

The two systems check heartbeats but that's at a completely different level than IPsec.

For 99.9 of people it works fine as-is.  For someone with a misconfigured network it'll have a problem, like you had, but there is so little benefit to "solving" this corner case it's just not worth doing. It could negatively impact cases that are working fine now.

Thank you viragomann, it make sense :D, I'll start from CARP.

SenseRider

Yes, you can drive 4 pfSense in CARP mode and sync configuration from one to the other, but not over internet, this doesn't make any sense anyway.

For CARP all interfaces sharing the same VIP has to be connected to the same switch. On each box you have to set a different skew value for the VIP, that one with the lower skew has the higher priority.

For syncing you can only sync from one the another, so you can sync from the first (master) to the second and from the second to the third and so on.

I decided to look at the physical components and have the following (brief) layout but I no longer think the issue is with pfSense hence the briefness:

pfSense VMs /w CARP (XenServer) -> Cisco L3 -> Netgear L2 -> Client

Both pfSense VMs are accessible when connected to a VLAN 20 port on the L3 switch. Now the issue, when you get to the VLAN 20 port on the L2 netgear switch it's no longer possible to reach one of the two gateways on any VLAN. If this happens to be the Slave then the VIP and Master are reachable; if it happens to be the Master then only the Slave is reachable.

This is most likely an issue between the cisco and netgear switches and will require further investigation. I'm not sure if you have any insight Derelict but the issue is no longer as originally described.

EDIT: Upon even further investigation, it seems like the L2 is basically useless in this case. It has no clue how to handle the gateways correctly and does not come with any kind of SSH let alone HTTPS. Fun times…

I believe this may have been the solution. I seem to be able to pull the WAN or LAN cables from Primary now and get a full failover.

Do you sync the VIP's? If so that could be the cause…
Had some issues with that in the past, see: https://forum.pfsense.org/index.php?topic=102740.msg572905#msg572905

This looks like a solved problem. Wouldn't restoring the master offline and enabling persistent maintenance mode solve this?

Restore offline
Enable maintenance mode
shutdown
Put back in line
start
Let everything settle, all CARP should be BACKUP
Check all your packages/services
Disable maintenance mode

Or just add this to the bottom of your config and restore inline:

<virtualip_carp_maintenancemode></virtualip_carp_maintenancemode>