In case anyone else needs an answer to a similar problem, while searching for something entirely unrelated, I came across this link
https://forum.pfsense.org/index.php?topic=42532.0

In the final post was the solution to my problem which I have cut and pasted from there to here for ease of reference and added some notes of my own relating to the key points.

The following also pertained to my situation…
"The router had already been in production for a while and had some NAT port forwards configured"
"I assumed those rules would carry right over to the CARP setup because the destination was WAN."

The following is what got me on the right track ....
"I went to make a new rule for some reason or another and noticed that there was a new destination choice called WAN CARP (what I had named that VIP).  When I realized the firewall was discriminating between real IPs and virtual IPs, I had my answer."

This bit summed up my situation perfectly too ...
"I guess I just assumed that my rules were all per-interface, but they're actually more granular than that.  Changed all my regular stuff to the CARP destination"

When I did the above - it worked.  As the original poster sad, it is worth noting that the NAT rules are quite so granular ...

Build all the interfaces first. Make them exactly match the primary, in the same order, but with a different interface address, obviously.

Sorry, my mistake.

I missed one thing clearly written on the ufficial guide: the states syncronization MUST be enabled on the slave node too!
After enabling this everything workey, now my OpenVPN/SSH connections remain up&running even if I shutdown the primary node, pretty impressive :)

Thanks all for your help!

We use pfSense version '2.3.2_1' and pfBlockerNG '2.1.1_4', we don't experience this issue, check you're on the latest version.

If you're using a dedicated NIC for XMLRPC (recommended) you should use a cross-over cable and set the speed\duplex to 100base TX with static IPs.

Enter the following settings for pfBlockerNG -> Sync (might be an issue with it using the generic settings):
Enable Sync: Sync to hosts defined below
Protocol: HTTPS
Target IP/Hostname: backup pfSense IP
Target Port: 443
Target Password: your admin password

We had a few problems when we initially built our environment on Broadcom NICs, we now use Intel.

Hope this helps.

That ticket was referring to the keep-alive pinger process, which is what I already mentioned.

The two systems check heartbeats but that's at a completely different level than IPsec.

For 99.9 of people it works fine as-is.  For someone with a misconfigured network it'll have a problem, like you had, but there is so little benefit to "solving" this corner case it's just not worth doing. It could negatively impact cases that are working fine now.

Thank you viragomann, it make sense :D, I'll start from CARP.

SenseRider

Yes, you can drive 4 pfSense in CARP mode and sync configuration from one to the other, but not over internet, this doesn't make any sense anyway.

For CARP all interfaces sharing the same VIP has to be connected to the same switch. On each box you have to set a different skew value for the VIP, that one with the lower skew has the higher priority.

For syncing you can only sync from one the another, so you can sync from the first (master) to the second and from the second to the third and so on.

I decided to look at the physical components and have the following (brief) layout but I no longer think the issue is with pfSense hence the briefness:

pfSense VMs /w CARP (XenServer) -> Cisco L3 -> Netgear L2 -> Client

Both pfSense VMs are accessible when connected to a VLAN 20 port on the L3 switch. Now the issue, when you get to the VLAN 20 port on the L2 netgear switch it's no longer possible to reach one of the two gateways on any VLAN. If this happens to be the Slave then the VIP and Master are reachable; if it happens to be the Master then only the Slave is reachable.

This is most likely an issue between the cisco and netgear switches and will require further investigation. I'm not sure if you have any insight Derelict but the issue is no longer as originally described.

EDIT: Upon even further investigation, it seems like the L2 is basically useless in this case. It has no clue how to handle the gateways correctly and does not come with any kind of SSH let alone HTTPS. Fun times…

I believe this may have been the solution. I seem to be able to pull the WAN or LAN cables from Primary now and get a full failover.

Do you sync the VIP's? If so that could be the cause…
Had some issues with that in the past, see: https://forum.pfsense.org/index.php?topic=102740.msg572905#msg572905

This looks like a solved problem. Wouldn't restoring the master offline and enabling persistent maintenance mode solve this?

Restore offline
Enable maintenance mode
shutdown
Put back in line
start
Let everything settle, all CARP should be BACKUP
Check all your packages/services
Disable maintenance mode

Or just add this to the bottom of your config and restore inline:

<virtualip_carp_maintenancemode></virtualip_carp_maintenancemode>

So after pulling my hair out for a few more hours, going back to a stock configuration, removing options in /boot/config.conf.local, and trying a few more things I finally figured out what's going on.

My setup is perfect. This is validated by changing my public IP address from 'CARP' to 'IP Alias'. Everything works great in 'IP Alias' mode, but of course this lacks failover for the WAN.

Everytime I switched to 'CARP' for my public virtual IP I would lose connectivity to my public gateway and I would see this in the system.log:

pfsense01 kernel: arpresolve: can't allocate llinfo for <public ip="" address=""> on igb0</public>

This led me to some more googling around and stumbling upon this: http://marc.info/?l=pfsense-support&m=129985175716097

Here's an update on this issue that may prove helpful for other
customers that reside on a passive optical network and want to use
CARP.  The service provider, service provider hardware vendor, and I
have performed extensive troubleshooting on why CARP was not working
over the passive optical network, and we have gotten it working. 
Basically, there were 2 issues on the Service Provider side that was
preventing the CARP IP's from being accessible from beyond the ONT.

First the typology.  The redundant set of pfsense boxes were plugged
into a managed layer 2 switch.  The ONT was plugged into this switch
also.  The ONT terminated at the service provider's local point of
presence into a managed layer 2 switch.  Up-links from the service
provider's switch led to a router in their core network.

2.  The Service provider, in order to make its standard residential-type
configuration work efficiently, had proxy-arp enabled on its router. 
This being enabled was causing the original problem of the secondary
firewall not being able to ping the CARP IP's.  Disabling proxy-arp for
the Pfsense and CARP IP's on the service provider's router fixed this
initial issue, but we could still not communicate with the CARP IP's
from a remote network or from inside the Service Provider's network.

3. The Service provider, as a security item, had its local point of
presence switch configured to not allow communication between switch
ports (even on the same VLAN).  Any local traffic bound for other local
traffic on the switch had to first travel to the provider's core
network.  Unfortunately, this broke CARP by preventing the core router
from asking for which MAC's belonged to the CARP IP's.  The service
provider removed this restriction for the IP's in use by the PFsense box
and CARP, and things started working correctly.

I hope this helps others who may have Internet Service over a Passive
Optical Network and are having trouble getting CARP working.

This makes perfect sense as to why CARP fails for me. My ISP is a local telcom that just performed a fiber rollout previously to mostly residential customers. I'm now going to reach out to my ISP to see if we can get CARP working properly as I'm 99% sure the problem is due to my ISP.

carp_vs_alias.png
carp_vs_alias.png_thumb

Hi jimp,

thanks for the answer.

Will try that. Unfortunately this means that I cannot add vlans/interfaces on the fly, because all ipsec tunnels, openvpn tunnels and so on will get re-established two times.

Greets

Sorry long weekend with this upgrade.
Well so far the upgrade worked, but, some of the packages didn't do so well.
Squid and Squidguard didn't want to work. I had to run some command to remove pbi links of some sort.
At this point I'm still under 2.2.6 because we still doing testing, and because syslog-ng will NOT run under 2.3.2.
I was able to get syslog-ng running with no issues in 2.3.1. The moment I updated with the latest patch to 2.3.2, syslong-ng stopped working and I cannot see any logs showing any errors about it.
And its driving me nuts. I don't know how to fix that. I was able to post a question in another thread and I'm hoping someone could help me with that.
But the upgrade was quick about 35 mins and when I did flip from Master to Backup, no ping loss, worked flawlessly. Unfortunately I had to flip back to Master firewall due to too many issues, which I hope I can resolve soon.
Thanks for all your help guys…

Cem,

I know you emailed me privately, but I figured since you also posted here I would reply again on the public forum in case others would benefit from the discussion. As I said in my private email, I highly suggest you try OpenVPN if you are dealing with multi-wan (and maybe dynamic IPs?).  It is just more suited to your task than IPSEC at this point.  If you must use IPSEC then as dotdash mentioned, you can use a DynDNS-type service tied to a gateway group so that your endpoints will get updated automatically if one link goes down.  Keep in mind that even if your DNS provider allows for very short TTL's (5 minutes is basically the practical lower limit) you will have some downtime before this failover happens until DNS propagates and adjusts.  It could be anywhere from 1-10 minutes.  I have done this and yes it does work but it is not ideal and sometimes a simple alert & manual intervention can be faster.

Good luck (kolay gelsin) ;)

Not a supported config. If it's virtual, why don't you just bump the resources on the VM so it can handle the full load? Are they on the same underlying hardware? If it's active/passive, the backup won't use much cpu/memory.

thanks for the explanation of doubts ..
Tomorrow I will try to contact an engineer from ISP ..
At the moment, each of pfsensów is plugged into a separate port of switch DCN DCS-4500-10C, which is owned by ISP
Finally as part of the test can switch between the ISP plug in any Cisco (eg. C3750) and check if the variations in work

–-----

thanks for the clarification and draw attention to the configuration of the switch .. now everything is OK
ISP filtering, turn on your switch by default GVRP and GMRP on ports clients
And that was the problem .. after filtering off GVRP and GMRP on ports which I used everything behaves correctly with 27-bit mask set in CARP
Another new experience, a man learns his whole life :-)
So far I've used in a production environment several devices F5 Networks that work in the HA cluster quietly use probably just CARP and this combined with cisco switches work always without a problem even when the aggregation ports and support for multiple VLANs ..