So after pulling my hair out for a few more hours, going back to a stock configuration, removing options in /boot/config.conf.local, and trying a few more things I finally figured out what's going on.

My setup is perfect. This is validated by changing my public IP address from 'CARP' to 'IP Alias'. Everything works great in 'IP Alias' mode, but of course this lacks failover for the WAN.

Everytime I switched to 'CARP' for my public virtual IP I would lose connectivity to my public gateway and I would see this in the system.log:

pfsense01 kernel: arpresolve: can't allocate llinfo for <public ip="" address=""> on igb0</public>

This led me to some more googling around and stumbling upon this: http://marc.info/?l=pfsense-support&m=129985175716097

Here's an update on this issue that may prove helpful for other
customers that reside on a passive optical network and want to use
CARP.  The service provider, service provider hardware vendor, and I
have performed extensive troubleshooting on why CARP was not working
over the passive optical network, and we have gotten it working. 
Basically, there were 2 issues on the Service Provider side that was
preventing the CARP IP's from being accessible from beyond the ONT.

First the typology.  The redundant set of pfsense boxes were plugged
into a managed layer 2 switch.  The ONT was plugged into this switch
also.  The ONT terminated at the service provider's local point of
presence into a managed layer 2 switch.  Up-links from the service
provider's switch led to a router in their core network.

2.  The Service provider, in order to make its standard residential-type
configuration work efficiently, had proxy-arp enabled on its router. 
This being enabled was causing the original problem of the secondary
firewall not being able to ping the CARP IP's.  Disabling proxy-arp for
the Pfsense and CARP IP's on the service provider's router fixed this
initial issue, but we could still not communicate with the CARP IP's
from a remote network or from inside the Service Provider's network.

3. The Service provider, as a security item, had its local point of
presence switch configured to not allow communication between switch
ports (even on the same VLAN).  Any local traffic bound for other local
traffic on the switch had to first travel to the provider's core
network.  Unfortunately, this broke CARP by preventing the core router
from asking for which MAC's belonged to the CARP IP's.  The service
provider removed this restriction for the IP's in use by the PFsense box
and CARP, and things started working correctly.

I hope this helps others who may have Internet Service over a Passive
Optical Network and are having trouble getting CARP working.

This makes perfect sense as to why CARP fails for me. My ISP is a local telcom that just performed a fiber rollout previously to mostly residential customers. I'm now going to reach out to my ISP to see if we can get CARP working properly as I'm 99% sure the problem is due to my ISP.

carp_vs_alias.png
carp_vs_alias.png_thumb

Hi jimp,

thanks for the answer.

Will try that. Unfortunately this means that I cannot add vlans/interfaces on the fly, because all ipsec tunnels, openvpn tunnels and so on will get re-established two times.

Greets

Sorry long weekend with this upgrade.
Well so far the upgrade worked, but, some of the packages didn't do so well.
Squid and Squidguard didn't want to work. I had to run some command to remove pbi links of some sort.
At this point I'm still under 2.2.6 because we still doing testing, and because syslog-ng will NOT run under 2.3.2.
I was able to get syslog-ng running with no issues in 2.3.1. The moment I updated with the latest patch to 2.3.2, syslong-ng stopped working and I cannot see any logs showing any errors about it.
And its driving me nuts. I don't know how to fix that. I was able to post a question in another thread and I'm hoping someone could help me with that.
But the upgrade was quick about 35 mins and when I did flip from Master to Backup, no ping loss, worked flawlessly. Unfortunately I had to flip back to Master firewall due to too many issues, which I hope I can resolve soon.
Thanks for all your help guys…

Cem,

I know you emailed me privately, but I figured since you also posted here I would reply again on the public forum in case others would benefit from the discussion. As I said in my private email, I highly suggest you try OpenVPN if you are dealing with multi-wan (and maybe dynamic IPs?).  It is just more suited to your task than IPSEC at this point.  If you must use IPSEC then as dotdash mentioned, you can use a DynDNS-type service tied to a gateway group so that your endpoints will get updated automatically if one link goes down.  Keep in mind that even if your DNS provider allows for very short TTL's (5 minutes is basically the practical lower limit) you will have some downtime before this failover happens until DNS propagates and adjusts.  It could be anywhere from 1-10 minutes.  I have done this and yes it does work but it is not ideal and sometimes a simple alert & manual intervention can be faster.

Good luck (kolay gelsin) ;)

Not a supported config. If it's virtual, why don't you just bump the resources on the VM so it can handle the full load? Are they on the same underlying hardware? If it's active/passive, the backup won't use much cpu/memory.

thanks for the explanation of doubts ..
Tomorrow I will try to contact an engineer from ISP ..
At the moment, each of pfsensów is plugged into a separate port of switch DCN DCS-4500-10C, which is owned by ISP
Finally as part of the test can switch between the ISP plug in any Cisco (eg. C3750) and check if the variations in work

–-----

thanks for the clarification and draw attention to the configuration of the switch .. now everything is OK
ISP filtering, turn on your switch by default GVRP and GMRP on ports clients
And that was the problem .. after filtering off GVRP and GMRP on ports which I used everything behaves correctly with 27-bit mask set in CARP
Another new experience, a man learns his whole life :-)
So far I've used in a production environment several devices F5 Networks that work in the HA cluster quietly use probably just CARP and this combined with cisco switches work always without a problem even when the aggregation ports and support for multiple VLANs ..

did you find any solution to this?
I am also looking for same thing.

I manually change all skew to 0 on master. On secondary sever skews changes to 100 automatically. But secondary server stop show CARP status. After reboot secondary it normally show status, and master role normally migrate from master to backup and reverse.

Thank you!

It will only work when that unit is Master. The other side should be configured to connect to the CARP VIP. They will get whichever unit is Master at the time.

Outbound NAT rules should be source-address limited to the addresses you actually want to NAT. If those are any that is almost never right.

Outbound NAT should almost always be set to have a NAT address of the CARP VIP, too. But those rules are for IPsec passthrough from clients behind the firewall, not for Site-to-site connections from the firewall itself.

Update: 2 rounds of rebooting both boxes seems to have gotten everything working.

Problem solved –- had to enable rp_filter in linux.

The first thing to check is if the secondary can resolve names, check for updates, etc while in backup status. And if not why not.

You probably want to start another thread and provide more details there.

Failover Groups I can not work like this allows me to answer.

I was with a feedback as you like.

Thanks for the reply, Derelict.

Sorry for my slow reply. I went in for surgery, and am just back on my feet again.

It is indeed unusual. I can see the connection in the state table of the master node, with TIME_WAIT:TIME_WAIT, and 9/4 packets, but the browser tells me the connection was reset, and indeed telnet to the management port is denied as well. ARP of the pfSense VIP is correct on the pinging machine, and ARP of the pinging machine is correct on the pfSense box. It seems like pfSense is blocking the connections, even though it's been told to allow them through (my allow rule uses an alias that includes the VIP and both real IPs of the pfSense boxes). Very, very strange.

I'll walk through the Network Connectivity process and see if it turns up anything.

In a last ditch effort to get things running again, I blew reset the config on FW2 and started over.  Since this is an HA pair, I just did the initial setup and had FW1 sync over the settings.  This seems to have fixed the problem.  The secondary FW is in BACKUP mode and the traffic is very minor (16KB/sec).

Not sure what happened, but something must have gone wrong during the upgrade from 2.2.6 to 2.3.2.  I might consider doing the same on FW1 (reset to factory then sync from FW2).

Thanks again for helping out!

How is your Outbound NAT configured?  I had a similar problem whereby my Outbound NAT was tied to the interface and not the VIP.  Here is what I did:

Firewall -> NAT Click Outbound NAT Make sure your "NAT Address" for the LAN subnet is tied to the external CARP IP address instead of interface address

Hope this helps.