@whitwye: If not, and you know where to look on the CLI to check that, I can put together a plugin to watch that. We need it here. Thanks! I got a few for Check_MK you can adopt. Basically, neither CARP nor uCARP give good monitoring interfaces. This is what I used, and also some time had adopted for pfSense (I think) https://github.com/FlorianHeigl/nagios/tree/master/check_mk/ucarp_status The main thought behind this is to monitor the initial role of both nodes and then compare that.

And sorry for late responses - my settings are set to notify me on reply and I'm not getting them. I just turned it off and back on. Hopefully I will catch these quicker. Again - thanks for everyones input.

Why should promisc have to be enabled?  Not making any sense.. Is this on some sort of virtual distributed switch?

@xonacs: When using private IPs, the secondary (standby) unit never has internet access until failover occurs.  Therefore, this issue seems to be related to the standby unit not having internet and/or not reaching the gateway. That's likely the entire issue. Which is why we don't recommend using that style of configuration on a primary WAN. For a non-default/secondary WAN it can be OK, or for internal interfaces, but both units need to have functioning Internet access, or at least functioning DNS. Now if your private IP addresses on WAN can get out (upstream does NAT, for example), and your NAT rules on WAN are OK, then it's possible the units themselves could get out and be OK. If traffic leaving the firewall must use the CARP VIP to exit, then probably not. You might try spinning up a local DNS server off the firewalls and then point DNS on the firewalls to that, see if it helps.

sounds like a can of worms i dont really want to be opening on myself! Its a single site with remote vpn users, long as the SG-4860's rock solid, we should be fine. Cheers JimP

Thanks for your reply, the "VLAN" thing would be one alternative without an additional network card… but at the moment we do not have any VLANS and no switches which support VLANS. Meanwhile I have contacted our provider: the only possibility with our line solution are two network interfaces and two switches for WAN access. Every provider line(Router) is connected to a MASTER and a BACKUP switch. The switches are connected together. Because we also use cheap switches the solution for us is to use the LAGGS in pfsense (we already have them configured because of CARP and pfsync).  So we will use a second network interface in the LAGG in failover mode for WAN access and both pfsense nodes are connected to both switches. The only problem is to get some old supported PCI dual network cards... because the hardware is ancient  ;D I found this old compatibility list https://forum.pfsense.org/index.php?topic=25.msg58#msg58  ....

I would fix the source of the problem (your layer 2 gear sending its own advertisements back to you.) instead of suppressing the logs. They are telling you there is a problem.

You can have both uplinks active if you enable this advanced host parameter: Net.ReversePathFwdCheckPromisc  (see pfSense Troubleshooting guide) By the way I discovered today that if your VM has "VM DirectPath IO" enabled it bypass this parameter and you will have duplicated packet again.

active/active is not a supported configuration. All VIPs on one node should be MASTER. All VIPs on the other should be BACKUP. If not your configuration is invalid. Promiscuous mode is not required to receive CARP heartbeats. Promiscuous mode in the hypervisors is so the hypervisor will pass the traffic to the VM for alternate MAC addresses and really has nothing to do with pfSense, but the "switch" in that case. Which is what has been pointed out to you as the almost certain cause of your problems multiple times regarding your environment but you refuse to listen. You will not find a list of all the stupid things people try to do that they can't do in the book. It would be a billion pages long.

Found the relevant docs for this https://doc.pfsense.org/index.php/Redundant_Firewalls_Upgrade_Guide and it does indeed say for anything before 2.2.5 upgrade the master first. thanks for the help

Yeah. CARP maintenance mode is your friend there.

Problem Solved using a good example from the you tube https://www.youtube.com/watch?v=VjDL8T99_c8&t=1235s

You can't do that.  Please see: https://forum.pfsense.org/index.php?topic=132909.0

CARP is done on the interfaces themselves. There is a far-too-common misconception that the SYNC interface has something to do with CARP. It does not. It is generally used for state sync (pfsync) and configuration sync (XMLRPC sync). If you have two WAN interfaces, each with an address and sharing a CARP VIP, those interfaces themselves need to be able to exchange CARP heartbeats. The same is true for all CARP/HA interfaces. These are multicast using 224.0.0.18. https://portal.pfsense.org/docs/book/highavailability/index.html A useful troubleshooting tool is to packet capture for CARP traffic on the interface you think should be BACKUP but is MASTER instead. The built-in packet capture will decode CARP and show you the base/advskews, etc. In the default configuration the base/advskew of the primary should be 1/0. It should be 1/100 on the secondary. You will probably see nothing but 1/100 being sent by the secondary where it should be receiving the 1/0 from the primary but it is not. If it was, it would assume BACKUP status and stop the 1/100 heartbeats. A very common misconfiguration is adding an interface but not tagging the new VLANs through all the way between interfaces, etc.

Yes, there is. Configure your machines as real HA with CARP as it should be: https://doc.pfsense.org/index.php/Configuring_pfSense_Hardware_Redundancy_(CARP) And then set up a Multi-WAN configuration with the two ISPs: https://doc.pfsense.org/index.php/Multi-WAN

You don't configure a CARP cluster like that. You configure everything on the primary and duplicate it on the secondary, preferably by letting XMLRPC sync do the duplication work. The secondary does nothing until a failover event occurs. Best source of info: https://portal.pfsense.org/docs/book/highavailability/multi-wan-with-ha.html See sig for a link to get access to the book cheap.