Thank you guys for the clarification!

So just one client on the network was impacted? Probably that client has something hardcoded pointed at the master and not a CARP VIP (e.g. its gateway or DNS servers)

Yes, I guess I should have done that to begin with. When things are working vs not,``` ifconfig igb1 working: inet 108.245.XXX.XXX netmask 0xfffffc00 broadcast 108.245.XXX.255 inet 192.168.10.2 netmask 0xffffff00 broadcast 192.168.10.255 The obfuscated 108.245 address is the modem's address on the internet. 192.168.10.2 is the VIP, used for modem administration as the modem is at 192.168.10.1 In this scenario, 108.245.xxx.xxx is what will be used by "WAN" for firewall rules, OpenVPN, etc "Status->Interfaces" shows this address for WAN When things go wrong, I see: inet 192.168.10.2 netmask 0xffffff00 broadcast 192.168.10.255 inet 108.245.XXX.XXX netmask 0xfffffc00 broadcast 108.245.XXX.255 "Status->Interfaces" shows 192.168.10.2 address for WAN, and things are broken.

@Derelict: All of those should be changed to the CARP VIP. Thank you! It is working now. Latest upgrade to 2.3.4-RELEASE-p1 worked fine as well. Again, thanks for your help!  :D

Hooray! Confirmed working on 2.4.0-RC

Thanks jimp for the answer. I am gonna investigate haproxy ;-)

Is this a client or a server? OpenVPN servers bound to CARP VIPs are kept shut down on whichever unit has the VIP in a BACKUP state. So it's normal for them not to be running on the secondary. When the VIP state transitions to MASTER they are started automatically.

@whitwye: If not, and you know where to look on the CLI to check that, I can put together a plugin to watch that. We need it here. Thanks! I got a few for Check_MK you can adopt. Basically, neither CARP nor uCARP give good monitoring interfaces. This is what I used, and also some time had adopted for pfSense (I think) https://github.com/FlorianHeigl/nagios/tree/master/check_mk/ucarp_status The main thought behind this is to monitor the initial role of both nodes and then compare that.

And sorry for late responses - my settings are set to notify me on reply and I'm not getting them. I just turned it off and back on. Hopefully I will catch these quicker. Again - thanks for everyones input.

Why should promisc have to be enabled?  Not making any sense.. Is this on some sort of virtual distributed switch?

@xonacs: When using private IPs, the secondary (standby) unit never has internet access until failover occurs.  Therefore, this issue seems to be related to the standby unit not having internet and/or not reaching the gateway. That's likely the entire issue. Which is why we don't recommend using that style of configuration on a primary WAN. For a non-default/secondary WAN it can be OK, or for internal interfaces, but both units need to have functioning Internet access, or at least functioning DNS. Now if your private IP addresses on WAN can get out (upstream does NAT, for example), and your NAT rules on WAN are OK, then it's possible the units themselves could get out and be OK. If traffic leaving the firewall must use the CARP VIP to exit, then probably not. You might try spinning up a local DNS server off the firewalls and then point DNS on the firewalls to that, see if it helps.

sounds like a can of worms i dont really want to be opening on myself! Its a single site with remote vpn users, long as the SG-4860's rock solid, we should be fine. Cheers JimP

Thanks for your reply, the "VLAN" thing would be one alternative without an additional network card… but at the moment we do not have any VLANS and no switches which support VLANS. Meanwhile I have contacted our provider: the only possibility with our line solution are two network interfaces and two switches for WAN access. Every provider line(Router) is connected to a MASTER and a BACKUP switch. The switches are connected together. Because we also use cheap switches the solution for us is to use the LAGGS in pfsense (we already have them configured because of CARP and pfsync).  So we will use a second network interface in the LAGG in failover mode for WAN access and both pfsense nodes are connected to both switches. The only problem is to get some old supported PCI dual network cards... because the hardware is ancient  ;D I found this old compatibility list https://forum.pfsense.org/index.php?topic=25.msg58#msg58  ....

I would fix the source of the problem (your layer 2 gear sending its own advertisements back to you.) instead of suppressing the logs. They are telling you there is a problem.

You can have both uplinks active if you enable this advanced host parameter: Net.ReversePathFwdCheckPromisc  (see pfSense Troubleshooting guide) By the way I discovered today that if your VM has "VM DirectPath IO" enabled it bypass this parameter and you will have duplicated packet again.

active/active is not a supported configuration. All VIPs on one node should be MASTER. All VIPs on the other should be BACKUP. If not your configuration is invalid. Promiscuous mode is not required to receive CARP heartbeats. Promiscuous mode in the hypervisors is so the hypervisor will pass the traffic to the VM for alternate MAC addresses and really has nothing to do with pfSense, but the "switch" in that case. Which is what has been pointed out to you as the almost certain cause of your problems multiple times regarding your environment but you refuse to listen. You will not find a list of all the stupid things people try to do that they can't do in the book. It would be a billion pages long.

Found the relevant docs for this https://doc.pfsense.org/index.php/Redundant_Firewalls_Upgrade_Guide and it does indeed say for anything before 2.2.5 upgrade the master first. thanks for the help