Following up on this topic.  I never got anywhere with it.  After reading many posts which suggested solutions such as utilizing STP to handle the potential issues that would arise from the mac address changes on the same physical interfaces - loops, etc, I've had to abandon the idea of using pfSense as a high availability solution when bridging. There just doesn't seem to be a way to really handle it without complex networking and/or adding more hardware across each side.  pfSense seems to need a bridge-specific solution to make this work without significant effort or alteration in my environment - so I've had to end the search and just use pfSense as a router as well.

To start, your provider is probably not CARP compatible, or it would likely be working. When it comes to CARP VIPs and ISPs there are two general principles that they must support. CARP advertisements egress sourced from the CARP MAC address. This performs two tasks: The switch sees the CARP MAC address and adds it to its MAC address table The BACKUP CARP node sees the advertisement and does not switch to MASTER The ISP, having traffic for the CARP VIP address does an ARP request. The pfSense WAN responds to the ARP "WHO HAS" from the interface MAC address but says the address IS AT the CARP MAC. This directs traffic from upstream to the CARP VIP to the CARP MAC address which has previously been installed in the switches MAC address table by virtue of the CARP advertisements. Upstream has to support multiple MAC addresses and multicast for CARP to function. ISP gear does some silly-ass crap. Especially residential gear. There has to be solid layer 2 between the two CARP nodes and the upstream.

If you cannot upgrade the secondary, backup the configuration, install 2.4.2 fresh, and restore the configuration. It will be a surprise to nobody that you are experiencing problems with that disparity between node versions.

Think of the DMZ as another LAN segment. It will need a CARP VIP to float between the firewalls. The Public IPs you are using for 1-1 NAT will just be CARP VIPs off the WAN.

In all honesty, I would go to 2.2.6 first. It is much more tolerant of being installed with the WAN disconnected. After you can do it in Maintenance mode with the WAN connected the other upgrades will go a lot smoother.

So, I may have solved my own problem. I created a new CSR and created a cert with an Alternate name, so that the cert would work for both nodes. It seems to work. I'll report any oddities. John

Thanks for pointing me to that bug.

You can certainly use LACP and VLANs to do LAN and WAN in the lab. Many people (me included) do not like mixing inside and outside traffic on one switch/stack. Many people (me included) do it anyway. I have not seen a recent, credible case of VLAN hopping with the exception maybe of TP-Link's VLAN1 nonsense. Even less of a reason to be concerned in the lab. But in your case, I would probably do a lag for the outside and a lag for the inside, with two interfaces each even if they are to the same stack, and one of the add-on ports for SYNC.

Glad it's working. Status > Interfaces is probably the best tool to use for this since it lists all of the interface elements in play in order in one place.

Any time there is an upgrade, especially across operating system versions, there is always a possibility that will happen. It doesn't always affect everyone, but you can never rely on pfsync working during a significant OS update.

Ok, I found out the following: The described problem only occurs to VM's hosted our 2 ESX 6.5 hypervisors. All bare metal servers will work completely fine (on-link and directly connected), only VM's are affected. On Vmware the vSwitches are configured including the following settings: Promiscuous mode enabled; MAC Address changes enabled; Forged transmits enabled; However, I don't think this is strictly needed since the firewalls are physical devices. Is someone aware of a required setting that is required in VMware / pfSense to get this correctly working? Thanks anyway :)

Got an answer from OVH that CARP is not possible for their hardware dedicated servers due to network design. I've solved this using OVH Control Panel API - https://api.ovh.com buy some OVH failover IP's (one or subnet block ) and assign them to "master" firewall in OVH Control Panel create identical "IP alias(es)" for OVH failover IP's attached to WAN interfaces on both "master" and "backup" firewalls.     Yes, create identical IP Aliases - no IP conflict will ever happen. wrote a Python script that moves above OVH failover IP's to "backup" server in case "master" firewall stops responding for let's say 10 seconds     Script can work on backup server on any other Linux/Windows server anywhere. Works just fine - API failover IP move takes about 50-55 seconds to finish. So, if scripts timeout for your "master" firewall is set to 10 seconds - you are looking at max 60-65 seconds outage for your services. Boom.

Got an answer from OVH that CARP is not possible for their hardware dedicated servers due to network design. I've solved this using OVH Control Panel API - https://api.ovh.com buy some OVH failover IP's (one or subnet block ) and assign them to "master" firewall in OVH Control Panel create identical "IP alias(es)" for OVH failover IP's attached to WAN interfaces on both "master" and "backup" firewalls.     Yes, create identical IP Aliases - no IP conflict will ever happen. wrote a Python script that moves above OVH failover IP's to "backup" server in case "master" firewall stops responding for let's say 10 seconds     Script can work on backup server on any other Linux/Windows server anywhere. Works just fine - API failover IP move takes about 50-55 seconds to finish. So, if scripts timeout for your "master" firewall is set to 10 seconds - you are looking at max 60-65 seconds outage for your services. Boom.

No, the drives don't have to match. But you really ought to get on a somewhat recent version…