I created a new vm, assigned the offending IP to that VM, let it hang out for a day, deleted the VM and the DNS records in Infoblox. Finally, after the weekend the record has cleared in Infoblox and presumably the switch. Switch weirdness in the end, I suppose. Thanks to Derelict for the suggestions in trying to track this down.

That's called out in the documentation in several places https://docs.netgate.com/pfsense/en/latest/book/highavailability/high-availability-troubleshooting.html#issues-inside-of-virtual-machines-esx https://docs.netgate.com/pfsense/en/latest/highavailability/configuring-high-availability.html#vmware-esx-users https://docs.netgate.com/pfsense/en/latest/highavailability/troubleshooting-high-availability-clusters.html#hypervisor-users-especially-vmware-esx-esxi [...]

As long as the L1/L2 between them is clean and fast that should be OK. You can adjust advbase if it isn't. With a higher advbase it won't fail over as fast, but it is more tolerate of latency between the nodes. Having them in different places in the same building is not unheard of, or even separate buildings on the same campus. It's not ideal but it can work. I think we even had someone try to use CARP for HA to a node in a DC as some attempt at DR, but I can't remember if that ended up working in the end. When working with skew and base values for CARP VIPs, Skew values are 1/256th of a second, and base values add whole seconds. So if you set advbase to 1, then it would wait 1 second + skew, rather than only the skew time.

In other words: Using the CARP VIP you get guaranteed failover and consistent behavior across all client platforms. Using both you are completely reliant upon the client to behave in specific ways, which only gets worse on networks with many different types of clients.

@Derelict Thanks for the confirmation that the behaviour is as expected. Very much appreciated.

I think I'm at least partially encountering this bug, which I updated with what I'm seeing: https://redmine.pfsense.org/issues/6579

Thanks for the reply

Thanks for contacting me. We did mostly static reservations but then it got too much overhead with devices coming online. And yes, when the backup dhcp answers it registers with unbound on the backup DNS where I can see the entry. but the backup DNS does not sync back to the master, and when there is an update from the master the entry in the backup gets wiped (i think, didn't verify). I thought I had a solution by blocking dhcp from 0.0.0.0 to lan-address:67 which gets propagated to the backup. and dhcp traffic is only received via the carp address. That seemed to work but then I found a client with which it did not work haven't found why. The proper way is to turn DDNS and have a separate DNS server to take the registration from either dhcp server. This will also solve one deficiency of pfsense where it can not resolve/access DNS servers on the other side of an ipsec tunnel (if you have branch offices) We had to resort to having a second caching DNS server for that purpose to forward inquiries too.

@JeGr Hi JeGr Thank you for you explanation. I've talked to my provider and they can supply me with a transit network and route a /29 through it. Though their /29 is more expensive than renting a /24 from a provider. My concern is if they will be willing to announce this /24, if they have to or they can refuse? the price they will charge for it I will clarify it tomorrow.

PROBLEM SOLVED: Call on OVH to activate the promiscuity mode on our WAN interfaces. From now on everything is working, thank you for your help.

OMG. My bad! I have protected the HTTP directory password. The password was stored on the external system in the browser. So LE could not access it. Sorry for my misfortune :-(

No. You set up outbound NAT on the inside interface of the HA pair. You need connections to the backup node to appear as they are coming from the master node's inside interface. That way reply traffic is same-subnet so it will be routed correctly. This should be configured in both directions since you might want to access the primary while the secondary is master.

René, Depends on your modem. You can. Ask your isp for the instructions. But to you can try to log on to the modem and look for DMZ host or forwarding host and have it send all the data to your internal private specific IP address. Niels