Solved. (Although I still have temporary outages after each DHCP configuration change.)

TLDR; I had the skew on my VIP addresses set to 100 and 200 instead of 0 and 100. Unfortunately I hadn't noticed this since the DHCP failover was working as expected prior to the config change. The notes in troubleshooting HA DHCP failover are worth careful study.

The smoking gun in my post above is that all the communication is on port 519 and not a mix of 519 and 520. This was caused by both of my pfSense units believing they were secondary (since the skews were high). This was found by browsing /var/dhcpd/etc/dhcpd/conf and looking under the clause marked "failover peer".

Thanks! Daryl

The opt number for the interface is probably the most important value. Everything else references that, firewall rules, NAT rules etc.

That's why you can reassign the interface to a new NIC or rename it and all the config will follow it. If you change the opt number you have to change everything that references it or recreate everything else.

The opportunity for typo'ing something here is large!

@viragomann TNX, i forgot it was there.

@Igor-Moura Happy to help.

I'm still not coming up with any reason this should be happening, quite odd if I'm being honest. It sounds like a configuration thing but I'm not sure what would actually cause that, if it were a bug though I would imagine my test or prod HA environments would be seeing it too.

I'll keep thinking on this and come back if I have any other ideas.

@stephenw10 any thoughts on this thread here? Nothing is immediately coming to mind that would cause traffic to end up on both nodes.

@SteveITS Yeah this is correct, no need for a managed switch, personally though I usually use an existing switch, setup the VLAN with tags at the port level, this way I don't need a different switch for WAN. So that's why I suggested that, but only works if you have available ports on an existing managed switch, otherwise an unmanaged one makes more sense.

Hi,

Already found out that there was a VLAN misconfiguration on slave pfSense, even if this shouldn't affect the wan and gateway functionality.
I'll check up all of the ports/cables as soon as I can.

@planetinse said in Can i use VIPs and CARP on non-HA configured firewalls?:

then i should be able to use more than two firewals participating in a fail over / loadbalance scenario - right?

Yes on VIPs.

No on this concept. This is only supported on two pfSense systems in HA.

@Luis-Cordero OK so looks like both hosts are running XCP-ng.

There are a lot of things that could cause this behavior, are the two hosts in a pool together or 2 separate pools? Either way they are probably communicating over a switch, maybe that physical switch doesn't have the right VLANs in place?

@jimp Thank you!

@Delegator5042 said in Setting up CARP Master and Backup on a per VLAN basis (Like VRRP): is it possible?:

I've read that you can use a switch on the ISP Ethernet connection so it could be shared with multiple routers, but I haven't tried this.

Yes, put a small switch into each line or even a VLAN capable switch and split it into two virtual switches.

Consider that for CARP, you need an IP on each pfSense and a third as VIP. So you should have 3 IPs on each.
If you haven't there is also a way to configure private IPs on the boxes, but this has some drawbacks.

I would still like to know if I could force a vlan (or subnet) to use a specific gateway and only when that gateway is down to send the traffic over to designated backup connection.

You can configure a gateway group and set this as default gateway.
For routing traffic other than according the default gateway you can do policy routing by stating a gateway in the firewall pass rule.

You your purposes you can configure an additional gateway group, say with inverted priorities, and use this in the policy routing rule.

@jimp Yea i know - but there is no other way when a single instance can not take the load, especially since it's a single CPU process only
(see load below) - other ways to solve this ? please enlighten me :-)
cce21a91-f3c8-4bdf-ab67-99f1a3fc7d85-image.png

I have handled this in the past by simply unlink CARP sync and manually set skew for VIP's to loadbalance load over two HA's

Example:
so some customers has fw1 as primary and some other customers has fw2 as primary - failover still works.

@johnpoz Thank you so much. I configured followed your comment.

@viragomann said in ACL conditions:

So are these IPs behind pfSense or are these IPs assigned to WAN?

Ok, so we are using this in L4 mode, and it's working fine. To reply to you we have no need to hide those public IPs, for us those are on a DMZ assigned to our WAN, so it's ok.

Thanks for your support Virago, sadly we are struggling a bit on this.

@nouman786
Which OpenVPN client are you talking about? Incoming connections to the firewall or outgoing from behind pfSense?
If the issue is on incoming connections, do you connect to the WAN CARP VIP?

Is your HA setup working properly, so that all interfaces are in master state on the primary and in backup on the secondary?
Are all devices inside your network using the CARP VIP as their default gateway?

@jimp
Sorry for my late reply. I performed several tests and CARP is working fine now :) Thanks for your help!