@viragomann
THANK YOU!

I will study every word of your message and continue my inspection.

I'll come back here with the rest, and if I finally come up with a solution that works I'll put it here completely for others too.

@YannL I have experienced similar (or same) issues since years and could not fix it. All investigation I did, did not find the real issue.

dedicated HA sync interface which is officially supported (in our case an additional LAN card with intel chipset instead of built in RJ45 ports), MTU lowered to 1360 instead of default 1500 (https://forum.netgate.com/topic/190990/ha-sync-does-not-work-error-operation-timed-out/2) Interface physical connection is stable (permanent ping with high frequency, big packets, etc. all stable and fast iperf3 tests forward and backwards are close to the theoretical maximum of 1 GBit/s firewall rules: allow any - any on each HA interface in both pfsense cluster members user login credentials and permissions for the ha user checked webconfigurator processes set to 500 (max allowed value) cpu is idle (never less than 95% free) ram is > 90% free mbuffers increased io statistics of SSDs on both sides are very low bandwidth usage on ha interfaces during ha sync / xmlrpc is very low (just some kbit/s) checked nginx logs, system logs, error logs => no specific reason found checked the ha documentation multiple times: no error in our config found. each time ha sync / xmlrpc is happening, I can see on the console of the backup/passive member messages that port bindings of the captive portal ports fail: Message from syslogd@srvwlan02 at Apr 10 01:39:53 ... nginx: 2025/04/10 01:39:53 [emerg] 44580#101678: bind() to [::]:8006 failed (48: Address already in use)

maybe you have some idea what you can check further or not. We are lost...

@RobertK-1 You know...I had not considered that, you may be right. either way. it doesnt switch smooth without some drop, so i've just accepted that, and it does work well as it is. I was just chasing the possibility if a fail over with no network loss..pretty hard.

Thanks, yeah, makes sense, but using the gateway of SUBNET1 IP 3.x.x.3/26 on the .20 server disables access to the server all together. The route to the public SUBNET1 from the data center flows to the WAN Virtual IP address. Perhaps for these reasons, I am not able to do what I'd like. I'm looking for a way to pfSense to handle the responses when the server itself is always going to do that. The server is a TrueNAS server, perhaps I'll need see if there is a way to route back to the two different routers depending on which one it came from? There is a layer 3 switch between the two as well, maybe something there could help.

If you have OpenVPN server running you might be affected by this bug: 13569

The workaround:

Commenting out the line /sbin/pfctl -i $1 -Fs in /usr/local/sbin/ovpn-linkdown "fixes" the issue

@Xando said in Incorrect definition of CARP roles:

What is your version of pfsense?

It does run on 2.7.2 CE, I really suspect Hyper-V - QEMU combination.

Do you have the patients and/or time to setup the backup node on Hyper-V (export the config of the backup node, import on another Hyper-V machine)?

Add: Or a package capture, although I haven't done that for CARP and don't know what to expect.

@jason0 I am just facing the same issue! Any tricks to get GIF over CARP?

@awebster

You must uncheck Synchronize Vitual IPs in the System -> High Avail. Sync, otherwise the MASTER will keep overwriting the ADVBASE value.
This also means you must manually configure the VIP address on each box, initially check the Synchronize Vitual IPs when you do the setup, then uncheck it to go into production, and never check it again.

In the year 2025 I found this comment which resolved my problem!

@michmoor said in CARP alternative:

if the secondary firewall needs to install patches/packages, is that when you just flip it to Master (One WAN IP being shared).

It needs to have packages and updates at all times, not just when it's master. Otherwise you'd have to fail over to it to do any sort of maintenance, which defeats the idea of HA to reduce disruptions.

@bp81 I ended up performing the addition using the first method you mentioned, plus the firewall configuration, and then it worked like a charm. It even added all of the packages to the new node.

Now, I'm dealing with an IPSec speed issue, but that is a whole separate issue and one I have already opened another thread on.

Thank you for taking the time to reply; it is much appreciated, and now if someone else is looking for the same thing, they have some really good options!

Have an excellent day!

TSoF

@Nyxtorm said in Different CARP LAN - WAN unplug behavior:

I'm replying to myself in case anyone has the same case.

On my WAN interfaces, I had a static IPv4 (on the local subnet of my Livebox (French ISP router)), and an IPv6 in DHCP6.

Once I've also set the IPv6, and the gateway v6 to static, the behavior is fine when I disconnect the WAN: the WAN interface goes to INIT, the others to BACKUP on the primary, and the secondary recovers MASTER status on all interfaces.

I believe that this behavior is what you would typically see if one of your WAN interfaces (your IPv6 gateway in this case) is set to DHCP instead of static addressing. CARP/HA doesn't tend to work that well with WANs using DHCP instead of static addressing.

We are also seeing this since we upgraded to 24.11 with all patches applied.
"A communications error occurred while attempting XMLRPC sync." on primary node.

Accessing the webgui on secondary node hangs the firewall after 5-10 seconds.
If we access the CLI everything seems fine and no hangs unless we initiate a reboot, Then the secondary hangs and we need to pull the power to recover.

This happens usually after 5-14 days of uptime of secondary node.

It's helpful thanks for sharing.

I am seeing a similar problem on pfSense+ 24.11 (patches applied).

The ADMIN group being REMOVED from user rights assignments on secondary/backup HA cluster members any time the password is changed on the primary member.

I am having to logon to the secondary members and manually add the user(s) back to the ADMIN group.

This is not desired behavior, and I confirmed it is not happening on CE 2.7.2 (patches applied).