@SteveITS Thanks for the link. I had not considered the VIP stacking idea. I'm not sure how much of an issue my setup will have with VIP multicast traffic on my WAN link but good to know this technique is available to reduce some of that. Again, thanks for the link.

@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Well it would remove load from the firewall. So if you were under a DDoS attack and needed to still route between internal subnets that could be useful. But it wouldn't help with the attack itself much.

Agree! Anyway for middle/big DDoS better to deal on local ISPs + CloudFlare level. Here no room for edge FW… :)

Let me to note, if thinking in “Zero thrust” direction, also FW on end local node/service as “fine tuning firewalling” would be great, because each end node better know what particular (and how) need to be secured.

So at the end we build 3-layered (as minimum) defense:

ACLs on edge ASIC-based switches; pfSense as edge FW; PF/IPF/IPFW FW (sertificates, tokens, etc…) on end node/service;

What do You think about this 3-layered scheme, @stephenw10 ?

Happy to report that I was able to upgrade both of my servers to CE 2.8 with no issues. Everything still works as expected. 👍

So thought about this a bit and realize I'd need to have the CARP VIP on the LAN to if nothing more facilitate the failover state on the WAN in the case of a LAN failure. CARP on here fails together so would still want the CARP VIP IP on the LAN even if I don't technically need it for routing traffic.

Still could use some help on getting the next hop for the default route learned by my switches to be the VIP address and not the LAN interface IP.

I thought there was a doc page on this but can't find it. Maybe it was a forum post. All I can say is, it's supposed to move both.

https://docs.netgate.com/pfsense/en/latest/highavailability/test.html#test-carp-failover
notably, "Unplug the WAN or LAN cable" (my bold)

I tried a quick search and found some really old stuff like https://www.reddit.com/r/PFSENSE/comments/4yebk5/comment/d6s45xk/ but note Jim-P I'm pretty confident is https://www.netgate.com/blog/author/jim-pingle.

@Yathus said in [SOLVED]CARP Cluster, what will happen if i bring back my backup online after configuration change ?:

OK we started this night and backup node tried to take "Primary" on the two CARP IP, backup node won't came up as backup status so...

I think this is expected behaviour, untill it synced, if it possible to sync at all.

I'm glad everything worked out for you.

I'd like to slightly correct your terminology, which is also referenced in the documentation. Refer to the firewalls as Primary and Secondary—these are their permanent roles. Only their status changes, which can be either Master or Backup.

And for the future, if everything is set up correctly, there's no need to power off the Secondary firewall at all. It should properly synchronize what it needs to. If synchronization of certain settings isn't possible, use Maintenance Mode or Disable CARP, provided it doesn't cause conflicts in the network.

@bp81

https://forum.netgate.com/topic/185693/ha-setup-with-multi-wan-and-dhcp-guide this is an excellent way to actualy get failover to occur with single ip dhcp WAN side.

if you already have the spare hardware, this would be the optimal solution. full failover, (CARP only on LAN side)

@t04s

not sure, but i suggest you look at this one:

https://forum.netgate.com/topic/185693/ha-setup-with-multi-wan-and-dhcp-guide

you can get the devd system to only run on specific vhid events.

I recently followed the same guide and stumbled over that thread because I have the same issue.
Is it support now?
thanks,

@viragomann
THANK YOU!

I will study every word of your message and continue my inspection.

I'll come back here with the rest, and if I finally come up with a solution that works I'll put it here completely for others too.

@YannL I have experienced similar (or same) issues since years and could not fix it. All investigation I did, did not find the real issue.

dedicated HA sync interface which is officially supported (in our case an additional LAN card with intel chipset instead of built in RJ45 ports), MTU lowered to 1360 instead of default 1500 (https://forum.netgate.com/topic/190990/ha-sync-does-not-work-error-operation-timed-out/2) Interface physical connection is stable (permanent ping with high frequency, big packets, etc. all stable and fast iperf3 tests forward and backwards are close to the theoretical maximum of 1 GBit/s firewall rules: allow any - any on each HA interface in both pfsense cluster members user login credentials and permissions for the ha user checked webconfigurator processes set to 500 (max allowed value) cpu is idle (never less than 95% free) ram is > 90% free mbuffers increased io statistics of SSDs on both sides are very low bandwidth usage on ha interfaces during ha sync / xmlrpc is very low (just some kbit/s) checked nginx logs, system logs, error logs => no specific reason found checked the ha documentation multiple times: no error in our config found. each time ha sync / xmlrpc is happening, I can see on the console of the backup/passive member messages that port bindings of the captive portal ports fail: Message from syslogd@srvwlan02 at Apr 10 01:39:53 ... nginx: 2025/04/10 01:39:53 [emerg] 44580#101678: bind() to [::]:8006 failed (48: Address already in use)

maybe you have some idea what you can check further or not. We are lost...

@RobertK-1 You know...I had not considered that, you may be right. either way. it doesnt switch smooth without some drop, so i've just accepted that, and it does work well as it is. I was just chasing the possibility if a fail over with no network loss..pretty hard.

Thanks, yeah, makes sense, but using the gateway of SUBNET1 IP 3.x.x.3/26 on the .20 server disables access to the server all together. The route to the public SUBNET1 from the data center flows to the WAN Virtual IP address. Perhaps for these reasons, I am not able to do what I'd like. I'm looking for a way to pfSense to handle the responses when the server itself is always going to do that. The server is a TrueNAS server, perhaps I'll need see if there is a way to route back to the two different routers depending on which one it came from? There is a layer 3 switch between the two as well, maybe something there could help.

If you have OpenVPN server running you might be affected by this bug: 13569

The workaround:

Commenting out the line /sbin/pfctl -i $1 -Fs in /usr/local/sbin/ovpn-linkdown "fixes" the issue

@Xando said in Incorrect definition of CARP roles:

What is your version of pfsense?

It does run on 2.7.2 CE, I really suspect Hyper-V - QEMU combination.

Do you have the patients and/or time to setup the backup node on Hyper-V (export the config of the backup node, import on another Hyper-V machine)?

Add: Or a package capture, although I haven't done that for CARP and don't know what to expect.

@jason0 I am just facing the same issue! Any tricks to get GIF over CARP?