@benrichardson_insync So its is expected to have this behaviour. Carp interfaces must be on the same broadcast domain. The master sends regular advertisements to the backups. See here for more details about the mechanism https://www.netbsd.org/docs/guide/en/chap-carp.html

"Flush all states when a gateway goes down" has to be off, otherwise states are killed even when a gateway reaches the high watermark, that has been set to down.

what is the error message you received? make sure noting broken in L2 level

It can depend on the switch/router on the other end of the cable. For instance with Comcast routers often when replacing a router in an office (inside the Comcast router) I've found it's fastest to power off or reboot the Comcast router so it learns the IP has a new MAC. If you have the second router on, and are just plugging in cables, I would wonder if restarting the second router (or just leaving it off and powering it on) would help. But overall CARP set up properly works basically instantly so that would be preferred. https://docs.netgate.com/pfsense/en/latest/book/highavailability/index.html

Fix your sync configuration. The secondary needs to be configured to accept connections from the primary using the credentials defined. This might require a firewall rule on the sync interface to allow connections from the primary. Note that on successful sync this will be replaced by the rule on the primary so that rule has to pass the required traffic as well.

No, it isn't something we are currently considering.

I meant "before" the XFinity router, not after.

So I have done a bit more experimenting and it seems if I add a Network of other VIPs then in the 1 to 1 section it does nothing but in the outbound NAT section it expands the network entry out to all the individual entries. If I then check the "Disable expansion ....." checkbox it does not do that. However I do not care about outbound NAT as such, this subnet is only ever going to be used for 1 to 1 NAT entries so do I gain anything by using a Network entry rather than individual entries in the VIP section ?

Hi JeGr, yes the XXs are all the same and yes it is with downtime .... but i never checked the carp maintenance mode - at next maintenance time window i checks this way let me try do show screenshot later Best Regards

@viragomann said in Pfsense 2.4.5 CARP - Traffic dies when moving back to Master: Did you check "Synchronize states" in System > High Availability Sync on both nodes? Good tip and easy to miss, but this one was ok in settings. @jeppunen said in Pfsense 2.4.5 CARP - Traffic dies when moving back to Master: At the Slave -> System -> High Avail. Sync -> pfsync Synchronize Peer IP is not set, so the slave is using multicast. Should I consider changing to unicast and add Master's IP to the Slave Just give it a try. I've set the respective other nodes IP here and it fails over flawlessly in both directions. Your first tip gave me an idea and you might be onto something with your state-theory.. Master uses unicast to transfer states etc but Slave uses multicast (as there is no IP set). Even sync-interfaces are connected with direct cable, maybe it's possible that states are missing from Master when Master resumes.. Or they are missing for some other reason.. @jeppunen said in Pfsense 2.4.5 CARP - Traffic dies when moving back to Master: I have re-used VHID's, but only in differenct VLAN/subnets. Should I give every CARP-IP a different VHID, even they are in separate VLAN's? I'd rather go with unique VHIDs to be safe. The manual says: "The input validation in pfSense software will not permit using conflicting VHIDs on a single pair of systems". Because I have managed to use same VHID again and again, this mechanism should have prevented me to do crazy things? But if I'm using 254 as VHID, the MAC address is 00:00:5e:00:01:FE on all interfaces with same VHID. Even transfering from Master to Slave succeeds, maybe my switch does not like same MAC to be on multiple interfaces.. I don't know if this is an issue or not. I'll probably have to go through all interfaces and give them all an unigue vhid. Thanks for the insights @viragomann

AFAIR chelsio are the ones Netgate uses itself in the XG series thou I don't know exactly what model or revision, but I'd try them!

@jimp True story :)

There is no limit imposed by pfSense. Though depending on what you are trying to do, if you need that many, you probably have a design problem with how you intend to implement something, not a problem with the limit of VIPs.

This is old, I know, but I am throwing this out there in hopes of helping others. I have found another reason that needs to be added to the CARP troubleshooter on the Netgate site when it doesn't work under ESXi. Even if the security settings are all allowing per the documentation in a distributed vSwitch environment AND SR-IOV IS ENABLED it will not work. We had a few hosts that had this enabled on the physical NICs. After hours of trying to determine why CARP would work on some hosts, but not on others as we used vMotion to move them around we found SR-IOV was the cause. When we disabled SR-IOV CARP immediately, without reboot, started to ping on the virtual IPs. This is on ESXi 7. Hope this helps others.

@Derelict What is the best solution for this issue?

So, finally I've discovered that there is probably a bug in my HP SFP552 10GB cards that do not allow native Mac Learning to work on untagged port groups, moved to tagged port group, now everything it's working properly and I don't have anymore pfSense flooded with all traffic of promiscuous mode. I think documentation should be updated reflecting this improvement.

Hi All! I switched to the HAproxy package as suggested and it works like a charm with a 2 click configuration! Thank you all for the help! Bob