So I have done a bit more experimenting and it seems if I add a Network of other VIPs then in the 1 to 1 section it does nothing but in the outbound NAT section it expands the network entry out to all the individual entries. If I then check the "Disable expansion ....." checkbox it does not do that. However I do not care about outbound NAT as such, this subnet is only ever going to be used for 1 to 1 NAT entries so do I gain anything by using a Network entry rather than individual entries in the VIP section ?

Hi JeGr, yes the XXs are all the same and yes it is with downtime .... but i never checked the carp maintenance mode - at next maintenance time window i checks this way let me try do show screenshot later Best Regards

@viragomann said in Pfsense 2.4.5 CARP - Traffic dies when moving back to Master: Did you check "Synchronize states" in System > High Availability Sync on both nodes? Good tip and easy to miss, but this one was ok in settings. @jeppunen said in Pfsense 2.4.5 CARP - Traffic dies when moving back to Master: At the Slave -> System -> High Avail. Sync -> pfsync Synchronize Peer IP is not set, so the slave is using multicast. Should I consider changing to unicast and add Master's IP to the Slave Just give it a try. I've set the respective other nodes IP here and it fails over flawlessly in both directions. Your first tip gave me an idea and you might be onto something with your state-theory.. Master uses unicast to transfer states etc but Slave uses multicast (as there is no IP set). Even sync-interfaces are connected with direct cable, maybe it's possible that states are missing from Master when Master resumes.. Or they are missing for some other reason.. @jeppunen said in Pfsense 2.4.5 CARP - Traffic dies when moving back to Master: I have re-used VHID's, but only in differenct VLAN/subnets. Should I give every CARP-IP a different VHID, even they are in separate VLAN's? I'd rather go with unique VHIDs to be safe. The manual says: "The input validation in pfSense software will not permit using conflicting VHIDs on a single pair of systems". Because I have managed to use same VHID again and again, this mechanism should have prevented me to do crazy things? But if I'm using 254 as VHID, the MAC address is 00:00:5e:00:01:FE on all interfaces with same VHID. Even transfering from Master to Slave succeeds, maybe my switch does not like same MAC to be on multiple interfaces.. I don't know if this is an issue or not. I'll probably have to go through all interfaces and give them all an unigue vhid. Thanks for the insights @viragomann

AFAIR chelsio are the ones Netgate uses itself in the XG series thou I don't know exactly what model or revision, but I'd try them!

@jimp True story :)

There is no limit imposed by pfSense. Though depending on what you are trying to do, if you need that many, you probably have a design problem with how you intend to implement something, not a problem with the limit of VIPs.

This is old, I know, but I am throwing this out there in hopes of helping others. I have found another reason that needs to be added to the CARP troubleshooter on the Netgate site when it doesn't work under ESXi. Even if the security settings are all allowing per the documentation in a distributed vSwitch environment AND SR-IOV IS ENABLED it will not work. We had a few hosts that had this enabled on the physical NICs. After hours of trying to determine why CARP would work on some hosts, but not on others as we used vMotion to move them around we found SR-IOV was the cause. When we disabled SR-IOV CARP immediately, without reboot, started to ping on the virtual IPs. This is on ESXi 7. Hope this helps others.

@Derelict What is the best solution for this issue?

So, finally I've discovered that there is probably a bug in my HP SFP552 10GB cards that do not allow native Mac Learning to work on untagged port groups, moved to tagged port group, now everything it's working properly and I don't have anymore pfSense flooded with all traffic of promiscuous mode. I think documentation should be updated reflecting this improvement.

Hi All! I switched to the HAproxy package as suggested and it works like a charm with a 2 click configuration! Thank you all for the help! Bob

Lots of possibilities. I would simplify the configuration down to the basics and see if you can get it working. Key suspects for me would be network structure issue (have you accidentally introduced a loop and STP/RTSP is kicking in and disabling ports, causing weird routing? do you have a bad cable or ports that are auto-negotiating at the wrong speeds? etc). A CARP or Virtual IP configuration step you missed (wrong netmask on a virtual IP? left CARP in temporary maintenance mode, etc...)? Check out the pfSense system logs, check that node is MASTER on both WAN and LAN (if they split then of course you have routing issues) and the other is BACKUP on both, check your switch port status and logs to see if it gives you any hints...

https://forum.netgate.com/topic/151718/carp-with-single-ppoe-make-internet-working-from-the-slave-node

When i try to ping the other way around, it goes via the correct path via primary 10.x.x.196 Tracing route to 10.x.x.41 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms 10.x.x.196 2 <1 ms <1 ms <1 ms 10.x.x.41 Trace complete.

CARP/VRRP/etc. are using not only virtual IPs but also virtual MACs to make failover a smooth experience without clients or network equipment having to learn a new MAC address of a failover server like with only IP based configurations (early linux HA cluster for example). The VHID setting is influencing which MAC is handed out for that CARP style VIP. All of them are (IMHO) using the failover MAC space of 00:00:5E:00:01:XX so with changing the VHID you are also configuring the last "XX" segment of said MAC address. That's why it has to be unique on that network segment (L2) and you also have to watch out for other cluster/HA-grade setups, that are using VRRP or HSRP style VIP/MAC combinations. But if your pfSense cluster is the only cluster in that network segment, VHID 1 is commonly fine on all interfaces. We're using VHID 4 and 6 (for IP4 / IP6 VIPs on the same VLAN) over multiple VLANs just fine :)

No, only the WAN IPs of the two pfSense boxes have to be within the same subnet. A /29 is just the minimal network size for a common CARP setup with the two WAN IPs and the WAN VIP within a subnet. But it doesn't matter if the subnet is larger.