@derwin: problem 5 PC with WinXP and ALL raspberry (over 500) Not induced by CARP, no Windows or Linux versions have any issues with it. You're misdiagnosing whatever the real problem is there.

@ded_oa: Why only one? Because the others need to be assigned to the hosts that are using them. Only the gateway IP is assigned to the firewall.

Code level differences like that should really only be run for a minimum of time. Enough time to know everything's working, then update the other node to match. If not, fail back and restore the secondary to the working version. You will find that the closer the two nodes in the cluster are to each other (hardware, software, etc) the happier your cluster will be. The interface will respond with the interface MAC for ARP for the interface address. The unit that is CARP master will respond with the CARP MAC for the CARP VIP address. The ARP request will be for one IP address or the other. When you're looking at the ARP traffic, you see a WHO HAS X.X.X.X IP address. Only the MAC address that has that actual IP address will respond. Need more details about what you're really seeing, like specific IP addresses, MAC addresses, and probably packet captures showing what you're seeing to be of any sort of assistance. Both nodes please. And so we all are talking about the same things let's use the same terminology: Primary - the node that is usually Master and sends its config XMLRPC Sync to the other node. Secondary - the node that is usually Backup and does not send config XMLRPC sync to the other node. Master - the node that is currently CARP master Backup - the node that is currently CARP backup.

Yes, the states are bound to the hardware interface name. As I remember, this behaviour was different in the past and was changed with FreeBSD 10.1 and pfSense 2.2 and assigning a LAGG interface is a recommended workaround: https://doc.pfsense.org/index.php/Redundant_Firewalls_Upgrade_Guide#pfSense_2.2.x_and_pfsync But I was thinking, this should only be an issue during a failover, cause the states are not true at the other pfSense.

@gslongo: Can't you use static routes ? Depends on what exactly you're referring to, but generally speaking, no, static routes have no relevance to what's being discussed here.

@jimp: Traceroute will appear to respond from the interface address but that's a different concept entirely. Yes, and consistent with how any router or firewall with VRRP, HSRP, etc. works in that circumstance.

Hi bahsig, my problem was that I tried to start a 3rd party script (ElasticSearch Beats binary in my case) via the shellcmd package. As Beats is not a service by default it ran as a program and didn't provide an exit code to shellcmd. So in the end shellcmd waited to infinity for Beats exit code. Due to the daisy chaining of shellcmd in the PfSense / FreeBSD boot process it blocked the machine from booting. ;-) Once I killed the binary from the console / SSH booting finished and PfSense worked as expected with syncing, etc. Sadly there is no alert or system stat that shows you the FWs "boot state". ;-( Hope that helps. Michl

If you're not NATing to a CARP IP, your sessions will be lost on failover, which is why he asked. Other likely cause, if the server is pointing to the primary's IP rather than a CARP IP for its gateway.

i already found the problem.. it is due to i used windows text editor to make the script (CR LF) after saving the script file using UNIX (LF only) , the script successfully executed.. thx

The VIP MAC is used if VIP is used. Have you set your outbound NAT to translate to VIP?

Because CARP VIPs are static and those are the addresses that "swing" over to the secondary in the event of a failure. This means that Layer 3 stays intact for states, routes, client gateways, DNS servers, etc.

@jnevestdl: Hi vocatus, Congrats for the complicated setup. It is possible you to make us a tutorial with step by step or have screenshots of what is needed to configure this? Don't forget to hide the public IP's. Thanks. Hi jnevestdl, It's been quite a while since I designed this and I'm at a different position now, so I don't have access to the GUI to take screenshots. I can try to answer questions for you though if you have any.

This appears fixed in 2.3.1-RELEASE-p5.  (I can't spot anything that could be related in the release notes.)

That explains it. CARP works on every interface where a CARP VIP exists, not the sync interface. They'll switch over completely where the secondary knows it needs to take over and the primary sees that fact, but they have to be able to communicate on all the VLANs and interfaces for that to function.

That sounds like the interface stayed up but would not pass traffic any more, either due to something on either interface (primary or secondary) or something in layer 1 (bad pair out but not in maybe) or layer 2. It is not possible for HA to know what to do in that case. Disable CARP on the malfunctioning master or unplug the failed interface / shutdown the switch port and HA will shutdown CARP on all interfaces and swing to the backup. The answer is more redundancy like LAGG interfaces to stacked switches so traffic will continue to pass in a carrier-up-but-no-traffic-passing situation on one interface. This image is what I get when I change the VLAN on one interface's switch port so carrier stays up but traffic (including CARP) no longer passes between nodes.  

Simply ended up using VLANs for this situation.  Previously had been told that the switch did not support VLANs, found out otherwise. Further… configured VLANs on pfSense under: Interfaces > Assign > VLANs and Interface Assignments.