Is there any way to work around this issue?  Perhaps some script that I can modify which gets called any time dhcpv6.conf and radvd.conf is written?

That is expected behavior. When the two units have properly configured DHCP servers, they both hand out leases and they share lease information. They each agree on a portion of the address space to serve.

The above assumes you have filled in the "Failover Peer IP" on the master (and that it has been copied to the secondary during config sync). If you left that out, then filling it in will correct the problem.

As long as you can get a link from node to node it should work. Otherwise you might have to run it through a switch on an isolated VLAN. At heart, not that much different than any other NIC except for the physical connections.

To clarify: The above isn't mean to be rude, but a statement of experience. At my previous job I ran an HA pair for years that was bridged and it was a never-ending nightmare of babysitting switches, some things not working during a primary failure, mysterious network issues, etc. I bit the bullet and redesigned the entire network to use routing and that same setup has had zero problems since, other than an unrelated hardware failure.

That is expected behavior. When the two units have properly configured DHCP servers, they both hand out leases and they share lease information. They each agree on a portion of the address space to serve.

The above assumes you have filled in the "Failover Peer IP" on the master (and that it has been copied to the secondary during config sync). If you left that out, then filling it in will correct the problem.

Okay just tried that. The plot thickens.

Now the logs are reporting that the traffic being allowed. I also see traffic from my Windows DNS servesrs reaching out to Google's public resolvers being shown as "Passed". However, running nslookups and pinging anything that isn't LAN side isn't working :(

This is thoroughly mystifying. This was working only a week ago I believe.

If you check the ifconfig output from both units, it will likely be different in some way than it was when it was working. If, for example, the secondary unit didn't remove the IP Alias VIP from the interface, that might cause it to think the master had a problem ("I should be master because the other node forgot about this IP address").

Post what you have done. Not what you think you have done.

@derwin:

problem 5 PC with WinXP and ALL raspberry (over 500)

Not induced by CARP, no Windows or Linux versions have any issues with it. You're misdiagnosing whatever the real problem is there.

@ded_oa:

Why only one?

Because the others need to be assigned to the hosts that are using them. Only the gateway IP is assigned to the firewall.

Code level differences like that should really only be run for a minimum of time. Enough time to know everything's working, then update the other node to match. If not, fail back and restore the secondary to the working version. You will find that the closer the two nodes in the cluster are to each other (hardware, software, etc) the happier your cluster will be.

The interface will respond with the interface MAC for ARP for the interface address. The unit that is CARP master will respond with the CARP MAC for the CARP VIP address. The ARP request will be for one IP address or the other.

When you're looking at the ARP traffic, you see a WHO HAS X.X.X.X IP address. Only the MAC address that has that actual IP address will respond.

Need more details about what you're really seeing, like specific IP addresses, MAC addresses, and probably packet captures showing what you're seeing to be of any sort of assistance. Both nodes please.

And so we all are talking about the same things let's use the same terminology:

Primary - the node that is usually Master and sends its config XMLRPC Sync to the other node.
Secondary - the node that is usually Backup and does not send config XMLRPC sync to the other node.

Master - the node that is currently CARP master
Backup - the node that is currently CARP backup.

Yes, the states are bound to the hardware interface name. As I remember, this behaviour was different in the past and was changed with FreeBSD 10.1 and pfSense 2.2 and assigning a LAGG interface is a recommended workaround:
https://doc.pfsense.org/index.php/Redundant_Firewalls_Upgrade_Guide#pfSense_2.2.x_and_pfsync

But I was thinking, this should only be an issue during a failover, cause the states are not true at the other pfSense.

@gslongo:

Can't you use static routes ?

Depends on what exactly you're referring to, but generally speaking, no, static routes have no relevance to what's being discussed here.

@jimp:

Traceroute will appear to respond from the interface address but that's a different concept entirely.

Yes, and consistent with how any router or firewall with VRRP, HSRP, etc. works in that circumstance.

Hi bahsig,

my problem was that I tried to start a 3rd party script (ElasticSearch Beats binary in my case) via the shellcmd package.
As Beats is not a service by default it ran as a program and didn't provide an exit code to shellcmd.
So in the end shellcmd waited to infinity for Beats exit code.
Due to the daisy chaining of shellcmd in the PfSense / FreeBSD boot process it blocked the machine from booting. ;-)

Once I killed the binary from the console / SSH booting finished and PfSense worked as expected with syncing, etc.

Sadly there is no alert or system stat that shows you the FWs "boot state". ;-(

Hope that helps.

If you're not NATing to a CARP IP, your sessions will be lost on failover, which is why he asked.

Other likely cause, if the server is pointing to the primary's IP rather than a CARP IP for its gateway.

i already found the problem..
it is due to i used windows text editor to make the script (CR LF)
after saving the script file using UNIX (LF only) , the script successfully executed..

thx