HA issue

[bakisho]

I have a cluster of 2 pfsense physical machines.
Everything work well regarding HA:

1. I see pfsync packets over the "sync" interface.
2. I see carp broadcasts (once in each second) on each interface (wan1 / wan2 / lan ).
3. XMLRPC is working OK, whenever I create/change a rule it is reflected immidiatelly to the secondary.
   If I disable carp on the master, the secondary is promoted with downtime of a 1-5 seconds (which is OK for me).

The problem is whenever I restart the Master "unexpectedly".
On my last test I noticed that the following:

1. The secondary is promoted.
   • It is broadcasting carp packets on every interface.
   • I see it shows up as master in "Status -> Carp".
2. on the sync interface there was an arp "who has <master IP>".
   But nothing works until the master is back....

What can the root cause for that?
How can I troubleshoot it ?
Thanks

[viragomann]

Are you LAN devices using the LAN CARP VIP as default gateway?

Have you configured the outbound NAT to translate source addresses into the WAN CARP VIP instead of WAN address?

[Derelict]

on the sync interface there was an arp "who has <master IP>".

There should not be CARP configured on the sync interface so no idea what this is about.

[bakisho]

Yes I am using my carp VIP as a default gateway.
I did not understand, is it good or bad ?
Some of my servers have a 1:1 NAT.
Some of my servers do not have 1:1 Nat and use the WAN CARP VIP as the their public IP.

There is no CARP on the SYNC interface, only "Sync" packets and "XMLRPC".

[viragomann]

Yes, that's OK, the CARP VIP should be used as upstream gateway.

Do you syncing the states?

When the second box is master and upstream traffic is blocked, what does the filter log show?