CARP sync failure
During my attempts to configure CARP between 2 identical PFsense boxes i am seeing a very weird issue.
Below is my setup (please let me know if you need more info ill be happy to provide):
- 2x Super Micro C2758 for PFsense
- 2x Cisco 2960x 48 port switches same model
We have 4 vlans in our network. Each vlan on each PFsense was configured with x.x.x.2 & x.x.x.3 with CARP VIP of x.x.x.1 as the gateway. The DHCP server is configured to handout the x.x.x.1 as the default gateway.
CARP has a dedicated sync interface between the 2 PFsense.
PFsense1 has a LAG configured as LACP between the one of the Cisco switches to it. Everything is working well with no issues.
Before activating CARP i configured all the VLANs with permit any any on the sync interface.
The issue is when CARP is starting to work it is syncing the rules and everything else to the wrong vlans. Example:
Rules from management vlan are actually being synced to the guest vlan on the secondary pfsense box and from guest vlan to the internal vlans etc. This happens also to DHCP configuration,
Logs are not really telling me anything aside from the XML sync of CARP was completed.
I am aware of the states but explained here: http://phil.lavin.me.uk/2016/08/solved-pfsense-pfsync_undefer_state-unable-to-find-deferred-state/ & https://redmine.pfsense.org/issues/4310
For the sake of having the business up and running i had to restore the primary pfsense with the backup file prior to all these changes
Any advice would be appreciated.
Look at Status > Interfaces on both nodes.
OPT1 Interface (opt1, xn2)
Everything: The name (OPT1) the internal interface name (opt1) and the physical interface (xn2) must match exactly across both nodes.
Create the interfaces on the secondary in the exact order they were created on the primary.
Thank you for the reply.
Did that. I started from scratch factory default. Upgraded both to 2.3.4. Created interfaces on both boxes with matching op1, op2 etc.
I am still seeing this issue. I do not have any multicast blocking rule on any vlan interface and also enabled ip igmp snooping globally on the cisco switches to enable multicast support.
The only way for a rule to go on the wrong interface on the secondary is if they do not match. (Rules on opt1 go to opt1 on the other node).
This has nothing to do with multicast. This is XMLRPC sync presumably on the dedicated sync interface. If the rules are making it across to the secondary but going on the wrong interface there, then XMLRPC sync is working and the interfaces are mismatched.
I will give it another shot and will report back with the results when its completed.
My apologies for the really late reply.
Everything was configured and is working well!
I guess sometimes learning the hard way is the only way.
Derelict, you were right.
What I noticed is that I was not able to get it working with existing configuration even if I matched everything.
The only way was to start from scratch on both firewalls and reconfigure everything else once i confirmed that CARP was syncing rules and such.
Thank you for your support
Glad it's working.
Status > Interfaces is probably the best tool to use for this since it lists all of the interface elements in play in order in one place.