CARP on Bridges

jtryon

First off I know that running CARP on bridges is not recommended, however, I am OK with living dangerously.

Here is my test environment.

Firewall 1 with 2 port Chelsio 10GB card and 2 x onboard Intel NICs
EM0: WAN, no IP
EM1: LAN, 192.168.252.2 (management and sync)
CXL0: no IP address
CXL1: no IP address
BRIDGE0: 192.168.250.251 (CXL0 and CXL1 as members, RSTP turned on to CXL0 and CXL1)

Firewall 2 with 3 x onboard Intel NICx (this is the master firewall for HA sync)

EM0: 192.168.252.1 (for management and sync)
EM1: no IP address
EM2: no IP address
BRIDGE0: 192.168.250.249 (EM1 and EM2 as members, RSTP turned on to EM1 and EM2)

I have pfsync and XMLRPC sync working.

Here is my CARP settings i place on Firewall #2 (master)

interface BRIDGE0
IP address 192.168.250.250 /24
password: pfsense
vhid group: 1
advertising frequency Base: 1 Skew: 0

Save and apply.

When I check the VIPs on the other firewall my CARP setting have synced BUT instead of using the BRIDGE0 interface for my CARP IP on Firewall 1, it uses the CXL1 interface. If I attempt to go change the interface from CXL1 to BRIDGE0, I get an error message "This IP address is being used by another interface or VIP."

SO

What I did next was to go onto the CARP status page on Firewall 1 (slave) and temporarily disable CARP, go back to the CARP VIP settings and change the interface to BRIDGE0. This worked, so then I went back and enabled CARP on Firewall 1. CARP tested out just fine and master/backup reported correctly.

I then went and altered the firewall rules for BRIDGE0 on firewall 2 (master) saved and applied the settings and sent back to Firewall 1 (slave) and my CARP interface was changed back to CXL1 and the firewall policy that I placed on BRIDGE0 on Firewall 2 was synced to the firewall policy for CXL1 on Firewall 1.

It seems that CARP thinks CXL1 is my bridge interface instead of BRIDGE0 on firewall 1.

Anyone have any ideas?

edited because I hadmy firewall NICs backwards

jtryon

I was able to correct this.

Because my testing environment is using different hardware and interfaces I needed to setup the interfaces more carefully

What I discovered is that when CARP assigns matches interfaces, it must choose them in sequential order from the assign interfaces page, matching them with the other firewall.

What I had was
Firewall 1
#1 LAN
#2 EM1
#3 EM2
#4 Bridge0

Firewall 2
#1 WAN
#2 LAN
#3 CXL0
#4 CXL1
#5 Bridge0

So, what I think happened was that CARP was matching #4 from each list, so my bridge0 (#4 on Firewall 1) was being matched with CXL1 (#4 on Firewall 2)

Once I reassigned my interfaces and lined up the interface numbers CARP matched the correct interfaces.