CARP is driving me INSANE.

Numbski

Okay, so here I sit, one WRAP, one Soekris, and embedded Beta4 on each. The soekris is named lbfw1, WRAP lbfw2. Each has sis0-sis2. Named as follows:

sis0 WAN
sis1 LAN
sis2 pfSync

On each box, I go to firewall rules, enable opt1, name it pfSync, then create an "any any" rule. I take a short crossover cable and plug the matching sis2 interfaces together. I assign lbfw1 102.168.200.1/30, and lbfw 102.168.200.2/30. From terminal they can ping one to the other via these IP addresses. All is well thus far.

Now, the setup I'm doing is a no-nat situation, ie, the LAN interface is a routable network. I go into outbound NAT on each firewall and delete the default NAT rule.

Here are the WAN/LAN assignments for each machine:

LBFW1:
WAN - 206.80.68.18
LAN - 206.80.68.25

LBFW2:
WAN - 206.80.68.19
LAN - 206.80.68.26

On LBFW2, went to Virtual IP/CARP configuration, enabled synchronization, checked no other boxes, chose my pfSync interface and saved. On LBFW1, go to the same page, enabled synchronization, chose the pfSync interface, and also checked sync Virtual IP's, NAT, and rules. I entered the IP address 102.168.200.2 and the admin password of LBFW2, and saved the config.

I then visited the Virtual IP config page on LBFW1. I created a pair of Virtual IP's:

Labelled carp0 in terminal:
Type: CARP
Interface: WAN
IP Address: 206.80.68.20/29
Virtual ID Password: ************
VHID Group: 1
Advertising Frequency: 0
Description: WAN Core Virtual IP

Labelled carp1 in terminal:
Type: CARP
Interface: LAN
IP Address: 206.80.68.27/29
Virtual ID Password: ************
VHID Group: 2
Advertising Frequency: 0
Description: LAN Gateway Virtual IP

Save and Apply all changes, visit the CARP status page and make sure that carp is enabled.

When I go back to LBFW2, the Virtual IP page lists the two IP addresses, and are configured identically except that Advertising Frequency is listed at 100. When I go to the CARP Status page, CARP is disabled. I enable it, and at this point everything appears to be perfect. The interfaces on LBFW1 show up as Master, the interfaces on LBFW2 show up as Backup.

What happens next is what drives me nuts.

From the terminal on LBFW1:


# ifconfig carp1
carp1: flags=49 <up,loopback,running>mtu 1500
        inet 206.80.68.27 netmask 0xfffffff8 
        carp: MASTER vhid 2 advbase 1 advskew 0</up,loopback,running>

LBFW2:


carp1: flags=49 <up,loopback,running>mtu 1500
        inet 206.80.68.27 netmask 0xfffffff8 
        carp: BACKUP vhid 2 advbase 1 advskew 100</up,loopback,running>

LBFW1:


# ifconfig carp1 down
# ifconfig carp1
carp1: flags=8 <loopback>mtu 1500
        inet 206.80.68.27 netmask 0xfffffff8 
        carp: INIT vhid 2 advbase 1 advskew 0</loopback>

LBFW2:


# ifconfig carp1
carp1: flags=49 <up,loopback,running>mtu 1500
        inet 206.80.68.27 netmask 0xfffffff8 
        carp: MASTER vhid 2 advbase 1 advskew 100</up,loopback,running>

LBFW1:


# ifconfig carp1 up
# ifconfig carp1
carp1: flags=49 <up,loopback,running>mtu 1500
        inet 206.80.68.27 netmask 0xfffffff8 
        carp: MASTER vhid 2 advbase 1 advskew 0</up,loopback,running>

LBFW2:


# ifconfig carp1
carp1: flags=49 <up,loopback,running>mtu 1500
        inet 206.80.68.27 netmask 0xfffffff8 
        carp: BACKUP vhid 2 advbase 1 advskew 100</up,loopback,running>

Again, all looks golden. Where things go screwy is doing this same precise sequence on carp0…

LBFW1:


# ifconfig carp0
carp0: flags=49 <up,loopback,running>mtu 1500
        inet 206.80.68.20 netmask 0xfffffff8 
        carp: MASTER vhid 1 advbase 1 advskew 0</up,loopback,running>

LBFW2:


# ifconfig carp0
carp0: flags=49 <up,loopback,running>mtu 1500
        inet 206.80.68.20 netmask 0xfffffff8 
        carp: MASTER vhid 1 advbase 1 advskew 100</up,loopback,running>

Shortly after enabling carp, LBFW2 promotes itself to master on carp0. I can still ping between the crossover connected sis2 interfaces by IP address, and the firewall "any any" rule is working (otherwise my pings would fail). I can disable CARP on LBFW2, and when I re-enable it, it temporarily shows both carp0 and carp1 in state Backup, but within a few seconds, carp0 is a Master again, and LBFW1 still shows the interface as Master as well. ???

carp1 behaves flawlessly as expected. I have inspectedd and reinspected my config for carp0 on both boxes and I'm just not seeing my error. Help?

Numbski

My apologies. The switch for that interface needed to be reset to factory defaults. For whatever reason the two interfaces wound up on seperate VLANs, yet they could both reach the gateway (just not one another) with their frames. Bizarre. I cannot even begin to fathom that, but once set to defaults all was well.