Carp, Multi-lan and bridges… again. (With images)

[svensol]

Hi all,

Firstly, I know, I know… as taken from the book:

CARP (Chapter 24,Firewall Redundancy / High Availability) is not  compatible with bridging at this time — but, there are some manual hacks. Using CARP with networks that involve bridging is not generally recommended, but this kind of setup has worked for a number of individuals.

However this is exactly what we need to do.  We need to have multiple LANS (main lan, dmz, phones, guest.. etc etc.) as well as a bridged network for servers that must have real world IP addresses.

We're running pfSense 2.1-Release-64bit on 2 HP DL360-G5 (Dual Quad-core Xeon, 16GB RAM) servers.

I've attached two images, one is the logical layout and an image of the test rig.

NIC1:  WAN
NIC2:  Carp - direct attached to the backup sever
NIC3:  Main LAN and other VLANS
NIC4:  Servers with VMs

Bridge0: NIC1 and NIC4

I've been using 3Com/HP managed switches with the various VLANs added to them, spanning tree is enabled globally (RSTP) and a couple of trunk/aggregated ports between the two with VLANS 2-100 tagged across them.

I made up some VLANs to get me going:

1. LAN
2. WAN
3. Servers
4. Phones
5. DMZ
6. Guest network
7. Project 1
8. Project 2

DHCP, routing, OpenVPN and everything works as expected.

As soon as I attach the 2nd cable for the bridge on the backup firewall, low and behold, the layer2 loop kicks in and all the Carp IPs fluctuate and eventually bring the network to a halt.

I have tried the two methods of described in the book:

• Using the /etc/devd.conf hooks and scripts
• Cron to check on the state and bring the bridge up

Neither of which worked reliably.

I know I have to get the (R)STP working.  However I'm stuck, I just don't know how to configure these switches/bridges to enable the STP correctly.  I know that I have to enable the option on pfSense, but I don't know what magic runes I need to cast (STP Interfaces, PTP/Sticky Ports..) and with which combinations on the pfSense boxes and the switches.

I'd be grateful for any help or pointers anyone can give.

Thanks in advance,

Sven.