CARP, Bridge & Link failures

markvr

Hey,

I've read most of the docs on CARP (I think!), but I can't figure out how it detects a failure.
eg, the diagram at:
http://olddoc.pfsense.org/index.php/Setting_up_CARP_with_pfSense
If the primary box dies (eg power failure), then fine, the backup takes over.

But what happens if one of the primary links in/out of the primary fails? Where exactly is it heartbeating over? If it's the dedicated pfsync link then this is no use as it won't detect a link failure. Apparently ifstated would do this, and used to be a package for pfSense, but isn't on the package list anymore.

Also, I am intending to run this as a bridge, which introduces it's own fun. I've read the sticky post about this at the top. Is this really the best way of doing this? Has anyone else any experience of running CARP on bridges other than as described in this post?

Cheers, Mark

morbus

As far as I know each node sends out Advertising on each interface a CARP VIP is defined with the master having the greatest Advertising Frequency.

If the master fails it stops Advertising hence the box with the next highest advertising frequency becomes the master. If just the one interface (eg LAN) on the master fails this will cause it to fail all interfaces to the next node.

The sync link is only used for state table copying between the nodes and is not entirely necessary as you can do thin on another interface (eg LAN) but the pfsync system multicasts the state changes in clear text to the network therefore this is not very secure so most of us tend to have a dedicated link for the pfsync

markvr

According to the sticky post, it says:
"Next I created a virtual IP address on the management interface so CARP has something to work with.
This IP is not used for anything except to create a carp0 interface."

Which to me implies that the heartbeating is done over the management interface.

So if i want it to heartbeat over both the WAN & Lan interfaces, I create a Virtual IP on each interface as per morbus above. At the moment neither of them has an IP as they are bridging. (I need it to bridge as this will have to drop seamlessly into an existing structure).

I've read in a few places that CARP doesn't work on bridges, but other places say it does so I'm all a bit confused. I've also read (http://www.countersiege.com/doc/ifstated/) that you need to use ifstated to detect link failures, but this may only apply to actual routing interfaces not bridges?

Does pfSense support STP, I can't find it mentioned anywhere? If it does, does anyone know if it will work if I can't control the WAN switch? ie, if only the pfSense boxes are running STP will this still work?

Arghhh all confused! ???

morbus

You are right on a bridge you would have to create a CARP vip on the management interface as you don't have IPs on the lan and wan to allow carp to use.

Someone else will have to chime in on how pfSense detects a single interface failure but I have tested it and if you pull the plug on the masters lan or wan it will failover.

I think you wouldn't need STP "support" as such. STP prevents round routes networks confusing the network and pfsense in bridge mode would be 2 routes to the same place so STP should prevent this becoming a problem. STP is a little slow to react though (much slower than CARP)

The sticky post is one way to make sure you don't end up with two routes.

Any less confused??

markvr

Regarding link failures, as the page at http://www.countersiege.com/doc/ifstated/ explains now I've read it more carefully, if one of your WAN / LAN links fail, CARP will failover that link, but you actually need it to failover BOTH WAN & LAN links simultaneously, not just the failed one. Hence ifstated is needed to detect the change in CARP status on the failed link, and then also take over the non-failed link. Read the page for more detail.

I think I'm starting to get slightly less confused! As I see it, there are three options:

Assign IPs to the WAN/LAN NICs, and use carp to detect the failure, then use ifstated to detect the CARP status change on the interfaces, and have a script, (similar to the one in the sticky post) that shuts the bridge down on the master, and brings it up on the slave. Should failover quite fast. (This assumes that you can assign IPs to the interfaces whilst they are bridging, I think this can be done….) I think this is a better solution than in the sticky post as it isn't using a cron task to periodically check ever minute, so should failover faster. Also, if you have a dedicated pfsync line, you could configure passwordless ssh over this line, so the backup can shutdown the bridge on the master via ssh, as a backup to ifstated doing it.
Forget about CARP, set up both bridges as normal, and hope the spanning tree takes one down in the event of a failure. In such a simple config, it may be possible to tune STP to fail faster than normal.
Go with the setup as described in the stickypost, and hope that a link doesn't fail. Not ideal!

Is it possible to assign IPs to an interface, whilst it is still bridging??

cmb

We enable preemption by default, and don't provide any means of turning it off, so what happens is upon any single interface failure (if the NIC fails, or is unplugged, etc.) everything will fail over to the secondary.

markvr

Ah I see, cheers. Makes things simpler