CARP with distribution switch cross connects

[teslamad]

So I have finally decided to implement CARP now that the new pfSense has rolled out. A few features have been released in the past few versions that make it worth while for me to give it a go. So I decided to give it a test run in a dev environment first. I create a few VM's, did some sloppy configs, and had some successful results. Thus I decided to move forward with some bare metal installations. I grabbed two identical systems off the shelf (x2 PowerEdge 750's) that will eventually replace the single PowerEdge 750 currently in the rack as the lone pfSense box. Yes I know, could have just mirrored the first and called it a day. It was a chance to reconfigure, clean out, etc.

I setup the basic/recommended configuration first as outlined in the old pfSense book and all was happy. Notice here in this first example (Attachment: Basic.png) I have three public IP's. Two for the interfaces and one for the VIP (Red). I also have the pfSync network setup (Magenta). Finally, I am doing intraVLAN routing, thus I setup an address per VLAN per interface with a VIP for each VLAN. There is also a trunk line (irrelevant for this discussion) between the two switches in case a gateway goes down. Now I have not tested this yet as I am fairly confident everything will work as expected. I would rather move on to the real question.

So ideally, what I would like to do is implement a Cisco-esk HSRP style topology. Now either it has been too many years since my CCNP and I am just forgetting basic concepts, or pfSense really does do something different. In my more complex example (Attachment: Advanced.png), you can see how each switch has a redundant connection to each router. This way you can lose both opposing switch and router (ie diagonally) and yet still have a route out. I have depicted an example scenario as well (Attachment: Scenario.png) in case my description was not clear enough. This is great, although the problem arises that now I technically need to create four interfaces that all share the same VIP. It sounds simple at first. Lets just take the first VLAN as an example. The VIP would stil be .1.1. We would then assign the four interfaces as .1.2, .1.3, .1.4, and .1.5. Sounds easy at first. This is where the trick comes in, and I'm pretty sure I know what you're all going to say. "You can't do that!", "You can't assign two interfaces to the same VIP on a single host!", and finally "For this to work as you are describing it, you need to enable layer 3 routing on those two core or distribution switches!".

Can someone please either confirm that my suspicions are correct, or that I am just doing something else wrong in pfSense with regards to CARP configuration.

Thanks,
Andrew

[ajrg]

You can use LAGG on the routers to achieve this. Depending on what your switches support;

Create a LAGG interface from two ports on each router.
If your switches support cross-stack LACP, enable this and set the LAGG interface to use active LAGG.
If your switches don't support cross-stack LACP, just configure the switch ports as normal and make sure RSTP (or similar) is turned on, and set the LAGG interface to use failover mode.

Then, connect everything up, make sure you're not getting any duplicate packets, and then test by randomly disconnecting power and uplinks from things :)

[teslamad]

Interesting solution. Sounds like a workaround solution though :(

Any specific reason you recommended RSTP here instead of PVST+?

[ajrg]

If your switches support PVST+, use that - it's far superior.

In theory, you shouldn't actually need any kind of spanning tree when you're using passive/failover LAGG - but I consider it good clean living to use it where you're not using active/LACP mode.

Given your switches support PVST+, I'll assume they're newer Catalysts. In that case (assuming your switches are stacked);

On pfSense, create an LAGG across the interfaces you want to use, and set LACP mode.
On the switches, create two Port-channel interfaces across the two pairs of ports you want to use (ie. one port for each pfSense box on each switch - you can aggregate links across the stack), and then on the switchports, use channel-group x mode active to enable LACP.

The stack modules and cables are (relatively) cheap compared to the switches if you don't have them, and it eliminates relying on spanning tree for those interfaces, which means you can use portfast, so those ports come up quickly.

Somebody else may weigh in with another solution - so if this one doesn't sound palatable, hopefully you'll have a few choices!

[teslamad]

Yeah I was wondering why you would make that recommendation. These are all brand new Catalysts.

Please help me visualize the pfSense side interface logic.

I'm not dismissing this solution, but it does seem like an unorthodox solution to a problem that is usually just handled with L3 switches. One way or another though, The L3 solution wont work for me as there is no need in my opinion and I'm too cheap right now to add two L3 switches to an already bloated topology.

So either I test your solution out (in progress), or just suck it up and stick with the first basic setup I proposed.

Thanks

[ajrg]

I'll draw up a quick diagram of how we have ours set up when I get out of work.

We're using Catalyst 3650s with the stacking modules, so it should be reasonably similar to what you are looking to achieve.

[teslamad]

Any updates? Thanks

[ajrg]

Hi, sorry! Been a lot busier than I expected.

Attached a quick and dirty diagram - this relies on you having stacking modules for your switches, so you can make port-channels across the stack.
This way, you can have any pfSense box or any switch die (even at 'diagonals' on the diagram) and still have a functioning system, with only a minor blip.

If you can't use the stacking modules, you would need to use LAGG Failover mode, and configure spanning-tree on the switchports.


[matwolb]

Good morning forum,

I'm just suffering the same question as Andrew M. Robinson. The schema he's proposing seems to be the best one when HA is required both at filtering level (pfSense) and routing level (switches behind pfSense, L3 maybe?).

After looking at this thread, it seems that it's posible to create a LAGG link (2 links from pfSense box1 to switch box1, 1 link from pfSense box1 to switch box2 - and same for pfSense box2, 2 connections from pfSense box2 to switch box2 and another one to switch box1), but apparently you would need to have stacking kit between those Catalyst.

Question is: is really stacking kit needed here or is it possible to do cross-stack LAGG by just creating an LACP trunk link between the switches? (simulating the stacking kit).

Thank you very much, kind regards

David