Fully redundant network setup

  • Hello, and thanks in advance for any assistance.

    I've been working on a network configuration for use in a leased datacenter rack. The primary goal is to have full hardware redundancy, such that any single router or switch could fail without causing a network outage. The datacenter provider gives us two cat6 ethernet drops, providing what they have described as "blended internet service". We're still working on the exact details of that, but what they've indicated is that if we setup two interfaces in a bridge configuration, connect each drop to one of the bridged interfaces, it will work. For the context of this thread, assume this part of the network is functioning properly.

    We are using this space to host networks for clients, which will connect using their own routers downstream from our primary routers. The example I'll use is another pair of pfSense routers setup with CARP failover. What I'm looking to figure out is what kind of network will best accommodate connecting these "internal" routers to our "external" routers, using redundant switches. I've included a diagram I've drawn up which illustrates the situation. My understanding is that CARP cannot be configured to failover on issues which occur outside of pfSense (I.E. if one of the "Internal WAN" switches goes down), so for this to work we will need both routers, on both the "internal" and "external" side, connected to both switches. This "internal" network is a /27 WAN IP block which is routed through the "External" routers, which connect to the ISP service through a separate /28 WAN IP network.

  • LAYER 8 Global Moderator

    What is the point of the external and internal routers?  Just more hardware to pay for and power, and something else that can fail.. Along with more switches, etc.  What are you accomplishing what what looks to be just a transit..

    Are they switches in a stack?  Is this why your showing them with a connection to each other?

    Really need to understand what this "blended" internet is.. They want you to use a bridge to connect to them?  Are these external wan switches also yours - or the ISP?

  • I was lost at "A series of tubes".  Looks nice though.  Symmetrical.  (-:

  • Sorry for some of the things which I wasn't very clear about.

    @kejianshi: "A Series of Tubes" is a joke/reference (see https://en.wikipedia.org/wiki/Series_of_tubes) I should have left out seeing as it caused some confusion. It's just there to represent "upstream to the internet". For clarity I should have labeled it as "upstream gateway(s)".

    @johnpoz: I had (over) simplified my diagram to highlight the specific problems I'm trying to solve, but there are a lot more pieces to this puzzle, I'll describe the ones which help clarify your questions.

    The point of having the internal and external routers is that we're trying to build a datacenter environment we can use to host dedicated networks for multiple clients, using their own routers and providing them with their own WAN IP addresses. The client's equipment will start at the "internal routers" in my diagram, which illustrates as if we only had one client present in the rack, but really there will be many.

    Going slightly out of order, the external WAN switches are ours. Logically this wouldn't make a difference, but to clarify there will actually only be two physical switches, but we will use VLANing to make them function like two separate devices as depicted in the diagram. 3 ports on (internal) switch A will be carved out into a VLAN to function as external switch A, and the same thing for internal/external switch B.

    I agree with the confusion regarding "blended internet service", I've been trying to get specific details from he datacenter network engineers but they haven't yet gotten through to me on exactly how this works. I have done a bit more research since my original post, and I believe that the terminology "bridge" means something very different in pfSense than it does for Cisco, which is what I believe they were referring to when they said "bridge". After discussing that part in depth with my boss, we're expecting that we're just going to have to figure that out once we're onsite with the routers.

    The engineers at the datacenter provided me with a diagram that helps illustrate their WAN service, and how they expect clients to configure their routers. I've sent an email asking if they are OK with me posting it here. If I get back confirmation from them, I'll provide it here, which may help illustrate the external WAN side (and clarify "blended service")

    My main goal in this thread is to figure out the best way to configure the "internal WAN" (blue connections on the diagram), which ties into the remaining question regarding stacked switches. That part of the network is entirely open to suggestions/recommendations, and able to be changed. I actually talked to my boss about the direct link between the two switches, and he said that is necessary for some of what we'll be doing with the other client networks, which is in particular Windows servers using NIC Teaming. I'll get some more details on that as well and post it here soon.

    The real goal is to have the switches redundant - I.E. figure out a configuration where a server can be connected to both switches, and if either switch goes out, internet connectivity stays alive.

  • LAYER 8 Global Moderator

    "The real goal is to have the switches redundant"

    I am walking out the door for some beers… But real quick on that statement - your going to want to go with stack switches.  You could do it the hillbilly way with stp or rstp, which is kind of how you have it drawn..  But that would not really allow you to do etherchannel/portchannel, ie a lacp lagg from your windows machines..

    Stacked switches with dual power supplies in both or with stacked power would be the way to go.. We don't ever do non stacked switches, even at the access layer.  We normally do cisco 3850s, 24 or 48 ports stacked.. Sometimes even 3 or 4 of them depending on the port density needed at a access location, etc.

  • Hi,

    you might take a look into a lagg. This is what I have done.
    Two interface going to two switches. This are bound in a failover. On top of this you can setup carp.

    This eliminates the bridging and the problem that brings it in.

Log in to reply