Planning to use PFsense with Cisco L3 core router and Unifi for L2... does this look ok? suggestions?

JKnott

@Jpub said in Planning to use PFsense with Cisco L3 core router and Unifi for L2... does this look ok? suggestions?:

It's the MS stuff that gives me headache

I'm allergic to it too.

Jpub

Trying to wrap my head around this a bit more. Found some tutorials that talk about this solution and "simple, just use VLAN TRUNKING!"... but... VLAN TRUNK would conflict with the /30 static route between the L3 switch and PFSENSE we're talking about here, no? See image... it's one (VLAN TRUNK) or the other (STATIC /30 route) right?

Assume orange connections are 10GBE and blue are 1GBE.

I would presume if you VLAN trunk the 10GBE vlans like that, then potentially the pipe to pfsense could get flooded with traffic as well.

Feels like I'm putting a bicycle on top of a car and riding the bike. :(

johnpoz

You can use either - you can have pfsense route the vlans, ie trunk all of them up to pfsense... Now your switch is only L2 and not doing any L3..

You do can do either - there is not right or wrong way.. You use the one that best suites your needs..

As to being a problem with hairpin - depends on how much intervlan traffic you have... But you can always use more uplink to router if sharing 1 physical link is a bottle neck, etc. etc..

But yes a bottle neck in required intervlan traffic is normally a reason to use a downstream router.

BTW: your going to trunk all your vlans up to pfsense, then the transit becomes moot.

Derelict

Jpub

@johnpoz said in Planning to use PFsense with Cisco L3 core router and Unifi for L2... does this look ok? suggestions?:

You can use either - you can have pfsense route the vlans, ie trunk all of them up to pfsense... Now your switch is only L2 and not doing any L3..

You do can do either - there is not right or wrong way.. You use the one that best suites your needs..

I guess my overnight concern was if I use the /30 between L3 switch and PFSENSE, what's that going to look like on PFSENSE configuration side.

Will I still be able to define gateways for VLAN's (i.e. 10.0.10.1, 10.0.20.1, etc.) at/in pfsense? Will I see see the VLAN tagged traffic and be able to setup firewall rules per-device-in-a-vlan (i.e. a windows wsus.mydomain.com server in VLAN20 "SERVERS" can only go outbound to wsus.microsoft.com port 80, etc.)

johnpoz

Dude I already went over this your going to let your downstream do all the routing then there are NO VLANS on pfsense... all there is the transit, and routes... You do not create any vlan interfaces on pfsense when your using downstream router to get to those networks - it will NEVER see the tags..

Your gateways for vlans will be on your L3 doing routing via SVI setups on it..

And yes you can still firewall them, you would just do all the firewall on the transit interface - and you would not have any control over devices talking intervlan - only control of them going to the internet.

JKnott

@Jpub said in Planning to use PFsense with Cisco L3 core router and Unifi for L2... does this look ok? suggestions?:

VLAN TRUNK would conflict with the /30 static route between the L3 switch and PFSENSE we're talking about here, no?

Why would they conflict? They'd conflict only if you tried assigning the same addresses to multiple connections. Each VLAN would have it's own address range.

Jpub

I dug into the Layer 3 CISCO SG550X we have as "Core Switch" and capabilities for layer 3 VLAN routing and not too excited about managing the intervlan routing there as layer 3 on the Cisco. So looking back now at your suggestion @johnpoz to pass everything up from core switch to Pfsense as L2 / tagged VLANs. If I have to I can beef up and/or upgrade PFsense appliance to support more traffic (Chelsio 10GBE nics are cheap!)

But then I remembered why we were planning to route VLANs via L3 at the core switch. There may be a problem with this layout below and potential broadcast storms? See below. We have fiber going between buildings on site, all the fiber between the buildings goes to central location where the core switch is....

Problem?

Derelict

Please don't take offense. Do you even know what Layers 2 & 3 are?

Where do you want the routing to occur (Layer 3)? On the firewall or on the switch? Look carefully at the diagram I already posted. Pay particular attention to where the IP addresses (Layer 3 addresses) are on the various interfaces.

In your example the "Core Switch" is functioning as a Layer 2 switch.

You would typically make the determination to let the switch route the traffic for performance reasons. You would typically want the firewall to route between subnets if you need firewalling services between the segments, such as a DMZ. You can do a combination of both, such as in the diagram I posted above.

Jpub

@Derelict said in Planning to use PFsense with Cisco L3 core router and Unifi for L2... does this look ok? suggestions?:

In your example the "Core Switch" is functioning as a Layer 2 switch.

You'll have to read the comment. Yes, that was the intent of this iteration. In this example the "core switch" is functioning as layer 2 switch (ignoring its L3 capabilities) with routing the occurring in PFsense as johnpoz suggested early on. My concern with this iteration is due to our physical layout requiring router>switch>switch>device.

Thanks for your suggestions and comments btw, I'm just trying to understands the pro's and con's of the different ways this can be accomplished.

Derelict

All you have to do is tag the VLANs from the core down to the edge switches.

JKnott

@Jpub said in Planning to use PFsense with Cisco L3 core router and Unifi for L2... does this look ok? suggestions?:

There may be a problem with this layout below and potential broadcast storms? See below. We have fiber going between buildings on site, all the fiber between the buildings goes to central location where the core switch is....

Why do you think you will have a broadcast storm? Broadcast storms generally result from having loops. I don't see any loops and even if you had some, you'd use spanning tree protocol to block them. That the L2 switches are in another building is irrelevant. They'd behave exactly the same as if they were in the same rack. If you use L3 (routing) to separate the sites, then you will need separate IP address ranges for each VLAN on each site.

Jpub

@JKnott said in Planning to use PFsense with Cisco L3 core router and Unifi for L2... does this look ok? suggestions?:

Why do you think you will have a broadcast storm?

Someone who knows more about this stuff than me (but perhaps less than you all) suggested it could be a big issue here due to the chaining of switches. I also read about STP back then it would seem that is exactly what STP sets to mitigate....with STP traffic would find it's way through the trunks to the best known destinations?

Derelict

In that configuration you should be more worried about the core failing and taking your whole network with it than loops.

Jpub

@Derelict said in Planning to use PFsense with Cisco L3 core router and Unifi for L2... does this look ok? suggestions?:

All you have to do is tag the VLANs from the core down to the edge switches.

Yeah I think I'm going to do it this way.... Trunking all the VLAN's through a 10GBE connection or two to a pfsense box.

Problems I see with the other solution, the "/30 to pfsense and L3 on the Cisco" solution, is if I use pfsense at all on the edge as firewall, then I am already there in that GUI managing rules for WAN and whatever else. Then I am also digging around, probably a lot, on the Cisco in not-quite-IOS (because it is Small Business/SG series) maintaining a bunch of firewallish rules as ACL's between the VLAN's. From what I've read there (Cisco ACL's on SG*), people really have a hard time accomplishing what they want between the vlans because what they really need is..... a firewall.

Long story short... I've ordered a 10GBE nic for the test pfsense box so that it can have a larger connection to network.

JKnott

@Jpub said in Planning to use PFsense with Cisco L3 core router and Unifi for L2... does this look ok? suggestions?:

@JKnott said in Planning to use PFsense with Cisco L3 core router and Unifi for L2... does this look ok? suggestions?:

Why do you think you will have a broadcast storm?

Someone who knows more about this stuff than me (but perhaps less than you all) suggested it could be a big issue here due to the chaining of switches. I also read about STP back then it would seem that is exactly what STP sets to mitigate....with STP traffic would find it's way through the trunks to the best known destinations?

There are a couple of issues here. The first, loops, we can disregard as you don't have any (that I can see). The other is the more devices, the more broadcast/multicast traffic there will be. This is entirely dependent on the number of devices connected, not the number of switches. It was even a concern back in the dark ages, when Ethernet meant a coaxial cable linear bus and there were no switches. The problem with broadcasts, but not so much multicasts, is that every devices is interrupted by every broadcast packet on the network. This means each device has to stop what it's doing, receive the broadcast and then determine if it has to do anything. This is greatly reduced with multicasts, where a device only listens for the multicasts it's interested in. So, for example, a regular computer would not be listening to "all routers" multicasts. Also, some switches can filter multicasts, so that they only pass them to devices that are actually listening for those multicasts.

What spanning tree does is block any redundant paths, so that loops can't form. It will also respond to changes in the network, so that if one path fails a previously blocked redundant path is opened. It does this by constantly evaluating the various paths to a "root" switch and the best path is automatically enabled. All other paths are blocked. So, unless you have redundant paths, spanning tree does nothing.

Jpub

@Derelict said in Planning to use PFsense with Cisco L3 core router and Unifi for L2... does this look ok? suggestions?:

In that configuration you should be more worried about the core failing and taking your whole network with it than loops.

Definitely. Although, the SG550X we have is stackable. If only we have the money to purchase the other one. :) Soon. But yeah, that's the plan. In the mean time, I guess Cisco has a pretty quick replacement program for them if it were to fail. If it were to fail and we have to wait 24-72 hours for replacement, that'll be the day I guarantee we get the 2nd for stackable (and probably a HA setup for pfsense at the same time).

Derelict

Depends on how much money the downtime would cost.

I would be more comfortable with a stacked core and LACP to each edge, HA on the firewall, and a cold spare for each.

JKnott

@Jpub said in Planning to use PFsense with Cisco L3 core router and Unifi for L2... does this look ok? suggestions?:

Definitely. Although, the SG550X we have is stackable. If only we have the money to purchase the other one. :) Soon. But yeah, that's the plan

Then you will have potential loops and need spanning tree. You'll also need 2 connections to each building switch, 1 from each core switch. You'll also have to configure spanning tree priority to determine the root. I expect you'd make the 2 core switches root.

Jpub

@Derelict said in Planning to use PFsense with Cisco L3 core router and Unifi for L2... does this look ok? suggestions?:

I would be more comfortable with a stacked core and LACP to each edge, HA on the firewall, and a cold spare for each.

Yes me too. LACP possible with these (SG550XG-8F8T) we'll just need to first hit those unexpected hiccups OR have revenue/operations enough to pay for HA and spares.
First "availability"... then "high availability". Budget conscious startups are hard sometimes. You can only lay out the risks and suggest what can be done ahead of time to avoid them. When SHTF, then they'll remember and break out the checkbook to buy equipment. Right now it's more like "Oh that's great that we'll have that option in the future." :)