L3 Switch and pfSense design advise

phil80

Hi,

I need to migrate my Local network to VLAN mainly to add VLAN port security and to simplify the isolation.

I am limited my the physical locations of my devices as in the diagram.
I need 10 Gb wiring between VLANS 5, 10 and 20, so inter-VLAN routing at 10 Gbps should be done on the L3 switch
I do not afford running a dedicated DHCP server right now. I also want to keep DHCP server on pfSense and not on the Cisco Switches (they both can handle it) because they lack the DNS by lease/subnet and the pfSense convenience. In a 2-3 years, when changing hardware, I plan to install a dedicated DHCP server ans simplify my layout.
Many devices need the DLNA stream from the server. VLAN5 will be a backup, isolated

Internet Network Diagram Template 2.jpg

Currently, I use a flat network with IP ranges based on MAC and assigned by DHCP. Isolation is done by a bunch of ACL rules in the switch to lock traffic based on IP ranges.

I have a pfsense box with 2 built in NICs currently used for LAN and WAN.
I ordered a Quad LAN I305-T4 Intel card for my layout. The idea is to optimize bandwidth and keep the DHCP server on pfSense.

Any second advice on this layout in the diagram, mainly these:

Switch SG350X doing all the interVLAN routing (10.0.10.254) would need for each VLAN a static route pointing to the SG350 (10.0.20.254)
Switch SG350 (10.0.20.254) would need for each VLAN (except V20) a static route pointing to pfSense (10.0.20.1)
pfSense will need gateways pointing to the SG350 (10.0.20.254) for each interface + ACL/NAT rules to allow internet traffic for each of the subnets I guess

Only VLAN 5 and VLAN 40 won't have internet access or DHCP, but I do not need both.

Thank you for the help

johnpoz

@elodie80 said in L3 Switch and pfSense design advise:

keep the DHCP server on pfSense.

Unless there was a change I missed in the 2.5/21.02 release - pfsense can not provide dhcp for networks its not directly attached to.

So if you have some downstream networks via a L3 switch (router).. Pfsense would not be able to provide dhcp for these downstream networks.

Your also going to want some transit network to use to get to the downstream network, if you try and use one of your existing vlans as the transit with hosts on it - you will most likely run into asymmetrical routing.

phil80

@johnpoz
Thank you, but the V10, V20, V30 and V50 links in my diagram will be physical cables connected to a dedicated NIC port on the pfSense box. I have 6x LAN ports on the pfSense box. I can assign DHCP to each of the physical interfaces without configuring VLANs on pfSense, right ?

I thought that way I have no issues at all with asymmetrical routing since pfSense won't do any routing between the different subnets. Hope I am not wrong

johnpoz

@elodie80 said in L3 Switch and pfSense design advise:

I thought that way I have no issues at all with asymmetrical routing since pfSense won't do any routing between the different subnets. Hope I am not wrong

If pfsense is not doing any routing between any vlans - then why would you have them attached to pfsense? If all your routing is done via downstream.. Pfsense wouldn't have an interface in those networks. There would only be a transit between pfsense and the downstream.

You will have asymmetrical traffic flow if pfsense is attached to these networks, and any traffic flows through pfsense - ie to get to the internet. The return traffic would not go back to your downstream, but directly from the pfsense interface attached to that network.

phil80

@johnpoz
I am really confused here.
If I want pfSense to do the DHCP, I can define the VLANs in pfSense + use a trunk. But I will fall in asymmetrical routing again because pfSense would do the interVLAN routing

I thought that if I do not define the VLANs, add I add the physical interfaces in pfSense, I could keep the DHCP server in pfSense while allowing the downstream route from pfSense to the first switch ? pfSense should not do the routing between the subnets by default in my understanding ?

johnpoz

@elodie80 said in L3 Switch and pfSense design advise:

I thought that if I do not define the VLANs, add I add the physical interfaces in pfSense,

Doesn't matter if the networks are vlans or native (untagged).. You can not connect them like that and let some downstream router do the routing.. But pfsense is used for traffic to get to the internet. The return flow will go direct to the device vs being sent back to the downstream router..

Your downstream could route "some" vlans that are behind it, and pfsense could route other networks vlan or native. And then these routers are connected via a transit between pfsense and the downstream.

If you want your 10ge L3 switch to do all the routing - then the only thing connected to pfsense woudl be the transit network between these 2 routers. Pfsense can not do dhcp for downstream networks it not attached to.

So you would need some other dhcp server that can do that.

phil80

@johnpoz
Thank you even if I am really disappointed now.
I will return the 4x NIC card I ordered I guess !

By transit route, you mean a trunk with all VLANs defined in it ? And adding all the VLAN interfaces to pfSense ?
I am not sure how to add a transit route for an interface not defined in pfSense, either physically or by VLAN

johnpoz

No a transit (NO HOSTS).. If your downstream router is doing the routing - pfsense has no interface in these downstream networks. Be it a vlan or native.

The way he knows about these downstream or upstream networks is via routing, and he gets to those networks via his transit network.

When you put devices on a network that attaches 2 or more routers - you will most always run into asymmetrical routing if there is any communication between any of these devices and any other vlan. Unless you specifically route on these hosts that reside in the transit.

A transit network is a network that attaches routers together. You do not put hosts on this type of network.

edit: Here is a transit

The 172.16.0.0/30 would be used to get between your networks attached to your 2 routers.

phil80

@johnpoz
To define a transit rout in pfSense, is this the way to do it:
System/Routing/Static Routes/Edit

The "Gateway" would be the physical interface between pfSense and the SG350 L2 switch in my case ?
I could use the V10, V20, V30 and V50 NICs to optimize bandwidth I guess ?

johnpoz

Yes you create a gateway, the downstream or upstream router via the transit network IP. And then create routes pointing to that gateway, or use a routing protocol to exchange routes, etc.

But you can not put devices on this network - or your going to run into asymmetrical routing. And once you throw a stateful firewall in there.. You have even more problems, even if you didn't care about the asymmetrical traffic flow.

If your going to connect routers, be it a real router or a L3 doing routing. The network that attaches these routers together would have not devices on it - other than say another router to gets to other networks.

Your transit can be native or a vlan.. Doesn't matter.. Other than if vlan you might/could/would be hairpining traffic etc..

edit:
You could for sure allow your L3 to do all the routing for all your networks.. And then pfsense would just be an upstream router via a transit to get to the internet. But pfsense can not do dhcp for these downstream networks.

But to be honest if you have a VM host.. You could run dhcp on some VM, and let it do all the dhcp for all your networks/vlans. Or what is running on your backup server or dlna server for OS.. One of those could prob run a dhcp server.

phil80

@johnpoz
Again, many thanks for the detailed explanation.
I will rewrite my drawing and post again.

However, I still have one question: I was always told that, by default, pfsense will not route any traffic between interfaces except if it comes from its own.

I guess this is wrong?

In this case, can't I keep the interfaces in pfSense (for DHCP), define a transit route to the L3 switch (without devices in that route) like you suggested AND explicitly define firewall rules in pfSense to exclude any routing between the interfaces. That way, descending traffic would still be instructed to go down to the L3 Switch. Or it is just impossible?

johnpoz

@elodie80 said in L3 Switch and pfSense design advise:

pfsense will not route any traffic between interfaces except if it comes from its own.

Not sure where you heard such a thing to be honest. By "default" no it wouldn't because normally rules have source traffic limited to the network its attached to.. Say "lan net" or "optX net" etc.. So traffic from some downstream network wouldn't even be allowed into the interface.

But even if you did allow it - how would it no how to get back to some unknown network. Without a route.

But its very simple to create routing for networks that are not attached directly to pfsense. You just need to allow for them on your transit network interface.. And pfsense needs to know where to send traffic via routing.

edit: Here is a great drawing Derelict had put up and even had in his signature for long time that lays it out pretty good how to do a downstream router

edit2: Yeah he mentions the outbound nat and downstream networks. But pfsense will normally do this for you if you have outbound nat on auto. Where it might come into play where you would have to manual do the outbound nats for your downstream is if you switch to manual mode on your outbound nat (say followed some idiot vpn guide that said to do this)

phil80

@johnpoz said in L3 Switch and pfSense design advise:

Not sure where you heard such a thing to be honest. By "default" no it wouldn't because normally rules have source traffic limited to the network its attached to.. Say "lan net" or "optX net" etc.. So traffic from some downstream network wouldn't even be allowed into the interface.

But even if you did allow it - how would it no how to get back to some unknown network. Without a route.

I totally got your idea and the last drawing, thanks a lot, really.
But just about the part I quoted from your answer, and I still cannot understand it properly:

When the interfaces are added to the pfSense router, is there still a possibility to ask pfSense:

not to route between the interfaces at the pfSense layer
instead, route that incoming WAN traffic for the interfaces (defined in pfSense, so pfSense knows about the interface) to a transit route directed to the downstream L3 switch

That's the part I am not understanding the reason. Why the router cannot be asked to forward traffic to a static route instead of directly routing it between the interfaces he knows about ?

johnpoz

@elodie80 said in L3 Switch and pfSense design advise:

Why the router cannot be asked to forward traffic to a static route instead of directly routing it between the interfaces he knows about ?

Directly attached is ALWAYS going to be better than some route that is 1 hop away.

phil80

@johnpoz said in L3 Switch and pfSense design advise:

Directly attached is ALWAYS going to be better than some route that is 1 hop away.

I understand.

Now just for me to understand it even better, I did the below test setup:

added interfaces with VLAN 10 (10.0.10.1/24), VLAN 20 (10.0.20.1/24) and a Transit VLAN (172.26.1.1/24).
I added the Gateways to those interfaces to point to the Switch VLAN interface
pfSense automatically added the outbound NAT rules to WAN for those interfaces

Here's the config setup:

Define VLAN 10, 20 and a Transit VLAN
Assign the interfaces (Transit would be assigned to a physical interface in a real setup, but in fact it is not even needed)

~~3. Add the VLAN 10, 20 and transit gateways to point to the corresponding switch interface address (*.2)~~ This was sick
-deleted image-

With the gateway set this way: if I understand you well, pfSense would still prioritize the interVLAN routing directly in pfSense for inbound traffic causing asymmetrical routing, right ?

But if no such traffic ever comes: not possible from WAN as long as I know AND no device directly connected to pfSense, how could I end up with asymmetrical routing at all ? And in this config, the transit route is not needed and can be used as a regular VLAN !

johnpoz

@elodie80 said in L3 Switch and pfSense design advise:

But if no such traffic ever comes

What is the point of them - what do you think creating them does?

I have no clue what your trying to do.. Makes ZERO sense..

You only need 1 gateway - and then routes for networks that are reachable via that gateway..

Creating some vlan or interface with a network on it, isn't going to allow dhcp on that network. if the network doesn't see the broadcast for the dhcp discover..

phil80

@johnpoz said in L3 Switch and pfSense design advise:

does

Sorry for insisting, just couldn't understand what makes the setup in my last post (router-on-a-stick) cause an asymmetrical routing when no devices are connected to pfSense directly.

I am just trying to be sure there is no way to get DHCP working on pfSense for now without causing asymmetrical routing issues.

DHCP: since the Switch has a port on the same subnet as the interface on pfSense, it can see the DHCP server in pfSense

I really couldn't find explanations with a real example failing this (asymmetrical issue) and many guides advocate such a router-on-a-stick setup (but I do believe you and your explanations, just trying to get to the core issue)

phil80

@johnpoz said in L3 Switch and pfSense design advise:

What is the point of them - what do you think creating them does?

I have no clue what your trying to do.. Makes ZERO sense..

That was sick on my side, yes. There is no point to creating a Gateway pointing to the same subnet in this case. Better I go to sleep... a bit late on my side I guess

I understood the best practice you suggested. It is ok and thank you for that. I know I have to find the resources to run my DHCP server as I really hate the Cisco DHCP server in the switch.

I am just trying to understand what kind of TCP traffic in my case would cause asymmetrical routing issues when pfsense has track of each interface directly attached in it.

Here are two scenarios for asymmetric routing, credits to https://skminhaj.wordpress.com/2016/02/15/the-concept-of-asymmetric-routing/

And between User PC and www cloud server:

With my two switches and setup, return traffic would never pass from one location to another using a different route. Or am I missing such a case ?

So, if I take care at that aspect, can I run DHCP on pfSense without asymmetric routing until I can afford a dedicated DHCP server and remove any future potential asymmetric issues by using a transit route like you suggested ? Simply, am I missing some asymmetric scenario in my layout ?

Sorry again for insisting.

phil80

@johnpoz
I tested all possibilities and you are right. The transit route is what works the best.
I could test it on each device separately. I will go with the DHCP server on the L3 switch

However, I could not find a scenario where the L2 switch is facing the firewall while the L3 is downstream

I am limited by the physical locations as in the below diagram. Also, I only have 2 wall LAN plugs joining the rooms between the L2 and L3 switches. The desktop in L2 room needs a 10 Gb switching speed with the servers that the L2 switch cannot provide. I cannot invert the switch rooms for now.

Internet Network Diagram Template 3.jpg

Is it safe to run in this configuration ?
Static routes from pfSense wouzld point to the L2 switch and the default next hop in L3 switch would point to the transit interface in pfSense ? I guess the trunk between L2 and L3 switch would be on the same VLAN IP as the transit in that case ?

Hope you can explain me this specific Firewall -> L2 -> L3 layout

Best regards

johnpoz

@elodie80 said in L3 Switch and pfSense design advise:

With my two switches and setup, return traffic would never pass from one location to another using a different route.

Dude.. What is you don't get? Here is your basic network.. And how and why its asymmetrical when you put clients on a transit.

If something on the 192.168.1 network wants to talk to something on 192.168.2 network.. Sends syn to its gateway, gets sent down fine.. And sure you have a state in that direction. But if the box on the .2 network is trying to talk to something on .1 network - then the return traffic being sent to upstream router would be syn,ack - and this firewall doesn't have a state and would block the traffic.

You do not but hosts on a network that connects routers - you don't!! If you do - your going to have problems!!! unless you host route on this host telling it which router to talk to get to each network.. Or you nat on the downstream..

Also depending on your client it might not like that return traffic coming from a different mac.. It could say hey I sent traffic to 1.2.3.4 to mac abc.. Why is my return traffic coming from mac xyz. So again you do not put devices on a network that connects routers - or your going to have a bad day! ;)

Again in your last drawing - pfsense CAN NOT!!! CAN NOT do dhcp for a network its not directly attached too..

While from quick glance that looks fine - pfsense is just upstream router to a downstream router with the downstream router being the gateway for all networks, and routing.. Connected via a transit.. This is yes basic 101 networking. But pfsense can not provide dhcp services for those downstream networks.. It just can't..