L3 Switch and pfSense design advise
-
@elodie80 said in L3 Switch and pfSense design advise:
I thought that if I do not define the VLANs, add I add the physical interfaces in pfSense,
Doesn't matter if the networks are vlans or native (untagged).. You can not connect them like that and let some downstream router do the routing.. But pfsense is used for traffic to get to the internet. The return flow will go direct to the device vs being sent back to the downstream router..
Your downstream could route "some" vlans that are behind it, and pfsense could route other networks vlan or native. And then these routers are connected via a transit between pfsense and the downstream.
If you want your 10ge L3 switch to do all the routing - then the only thing connected to pfsense woudl be the transit network between these 2 routers. Pfsense can not do dhcp for downstream networks it not attached to.
So you would need some other dhcp server that can do that.
-
@johnpoz
Thank you even if I am really disappointed now.
I will return the 4x NIC card I ordered I guess !By transit route, you mean a trunk with all VLANs defined in it ? And adding all the VLAN interfaces to pfSense ?
I am not sure how to add a transit route for an interface not defined in pfSense, either physically or by VLAN -
No a transit (NO HOSTS).. If your downstream router is doing the routing - pfsense has no interface in these downstream networks. Be it a vlan or native.
The way he knows about these downstream or upstream networks is via routing, and he gets to those networks via his transit network.
When you put devices on a network that attaches 2 or more routers - you will most always run into asymmetrical routing if there is any communication between any of these devices and any other vlan. Unless you specifically route on these hosts that reside in the transit.
A transit network is a network that attaches routers together. You do not put hosts on this type of network.
edit: Here is a transit
The 172.16.0.0/30 would be used to get between your networks attached to your 2 routers.
-
@johnpoz
To define a transit rout in pfSense, is this the way to do it:
System/Routing/Static Routes/EditThe "Gateway" would be the physical interface between pfSense and the SG350 L2 switch in my case ?
I could use the V10, V20, V30 and V50 NICs to optimize bandwidth I guess ? -
Yes you create a gateway, the downstream or upstream router via the transit network IP. And then create routes pointing to that gateway, or use a routing protocol to exchange routes, etc.
But you can not put devices on this network - or your going to run into asymmetrical routing. And once you throw a stateful firewall in there.. You have even more problems, even if you didn't care about the asymmetrical traffic flow.
If your going to connect routers, be it a real router or a L3 doing routing. The network that attaches these routers together would have not devices on it - other than say another router to gets to other networks.
Your transit can be native or a vlan.. Doesn't matter.. Other than if vlan you might/could/would be hairpining traffic etc..
edit:
You could for sure allow your L3 to do all the routing for all your networks.. And then pfsense would just be an upstream router via a transit to get to the internet. But pfsense can not do dhcp for these downstream networks.But to be honest if you have a VM host.. You could run dhcp on some VM, and let it do all the dhcp for all your networks/vlans. Or what is running on your backup server or dlna server for OS.. One of those could prob run a dhcp server.
-
@johnpoz
Again, many thanks for the detailed explanation.
I will rewrite my drawing and post again.However, I still have one question: I was always told that, by default, pfsense will not route any traffic between interfaces except if it comes from its own.
I guess this is wrong?
In this case, can't I keep the interfaces in pfSense (for DHCP), define a transit route to the L3 switch (without devices in that route) like you suggested AND explicitly define firewall rules in pfSense to exclude any routing between the interfaces. That way, descending traffic would still be instructed to go down to the L3 Switch. Or it is just impossible?
-
@elodie80 said in L3 Switch and pfSense design advise:
pfsense will not route any traffic between interfaces except if it comes from its own.
Not sure where you heard such a thing to be honest. By "default" no it wouldn't because normally rules have source traffic limited to the network its attached to.. Say "lan net" or "optX net" etc.. So traffic from some downstream network wouldn't even be allowed into the interface.
But even if you did allow it - how would it no how to get back to some unknown network. Without a route.
But its very simple to create routing for networks that are not attached directly to pfsense. You just need to allow for them on your transit network interface.. And pfsense needs to know where to send traffic via routing.
edit: Here is a great drawing Derelict had put up and even had in his signature for long time that lays it out pretty good how to do a downstream router
edit2: Yeah he mentions the outbound nat and downstream networks. But pfsense will normally do this for you if you have outbound nat on auto. Where it might come into play where you would have to manual do the outbound nats for your downstream is if you switch to manual mode on your outbound nat (say followed some idiot vpn guide that said to do this)
-
@johnpoz said in L3 Switch and pfSense design advise:
Not sure where you heard such a thing to be honest. By "default" no it wouldn't because normally rules have source traffic limited to the network its attached to.. Say "lan net" or "optX net" etc.. So traffic from some downstream network wouldn't even be allowed into the interface.
But even if you did allow it - how would it no how to get back to some unknown network. Without a route.
I totally got your idea and the last drawing, thanks a lot, really.
But just about the part I quoted from your answer, and I still cannot understand it properly:When the interfaces are added to the pfSense router, is there still a possibility to ask pfSense:
- not to route between the interfaces at the pfSense layer
- instead, route that incoming WAN traffic for the interfaces (defined in pfSense, so pfSense knows about the interface) to a transit route directed to the downstream L3 switch
That's the part I am not understanding the reason. Why the router cannot be asked to forward traffic to a static route instead of directly routing it between the interfaces he knows about ?
-
@elodie80 said in L3 Switch and pfSense design advise:
Why the router cannot be asked to forward traffic to a static route instead of directly routing it between the interfaces he knows about ?
Directly attached is ALWAYS going to be better than some route that is 1 hop away.
-
@johnpoz said in L3 Switch and pfSense design advise:
Directly attached is ALWAYS going to be better than some route that is 1 hop away.
I understand.
Now just for me to understand it even better, I did the below test setup:
- added interfaces with VLAN 10 (10.0.10.1/24), VLAN 20 (10.0.20.1/24) and a Transit VLAN (172.26.1.1/24).
- I added the Gateways to those interfaces to point to the Switch VLAN interface
- pfSense automatically added the outbound NAT rules to WAN for those interfaces
Here's the config setup:
-
Define VLAN 10, 20 and a Transit VLAN
-
Assign the interfaces (Transit would be assigned to a physical interface in a real setup, but in fact it is not even needed)
3. Add the VLAN 10, 20 and transit gateways to point to the corresponding switch interface address (*.2)This was sick
-deleted image-With the gateway set this way: if I understand you well, pfSense would still prioritize the interVLAN routing directly in pfSense for inbound traffic causing asymmetrical routing, right ?
But if no such traffic ever comes: not possible from WAN as long as I know AND no device directly connected to pfSense, how could I end up with asymmetrical routing at all ? And in this config, the transit route is not needed and can be used as a regular VLAN !
-
@elodie80 said in L3 Switch and pfSense design advise:
But if no such traffic ever comes
What is the point of them - what do you think creating them does?
I have no clue what your trying to do.. Makes ZERO sense..
You only need 1 gateway - and then routes for networks that are reachable via that gateway..
Creating some vlan or interface with a network on it, isn't going to allow dhcp on that network. if the network doesn't see the broadcast for the dhcp discover..
-
@johnpoz said in L3 Switch and pfSense design advise:
does
Sorry for insisting, just couldn't understand what makes the setup in my last post (router-on-a-stick) cause an asymmetrical routing when no devices are connected to pfSense directly.
I am just trying to be sure there is no way to get DHCP working on pfSense for now without causing asymmetrical routing issues.
DHCP: since the Switch has a port on the same subnet as the interface on pfSense, it can see the DHCP server in pfSense
I really couldn't find explanations with a real example failing this (asymmetrical issue) and many guides advocate such a router-on-a-stick setup (but I do believe you and your explanations, just trying to get to the core issue)
-
@johnpoz said in L3 Switch and pfSense design advise:
What is the point of them - what do you think creating them does?
I have no clue what your trying to do.. Makes ZERO sense..
That was sick on my side, yes. There is no point to creating a Gateway pointing to the same subnet in this case. Better I go to sleep... a bit late on my side I guess
I understood the best practice you suggested. It is ok and thank you for that. I know I have to find the resources to run my DHCP server as I really hate the Cisco DHCP server in the switch.
I am just trying to understand what kind of TCP traffic in my case would cause asymmetrical routing issues when pfsense has track of each interface directly attached in it.
Here are two scenarios for asymmetric routing, credits to https://skminhaj.wordpress.com/2016/02/15/the-concept-of-asymmetric-routing/
And between User PC and www cloud server:
With my two switches and setup, return traffic would never pass from one location to another using a different route. Or am I missing such a case ?
So, if I take care at that aspect, can I run DHCP on pfSense without asymmetric routing until I can afford a dedicated DHCP server and remove any future potential asymmetric issues by using a transit route like you suggested ? Simply, am I missing some asymmetric scenario in my layout ?
Sorry again for insisting.
-
@johnpoz
I tested all possibilities and you are right. The transit route is what works the best.
I could test it on each device separately. I will go with the DHCP server on the L3 switchHowever, I could not find a scenario where the L2 switch is facing the firewall while the L3 is downstream
I am limited by the physical locations as in the below diagram. Also, I only have 2 wall LAN plugs joining the rooms between the L2 and L3 switches. The desktop in L2 room needs a 10 Gb switching speed with the servers that the L2 switch cannot provide. I cannot invert the switch rooms for now.
Is it safe to run in this configuration ?
Static routes from pfSense wouzld point to the L2 switch and the default next hop in L3 switch would point to the transit interface in pfSense ? I guess the trunk between L2 and L3 switch would be on the same VLAN IP as the transit in that case ?Hope you can explain me this specific Firewall -> L2 -> L3 layout
Best regards
-
@elodie80 said in L3 Switch and pfSense design advise:
With my two switches and setup, return traffic would never pass from one location to another using a different route.
Dude.. What is you don't get? Here is your basic network.. And how and why its asymmetrical when you put clients on a transit.
If something on the 192.168.1 network wants to talk to something on 192.168.2 network.. Sends syn to its gateway, gets sent down fine.. And sure you have a state in that direction. But if the box on the .2 network is trying to talk to something on .1 network - then the return traffic being sent to upstream router would be syn,ack - and this firewall doesn't have a state and would block the traffic.
You do not but hosts on a network that connects routers - you don't!! If you do - your going to have problems!!! unless you host route on this host telling it which router to talk to get to each network.. Or you nat on the downstream..
Also depending on your client it might not like that return traffic coming from a different mac.. It could say hey I sent traffic to 1.2.3.4 to mac abc.. Why is my return traffic coming from mac xyz. So again you do not put devices on a network that connects routers - or your going to have a bad day! ;)
Again in your last drawing - pfsense CAN NOT!!! CAN NOT do dhcp for a network its not directly attached too..
While from quick glance that looks fine - pfsense is just upstream router to a downstream router with the downstream router being the gateway for all networks, and routing.. Connected via a transit.. This is yes basic 101 networking. But pfsense can not provide dhcp services for those downstream networks.. It just can't..
-
@johnpoz
L3 switch will do DHCP. There was a typo I forgot to change in my last diagram. Sorry for the confusion. You maybe missed this phrase in my last post:@elodie80 said in L3 Switch and pfSense design advise:
I tested all possibilities and you are right. The transit route is what works the best.
I could test it on each device separately. I will go with the DHCP server on the L3 switchI decided to follow your advices.
- L3 interface for the trunk would be 172.26.1.5
- L2 interface for the transit/trunk: 172.26.1.4
- pfSense interface for the transit: 172.26.1.1
Not sure for me:
- pfSense static routes: 10.0.* --> 172.26.1.5 (L3) or 172.26.1.4 (L2) ??
- L2 next hop router: pfSense 172.26.1.1
- L3 next hop router: pfSense 172.26.1.1
Can you confirm setup please ?
I will update the diagram -
@elodie80 said in L3 Switch and pfSense design advise:
L2 interface for the transit/trunk: 172.26.1.4
Huh - you don't put an IP on a L2.. IP is L3.
-
@johnpoz
Thank you again.
I will push experiments to follow all your advices connecting the 2 switches in that wired order. I will give a feedback and a proper diagram once done.Once I can afford one of the new Cisco 10Gb fanless switches, I plan to replace the SG350 with it for the L3 part where I need that bandwidth.
Best regards
-
@johnpoz
Thank you for all your advices.
I managed to properly set a transit route and VLANs on the L3 Switch + static routes on pfSense. DHCP on the Switch. That works as expected.Now, and hope you don't loose your mind, I wanted to get my self some learning.
I managed to cascade the L3 switch and pfSense + use the DHCP on pfSense:
- tracert shows all interVLAN traffic not reaching pfSense
- internet works for all VLANs
- pfSense DNS resolver and DHCP work for each VLAN and I can assign specific DNS servers (openDNS) to given clients which is a big plus for my needs
- I can properly RDP between VLANs and from WAN to VLANs
- speed tests and regular use show no issues on teh WAN side
Clients manual config:
- Gateway: 10.0.x.2 (L3 Switch)
- DNS: 10.0.x.1 (pfSense)
- DHCP server: 10.0.x.1 (pfSense)
- (optional: set by DHCP)
L3 Switch:
- Static route: 0.0.0.0/0 > 172.26.1.1 (configured for internet access)
pfSense:
- No static routes since the VLAN interfaces are defined
- DHCP for each interface points to L3 Switch interface as the gateway
- Firewall rules on the Transit interface to allow WAN traffic for all VLAN interfaces
- Firewall rules on the VLAN interface to allow/block traffic if needed
Trace from VLAN 20 workstation:
tracert 9.9.9.9 Tracing route to dns9.quad9.net [9.9.9.9] over a maximum of 30 hops: 1 1 ms 1 ms 1 ms 10.0.20.2 2 <1 ms <1 ms <1 ms pfSense.intranet [172.26.1.1] 3 9 ms 11 ms 9 ms 77-56-224-1.dclient.hispeed.ch [77.56.224.1] 4 11 ms 11 ms 9 ms 217-168-54-9.static.cablecom.ch [217.168.54.9] 5 14 ms 11 ms 15 ms ch-nax01a-rc1-ae-55-0.aorta.net [84.116.204.249] 6 14 ms 10 ms 9 ms ch-gva01a-ri1-ae-0-0.aorta.net [84.116.133.154] 7 10 ms 10 ms 15 ms cixp-packet-clearing-house-as42.cern.ch [192.65.185.182] 8 9 ms 9 ms 12 ms dns9.quad9.net [9.9.9.9] Trace complete.Trace between VLAN 10 <-> 20 workstations:
tracert 10.0.20.10 Tracing route to 10.0.20.10 over a maximum of 30 hops 1 1 ms 1 ms 1 ms sg350.intranet [10.0.20.2] 2 <1 ms <1 ms <1 ms 10.0.20.10 Trace complete. tracert 10.0.10.10 Tracing route to 10.0.10.10 over a maximum of 30 hops 1 1 ms 1 ms 1 ms sg350.intranet [10.0.10.2] 2 <1 ms <1 ms <1 ms 10.0.10.10 Trace complete.InterVLAN Traffic doesn't reach pfSense and is no longer asymmetric in this way.
However, I see a potential asymmetric issue with WAN traffic because downstream traffic will probably not pass through the transit, but directly to the VLAN, right ?For a controlled use, is that a real concern ? Do you have examples of failed use / issues so that I see if I should be concerned.
Sure the transit only method is the most conventional and clear. I fully agree. However, I would really like a feedback on this setup.
Thank you for your patience
-
Hi,
Here's the final layout that is working perfectly in daily use. Obviously, you cannot use pfSense interface as a gateway for any inter VLAN communication
pfSense is properly providing DHCP and DNS Server services for all teh downstream VLANs
- pfSense is the WAN <-> LAN Firewall
- VLANs isolation must be done at the L3 routing Switch with ACL rules
- VLAN 1 dedicated interface for the management LAN is optional and can be untagged with the main Trunk if there is no room for physical interfaces.
- In pfSense, the only interface having internet access is the Transit interface: this simplifies internet control
- In pfSense, define all VLAN interfaces (10.0.x.1) and set these rules for each interface:
- block access to management ports (80, 443, 22) from "any" to "This Firewall"
- allow access from "VLAN net" to "VLAN address" for DNS ports (53, 853 as needed)
- optionally allow UDP access from "VLAN net" to broadcast addresses (10.0.x.255 and 255.255.255.255)
- the remaining is default deny
- In pfSense, for the Transit interface:
- block access to management ports (80, 443, 22) from "any" to "This Firewall"
- add any specific internet rules you need (DNS resolver NAT rules are done on this interface if DNS restrictions are needed)
- Allow IPv4+IPv6 access from "any" to "non RFC1918 Private Networks"
- pfSense DHCP server for each VLAN must set the gateway of the clients to the Routing Switch VLAN interface (10.0.x.2)
- Clients with a manually set IP, must point to the Routing Switch VLAN interface (10.0.x.2) as a gateway and pfSense VLAN interface (10.0.x.1) as DNS Server
- L3 routing switch (SG350):
- define all VLAN interfaces with static IP 10.0.x.2
- enable inter VLAN routing
- define the static route interface or VLAN 172.26.1.2
- define the default static route for internet and non-VLAN traffic: 0.0.0.0/0 -> 172.26.1.1
- define all ACL rules needed to isolate VLANs
- L3 / L2 switch (SG350X)
- the SG350X cannot be set to L2, so just disable inter VLAN routing
- define the management interface VLAN 1 (10.0.1.3)
- optional: I added the VLAN 10 interface (10.0.10.3 ) to be able to manage the switch from VLAN 10 workstation without needing the VLAN 1 interface
- no need to define other VLAN interfaces (just like an L2 switch) neither static routes
- The trunk linking the 2 switches must have tagged all the VLANs specified in the L2/L3 downstream switch. The same VLANs must be defined on both ends of the Trunk obviously
- Unifi AP :
- trunk is untagged to the default management interface of the AP. Can be tagged also depending on the controller software versions
- define in the trunk all the VLANs used in the AP for SSID broadcast
Hope this can help other people trying to get pfSense to work as DHCP server with a downstream L3 routing switch
And many thanks for all your help, even if I did not end up following your advice.
WARNING: as long as the user is aware that there are risks of asymmetric routing, this setup is perfectly fine if teh proper gateway is specified.
Side Note and question: Anyone can explain why pfSense is allowing the client <-> WAN access in this setup ? Normally it should be the only asymmetric part in the above design: outgoing traffic uses transit interface, but return traffic should use the VLAN interface as I thought. However it is not the case and there are no issues with remote WAN traffic at all. I thought a firewall wouln't allow this but maybe I misunderstood it.
