L3 Switch and pfSense design advise
-
Yes you create a gateway, the downstream or upstream router via the transit network IP. And then create routes pointing to that gateway, or use a routing protocol to exchange routes, etc.
But you can not put devices on this network - or your going to run into asymmetrical routing. And once you throw a stateful firewall in there.. You have even more problems, even if you didn't care about the asymmetrical traffic flow.
If your going to connect routers, be it a real router or a L3 doing routing. The network that attaches these routers together would have not devices on it - other than say another router to gets to other networks.
Your transit can be native or a vlan.. Doesn't matter.. Other than if vlan you might/could/would be hairpining traffic etc..
edit:
You could for sure allow your L3 to do all the routing for all your networks.. And then pfsense would just be an upstream router via a transit to get to the internet. But pfsense can not do dhcp for these downstream networks.But to be honest if you have a VM host.. You could run dhcp on some VM, and let it do all the dhcp for all your networks/vlans. Or what is running on your backup server or dlna server for OS.. One of those could prob run a dhcp server.
-
@johnpoz
Again, many thanks for the detailed explanation.
I will rewrite my drawing and post again.However, I still have one question: I was always told that, by default, pfsense will not route any traffic between interfaces except if it comes from its own.
I guess this is wrong?
In this case, can't I keep the interfaces in pfSense (for DHCP), define a transit route to the L3 switch (without devices in that route) like you suggested AND explicitly define firewall rules in pfSense to exclude any routing between the interfaces. That way, descending traffic would still be instructed to go down to the L3 Switch. Or it is just impossible?
-
@elodie80 said in L3 Switch and pfSense design advise:
pfsense will not route any traffic between interfaces except if it comes from its own.
Not sure where you heard such a thing to be honest. By "default" no it wouldn't because normally rules have source traffic limited to the network its attached to.. Say "lan net" or "optX net" etc.. So traffic from some downstream network wouldn't even be allowed into the interface.
But even if you did allow it - how would it no how to get back to some unknown network. Without a route.
But its very simple to create routing for networks that are not attached directly to pfsense. You just need to allow for them on your transit network interface.. And pfsense needs to know where to send traffic via routing.
edit: Here is a great drawing Derelict had put up and even had in his signature for long time that lays it out pretty good how to do a downstream router
edit2: Yeah he mentions the outbound nat and downstream networks. But pfsense will normally do this for you if you have outbound nat on auto. Where it might come into play where you would have to manual do the outbound nats for your downstream is if you switch to manual mode on your outbound nat (say followed some idiot vpn guide that said to do this)
-
@johnpoz said in L3 Switch and pfSense design advise:
Not sure where you heard such a thing to be honest. By "default" no it wouldn't because normally rules have source traffic limited to the network its attached to.. Say "lan net" or "optX net" etc.. So traffic from some downstream network wouldn't even be allowed into the interface.
But even if you did allow it - how would it no how to get back to some unknown network. Without a route.
I totally got your idea and the last drawing, thanks a lot, really.
But just about the part I quoted from your answer, and I still cannot understand it properly:When the interfaces are added to the pfSense router, is there still a possibility to ask pfSense:
- not to route between the interfaces at the pfSense layer
- instead, route that incoming WAN traffic for the interfaces (defined in pfSense, so pfSense knows about the interface) to a transit route directed to the downstream L3 switch
That's the part I am not understanding the reason. Why the router cannot be asked to forward traffic to a static route instead of directly routing it between the interfaces he knows about ?
-
@elodie80 said in L3 Switch and pfSense design advise:
Why the router cannot be asked to forward traffic to a static route instead of directly routing it between the interfaces he knows about ?
Directly attached is ALWAYS going to be better than some route that is 1 hop away.
-
@johnpoz said in L3 Switch and pfSense design advise:
Directly attached is ALWAYS going to be better than some route that is 1 hop away.
I understand.
Now just for me to understand it even better, I did the below test setup:
- added interfaces with VLAN 10 (10.0.10.1/24), VLAN 20 (10.0.20.1/24) and a Transit VLAN (172.26.1.1/24).
- I added the Gateways to those interfaces to point to the Switch VLAN interface
- pfSense automatically added the outbound NAT rules to WAN for those interfaces
Here's the config setup:
-
Define VLAN 10, 20 and a Transit VLAN
-
Assign the interfaces (Transit would be assigned to a physical interface in a real setup, but in fact it is not even needed)
3. Add the VLAN 10, 20 and transit gateways to point to the corresponding switch interface address (*.2)This was sick
-deleted image-With the gateway set this way: if I understand you well, pfSense would still prioritize the interVLAN routing directly in pfSense for inbound traffic causing asymmetrical routing, right ?
But if no such traffic ever comes: not possible from WAN as long as I know AND no device directly connected to pfSense, how could I end up with asymmetrical routing at all ? And in this config, the transit route is not needed and can be used as a regular VLAN !
-
@elodie80 said in L3 Switch and pfSense design advise:
But if no such traffic ever comes
What is the point of them - what do you think creating them does?
I have no clue what your trying to do.. Makes ZERO sense..
You only need 1 gateway - and then routes for networks that are reachable via that gateway..
Creating some vlan or interface with a network on it, isn't going to allow dhcp on that network. if the network doesn't see the broadcast for the dhcp discover..
-
@johnpoz said in L3 Switch and pfSense design advise:
does
Sorry for insisting, just couldn't understand what makes the setup in my last post (router-on-a-stick) cause an asymmetrical routing when no devices are connected to pfSense directly.
I am just trying to be sure there is no way to get DHCP working on pfSense for now without causing asymmetrical routing issues.
DHCP: since the Switch has a port on the same subnet as the interface on pfSense, it can see the DHCP server in pfSense
I really couldn't find explanations with a real example failing this (asymmetrical issue) and many guides advocate such a router-on-a-stick setup (but I do believe you and your explanations, just trying to get to the core issue)
-
@johnpoz said in L3 Switch and pfSense design advise:
What is the point of them - what do you think creating them does?
I have no clue what your trying to do.. Makes ZERO sense..
That was sick on my side, yes. There is no point to creating a Gateway pointing to the same subnet in this case. Better I go to sleep... a bit late on my side I guess
I understood the best practice you suggested. It is ok and thank you for that. I know I have to find the resources to run my DHCP server as I really hate the Cisco DHCP server in the switch.
I am just trying to understand what kind of TCP traffic in my case would cause asymmetrical routing issues when pfsense has track of each interface directly attached in it.
Here are two scenarios for asymmetric routing, credits to https://skminhaj.wordpress.com/2016/02/15/the-concept-of-asymmetric-routing/
And between User PC and www cloud server:
With my two switches and setup, return traffic would never pass from one location to another using a different route. Or am I missing such a case ?
So, if I take care at that aspect, can I run DHCP on pfSense without asymmetric routing until I can afford a dedicated DHCP server and remove any future potential asymmetric issues by using a transit route like you suggested ? Simply, am I missing some asymmetric scenario in my layout ?
Sorry again for insisting.
-
@johnpoz
I tested all possibilities and you are right. The transit route is what works the best.
I could test it on each device separately. I will go with the DHCP server on the L3 switchHowever, I could not find a scenario where the L2 switch is facing the firewall while the L3 is downstream
I am limited by the physical locations as in the below diagram. Also, I only have 2 wall LAN plugs joining the rooms between the L2 and L3 switches. The desktop in L2 room needs a 10 Gb switching speed with the servers that the L2 switch cannot provide. I cannot invert the switch rooms for now.
Is it safe to run in this configuration ?
Static routes from pfSense wouzld point to the L2 switch and the default next hop in L3 switch would point to the transit interface in pfSense ? I guess the trunk between L2 and L3 switch would be on the same VLAN IP as the transit in that case ?Hope you can explain me this specific Firewall -> L2 -> L3 layout
Best regards
-
@elodie80 said in L3 Switch and pfSense design advise:
With my two switches and setup, return traffic would never pass from one location to another using a different route.
Dude.. What is you don't get? Here is your basic network.. And how and why its asymmetrical when you put clients on a transit.
If something on the 192.168.1 network wants to talk to something on 192.168.2 network.. Sends syn to its gateway, gets sent down fine.. And sure you have a state in that direction. But if the box on the .2 network is trying to talk to something on .1 network - then the return traffic being sent to upstream router would be syn,ack - and this firewall doesn't have a state and would block the traffic.
You do not but hosts on a network that connects routers - you don't!! If you do - your going to have problems!!! unless you host route on this host telling it which router to talk to get to each network.. Or you nat on the downstream..
Also depending on your client it might not like that return traffic coming from a different mac.. It could say hey I sent traffic to 1.2.3.4 to mac abc.. Why is my return traffic coming from mac xyz. So again you do not put devices on a network that connects routers - or your going to have a bad day! ;)
Again in your last drawing - pfsense CAN NOT!!! CAN NOT do dhcp for a network its not directly attached too..
While from quick glance that looks fine - pfsense is just upstream router to a downstream router with the downstream router being the gateway for all networks, and routing.. Connected via a transit.. This is yes basic 101 networking. But pfsense can not provide dhcp services for those downstream networks.. It just can't..
-
@johnpoz
L3 switch will do DHCP. There was a typo I forgot to change in my last diagram. Sorry for the confusion. You maybe missed this phrase in my last post:@elodie80 said in L3 Switch and pfSense design advise:
I tested all possibilities and you are right. The transit route is what works the best.
I could test it on each device separately. I will go with the DHCP server on the L3 switchI decided to follow your advices.
- L3 interface for the trunk would be 172.26.1.5
- L2 interface for the transit/trunk: 172.26.1.4
- pfSense interface for the transit: 172.26.1.1
Not sure for me:
- pfSense static routes: 10.0.* --> 172.26.1.5 (L3) or 172.26.1.4 (L2) ??
- L2 next hop router: pfSense 172.26.1.1
- L3 next hop router: pfSense 172.26.1.1
Can you confirm setup please ?
I will update the diagram -
@elodie80 said in L3 Switch and pfSense design advise:
L2 interface for the transit/trunk: 172.26.1.4
Huh - you don't put an IP on a L2.. IP is L3.
-
@johnpoz
Thank you again.
I will push experiments to follow all your advices connecting the 2 switches in that wired order. I will give a feedback and a proper diagram once done.Once I can afford one of the new Cisco 10Gb fanless switches, I plan to replace the SG350 with it for the L3 part where I need that bandwidth.
Best regards
-
@johnpoz
Thank you for all your advices.
I managed to properly set a transit route and VLANs on the L3 Switch + static routes on pfSense. DHCP on the Switch. That works as expected.Now, and hope you don't loose your mind, I wanted to get my self some learning.
I managed to cascade the L3 switch and pfSense + use the DHCP on pfSense:
- tracert shows all interVLAN traffic not reaching pfSense
- internet works for all VLANs
- pfSense DNS resolver and DHCP work for each VLAN and I can assign specific DNS servers (openDNS) to given clients which is a big plus for my needs
- I can properly RDP between VLANs and from WAN to VLANs
- speed tests and regular use show no issues on teh WAN side
Clients manual config:
- Gateway: 10.0.x.2 (L3 Switch)
- DNS: 10.0.x.1 (pfSense)
- DHCP server: 10.0.x.1 (pfSense)
- (optional: set by DHCP)
L3 Switch:
- Static route: 0.0.0.0/0 > 172.26.1.1 (configured for internet access)
pfSense:
- No static routes since the VLAN interfaces are defined
- DHCP for each interface points to L3 Switch interface as the gateway
- Firewall rules on the Transit interface to allow WAN traffic for all VLAN interfaces
- Firewall rules on the VLAN interface to allow/block traffic if needed
Trace from VLAN 20 workstation:
tracert 9.9.9.9 Tracing route to dns9.quad9.net [9.9.9.9] over a maximum of 30 hops: 1 1 ms 1 ms 1 ms 10.0.20.2 2 <1 ms <1 ms <1 ms pfSense.intranet [172.26.1.1] 3 9 ms 11 ms 9 ms 77-56-224-1.dclient.hispeed.ch [77.56.224.1] 4 11 ms 11 ms 9 ms 217-168-54-9.static.cablecom.ch [217.168.54.9] 5 14 ms 11 ms 15 ms ch-nax01a-rc1-ae-55-0.aorta.net [84.116.204.249] 6 14 ms 10 ms 9 ms ch-gva01a-ri1-ae-0-0.aorta.net [84.116.133.154] 7 10 ms 10 ms 15 ms cixp-packet-clearing-house-as42.cern.ch [192.65.185.182] 8 9 ms 9 ms 12 ms dns9.quad9.net [9.9.9.9] Trace complete.Trace between VLAN 10 <-> 20 workstations:
tracert 10.0.20.10 Tracing route to 10.0.20.10 over a maximum of 30 hops 1 1 ms 1 ms 1 ms sg350.intranet [10.0.20.2] 2 <1 ms <1 ms <1 ms 10.0.20.10 Trace complete. tracert 10.0.10.10 Tracing route to 10.0.10.10 over a maximum of 30 hops 1 1 ms 1 ms 1 ms sg350.intranet [10.0.10.2] 2 <1 ms <1 ms <1 ms 10.0.10.10 Trace complete.InterVLAN Traffic doesn't reach pfSense and is no longer asymmetric in this way.
However, I see a potential asymmetric issue with WAN traffic because downstream traffic will probably not pass through the transit, but directly to the VLAN, right ?For a controlled use, is that a real concern ? Do you have examples of failed use / issues so that I see if I should be concerned.
Sure the transit only method is the most conventional and clear. I fully agree. However, I would really like a feedback on this setup.
Thank you for your patience
-
Hi,
Here's the final layout that is working perfectly in daily use. Obviously, you cannot use pfSense interface as a gateway for any inter VLAN communication
pfSense is properly providing DHCP and DNS Server services for all teh downstream VLANs
- pfSense is the WAN <-> LAN Firewall
- VLANs isolation must be done at the L3 routing Switch with ACL rules
- VLAN 1 dedicated interface for the management LAN is optional and can be untagged with the main Trunk if there is no room for physical interfaces.
- In pfSense, the only interface having internet access is the Transit interface: this simplifies internet control
- In pfSense, define all VLAN interfaces (10.0.x.1) and set these rules for each interface:
- block access to management ports (80, 443, 22) from "any" to "This Firewall"
- allow access from "VLAN net" to "VLAN address" for DNS ports (53, 853 as needed)
- optionally allow UDP access from "VLAN net" to broadcast addresses (10.0.x.255 and 255.255.255.255)
- the remaining is default deny
- In pfSense, for the Transit interface:
- block access to management ports (80, 443, 22) from "any" to "This Firewall"
- add any specific internet rules you need (DNS resolver NAT rules are done on this interface if DNS restrictions are needed)
- Allow IPv4+IPv6 access from "any" to "non RFC1918 Private Networks"
- pfSense DHCP server for each VLAN must set the gateway of the clients to the Routing Switch VLAN interface (10.0.x.2)
- Clients with a manually set IP, must point to the Routing Switch VLAN interface (10.0.x.2) as a gateway and pfSense VLAN interface (10.0.x.1) as DNS Server
- L3 routing switch (SG350):
- define all VLAN interfaces with static IP 10.0.x.2
- enable inter VLAN routing
- define the static route interface or VLAN 172.26.1.2
- define the default static route for internet and non-VLAN traffic: 0.0.0.0/0 -> 172.26.1.1
- define all ACL rules needed to isolate VLANs
- L3 / L2 switch (SG350X)
- the SG350X cannot be set to L2, so just disable inter VLAN routing
- define the management interface VLAN 1 (10.0.1.3)
- optional: I added the VLAN 10 interface (10.0.10.3 ) to be able to manage the switch from VLAN 10 workstation without needing the VLAN 1 interface
- no need to define other VLAN interfaces (just like an L2 switch) neither static routes
- The trunk linking the 2 switches must have tagged all the VLANs specified in the L2/L3 downstream switch. The same VLANs must be defined on both ends of the Trunk obviously
- Unifi AP :
- trunk is untagged to the default management interface of the AP. Can be tagged also depending on the controller software versions
- define in the trunk all the VLANs used in the AP for SSID broadcast
Hope this can help other people trying to get pfSense to work as DHCP server with a downstream L3 routing switch
And many thanks for all your help, even if I did not end up following your advice.
WARNING: as long as the user is aware that there are risks of asymmetric routing, this setup is perfectly fine if teh proper gateway is specified.
Side Note and question: Anyone can explain why pfSense is allowing the client <-> WAN access in this setup ? Normally it should be the only asymmetric part in the above design: outgoing traffic uses transit interface, but return traffic should use the VLAN interface as I thought. However it is not the case and there are no issues with remote WAN traffic at all. I thought a firewall wouln't allow this but maybe I misunderstood it.
-
@elodie80 said in L3 Switch and pfSense design advise:
WARNING: as long as the user is aware that there are risks of asymmetric routing, this setup is perfectly fine if teh proper gateway is specified.
Sorry but no.. YOUR DOING IT WRONG!!! If you on purpose create asymmetrical traffic flow.. Completely and utterly borked.. Not fine at all.. And you create a bypass of your firewall as well.. So the whole point of your setup provides no security.. Its BORKED!!!
Please refrain from suggesting any one do anything off the sort.
All of this mess because you want to use pfsense for dhcp?
-
@johnpoz
With all the respect I have, and the fact that I am far from a Network expert, I just cannot see why it is BORKED when it works in all the scenarios I tested
I asked for help on other forums, and the concensus was that as long as I am aware that the clients setup must account for the fact that pfSense is not the gateway, there is no issue.All my traceroutes show that there is no asymmetrical issues and that all routing is done on the SG350 and inter VLAN traffic NEVER reaches pfSense
The Transit route is still there and all internet traffic goes upstream through it. All the VLAN interfaces on pfSense have no internet access. The only VLAN with internet access is the Transit
RDP, speedtests... all work properly even from WAN to LAN
Logs show no asymmetrical packets dropped from WANI am maybe stubborn, but I'd kindly ask you a concrete example that I can reproduce to get a broken routing in my setup. Obviuosly if VLAN interface on pfSense is used as Gateway in this setup it will be broken
Thank you and I hope that you post an example of clients setup I can reproduce to show me it is broken
Note: yes, setting up a dedicated DHCP / DNS server is for now a limitation for me. It will be done in a couple of years and I would be able to use teh proper setup with only static routes in pfSense
Best regards
-
You have bypassed your firewall by putting 2 paths to get to a network, even host on the transit doesn't know about it..
You can do it, but its borked. If you also firewalled traffic at your downstream that you could remove the security issue.
You seem to be going out of your way to create a very complex setup for no other reason than you want to use pfsense as dhcpd for downstream networks which it can not do..
Run your dhcp on a VM is simple solution.. You clearly have boxes there your serveres to be able to run VM.. DNS doesn't matter where it is. ISC dhcp for example can be dhcpd for downstream networks. It just can not do it via pfsense and gui controls.
If you were going to put hosts on a transit, that can work too with simple host routing and not too much issue other than security.. But you have multiple transits with every vlan, so multiple paths for asymmetrical flow.. Even if pfsense sees traffic come from transit, since it has connection in the the vlan, it can send return traffic out that connection. Now the if the client was actually doing things securely - it would balk at such traffic..
You are creating a monster of a setup because you want to run dhcpd on pfsense - when you could just run it on your L3 switch, or pfsense and your L3 switch - or just on some vm running on 1 of your servers.
-
@johnpoz said in L3 Switch and pfSense design advise:
bypassed your firewall by putting 2 paths to get to a network,
Consider it a learning purpose please.
I need openDNS restrictions on some clients, so if the pfSense DNS resolver can still be used to restrict DNS queries on the transit route from subnets not directly attached, that would make it possible for me to use the DHCP server on the Switch.But still, why pfSense is allowing this sort of setup at WAN level instead of dropping the return traffic coming from WAN ?
The only explication I thought is that pfSense is keeping track that the initial request comes from the Transit interface and sends it back throgh that interface and not through the VLAN interface.I need the firewall for WAN <-> LAN currently. Is my setup breaking that part of the firewall ?
