L3 Switch and pfSense design advise
-
@johnpoz
I tested all possibilities and you are right. The transit route is what works the best.
I could test it on each device separately. I will go with the DHCP server on the L3 switchHowever, I could not find a scenario where the L2 switch is facing the firewall while the L3 is downstream
I am limited by the physical locations as in the below diagram. Also, I only have 2 wall LAN plugs joining the rooms between the L2 and L3 switches. The desktop in L2 room needs a 10 Gb switching speed with the servers that the L2 switch cannot provide. I cannot invert the switch rooms for now.
Is it safe to run in this configuration ?
Static routes from pfSense wouzld point to the L2 switch and the default next hop in L3 switch would point to the transit interface in pfSense ? I guess the trunk between L2 and L3 switch would be on the same VLAN IP as the transit in that case ?Hope you can explain me this specific Firewall -> L2 -> L3 layout
Best regards
-
@elodie80 said in L3 Switch and pfSense design advise:
With my two switches and setup, return traffic would never pass from one location to another using a different route.
Dude.. What is you don't get? Here is your basic network.. And how and why its asymmetrical when you put clients on a transit.
If something on the 192.168.1 network wants to talk to something on 192.168.2 network.. Sends syn to its gateway, gets sent down fine.. And sure you have a state in that direction. But if the box on the .2 network is trying to talk to something on .1 network - then the return traffic being sent to upstream router would be syn,ack - and this firewall doesn't have a state and would block the traffic.
You do not but hosts on a network that connects routers - you don't!! If you do - your going to have problems!!! unless you host route on this host telling it which router to talk to get to each network.. Or you nat on the downstream..
Also depending on your client it might not like that return traffic coming from a different mac.. It could say hey I sent traffic to 1.2.3.4 to mac abc.. Why is my return traffic coming from mac xyz. So again you do not put devices on a network that connects routers - or your going to have a bad day! ;)
Again in your last drawing - pfsense CAN NOT!!! CAN NOT do dhcp for a network its not directly attached too..
While from quick glance that looks fine - pfsense is just upstream router to a downstream router with the downstream router being the gateway for all networks, and routing.. Connected via a transit.. This is yes basic 101 networking. But pfsense can not provide dhcp services for those downstream networks.. It just can't..
-
@johnpoz
L3 switch will do DHCP. There was a typo I forgot to change in my last diagram. Sorry for the confusion. You maybe missed this phrase in my last post:@elodie80 said in L3 Switch and pfSense design advise:
I tested all possibilities and you are right. The transit route is what works the best.
I could test it on each device separately. I will go with the DHCP server on the L3 switchI decided to follow your advices.
- L3 interface for the trunk would be 172.26.1.5
- L2 interface for the transit/trunk: 172.26.1.4
- pfSense interface for the transit: 172.26.1.1
Not sure for me:
- pfSense static routes: 10.0.* --> 172.26.1.5 (L3) or 172.26.1.4 (L2) ??
- L2 next hop router: pfSense 172.26.1.1
- L3 next hop router: pfSense 172.26.1.1
Can you confirm setup please ?
I will update the diagram -
@elodie80 said in L3 Switch and pfSense design advise:
L2 interface for the transit/trunk: 172.26.1.4
Huh - you don't put an IP on a L2.. IP is L3.
-
@johnpoz
Thank you again.
I will push experiments to follow all your advices connecting the 2 switches in that wired order. I will give a feedback and a proper diagram once done.Once I can afford one of the new Cisco 10Gb fanless switches, I plan to replace the SG350 with it for the L3 part where I need that bandwidth.
Best regards
-
@johnpoz
Thank you for all your advices.
I managed to properly set a transit route and VLANs on the L3 Switch + static routes on pfSense. DHCP on the Switch. That works as expected.Now, and hope you don't loose your mind, I wanted to get my self some learning.
I managed to cascade the L3 switch and pfSense + use the DHCP on pfSense:
- tracert shows all interVLAN traffic not reaching pfSense
- internet works for all VLANs
- pfSense DNS resolver and DHCP work for each VLAN and I can assign specific DNS servers (openDNS) to given clients which is a big plus for my needs
- I can properly RDP between VLANs and from WAN to VLANs
- speed tests and regular use show no issues on teh WAN side
Clients manual config:
- Gateway: 10.0.x.2 (L3 Switch)
- DNS: 10.0.x.1 (pfSense)
- DHCP server: 10.0.x.1 (pfSense)
- (optional: set by DHCP)
L3 Switch:
- Static route: 0.0.0.0/0 > 172.26.1.1 (configured for internet access)
pfSense:
- No static routes since the VLAN interfaces are defined
- DHCP for each interface points to L3 Switch interface as the gateway
- Firewall rules on the Transit interface to allow WAN traffic for all VLAN interfaces
- Firewall rules on the VLAN interface to allow/block traffic if needed
Trace from VLAN 20 workstation:
tracert 9.9.9.9 Tracing route to dns9.quad9.net [9.9.9.9] over a maximum of 30 hops: 1 1 ms 1 ms 1 ms 10.0.20.2 2 <1 ms <1 ms <1 ms pfSense.intranet [172.26.1.1] 3 9 ms 11 ms 9 ms 77-56-224-1.dclient.hispeed.ch [77.56.224.1] 4 11 ms 11 ms 9 ms 217-168-54-9.static.cablecom.ch [217.168.54.9] 5 14 ms 11 ms 15 ms ch-nax01a-rc1-ae-55-0.aorta.net [84.116.204.249] 6 14 ms 10 ms 9 ms ch-gva01a-ri1-ae-0-0.aorta.net [84.116.133.154] 7 10 ms 10 ms 15 ms cixp-packet-clearing-house-as42.cern.ch [192.65.185.182] 8 9 ms 9 ms 12 ms dns9.quad9.net [9.9.9.9] Trace complete.Trace between VLAN 10 <-> 20 workstations:
tracert 10.0.20.10 Tracing route to 10.0.20.10 over a maximum of 30 hops 1 1 ms 1 ms 1 ms sg350.intranet [10.0.20.2] 2 <1 ms <1 ms <1 ms 10.0.20.10 Trace complete. tracert 10.0.10.10 Tracing route to 10.0.10.10 over a maximum of 30 hops 1 1 ms 1 ms 1 ms sg350.intranet [10.0.10.2] 2 <1 ms <1 ms <1 ms 10.0.10.10 Trace complete.InterVLAN Traffic doesn't reach pfSense and is no longer asymmetric in this way.
However, I see a potential asymmetric issue with WAN traffic because downstream traffic will probably not pass through the transit, but directly to the VLAN, right ?For a controlled use, is that a real concern ? Do you have examples of failed use / issues so that I see if I should be concerned.
Sure the transit only method is the most conventional and clear. I fully agree. However, I would really like a feedback on this setup.
Thank you for your patience
-
Hi,
Here's the final layout that is working perfectly in daily use. Obviously, you cannot use pfSense interface as a gateway for any inter VLAN communication
pfSense is properly providing DHCP and DNS Server services for all teh downstream VLANs
- pfSense is the WAN <-> LAN Firewall
- VLANs isolation must be done at the L3 routing Switch with ACL rules
- VLAN 1 dedicated interface for the management LAN is optional and can be untagged with the main Trunk if there is no room for physical interfaces.
- In pfSense, the only interface having internet access is the Transit interface: this simplifies internet control
- In pfSense, define all VLAN interfaces (10.0.x.1) and set these rules for each interface:
- block access to management ports (80, 443, 22) from "any" to "This Firewall"
- allow access from "VLAN net" to "VLAN address" for DNS ports (53, 853 as needed)
- optionally allow UDP access from "VLAN net" to broadcast addresses (10.0.x.255 and 255.255.255.255)
- the remaining is default deny
- In pfSense, for the Transit interface:
- block access to management ports (80, 443, 22) from "any" to "This Firewall"
- add any specific internet rules you need (DNS resolver NAT rules are done on this interface if DNS restrictions are needed)
- Allow IPv4+IPv6 access from "any" to "non RFC1918 Private Networks"
- pfSense DHCP server for each VLAN must set the gateway of the clients to the Routing Switch VLAN interface (10.0.x.2)
- Clients with a manually set IP, must point to the Routing Switch VLAN interface (10.0.x.2) as a gateway and pfSense VLAN interface (10.0.x.1) as DNS Server
- L3 routing switch (SG350):
- define all VLAN interfaces with static IP 10.0.x.2
- enable inter VLAN routing
- define the static route interface or VLAN 172.26.1.2
- define the default static route for internet and non-VLAN traffic: 0.0.0.0/0 -> 172.26.1.1
- define all ACL rules needed to isolate VLANs
- L3 / L2 switch (SG350X)
- the SG350X cannot be set to L2, so just disable inter VLAN routing
- define the management interface VLAN 1 (10.0.1.3)
- optional: I added the VLAN 10 interface (10.0.10.3 ) to be able to manage the switch from VLAN 10 workstation without needing the VLAN 1 interface
- no need to define other VLAN interfaces (just like an L2 switch) neither static routes
- The trunk linking the 2 switches must have tagged all the VLANs specified in the L2/L3 downstream switch. The same VLANs must be defined on both ends of the Trunk obviously
- Unifi AP :
- trunk is untagged to the default management interface of the AP. Can be tagged also depending on the controller software versions
- define in the trunk all the VLANs used in the AP for SSID broadcast
Hope this can help other people trying to get pfSense to work as DHCP server with a downstream L3 routing switch
And many thanks for all your help, even if I did not end up following your advice.
WARNING: as long as the user is aware that there are risks of asymmetric routing, this setup is perfectly fine if teh proper gateway is specified.
Side Note and question: Anyone can explain why pfSense is allowing the client <-> WAN access in this setup ? Normally it should be the only asymmetric part in the above design: outgoing traffic uses transit interface, but return traffic should use the VLAN interface as I thought. However it is not the case and there are no issues with remote WAN traffic at all. I thought a firewall wouln't allow this but maybe I misunderstood it.
-
@elodie80 said in L3 Switch and pfSense design advise:
WARNING: as long as the user is aware that there are risks of asymmetric routing, this setup is perfectly fine if teh proper gateway is specified.
Sorry but no.. YOUR DOING IT WRONG!!! If you on purpose create asymmetrical traffic flow.. Completely and utterly borked.. Not fine at all.. And you create a bypass of your firewall as well.. So the whole point of your setup provides no security.. Its BORKED!!!
Please refrain from suggesting any one do anything off the sort.
All of this mess because you want to use pfsense for dhcp?
-
@johnpoz
With all the respect I have, and the fact that I am far from a Network expert, I just cannot see why it is BORKED when it works in all the scenarios I tested
I asked for help on other forums, and the concensus was that as long as I am aware that the clients setup must account for the fact that pfSense is not the gateway, there is no issue.All my traceroutes show that there is no asymmetrical issues and that all routing is done on the SG350 and inter VLAN traffic NEVER reaches pfSense
The Transit route is still there and all internet traffic goes upstream through it. All the VLAN interfaces on pfSense have no internet access. The only VLAN with internet access is the Transit
RDP, speedtests... all work properly even from WAN to LAN
Logs show no asymmetrical packets dropped from WANI am maybe stubborn, but I'd kindly ask you a concrete example that I can reproduce to get a broken routing in my setup. Obviuosly if VLAN interface on pfSense is used as Gateway in this setup it will be broken
Thank you and I hope that you post an example of clients setup I can reproduce to show me it is broken
Note: yes, setting up a dedicated DHCP / DNS server is for now a limitation for me. It will be done in a couple of years and I would be able to use teh proper setup with only static routes in pfSense
Best regards
-
You have bypassed your firewall by putting 2 paths to get to a network, even host on the transit doesn't know about it..
You can do it, but its borked. If you also firewalled traffic at your downstream that you could remove the security issue.
You seem to be going out of your way to create a very complex setup for no other reason than you want to use pfsense as dhcpd for downstream networks which it can not do..
Run your dhcp on a VM is simple solution.. You clearly have boxes there your serveres to be able to run VM.. DNS doesn't matter where it is. ISC dhcp for example can be dhcpd for downstream networks. It just can not do it via pfsense and gui controls.
If you were going to put hosts on a transit, that can work too with simple host routing and not too much issue other than security.. But you have multiple transits with every vlan, so multiple paths for asymmetrical flow.. Even if pfsense sees traffic come from transit, since it has connection in the the vlan, it can send return traffic out that connection. Now the if the client was actually doing things securely - it would balk at such traffic..
You are creating a monster of a setup because you want to run dhcpd on pfsense - when you could just run it on your L3 switch, or pfsense and your L3 switch - or just on some vm running on 1 of your servers.
-
@johnpoz said in L3 Switch and pfSense design advise:
bypassed your firewall by putting 2 paths to get to a network,
Consider it a learning purpose please.
I need openDNS restrictions on some clients, so if the pfSense DNS resolver can still be used to restrict DNS queries on the transit route from subnets not directly attached, that would make it possible for me to use the DHCP server on the Switch.But still, why pfSense is allowing this sort of setup at WAN level instead of dropping the return traffic coming from WAN ?
The only explication I thought is that pfSense is keeping track that the initial request comes from the Transit interface and sends it back throgh that interface and not through the VLAN interface.I need the firewall for WAN <-> LAN currently. Is my setup breaking that part of the firewall ?
-
@johnpoz said in L3 Switch and pfSense design advise:
You have bypassed your firewall by putting 2 paths to get to a network, even host on the transit doesn't know about it..
Is my setup working because of a bug in pfSense ?
Why is pfSense allowing/tracking my VLAN <-> WAN connections ?Because I checked the states and pfSense is clearly redirecting the wan -> LAN traffic through the VLAN interface and not transit. However, pfSense is properly keeping track of sates it seems. I have no option enabled to Bypass firewall rules for traffic on the same interface. I can set all rules in Transit interface to control internet access and access to the VLAN address on pfSense is properly limited to DNS queries. No traffic is dropped ! Why ?
-
Here's the recommended proper setup with the Transit route, DHCP server on the switch, no asymmetric traffic from LAN to WAN:
The only changes:
- no VLAN interfaces on pfSense (use aliases for VLAN subnets to simplify managing rules in the transit interface)
- use a static route in pfSense 10.0.0.0/16 -> 172.26.1.2
- the DHCP server for clients is set on the routing Switch SG350
My thoughts:
- the changes are not really big between the 2 layouts
- setup complexity is the same: DNS rules must be all applied in the Transit interface for each subnet (instead of applying them in each VLAN interface in pfSense)
- I see no difference in my firewall states than previous setup !
pfSense will first route the traffic to the corresponding interface, then it will check the firewall rules and drop/pass traffic accordingly. So, I am not bypassing the firewall here since the rules will apply to the interface. For inter-VLAN traffic, ACLs are applied by the L3 switch
@johnpoz
Still my question remains: why pfSense is allowing sloppy states and the anti-spoofing rules are not triggered with my previous setup for LAN <-> WAN traffic ? I see no difference in the firewall states at all !Thank you again for your patience. I know this subject was addressed many times and is emotional. Hope you can still answer my last question.
Edit: fixed VLAN 1 interface range
-
@elodie80 If your intent is to learn networking then indeed an impressive effort
And to be honest I did not follow all the details in the thread above but I recently created VLANs for my home network and it was quiet a steep learning curve. Interestingly I also bought a Dell L3 switch on eBay (for $40!) and it has quiet the firepower in terms of handling traffic on 64 ports. But after much deliberation I decided not to use L3 switch and here are my learning experiences from this exercise:
-
First the basics that I need to recollect each time I am troubleshooting
- Physical Layer (L1) works on bits ....literally
- Data Link Layer (L2) works on frame....VLAN tags comes here
- Network Layer (L3) works on packets.....that's how pfSense does the IP address based routing and filtering
-
Since I have a dedicated hardware appliance (Qotom) for pfSense with 6 ports so adding L3 switch seemed like fragmenting the functionality of one device over two which may still make sense for a very large network (see next point)
-
My home network with multiple TVs, HD cameras was not generating enough traffic to warrant a L3 switch separate from pfSense appliance, the CPU usage on pfSense never exceeds 15% just to give you an idea!
So I ended up creating few VLANs directly on pfSense and life has been good for me
-
-
@pm_13
Yes, sure that's a possibility, just like the transit and dhcp on the switch or a dedicated host. I did it for learning and I tried flat, router on a stick and the transit with dhc outside pfSense.The only advantage I found in router on a stick is pfSense stateful firewall for controlling traffic between VLANs. My switch has no Advanced or Reflexive ACLs. But in my case, they are not a first need on the LAN side
I could later upgrade the pfSense box for better bandwidth and route on pfSense though.
For now, the complex setup with DHCP on pfSense works, ACLs on the LAN and pfSense doesn't seem to consider that wan traffic as spoofed. A bug or limitation? Not sure since no answer was given expect assumptions I cannot reproduce.
-
@elodie80
I could be wrong here but since you mentioned ACL few times let me share my personal experience. Earlier I had pfSense device (with 6 ports - 1 WAN + 5 LAN ports & no VLANs or managed switches anywhere in the network) connected to an unmanaged switch, each LAN port was running its own DHCP with a dedicated subnet and guess what - I had to add at least (40+ MAC addresses in the MAC deny list) for each LANs.
Does this sound familiar to your case? Please respond (yes or no) and I will share more details.
Thanks. -
@pm_13
No, I use ACLs binded to ports and interfaces (VLANs). That was the main reason to segment: secure by interface and not by IP/MAC only, both easy to spoof.But sure, ACLs are not stateful (on my switch), so depending on the needs...
I just setup "Services Servers" on a dedicated VLAN and limit access by port/protocol + have proper permissions on the Services Servers. You just cannot easily limit one direction only initiated traffic from one VLAN to another
-
@elodie80 said in L3 Switch and pfSense design advise:
@johnpoz
Still my question remains: why pfSense is allowing sloppy states and the anti-spoofing rules are not triggered with my previous setup for LAN <-> WAN traffic ? I see no difference in the firewall states at all !Well, finally i found the topic answering my only real question in all this discussion
https://forum.netgate.com/topic/142983/how-does-antispoof-in-pfsense-workSo, it is by design and explains why my setup is working without any issues in near 2 years. The anti spoofing rule is never triggered here on the transit interface because I do explicitly allow internet traffic on this transit interface from specified (or any) subnets
Despite being uncommon and that it would be broken on other firewalls, pfsense design of anti spoofing rule gives this flexibility
Hope it can help other users that for some reason do not need a dedicated DHCP server
-
P phil80 referenced this topic on
