L3 Switch and pfSense design advise
-
@johnpoz
With all the respect I have, and the fact that I am far from a Network expert, I just cannot see why it is BORKED when it works in all the scenarios I tested
I asked for help on other forums, and the concensus was that as long as I am aware that the clients setup must account for the fact that pfSense is not the gateway, there is no issue.All my traceroutes show that there is no asymmetrical issues and that all routing is done on the SG350 and inter VLAN traffic NEVER reaches pfSense
The Transit route is still there and all internet traffic goes upstream through it. All the VLAN interfaces on pfSense have no internet access. The only VLAN with internet access is the Transit
RDP, speedtests... all work properly even from WAN to LAN
Logs show no asymmetrical packets dropped from WANI am maybe stubborn, but I'd kindly ask you a concrete example that I can reproduce to get a broken routing in my setup. Obviuosly if VLAN interface on pfSense is used as Gateway in this setup it will be broken
Thank you and I hope that you post an example of clients setup I can reproduce to show me it is broken
Note: yes, setting up a dedicated DHCP / DNS server is for now a limitation for me. It will be done in a couple of years and I would be able to use teh proper setup with only static routes in pfSense
Best regards
-
You have bypassed your firewall by putting 2 paths to get to a network, even host on the transit doesn't know about it..
You can do it, but its borked. If you also firewalled traffic at your downstream that you could remove the security issue.
You seem to be going out of your way to create a very complex setup for no other reason than you want to use pfsense as dhcpd for downstream networks which it can not do..
Run your dhcp on a VM is simple solution.. You clearly have boxes there your serveres to be able to run VM.. DNS doesn't matter where it is. ISC dhcp for example can be dhcpd for downstream networks. It just can not do it via pfsense and gui controls.
If you were going to put hosts on a transit, that can work too with simple host routing and not too much issue other than security.. But you have multiple transits with every vlan, so multiple paths for asymmetrical flow.. Even if pfsense sees traffic come from transit, since it has connection in the the vlan, it can send return traffic out that connection. Now the if the client was actually doing things securely - it would balk at such traffic..
You are creating a monster of a setup because you want to run dhcpd on pfsense - when you could just run it on your L3 switch, or pfsense and your L3 switch - or just on some vm running on 1 of your servers.
-
@johnpoz said in L3 Switch and pfSense design advise:
bypassed your firewall by putting 2 paths to get to a network,
Consider it a learning purpose please.
I need openDNS restrictions on some clients, so if the pfSense DNS resolver can still be used to restrict DNS queries on the transit route from subnets not directly attached, that would make it possible for me to use the DHCP server on the Switch.But still, why pfSense is allowing this sort of setup at WAN level instead of dropping the return traffic coming from WAN ?
The only explication I thought is that pfSense is keeping track that the initial request comes from the Transit interface and sends it back throgh that interface and not through the VLAN interface.I need the firewall for WAN <-> LAN currently. Is my setup breaking that part of the firewall ?
-
@johnpoz said in L3 Switch and pfSense design advise:
You have bypassed your firewall by putting 2 paths to get to a network, even host on the transit doesn't know about it..
Is my setup working because of a bug in pfSense ?
Why is pfSense allowing/tracking my VLAN <-> WAN connections ?Because I checked the states and pfSense is clearly redirecting the wan -> LAN traffic through the VLAN interface and not transit. However, pfSense is properly keeping track of sates it seems. I have no option enabled to Bypass firewall rules for traffic on the same interface. I can set all rules in Transit interface to control internet access and access to the VLAN address on pfSense is properly limited to DNS queries. No traffic is dropped ! Why ?
-
Here's the recommended proper setup with the Transit route, DHCP server on the switch, no asymmetric traffic from LAN to WAN:
The only changes:
- no VLAN interfaces on pfSense (use aliases for VLAN subnets to simplify managing rules in the transit interface)
- use a static route in pfSense 10.0.0.0/16 -> 172.26.1.2
- the DHCP server for clients is set on the routing Switch SG350
My thoughts:
- the changes are not really big between the 2 layouts
- setup complexity is the same: DNS rules must be all applied in the Transit interface for each subnet (instead of applying them in each VLAN interface in pfSense)
- I see no difference in my firewall states than previous setup !
pfSense will first route the traffic to the corresponding interface, then it will check the firewall rules and drop/pass traffic accordingly. So, I am not bypassing the firewall here since the rules will apply to the interface. For inter-VLAN traffic, ACLs are applied by the L3 switch
@johnpoz
Still my question remains: why pfSense is allowing sloppy states and the anti-spoofing rules are not triggered with my previous setup for LAN <-> WAN traffic ? I see no difference in the firewall states at all !Thank you again for your patience. I know this subject was addressed many times and is emotional. Hope you can still answer my last question.
Edit: fixed VLAN 1 interface range
-
@elodie80 If your intent is to learn networking then indeed an impressive effort
And to be honest I did not follow all the details in the thread above but I recently created VLANs for my home network and it was quiet a steep learning curve. Interestingly I also bought a Dell L3 switch on eBay (for $40!) and it has quiet the firepower in terms of handling traffic on 64 ports. But after much deliberation I decided not to use L3 switch and here are my learning experiences from this exercise:
-
First the basics that I need to recollect each time I am troubleshooting
- Physical Layer (L1) works on bits ....literally
- Data Link Layer (L2) works on frame....VLAN tags comes here
- Network Layer (L3) works on packets.....that's how pfSense does the IP address based routing and filtering
-
Since I have a dedicated hardware appliance (Qotom) for pfSense with 6 ports so adding L3 switch seemed like fragmenting the functionality of one device over two which may still make sense for a very large network (see next point)
-
My home network with multiple TVs, HD cameras was not generating enough traffic to warrant a L3 switch separate from pfSense appliance, the CPU usage on pfSense never exceeds 15% just to give you an idea!
So I ended up creating few VLANs directly on pfSense and life has been good for me
-
-
@pm_13
Yes, sure that's a possibility, just like the transit and dhcp on the switch or a dedicated host. I did it for learning and I tried flat, router on a stick and the transit with dhc outside pfSense.The only advantage I found in router on a stick is pfSense stateful firewall for controlling traffic between VLANs. My switch has no Advanced or Reflexive ACLs. But in my case, they are not a first need on the LAN side
I could later upgrade the pfSense box for better bandwidth and route on pfSense though.
For now, the complex setup with DHCP on pfSense works, ACLs on the LAN and pfSense doesn't seem to consider that wan traffic as spoofed. A bug or limitation? Not sure since no answer was given expect assumptions I cannot reproduce.
-
@elodie80
I could be wrong here but since you mentioned ACL few times let me share my personal experience. Earlier I had pfSense device (with 6 ports - 1 WAN + 5 LAN ports & no VLANs or managed switches anywhere in the network) connected to an unmanaged switch, each LAN port was running its own DHCP with a dedicated subnet and guess what - I had to add at least (40+ MAC addresses in the MAC deny list) for each LANs.
Does this sound familiar to your case? Please respond (yes or no) and I will share more details.
Thanks. -
@pm_13
No, I use ACLs binded to ports and interfaces (VLANs). That was the main reason to segment: secure by interface and not by IP/MAC only, both easy to spoof.But sure, ACLs are not stateful (on my switch), so depending on the needs...
I just setup "Services Servers" on a dedicated VLAN and limit access by port/protocol + have proper permissions on the Services Servers. You just cannot easily limit one direction only initiated traffic from one VLAN to another
-
@elodie80 said in L3 Switch and pfSense design advise:
@johnpoz
Still my question remains: why pfSense is allowing sloppy states and the anti-spoofing rules are not triggered with my previous setup for LAN <-> WAN traffic ? I see no difference in the firewall states at all !Well, finally i found the topic answering my only real question in all this discussion
https://forum.netgate.com/topic/142983/how-does-antispoof-in-pfsense-workSo, it is by design and explains why my setup is working without any issues in near 2 years. The anti spoofing rule is never triggered here on the transit interface because I do explicitly allow internet traffic on this transit interface from specified (or any) subnets
Despite being uncommon and that it would be broken on other firewalls, pfsense design of anti spoofing rule gives this flexibility
Hope it can help other users that for some reason do not need a dedicated DHCP server
-
P phil80 referenced this topic on
