4 Vlans - Use L3 Switch or PfSense?
-
Hi All, I'm using a VMware box with a PfSense vm (connected with 2 physical nics for that VM) and 4 Vlans going to a Ubiquiti Edgeswitch. This is all for home use and I was wondering if I should let PfSense handle all the interlvan routing rules or if I should use the switch? I have read a few different opinions but nothing really from this community so I'm curious what you think. This entire network will have maybe 60 devices on it at the end of the day.
Thanks!
-
It depends what your priorities are. You basically have a choice to favor performance or control. If you have a need (or want) to firewall your VLAN's, then create your VLANs on PFsense. Otherwise, create your VLANs on your switch… this way all intervlan traffic is handled by the switch instead of traversing your firewall.
-
^ exactly… Pfsense running on vm with all your vlans on 1 physical interface so all intervlan traffic is a hairpin as well, etc.. Prob doesn't provide for wire speed routing ;)
As Marvosa stated it comes down to what you want/need. That your creating the vlans in the first place prob means you want control, ie firewall rules between??
If your going to just use pfsense as edge, then make sure you connect it to your downstream layer 3 switch (router) with a transit network.
-
^ exactly… Pfsense running on vm with all your vlans on 1 physical interface so all intervlan traffic is a hairpin as well, etc.. Prob doesn't provide for wire speed routing ;)
As Marvosa stated it comes down to what you want/need. That your creating the vlans in the first place prob means you want control, ie firewall rules between??
If your going to just use pfsense as edge, then make sure you connect it to your downstream layer 3 switch (router) with a transit network.
That is an interesting point. While I do want to keep speeds high I, I have a need for atleast 1 of the Vlans to not have access to the rest of the network. A question however; if a connection starts in vlan 1 to access a client in vlan 2, and it gets routed through pfsense instead of my switch, once that connection has been established won't the switch just keep the connection alive instead of every packet having to go back to the router?
-
"once that connection has been established won't the switch just keep the connection alive instead of every packet having to go back to the router?"
No that is not how it works.. Think your mixing up a firewall state and switching or something - but no that is not how it works.
For vlan 1 to talk to vlan 3 ever single packet would have to go through pfsense.. Yes once the handshake has gone through ie syn, syn,ack then sure pfsense has a state open for those and packets now flow through the firewall.
If your going to want performance as close to wire speed as possible, vlans all on the same physical interface is not the best method anyway. You would want your vlans that are talking to each other to be on different physical interfaces. Sure they could be tagged or untagged vlans.. But when vlans talk to each other and they are the same physical interface you now have a hairpin and you auto cut your available bandwidth /2
Your sending the packet out the same physical interface it came in, The more vlans that share same physical interface that all talk to each other the lower your overall performance becomes. Its not that big of an issue if say your internet is 100Mb, and you have a few vlans on a gig lan interface all wanting to go to the internet. But when you have bunch of machines on vlan 1 all wanting to talk to vlan 2 and and both of those are on same physical interface then yeah its going to be a HIT, your not going to see gig… Best you could see is 500Mb.. Minus the routing/firewall hit, etc.
-
"once that connection has been established won't the switch just keep the connection alive instead of every packet having to go back to the router?"
No that is not how it works.. Think your mixing up a firewall state and switching or something - but no that is not how it works.
For vlan 1 to talk to vlan 3 ever single packet would have to go through pfsense.. Yes once the handshake has gone through ie syn, syn,ack then sure pfsense has a state open for those and packets now flow through the firewall.
If your going to want performance as close to wire speed as possible, vlans all on the same physical interface is not the best method anyway. You would want your vlans that are talking to each other to be on different physical interfaces. Sure they could be tagged or untagged vlans.. But when vlans talk to each other and they are the same physical interface you now have a hairpin and you auto cut your available bandwidth /2
Your sending the packet out the same physical interface it came in, The more vlans that share same physical interface that all talk to each other the lower your overall performance becomes. Its not that big of an issue if say your internet is 100Mb, and you have a few vlans on a gig lan interface all wanting to go to the internet. But when you have bunch of machines on vlan 1 all wanting to talk to vlan 2 and and both of those are on same physical interface then yeah its going to be a HIT, your not going to see gig… Best you could see is 500Mb.. Minus the routing/firewall hit, etc.
Thank you for the networking class reminder haha, I am totally scratching my head trying to remember this stuff. It's starting to come back to me now! One last thing. Right now when I transfer files between clients right now I get roughly about 105MB/s If I do this is it going to get cut down to 50-60? With the mentality of 1000/8 being my transfer speed, 500/8 doesnt seem to great.
-
How are you transferring files now - just over a switch?
To see 105MB yeah your doing roughly!!! 840mbps, which is gig sure.. yeah if you cut that down to 500 your not going to be able to get 105MB..
My pfsense running on esxi host (hp N40L) with not counting a hit for vlan hairpin can only do about 500mbps.. But then again I don't need intervlan speed.. I have my vlans isolated for control. My internet is only 80mbps anyway, and wifi can not do gig speeds anyway. My devices that need to transfer stuff where I really want speed are all wired on the same layer 2, etc.
As Marvosa correctly states do you want control or performance? Now if you scale up your router/firewall (pfsense) then sure its possible to get really close to wire speed and still maintain your control routing across segments. But find it unlikely your vm going to be able to do it, unless the host its running on is fast, etc. But for sure if you hairpin your not going to be able to do it.
If what you want is performance and still have to do routing at layer 3, then sure a layer 3 switch is going to be better than running it through a firewall. But your going to loose some control. Some switches that do routing do provide for some ability to control with ACL.. But its not going to be as easy as with pfsense, etc. Once you put in control, your going to take a hit on performance - how much that hit is depends on what is doing it. But it is going to be a hit be it a layer3 switch or router just routing or router/firewall, etc.
-
My storage server is being hosted in a VM on Esxi as well running on a physical NIC, it doenst use the vswitch. That goes to my L3 switch (basically running at L2 right now) and its connected to my hosts. When they grab files or upload stuff it gets a steady 100-110MB/s.
What I need/want to do is have a few hosts on my switch now on their own network but still access some of the info on the storage server. I do not want them to see the other hosts on another subnet.
My vm is running on an E3-1241 v3 with 32gb of ram so I think I have enough power (I hope lol). What I think what I will do is enable inter-vlan routing on the switch for some vlans and the ones that dont will have to go back to the router to get their instructions. Does that seem like a viable idea? Also, I want one of my clients (I have no idea how I'm going to do this yet) be able to access every vlan. I don't know if thats viable yet but seems like I will have to do some mac address firewall rules.
-
That is an interesting point. While I do want to keep speeds high I, I have a need for atleast 1 of the Vlans to not have access to the rest of the network.
You do not have to do all one or the other. You can do both.
You can tag a transit network to the switch and route your "trusted" networks to it. You can also tag a layer 2, pfSense VLAN interface to the switch as well.
The switch can route among as many different VLANs as you want locally.
Just don't put a VIF/SVI on the untrusted, layer 2 VLAN and set pfSense as their default gateway there. All traffic from the untrusted to the trusted networks will then have to go through the firewall.
-
So I'm using a combination of everything you all said. This might sound pretty stupid but I just added more virtual NICS to my VMS and I'm going to put the ones that need to be on multiple Vlans on them.
-
That generally gets pretty ugly and you end up with asymmetric routing problems but go for it.
-
^ yeah with Derelict here that is a HORRIFIC work around and yeah if your not very very careful your going to for sure have asymmetrical routing problems.
"do is enable inter-vlan routing on the switch for some vlans and the ones that dont will have to go back to the router to get their instructions. "
Yes as Derelict already went over this can be done.. You route what you want at the switch and stuff you want to firewall you can send to pfsense.
But I am curious if you have devices that you just want to route at the switch, so you not doing any firewalling - why can they not just be on the same layer 2 network anyway?? Why do you want to route between them?? In a work setup the reason you do this would lack of IPs.. Too many devices… You sure do not want to put 2000 machines on the same broadcast domain. But you have 2000 some servers that need to talk to each other. So you put them either on 4 /24s or 2 /23's etc. and route between them at the switch. Since you don't really care to firewall between these devices.
Or maybe different physical location and its just easier to route vs extend the layer 2, etc.
But in a home setup I don't see the reason to complicate the setup, especially for someone that isn't fully up to speed of all the stuff involved in the complication. I am all for network segregation in the home.. I have multiple networks that are all firewalled from each other for security reasons. My guest wifi for example doesn't need access to any of my other networks. My iot devices are isolated, etc. So while my even my wifi is eap-tls to access. It still doesn't have full access into my normal wired network, etc.
Why can you not just put the stuff that you need HIGH speed full wirespeed between on the same layer 2? Multihoming it is going to end up being way way more complicated than it needs to be for no reason at all.
-
^ yeah with Derelict here that is a HORRIFIC work around and yeah if your not very very careful your going to for sure have asymmetrical routing problems.
"do is enable inter-vlan routing on the switch for some vlans and the ones that dont will have to go back to the router to get their instructions. "
Yes as Derelict already went over this can be done.. You route what you want at the switch and stuff you want to firewall you can send to pfsense.
But I am curious if you have devices that you just want to route at the switch, so you not doing any firewalling - why can they not just be on the same layer 2 network anyway?? Why do you want to route between them?? In a work setup the reason you do this would lack of IPs.. Too many devices… You sure do not want to put 2000 machines on the same broadcast domain. But you have 2000 some servers that need to talk to each other. So you put them either on 4 /24s or 2 /23's etc. and route between them at the switch. Since you don't really care to firewall between these devices.
Or maybe different physical location and its just easier to route vs extend the layer 2, etc.
But in a home setup I don't see the reason to complicate the setup, especially for someone that isn't fully up to speed of all the stuff involved in the complication. I am all for network segregation in the home.. I have multiple networks that are all firewalled from each other for security reasons. My guest wifi for example doesn't need access to any of my other networks. My iot devices are isolated, etc. So while my even my wifi is eap-tls to access. It still doesn't have full access into my normal wired network, etc.
Why can you not just put the stuff that you need HIGH speed full wirespeed between on the same layer 2? Multihoming it is going to end up being way way more complicated than it needs to be for no reason at all.
Great question. The original reason I wanted to separate my networks at the end of the day would be for security reasons.
I have 4 networks that I've been messing around with.
-
public
-
private
-
servers
-
vpn
I want public and private networks to be able to talk to the servers and I want public and private to be able to talk to each other when private initiates the conversation, which I believe I setup correctly on PFsenses firewall.
My issue with public and private being on the same network is I don't want someone in public to be able to sniff the traffic of private.
Obviously VPN I want to be on a separate network all together with and isolate all traffic from everyone else. I am doing this right now by adding a network adapter to my PFsense box and that is connected to a L2 switch with the devices I need on the VPN. The only gateway for the interface on PFSense is to the vpn with firewall rules blocking all other traffic. That seems to be working right now.
My issue I'm at right now is the following:
The way my setup is currently is:
|PFSENSE|
–---|----
|Trunk port (All Vlans)
||L3 capable Switch1| ---------- Access port (Vlan10)
|
|Trunk port (All Vlans)
||L2 Switch2|--------- Access port (Vlan 10)
/
/
/
/ Access Port (Vlan10)
/
Access port (Vlan 5)The devices all get their IP addresses via DHCP correctly from PFSense and can ping the default gateway correctly. They can also Ping each other if they are connected to the same layer 3 switch. BUT as soon as a client on switch 2 needs to ping someon on switch 1, it breaks. And Vice versa.
The first switch is a ubiquiti edgeswitch and the 2nd is a Vmware Vswitch.
Any ideas?
-
-
Your transit network should be ONLY for transit traffic. There should be no hosts on it. Make another VLAN for the transit traffic and put VLAN 10 on the Layer 3 switch and tag it across with the rest of them.
If you need to ports on switch 1 to be on VLAN 10, put it behind switch 2. Two routers is going to give you asymmetric routing problems unless they all know all the routes necessary all the time.
Or you have to hairpin the traffic at pfSense, which is ugly. It'd be a shame to see that implemented in all the layer 3 switch goodness due to poor design.
-
when you say transit network, what are you referencing? The connection from pfsense to Switch1?
Just to clarify switch2 is a vmware Vswitch hosted on a virtual machine. I have 5 virtual machines connected to a vswitch with one physical network interface that goes back to switch1. I cannot put other devices on that switch that are not a part of the vm host.
I only have 1 router right now and its the pfsense router. The wan port goes out to the internet and I have 1 lan interface that goes back to switch 1 and switch 1 is connected to switch 2. I have access/general ports on both switches that are used for end devices.
-
Then why are you calling them L3 switches?
Please diagram your network properly.
-
Then why are you calling them L3 switches?
Please diagram your network properly.
The first L3 switch is a layer 3 switch; https://www.ubnt.com/edgemax/edgeswitch/
The 2nd switch isnt a full l3 smart switch but it supports full 802.1Q Vlan tagging. The 2nd switch does not support inter-vlan routing but that is not required if all the clients are on the same L2 subnet/Vlan.
-
A layer 3-capable switch that is not routing should be called a switch, not a layer 3 switch.
To do otherwise just confuses the people who are trying to help you and wastes everyone's time.
-
A layer 3-capable switch that is not routing should be called a switch, not a layer 3 switch.
To do otherwise just confuses the people who are trying to help you and wastes everyone's time.
ohhhhhh. Ok that makes sense. The next question would be… Should I use them as layer 3 switches :) ??
-
vswitch in esxi can not be layer 3 switches.. They can not route. And no you shoudn't be using layer 3 switches (downstream routing) in your network unless you have specific need for routing at wirespeed vs control. And when you do this then you need to connect your downstream routers with a transit network or your going to run into asymmetrical routing issues.
