VLAN or Multiport
I have a multi client building with numerous companies sharing a fiber connection. Currently the ISP delivers this fiber connection and the corresponding equipment terminating in a cisco router with an integrated 8 port managed switch configured as separate VLANS with public address subnets. Each client is attached to one of the ports and sits behind their own pfsense box which works well. The Cisco router is old and currently experiencing some hardware issues and needs to be replaced. The ISP would like to sell me a new cisco router, but I'd like to use PFSense to create a drop in replacement. I can't decide whether it makes more sense to do the switching inside of the pfsense box or outside. Is it best to outfit the pfsense box with 8 additional RG45 ports and configure each to correspond with the current network, or do I use VLANs and a managed switch attached to the lan port? Certainly the later would emulate the current setup but having 8 interfaces might provide some better control and perhaps some performance improvements. (Although I would think that the capabilities of the LAN port would be the same in both instances) I've done a bunch of searching and I can't find anything that argues for one or the other approach. Obviously, there is a hardware limit to the number of interfaces that can be installed in one box, whereas I could almost use an unlimited number of VLans, but I would think that the processing power required for both approaches would be almost equal. Any thoughts?
VLans are probably going to be cheaper depending on the type of switch you get (can you re-use the cisco switch?).
Separate interfaces can be more useful for trouble shooting, you can sniff for all traffic, and are potentially faster if you have a big WAN pipe. There is always a security question over separating subnets with VLans though I've yet to see an exploit or a failure. You can potentially get the config wrong and have traffic leak between them.I would go separate NICs if I had a choice. Be aware that when you start adding a lot of NICs to a box you might encounter some issues that don't arise otherwise. Like this: https://forum.pfsense.org/index.php/topic,69486.msg379897.html#msg379897
Just to give you the other side of the coin, I would go the pfSense and separate managed switch route. I have always preferred the router on a stick model better. I would have a couple of interfaces on my pfSense box but then I would create a lagg port going to my switch for performance and redundancy. I have not done testing but either way should be good, router on a stick in just my preference.
As a side note, you mentioned that your ISP provides you with a fiber connection. Is the hand off a fiber connection or do they provide some type of equipment to do the conversion? If it is fiber what NIC are you planning to get that is compatible with PfSense? Or do you plans to use some type of media converter?
I would go separate NICs if I had a choice. Be aware that when you start adding a lot of NICs to a box you might encounter some issues that don't arise otherwise. Like this: https://forum.pfsense.org/index.php/topic,69486.msg379897.html#msg379897
Heh, that box completely imploded when I added some 10Gbe ports on Friday. Even with queuing disabled in the igb and ix drivers I had to limit the box to 2 cores to get it to boot. My 2.1.1 box (backup in CARP pair) works without any tweaks.