Switching from virtual to physical home setup
I had a pfsense setup running for 5 years inside of ESXI at my house. It worked well for quite sometime but whenever the power went out (longer than UPS could handle) and I was on travel trying to explain logging into ESXI and starting VMs became way to much for my wife. To mitigate this I took the plunge and bought a Supermicro 5019D-FN8TP to try a physical pfsense configuration that's hopefully one power button to get internet back.
I have FIOS internet (1gbps WAN) and a Cisco 3750X with a 10gbe module installed that gives me 2 10gbe ports. Currently I have one 10gbe port connected to my ESXI and one to my NAS. This worked well because ESXI trunked everything to a vNIC and I could push ~6gbps through the VM between virtual and physical through the pfsense appliance. As I introduce the new server I'm stuck in a difficult spot because I'll now need 3x10gbe NICs (NAS, ESXI, and pfsense). So now I'm at a cross-roads with network design. The NAS is on my main VLAN with a large portion of the heavy consumers. The only users not on the main VLAN are my IP cameras which record to the NAS (100-200mbps). They need to stay on a separate VLAN with heavy fw rules around access.
The Supermicro server will add more 10gbe ports into the setup. So I'm wondering what you guys think would be an optimal setup. Should I move the ESXI up to the Supermicro 10gbe interface and/or the NAS as well then trunk everything back down to the Cisco for other physical consumers or some other combination?
My first thought would be to keep the NAS on the Cisco as an access port, use the other Cisco 10gbe to trunk up to pfsense, and tag all the ESXI traffic into one of the Supermicro ports. My other thoughts involved L3 between pfsense and Cisco, basically only using pfsense for WAN, but that doesn't seem as fun :)
What things do you need to filter between?
It's always better to keep things at the switch if you don't need to filter between them. If you are using the NAS as NFS storage for ESXi for example you almost certainly don't want any filtering there.
@stephenw10 As far as the main network I don't need to filter much most of that stuff is on the same vlan but auxiliary vlans like my IOT, guest wifi, a few vpns, and IP cameras I want to limit them to only a very specific set of IPs and ports. I definitely would want those to route through pfsense. For the main traffic I could use the Cisco for routing with out a huge amount of impacts. Are you suggesting I split routing duties between the Cisco and pfsense? Maybe only use a 1gbe link up from Cisco to pfsense?
The NAS doesn't serve ESXI NFS datastores but many of the VMs use NFS and CIFS mounts for storage.
It sounds like you want the ESXi and NAS connected to the switch 10G ports directly for best connectivity between them and the connection to pfSense can be 1G or probably a LAGG of several 1G ports. Disappointing not to have a 10G link to pfSense with that hardware but you could be crippling the VMs to do it otherwise.
That's what I was thinking. I wasn't sure if there was any kind of exotic configs that might work just as well. I now need a POE+ switch so I might be upgrading the 3750 at some point. I think Cisco changed their licensing model on the 3850s and the cat9k making it harder to deal with as a home user. I like Cisco but it is expensive for home use. I really do like L3 multicast though, so that helps justify it.