Using pfsense inside esxi cluster (routing/vlan help)

[thecoffeeguy]

Hey folks.

I just built out two new esxi for my home lab. Roughly, about 128gbs memory, 4 terabytes of storage and about (8)nic's combined. This setup is going to be used for a combination of work and home stuff (for work, i built out a lot of VM's, do some development. work with azure/aws/gcp) and for my home, use stuff like plex, grafana, openhab and any other goodies I find.

That said, I would like to setup basically a network or two within this setup. I want to completely isolate my work related VM's from my home network, but I need to be able to access it from a PC and laptop. It should not talk to anything else.

For my home stuff, I can drop it on my home network (with my wireless AP's, roku/firestick/chromecast).

I have a layer 2 24 port switch would should give me plenty of ports to work with.

Now, I have been reading about this but trying to figure out:

1.) Is this even possible with what I have.
2.) How can I do this?

A friend told me to check out pfsense as a option so here I am, reading away as much as I can.

I hope that makes sense and appreciate the help.

Cheers,

TCG

[jclear]

Depending on the hardware it should be very doable.

I'd create vSwitches for WAN, Work LAN, Home LAN, and and possibly Home WiFi (I prefer to keep mine somewhat isolated as it's inherently less secure than wired). pfSense would be attached to all 4 and give you a mechanism for tailoring your isolation as needed.

The hardware part is how you get those VLANs into the physical domain, which depends on the capabilities of your switch. Ideally you'd have a managed switch capable of VLAN tagging, and use that to connect the ESXi NICs. If the switch has no VLAN support, you could get by if you have multiple NICs on your ESXis.

[thecoffeeguy]

@jclear said in Using pfsense inside esxi cluster (routing/vlan help):

Depending on the hardware it should be very doable.

I'd create vSwitches for WAN, Work LAN, Home LAN, and and possibly Home WiFi (I prefer to keep mine somewhat isolated as it's inherently less secure than wired). pfSense would be attached to all 4 and give you a mechanism for tailoring your isolation as needed.

The hardware part is how you get those VLANs into the physical domain, which depends on the capabilities of your switch. Ideally you'd have a managed switch capable of VLAN tagging, and use that to connect the ESXi NICs. If the switch has no VLAN support, you could get by if you have multiple NICs on your ESXis.

Thanks for the help and feedback.
Yes, the switch I have is a 24 port layer 2 managed switch from Ubiquiti. From what I know, it should support VLANs, but i will double check.

I really like your layout for the VLANS. It gave me some things to think about.

With that said, two ESXI hosts managed by Vcenter, I need to brush up on the vmware networking, but need to setup a distributed switch and group I think. SOmething to learn.
Then i would attach the NICs to the pfsense VM to associate the vlans, i think?

Thanks for the help

TCG

[jclear]

If your license supports VDS, then that's the way to go. You create port groups for each of the guest VLANs (and perhaps additional ones for vSphere management, vMotion, etc.). You can do it with the standard virtual switches, it just adds extra steps as you have to create the port groups on each ESXi, instead of once. You can find many examples on the web, and the official VMware docs aren't too horrible. ;-)

When creating the pfSense VM, you'll attach virtual NICs for each VLAN and assign them to the equivalent port group on the virtual switch. That VM acts as both the firewall and router between all the VLANs. While I'm familiar with both vSphere and pfSense, I actually don't use them in the same place, so the following is informed speculation: When you add the vnics to the VM, the order will determine how pfSense sees them. The first will be "LAN", which you'll want to be which ever LAN you're going to manage it from. The second will be WAN, then OPT1, OPT2, etc. You can relabel the latter as say Work LAN and Home WiFi. If you have problems connecting to the Web interface initially. use the console for the pfSense VM to check the MAC addresses on the interfaces to verify the mapping.

[thecoffeeguy]

@jclear said in Using pfsense inside esxi cluster (routing/vlan help):

If your license supports VDS, then that's the way to go. You create port groups for each of the guest VLANs (and perhaps additional ones for vSphere management, vMotion, etc.). You can do it with the standard virtual switches, it just adds extra steps as you have to create the port groups on each ESXi, instead of once. You can find many examples on the web, and the official VMware docs aren't too horrible. ;-)

When creating the pfSense VM, you'll attach virtual NICs for each VLAN and assign them to the equivalent port group on the virtual switch. That VM acts as both the firewall and router between all the VLANs. While I'm familiar with both vSphere and pfSense, I actually don't use them in the same place, so the following is informed speculation: When you add the vnics to the VM, the order will determine how pfSense sees them. The first will be "LAN", which you'll want to be which ever LAN you're going to manage it from. The second will be WAN, then OPT1, OPT2, etc. You can relabel the latter as say Work LAN and Home WiFi. If you have problems connecting to the Web interface initially. use the console for the pfSense VM to check the MAC addresses on the interfaces to verify the mapping.

Perfect.
yes. I bought a VMUG license for testing/development work, so I do have access to the VDS switches.
That makes a lot of sense and I am looking at this visually. I think I just need to test it out, but I think you have given a good place to start, and hope that this will work. I initially thought that I might have to use a layer 3 switch, which can be expensive, but using pfsense as my router, using VLAN's sounds like a great opportunity for me.

For the pfsense VM, at a minimum, off the top of my head i would be attaching 3-4 NIC's, for the VLAN's? Sound about right?

Really appreciate your help. I have been wondering how to do this for a few weeks now.

Cheers
TCG

[jclear]

Unless you want to play around with passing tagged traffic to a VM and pfSense's VLAN handling, I'd at least start with one NIC per subnet/portgroup/VLAN.

At first that sounds like three NICs. But I'd look into dividing up the home network. For instance, do the streamers really need access to your PC, or just the "Interwebs"? Also pfSense would allow you to make the WiFi a DMZ. Your laptop, phones and guests coming in via WiFI can get to the Internet, but you'd have to fire up a VPN to pfSense to reach into the home or work VLANs.

Take it a piece at a time as you figure our pfSense and have fun.

[thecoffeeguy]

So I finally got around to doing some testing with my setup.
To quickly summarize:
(2)ESXi hosts
(9) total NIC's to use

Want to setup different networks to keep them separate (work, dev etc.)
I have done quite a bit of reading on vmware networking (still wrapping my head around it) and setup some distributed switches and port groups, per this thread.
I deployed pfsense as a VM and added the interfaces that I attached to it in Vcenter (LAN, DEVVLAN, DEVHOME).

I configured IP's on the interfaces, I can reach pfsense just fine from my PC, but having problems with the VM's attached to specific networks to get a IP from pfsense DHCP server.
My PFSense has three interfaces:

LAN: 192.168.55.1
DEVLAN: 10.0.3.1/24
DEVHOME: 10.0.2.1/24

I am testing with one VM right now, attached to the DEVVLAN.

Before I go to far down the rabbit hole, any suggestions on where to start troubleshooting?
I tried to manually assign a static IP to my test VM, but I can not ping the gateway off the pfsense box on that interface (10.0.3.1 in this case.)

Thinking through this, I have not done any configuration on my 24 port switch, so to my knowledge, the ports the esxi NICs are plugged into are still native 1.

My brain is hurting right now, so going to take a step back and rest.
I appreciate all the help.

Cheers,

TCG