Segregated VMs on separate box Help -> Double pFsense? Multiple quad port NICs? Virtualized solution? Vlans + Switch?

srytryagn

Looking to connect a Proxmox Box to my network. This Proxmox box will run several VMs that cannot talk to each other or any other LAN on the entire network but can reach out to the internet.

I am struggling with how to achieve this and have some questions.

Physical method -> connect an extra quad port NIC on pFsense box and a quad port nic on proxmox box. Make 4 lans on pFsense box. On the Proxmox box pass through each port from the installed quad port NIC to each of the four VMs . 4 phsyical wires with 4 Lans and each port passed through to each vm essentially.
Is there a way to achieve this with perhaps a second virtualized pFsense running on the proxmox box that keeps them segregated while providing internet access?
Vlans & Switch some how?

Very confused as to what is the most robust and secure way to achieve the desired result. The physical method could work, but will be limited to the amount of quad port NICs I can place in the proxmox box so not scalable really. Hoping there is a better double pFsense ( one physical and one virtualized) solution that could work for my purposes.

Really could use your help to get this up and running. Thanks in advanced !

viragomann

@srytryagn
I would take the 4th option:

Define 4 VLANs on pfSense.
On Proxmox create a bridge if you didn't already and check "VLAN aware" in its settings.
In the network interface settings of the VMs select the bridge and enter the proper VLAN tag.
Done.

On your pfSense you can then control the traffic of each unique VM.

srytryagn

@viragomann I suppose you are right and I feel a little foolish for not thinking of "option 4".

So it is basically no different from what you would do phsyically with a switch, trunk, and a bunch of clients. So just treat Proxmox switch as you would a physical switch connected to a pfsense appliance.

Couple of questions if you dont mind, please chiming in on:

Is options 4 any less secure than the other options ?
Seen many tutorials where people setup a virtual lab/network in proxmox, where pfsense is a VM that all other vms connect to. I have seen this with:
A) with a single natted ethernet (out of an ISP all in one router) into the proxmox box pfsense VM so double natted I guess
B) With a NIC passing through two ports Lan and Wan un-natted no different than a physical so easy to understand

NOW -> WIth A) and my lack of networking know how I am utterly confused as to what the pfsense firewall is actually protecting since it is not before the open web. If you can make sense of this for me I would appreciate it.

Under what circumstances is more than one pfsense needed ?
How do you make sure Vlans cant be hopped by the hypervisor, once the aggregated trunk is sent to the virtual switch, all trust of segregation between vlans remains within the capability of that switch and not pfsense. How to make sure there is no misconfig or funny business with the hypervisor ?
Is there a way to achieve my desired result so that the hypervisor does not know anything about the traffic on the vlan or its vms ? Would this require some pass through or a second virtual pfsense ?

viragomann

@srytryagn said in Segregated VMs on separate box Help -> Double pFsense? Multiple quad port NICs? Virtualized solution? Vlans + Switch?:

Is options 4 any less secure than the other options ?

There are often concerns about VLANs. But the whole VLAN stuff is managed on pfSense and Proxmox in this case. And I trust both systems more than a cheap switch.
The less trustworthy VMs can do nothing on their own to get access to packets, which are destined to other networks.
So I think, it's secure.

Seen many tutorials where people setup a virtual lab/network in proxmox, where pfsense is a VM that all other vms connect to. I have seen this with:
A) with a single natted ethernet (out of an ISP all in one router) into the proxmox box pfsense VM so double natted I guess
B) With a NIC passing through two ports Lan and Wan un-natted no different than a physical so easy to understand

NOW -> WIth A) and my lack of networking know how I am utterly confused as to what the pfsense firewall is actually protecting since it is not before the open web. If you can make sense of this for me I would appreciate it.

Yeah, passing through the NICs gives you a better isolation from the hypervisor machine, but the bigger advantage is that the VM directly runs the hardware drivers. Hence you save one software layer.

Under what circumstances is more than one pfsense needed ?

You mean a virtualized, I guess.
It would be meaningful if the VMs on different VLANs needs to communicate together with a high throughput.
In this case the hardware NICs and the cable could be a bottleneck.
But as you stated, the VMs need only internet access, the bottleneck would rather be the upstream connection to the ISP.

How do you make sure Vlans cant be hopped by the hypervisor, once the aggregated trunk is sent to the virtual switch, all trust of segregation between vlans remains within the capability of that switch and not pfsense. How to make sure there is no misconfig or funny business with the hypervisor ?

VLANs are well-established today. For sure it's not impossible that there might be an issue on the hypervisor, but that could be as well in pfSense (FreeBSD).
pfSense and Proxmox are both involved in the network segregation.

In my opinion there is a significantly higher risk of a possible miss-configuration than a failure in the networking software, however this is independently from using VLAN or not in my eyes.

Is there a way to achieve my desired result so that the hypervisor does not know anything about the traffic on the vlan or its vms ?

Even with VLANs the hypervisor doesn't really see the packets. He does only the networking, but he has no IP in one of these VLANs and hence cannot communicate with them. He does merely the job of a switch.

Would this require some pass through or a second virtual pfsense ?

So yes, with hardware pass-though the hypervisor not even do the networking. And so it might be a little bit more secure.

srytryagn

@viragomann Thank you very much for the responses!