Should I isolate my LAN from pfSense VM by giving it only virtual NIC ? (excluding WAN)
so I'm currently using pfSense 2.4.3-RELEASE-p1 as a Firewall/Gateway/Router in a VM, basically just emulating what I used to do with a dedicated hardware device - WAN and LAN connections, nothing else.
I also run two Windows DCs with pfSense. It's a pretty simple setup. E.g.
10.0.0.1 pfSense (gateway)
All of the above 3 VMs have vNICs pointing towards the LAN through VMXNET3 (I'm using ESXi 6.7) which is paravirtualized over a Chelsio T520-CR - kind of like they were just 'normal' physical machines.
I'm just realizing that there's a bunch of things I could do with virtualization that I hadn't thought of until now, and I'm wondering if it might improve outcomes for things like isolation and speed:
E.g. Should I only interface with pfSense through a virtual-only network and use another VM as a gateway to that virtual-only network? Could that make communication between these 3 VMs speedier and improve security from clients, maybe even make configuration easier? For instance:
WAN <--> pfsense <- internal vm-only-network -> DC01 <--> DC02 <--> LAN gateway VM <- physical LAN -> LAN clients
10.0.0.4 Gateway VM (internal VM network) +
172.16.0.1 Gateway VM (physical LAN interface IP)
Does anybody else do it that way, and why?
Also, would it be helpful to use another VM for routing to take some of the load off of pfSense? I am only getting about 2.5Gbps in iperf with 10Gbe hardware (probably for another thread...)