VM Cluster Single 'Virtual Network'
Apologies if this or parts of this are covered already on the forum. This is as much a general network architecture question as a specific pfSense L2/VLAN question.
I am currently trying to deploy a multi-node Proxmox (Qemu/KVM) based cloud. This will initially consist of 4 physical servers but will need to scale up to n (<100). Each server has two physical NICs- one internet facing NIC and one "private" NIC that is routed by the hosting provider as if each nic was plugged into the same L2 switch.
Using pfSense, I would like to create a single layer 3 network deployed via pfSense VM instances running on each physical node. The reason I need one single subnet is that the cluster computing platform I want to use (Apache Ignite) to run my application logic relies on each node being on the same multicast network for node discovery.
I would also like to route VM traffic out to the internet on its physical host's NIC as these only have 1gbps per NIC and therefore I cannot afford to route all traffic out of a single pfSense instance running on one host. That means at least 4 pfSenses somehow VPNing a single subnet on 1 L2 network with DHCP working, together with the default gateway route for each VM varying depending on which host it sits on.
Encryption on the 'private' network is essential as I will be handling financial information, not to mention user information and the private nature of the private NIC network is a fiction created by my hosting provider's routing.
I have looked (briefly) into layer 2 tunneling using OpenVPN as a way to create an ethernet bridge between the pfSense instances running on each physical node but am concerned that, with OpenVPN being single threaded, that I will not be able to saturate the 5gbps private port on each physical host (each host has dual xeon gold 24core and 192gb of ram).
I have also briefly read a little about ipsec tunneling but cannot get my head around it.
Finally, even thinking about how to get DHCP working reliably with the above setup has me scratching my head.
If anyone has any tips, it would be much appreciated.