So I tried both Debian and ESXI6.7.
On both it is possible to do a PCI Passthrough on of the network cards.
Both run pretty much the same, I don't think there's much of a difference in the resources they use.
Using Debian I can still use my PC as a media station / media server on the LAN. I could also install a few other things and have some LAN or WAN servers on it but I've never got to it.
Running ESXI I can have a web server, a file server and my own mail server (a long time obsession of mine) but I cannot use the unit for displaying any media to a TV / Monitor.
The issue with Debian is that it is set for automatic security updates (normal in my mind) and a couple of times already it rebooted on it's own and the VM does not come back up due to various issues, last incident it was something related to the display adapter for the VM.
ESXI on the other hand comes back up reliably, autostarts the VMs just fine but I loose the GPU basically.
Bottom line, there's no happy medium.
In another order of ideas, what is the difference security wise in running a bare metal hypervisor like ESXI compared to libvirt in Debian; all under the assumption that the network cards are being passed through and not bridged. It is my understanding that in a PCI passthrough situation the hardware is passed through with not underlying hypervisor interaction so in order to compromise anything one would have to exploit a firmware vulnerability in the NIC or a software vulnerability in the PfSense / OPNSense itself before getting to the host in any way.