A
Discussing this issue elsewhere I got this response. Basically in my case the HP T620 thin client can't do hardware passthrough on vmware which means you end up with a high level of interrupts when running pfsense hard inside of ESXi. Basically it's just a lot of work of the virtual NICS without passthrough in ESXi which the T620 does not support... I guess that may be the case with other CPU's hence the high CPU usage. Example below is for this device but will be applicable to any CPU which dies not have SR-IOV.
see below
a) While the GX420CA is a decent little chip (used in the Arista 10/40 GbE switches), it's not great for virtualization. The Jaguar/Puma cores are only so-so (around 2400 Passmarks, or around the performance of a Intel Core i5-540M), and that single channel RAM controller means that any memory copying operations simply, well, suck.
b) If you want to use hypervised pfsense, get something that can do SR-IOV / PCIe passthrough, so the hypervisor won't have to handle packets and flip it into the VMs - that's always computationally expensive, and running it as vmxnet3 simply makes it worse (since the e1000e emulation in VMWare is self-throttling, while vmxnet3 implies pushing it as hard as the hardware can handle). The GX420CA in the t620 Plus does not have SR-IOV support, so no PCIe passthrough is possible AFAIK. Technically the RX427BB APU in the t730 thin clients can do it (as can certain Core i5/i7s from, say, Haswell and up), but the problem is like this:
You need something that can support PCIe access control services (ACSCtl flag in lspci -vv), since you'll need that so SR-IOV can work with PCIe passthrough sanity check (you don't really want SR-IOV to start passing stuff to device groups that don't make sense). That's dependent on hardware capability and BIOS/UEFI implementations. Even if your CPU can do, say, VT-d or AMDVi, it doesn't mean that ACS is baked in. If it's not baked in, well, you'll need a Linux kernel patch to get around the lack of ACS. Also, due to no ACS implementation, well, you won't get Alternative Routing-ID Interpretation (ARI), so you can only pass up to 7 SRIOV Virtual Functions. So as you can guess, hackery like this is not supported by VMWare ESXi - this is strictly Linux. Both the t620 Plus and the t730 does not have VMDirectPath or passthrough support in ESXi. Not sure about the DFI DT122-BE (the cheap noisy industrial computer equivalent to the t730), but I am going to say...very unlikely.
Your network card must be able to hand out SRIOV Virtual Functions to the hypervisor so the hypervisor can dish it out to the guest VMs - that's supported in most Mellanox, Intel and Solarflare cards.
Your network card must have driver support in FreeBSD to work with virtual functions as a VM Guest.
That's fairly problematic - as far as I can tell, only Intel 82599/i350s and newer can work with them. Solarflares and Mellanoxes both don't seem to work well with VFs.
So yeah, you are much better off running the t620 Plus on bare metal pfsense, and if you really need the hypervisor capability, use bhyve to run VMs on it instead.
