Per the hardware requirements page (https://www.pfsense.org/hardware/#requirements), even on bare metal, in order to get over 500 Gbit, you need:
Multiple cores at > 2.0GHz are required. Server class hardware with PCI-e network adapters.
So, I would go multiple cores, 4GB RAM and a 64 bit install to start with.
Also, I've seen posts that suggest unless you're using the VMXNET3 driver you will see a ~400 Mbit limit.
It may also be worth mentioning that since you have an HP server and using ESXi, you might want to re-install with the HP specific ESXi image, so all the drivers are VMware certified. It may or may not make a difference, but it's just a thought.