Do you have performance tips for Proxmox virtualized pfSense?
-
After four or so years of running pfSense on a dedicated Protectli Vault (what a nice appliance!), I've recently moved to running pfSense CE 2.7 on a new, bigger Vault. The motivation was that the older core i5 Vault didn't have 2.5Gbps interfaces and the new one does. We're about to get something faster ISP service greater 1Gbps and I wanted to make sure we could take advantage of it.
I've enabled PCI passthrough on the new Vault for the pfSense WAN interface and that's working well.
And performance "feels" pretty good on the Proxmox pfSense VM and the devices on the LAN and OPTx networks. The usual throughput measurements are just about what they were on the physical appliance. But I can't help but wonder if there's still more performance I could eke out of the virtualized VM.
For those of you who always run virtualized, I wonder if you have any comments on the following questions and/or pointers to things I might have missed.
- I started with a 6GB VM and Proxmox showed 80% utilization. I upped the VM to 8GB -- and it still shows 80% utilization. How does one determine, for a given network, the optimal amount of memory for virtualized pfSense?
- Should I bother with physical switches so that the LAN and OPTx interfaces can run on physical PCI interfaces instead of Proxmox virtual bridges? Or is that really wasting time, money and energy when the bridge adapters are set to paravirtualized interfaces?
- Are there any well-known Proxmox settings that might improve packet throughput? FreeBSD settings that might be important?
Thanks.
-
S stephenw10 moved this topic from General pfSense Questions on
-
@yobyot said in Do you have performance tips for Proxmox virtualized pfSense?:
Should I bother with physical switches so that the LAN and OPTx interfaces can run on physical PCI interfaces instead of Proxmox virtual bridges?
It's all about your setup as to whether your requirement needs a switch or not. Experience shows having a smart switch opens more network control opportunities to a network administrator, especially expanding. I would recommend a smart switch that do layer3 and layer4. You can also use a virtual switch like vSwitch.
To help your speed, consider double stack.
-
@yobyot said in Do you have performance tips for Proxmox virtualized pfSense?:
2.5Gbps interfaces and the new one does.
@yobyot said in Do you have performance tips for Proxmox virtualized pfSense?:
VM and Proxmox showed 80% utilization.
@yobyot said in Do you have performance tips for Proxmox virtualized pfSense?:
Should I bother with physical switches so that the LAN and OPTx interfaces can run on physical PCI interfaces instead of Proxmox virtual bridges?
You are the best on to assess the performance impact, by trying it.
How may NICs does your new hardware have? Having separate physical NIC for separate high bandwidth logical interfaces may enable each to run on different sets of physical cores, increasing the efficiency of a multi-core CPU.
How may cores does your physical CPU have & how may are you allocating to pfsense? -
@Patch said in Do you have performance tips for Proxmox virtualized pfSense?:
You are the best on to assess the performance impact, by trying it.
How may NICs does your new hardware have? Having separate physical NIC for separate high bandwidth logical interfaces may enable each to run on different sets of physical cores, increasing the efficiency of a multi-core CPU.How may cores does your physical CPU have & how may are you allocating to pfsense?
A very interesting suggestion! Thanks.
My new-ish Protectli Vault is a VP4670 with a core i7 108010U -- it's fast enough to run pfSense plus a few Ubuntu VMs and an app-must-run-must-run-x64 Windows VM -- as long as you keep it cool enough to avoid thermal throttling. It's a 12-thread, six-core processor. pfSense is currently allocated 2 sockets and 4 cores. I have also converted the VM to the new Proxmox 8 x86-64-v2-AES type. pfSense says hardware crypo is enabled. See below.
It also has six Intel I225-V 2.5Gbs ports.
The end-design has one port connected to the ISP, another connected to a funky Netgear switch I am considering.
The Netgear switch would will support two Ubiquiti POE+ APs, upstream connection to the ISP, a NAS running at 1Gbps and, possibly, two wired connections.
The design (this is for a new condo) is focused on Wifi 6E to gain access to the 6Ghz band due to what I expect to be airwaves crowded with Costco-level Wifi routers from my neighbors. In my experience, apartment buildings are loaded up with Wifi APs that are used by non-technical people who want to economize, generating interference massive interference. It's the downside of unlicensed spectrum.
Based on my experience running pfSense CE 2.7 today in Proxmox with a 1Gbps ISP service to one a PCI Passthrough port on the Vault, pfSense doesn't even "breath hard" ( as measured by things like the s-tui stress test on Proxmox) when I load up traffic on that link.
But, once we move beyond the ISP limitation to 2.5Gbs, I don't want to lose that capability by using 1Gbps hardware -- and thus the question on how to make sure pfSense is as optimized as it could possibly be in a Proxmox VM.
So, the question is, does it make "pfSense" (ha!) to dedicate the LAN port using PCI passthrough to pfSense (
icgx) instead of creating a bridge and simply assigning the port to the bridge (vnetx).It seems like a no-brainer but the downside is that PCI passthrough devices are ineligible for Proxmox cluster migration. So, I'm trying to decide if that limitation is worth the price.
-
@yobyot said in Do you have performance tips for Proxmox virtualized pfSense?:
So, the question is, does it make "pfSense" (ha!) to dedicate the LAN port using PCI passthrough to pfSense (icgx) instead of creating a bridge and simply assigning the port to the bridge (vnetx).
May I suggest this guide: https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html
If you follow the above, you'll see that both WAN and LAN are required to pass-through. It's best to have a separate interface for Proxmox. So, in essence, the requirements is three Ethernet ports, two of which pass-through to the pfSense VM.
-
@NollipfSense said in Do you have performance tips for Proxmox virtualized pfSense?:
May I suggest this guide: https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html
If you follow the above, you'll see that both WAN and LAN are required to pass-through. It's best to have a separate interface for Proxmox. So, in essence, the requirements is three Ethernet ports, two of which pass-through to the pfSense VM.
Thanks.
But it’s not the _number _ of interfaces connected to pfSense. (Yes, it’s a minimum of three.)
The question I am asking is about the hardware configuration in Proxmox of those three (minimum) interfaces.
-
@yobyot said in Do you have performance tips for Proxmox virtualized pfSense?:
The question I am asking is about the hardware configuration in Proxmox of those three (minimum) interfaces.
When you were installing Proxmox, didn't you select an interface for Proxmox...so, I can fully understand the question. Didn't you configure this via the installer? There is not much to configure, just assign an IP and gateway. How I do mine is make pfSense LAN, Proxmox's gateway. Then, connect a smart switch to pfSense LAN, then connect an Ethernet cable from Proxmox's Ethernet port to the switch...it should be obvious that Proxmox Ethernet is on the same exact network as pfSense LAN.
-
@NollipfSense said in Do you have performance tips for Proxmox virtualized pfSense?:
When you were installing Proxmox, didn't you select an interface for Proxmox...so, I can fully understand the question. Didn't you configure this via the installer? There is not much to configure, just assign an IP and gateway. How I do mine is make pfSense LAN, Proxmox's gateway. Then, connect a smart switch to pfSense LAN, then connect an Ethernet cable from Proxmox's Ethernet port to the switch...it should be obvious that Proxmox Ethernet is on the same exact network as pfSense LAN.
Maybe a picture will help.
In the composite screenshot below, you can see that in pfSense a PCI passthrough Ethernet adapter on the physical host is passed through as
igc0. This connects to the ISP and could run at 2.5Gbps, if I had that service now, because the physical adapter on the appliance is 2.5Gbps capable. pfSense accesses this adapter directly, not through Proxmox. It is shown in the screenshot ashostpci0.The LAN interface for CIDR 192.168.2.0/24 ('vtnet0') is on a Proxmox bridge. This is configured in Proxmox as a Virtual I/O (paravirtualized) device (
enp2s0) to pfSense, which sees it asvtnet0.vtnet[x]is pfSense's device assignment to virtualized network hardware. IOW, it is not directly connected to pfSense; Proxmox is handling interrupts and data transfer into pfSense's memory.Proxmox is accessible on this bridge as I added an address for it (10.69.0.2). The
vmbr2bridge onenp3s0is solely for the case in which pfSense is down and therefore cannot be accessed. By adding a bridge onenp3s0and assigning a management address of 10.68.0.2 to Proxmox, I can simply connect a laptop to that port on the appliance, manually configure an address in 10.68.0.0/16 and connect to Proxmox when pfSense is crashed or won't start. I prefer not to have the Proxmox management address on the same interface as the LAN interface; losing a physical port to access it isn't an issue for me.So, the question -- or at least the networking portion of my question -- boils down to, "how much performance penalty might there for running pfSense LAN/OPTx interfaces paravirtualized in Proxmox if the connection to the ISP runs faster than 1Gbps?
I could dedicate additional PCI appliance Ethernet ports to pfSense LAN/OPTx interfaces, but there are two trade-offs (maybe more). First, you're locked to that device, making Proxmox node migrations harder. Second, if the FreeBSD kernel is handling I/O directly for a device on the PCI bus, what's the impact in pfSense's CPU and memory requirements vs. letting Proxmox handle that?
Finally, does anyone know if a paravirtualized FreeBSD interface (
vtnet[x]) can even run at 2.5Gbps?
-
@yobyot said in Do you have performance tips for Proxmox virtualized pfSense?:
VM to the new Proxmox 8 x86-64-v2-AES type.
I use processor type "Host" as I don't anticipate needing to live migrate my pfsense VM to another Proxmox host,
@yobyot said in Do you have performance tips for Proxmox virtualized pfSense?:
It also has six Intel I225-V 2.5Gbs ports.
Nice, I recently bought a similar unit.
Given the generous physical NIC we have, I pass through all NIC pfsense VM uses. Proxmox (and other VM's) connect to the pfsense VM via an external switch.@NollipfSense said in Do you have performance tips for Proxmox virtualized pfSense?:
If you follow the above, you'll see that both WAN and LAN are required to pass-through. It's best to have a separate interface for Proxmox.
Agree
@yobyot said in Do you have performance tips for Proxmox virtualized pfSense?:
The Netgear switch would will support two Ubiquiti POE+ APs, upstream connection to the ISP, a NAS running at 1Gbps and, possibly, two wired connections.
That's a bit weird. It is not clear why you are connecting the ISP to the Netgear switch.
Why are you not using a WAN connection:
ISP -> pfsense VM (optionally passed through) NICLan connection
pfsense VM (optionally passed through) NIC -> Netgear switch
(if pfsense Lan NIC is passed through, Proxmox & other VM will use a different physical NIC to connect to the Netgear switch / LAN. That also means if your Proxmox install has a problem you can easily connect it to another lan to fix it) -
@Patch said in Do you have performance tips for Proxmox virtualized pfSense?:
I use processor type "Host" as I don't anticipate needing to live migrate my pfsense VM to another Proxmox host,
Interesting. How do you get pfSense to then use hardware AES support, which is crucial for TLS performance?
That's a bit weird. It is not clear why you are connecting the ISP to the Netgear switch.
Why are you not using a WAN connection:
ISP -> pfsense VM (optionally passed through) NICI misstated it. I do plan to connect the ISP directly to the appliance. Not sure what I was thinking. :-)
Thanks.
-
@yobyot said in Do you have performance tips for Proxmox virtualized pfSense?:
How do you get pfSense to then use hardware AES support
My understanding is VM processor type "Host" means the VM is told it has the same processor as the Proxmox hypervisor is running on. So if the physical processor supports AES then the VM will be told that's the case.
-
@Patch said in Do you have performance tips for Proxmox virtualized pfSense?:
@yobyot said in Do you have performance tips for Proxmox virtualized pfSense?:
How do you get pfSense to then use hardware AES support
My understanding is VM processor type "Host" means the VM is told it has the same processor as the Proxmox hypervisor is running on. So if the physical processor supports AES then the VM will be told that's the case.
Hmmm....until I changed to the AES-specific host, none of the crypto showed as active in the pfSense summary. Weird.
-
@yobyot
Proxmox hardware settings for pfsense VM
pfsense GUI System information
-
@Patch said in Do you have performance tips for Proxmox virtualized pfSense?:
@yobyot
Proxmox hardware settings for pfsense VM
pfsense GUI System information
Hmmm…I wonder what the difference is between your appliance and mine is when it comes to the “Host” type.
-
@yobyot said in Do you have performance tips for Proxmox virtualized pfSense?:
Hmmm…I wonder what the difference is between your appliance and mine is when it comes to the “Host” type.
Looking at this screenshot below, WAN should be vtnet0 and LAN should be vtnet1...
Also, I have only followed the pfSense recipe quoted in earlier post and that required to use BIOS > OVMF for UEFI boot with machine Q35...
-
Yes. Use ESXi.
-
@NollipfSense said in Do you have performance tips for Proxmox virtualized pfSense?:
Looking at this screenshot below, WAN should be vtnet0 and LAN should be vtnet1...
Actually, no.
pfSense was running on an external Vault. When I migrated it to Proxmox, I put it on vmbr1. I haven't found a way to renumber the bridges so that it "looks" right and now I kinda like it.
-
@yobyot said in Do you have performance tips for Proxmox virtualized pfSense?:
Actually, no.
pfSense was running on an external Vault. When I migrated it to Proxmox, I put it on vmbr1. I haven't found a way to renumber the bridges so that it "looks" right and now I kinda like it.
Well, if it works for you, hooray...I just shared what the pfSense document says...like I installed using UEFI for pfSense on Proxmox, as well as, install Proxmox on ZFS.
