Help with High CPU Usage in OPNSense/Pfsense VM inside Proxmox

[saluteferux]

Hi everyone,
I'm experiencing an issue with high CPU usage in my OPNSense VM running inside Proxmox and could really use some help.
Setup Details:

Host Configuration:
    Proxmox Host
    64 GB RAM
    Intel i7-8700 (mitigations=off)
    IOMMU enabled (iommu=on and iommu=pt in /etc/kernel/cmdline)
    Proxmox installed on SSD NVME in ZFS RAID0
VM Configuration:
    OPNSense VM
    8GB RAM
    1 socket, 4 CPU threads (using host cpu type, testing with others doesn't change anything)
    Passing through 4 ports of Intel I226V 2.5GbE. PCI-E is on in the settings of pci devices.
    Q35, OVMF (UEFI)

Issue:
When I run a speed test, downloading at speeds of 2.1 Gbps to 2.5 Gbps, Proxmox reports 80-100% CPU usage. However, inside the OPNSense dashboard, the CPU usage is reported as 5-20%. (Same tested with pfsense)
I’m trying to understand why the CPU usage is so high on Proxmox.
Things I've Tried:

Ensured that IOMMU is enabled and configured properly.
Verified that CPU mitigations are turned off for performance.
Tested with Pfsense
Tested OpenWRT and the CPU Usage doesn't go above 10-15% at full 2.5Gb speed so this is a problem with the senses...

Questions:

Why is there such a discrepancy between CPU usage reported by Proxmox and OPNSense?
Is there anything I can do to optimize or reduce the CPU usage on Proxmox when running high-speed network tasks?

Any insights or suggestions would be greatly appreciated!
Thanks in advance!

[patient0]

@saluteferux I'm a bit confused as to why you would post this question here and not in the forum of the firewall software you use:

https://forum.opnsense.org

[saluteferux]

@patient0 Because the issue is occurring also in Pfsense when I tested it.

[Patch]

@saluteferux

• what physical NIC do you have and how many

• are you passing through the NICs or using Proxmox to create virtual switches?

My guess is that virtual switches in Proxmox are using your CPU cycles

[saluteferux]

@Patch So I have one onboard LAN I219-LM used for proxmox MAN/DEBUG port.

And then one Intel I226-V 4 ports PCI-E NIC, i'm passing through the all four ports of the I226-V to the OPNSense VM.

[saluteferux]

@Patch

I forgot to mention that the I226-V has correctly different IOMMU groups for each of the four ports.

And i'm also using proxmox bridges just for VM networks, and to give the proxmox host it's own dedicated network.

But the bridges do not have any bridge ports as there are just virtual switches for proxmox host, and his VM's.

[johnpoz]

@saluteferux said in Help with High CPU Usage in OPNSense/Pfsense VM inside Proxmox:

Because the issue is occurring also

And have you asked in proxmox forums? I personally would of worded your question better for where you were posting it - if on that other distro use their name, on this forum pfsense..

With a "footnote" only that also seeing this on the other distro, etc.

What version of proxmox, what versions of pfsense and that other distro.. Current version of pfsense is using freebsd 15 as base, isn't that other distro on 14?

While I am sure there are people here running proxmox, did I miss where you said what version you were running? This sure seems like something more suited for a forum dedicated to proxmox.

[Patch]

@saluteferux said in Help with High CPU Usage in OPNSense/Pfsense VM inside Proxmox:

Host Configuration:
Proxmox Host
64 GB RAM
Intel i7-8700

VM Configuration:
8GB RAM
1 socket, 4 CPU threads

speed test, downloading at speeds of 2.1 Gbps to 2.5 Gbps, Proxmox reports 80-100% CPU usage. However, inside the OPNSense dashboard, the CPU usage is reported as 5-20%. (Same tested with pfsense)

So your issue is why is 2.1 Gbps to 2.5 Gbps the maximum capacity for an Intel i7-8700 running pfsense (or OPNSense ) under Proxmox.

Depending on what you are doing in pfsense and the test type, this maybe near the capacity of that processor.

You say you are passing through the NICs used by pfsense, but I find your description of that contradictory.

• A NIC in Poxmox connected to a VM by a bridge is a 2 port virtual switch. This in NOT NIC pass through

• A NIC in Proxmox which is passed through to a VM is no longer accessible by the Proxmox host, it can not be connected to a Proxmox Bridge and so is not possible to be used by any other VM or the host.

• You say you have passed through "all four ports of the I226-V ". If that's actually the case none of the I226-V ports should be visible in Proxmox when the router VM is running leaving only the "one onboard LAN I219-LM " for Proxmox and all other VMs.

Is that actually what is happening?

[crimper]

Try turning off "Use tablet for pointer" under the Proxmox-VM-Options and see if it helps.

[saluteferux]

@Patch My issue is that when using full throughput testing with a speedtest either 1Gb on my desktop or 2.5Gb on the proxmox host for example.

The proxmox **Sense dashboard is reporting 95% usage of the CPU cores but in the **sense actual VM dashboard it doesn't go above 15% percent.

Yes i'm actually passing through the all 4 ports of the Intel I226-V NIC to the **sense VM and using additional bridges to assign an IP on the proxmox host and set the gateway to be the **sense VM.

Here are screenshots of the 2 dashboards reporting completely different metrics during a 1Gb speedtest.

Proxmox VM dashboard

**SenseVM Screenshot Dashboard

[Patch]

@saluteferux said in Help with High CPU Usage in OPNSense/Pfsense VM inside Proxmox:

and using additional bridges to assign an IP on the proxmox host

If any of those bridges are connected in Promox to any of the I226-V NICs then you are not using pass through of the I226-V.

• Pass through is done via Proxmox GUI -> pve -> VM -> Hardware -> Add -> PCI device
• Virtual switches / bridges via Proxmox GUI -> pve ->Network-> create (to create a virtual switch), then Proxmox GUI -> pve -> VM -> Network device -> Add (to plug the VM into the virtual switch).

But I assume you already knew this & I'm just not understanding what you are saying.

As for why there is a difference between VM load inside a VM compared to the Load measure outside the VM by the hypervisor, I believe the difference is overhead (including task switching, virtual device emulation, interrupt pre processing etc)