Pfsense 2.1 vmware cpu host high usage

arik010

Hello,
We have several pfsense 2.1 installed on vmware hosts 5.1 and 5.5
the pfsense show about 3% cpu usage but in vmware we see constantly about 35% cpu usage (4000+MHZ) per pfsense vm
we have latest open-vm tools instaled on each pfsense.

is there any way to solve this huge reoursce lose?

_Cyph3r_

Just to chime in on this thread: I also am facing this issue. I'm using ESXi 5.1 build 1157734 on a Dell PowerEdge R720 with Intel i350 NICs. I'm running pfSense 2.1 with Open-VM-tools installed, 6 E1000 NICs and 1vCPU.

pfSense itself does not detect high cpu load, but when you look at the performance graph in vSphere it shows a significantly higher cpu load. When generating high network load between two interfaces of the pfSense, the maximum throughput I get is 600Mbit and CPU is hitting 100% in VMWare. CPU load in pfsense is like 20-25%.

Another thread on this forum (http://forum.pfsense.org/index.php/topic,41647.75.html) indicates that it might be the VMWare vSwitches eating up the cpu cycles. Looking at the age of this topic, the problem is already around for quite a while.

Someone also opened a bug in redmine (https://redmine.pfsense.org/issues/2618), but the bug got rejected because apparently not every ESX install is experiencing the issue.

So, is this a VMWare bug in specific hardware environments? Would be great if someone from the pfSense support team could provide some additional info…

FuriouS

I am seeing this issue as well.

Pfsense with e1000 VNICS
Cisco UCS chassis with B200-M3 blades
40GB backplane

CPU use appears to be about 10x what is shown in the dashboard and RRD graphs for CPU use. Seems logical that the extra network traffic would take some CPU but the load seems high. only pushing 20Mb max over firewall, it should be able to handle that virtualized.

Corn

I'm seeing the same kind of problems on two ESXi hosts (5.1 and 5.5) and pfSense 2.1

~300 Mhz on VMWare while idle (0.3% load on pfSense), maxing out at ~4000 Mhz with about 500Mbit throughput. pfSense indicates the usage should be about 25%, but VMWare doesn't seem to agree.

The load also jumps up when transferring files over samba from a fileserver on the same host, probably due to promiscuous mode on the vm switch.

Host 1 is an AMD-based whitebox server, Host 2 is a Intel Nuc i5. Both hosts are connected through Intel network adapters, both pfSense vm's are handling 7 VLAN's over this interface. I tried booting the vm's without network adapters, but idle was still about 180 Mhz.

Tried installing VMWare tools and open-vm-tools, but noticed no improvement.

All suggestions are welcome :'(

bryan.paradis

@Corn:

I'm seeing the same kind of problems on two ESXi hosts (5.1 and 5.5) and pfSense 2.1

~300 Mhz on VMWare while idle (0.3% load on pfSense), maxing out at ~4000 Mhz with about 500Mbit throughput. pfSense indicates the usage should be about 25%, but VMWare doesn't seem to agree.

The load also jumps up when transferring files over samba from a fileserver on the same host, probably due to promiscuous mode on the vm switch.

Host 1 is an AMD-based whitebox server, Host 2 is a Intel Nuc i5. Both hosts are connected through Intel network adapters, both pfSense vm's are handling 7 VLAN's over this interface. I tried booting the vm's without network adapters, but idle was still about 180 Mhz.

Tried installing VMWare tools and open-vm-tools, but noticed no improvement.

All suggestions are welcome :'(

If you run up a new 2.1 instance without installing vm tools do you see the same problem?

deagle

Yup, just tried a brand new 32 and 64 bit. 10x CPU usage in esxtop vs the guest os.

jamesp81

I'm having similar issues running pfSense in virtual box on a Linux Mint 16 Petra host.

At first I thought it was because my hardware was underpowered. The machine in question is my homebrew nas / backup server and the hardware was chosen for low noise and low power consumption over performance.

Machine's specs:

1.86Ghz dual core Intel Atom processor
2GB Ram
1 TB 2.5" hard drive
Integrated graphics (don't remember the chipset)
Linux Mint 16 Petra, XFCE desktop
Oracle Virtualbox
pfSense 2.1 running in Virtualbox
3x NICs. One WAN, two lan, with each lan being served by a SOHO router that's acting solely as a switch

At first CPU usage was running around 90% when the VM was under load downloading large files. I installed virtio drivers on pfSense and got it to use paravirtualized NICs. This seemed to help it a little, but it still routinely sits on 80 percent while under network load, although at times it hovers more around 65 to 70 percent. It also matters which utility I'm using to measure CPU consumption. Mint's Task Manager application shows cpu usage hovering around 70 percent, but I think it aggregates from both cores. htop from the terminal tends to show somewhat heavier load, with one core fairly regularly pegging out at 95%+. The core that pegs out does change, it's not just the one that the VM is using.

The VM is configured to use a single core of the dual core CPU (I am not give an option to allocate the second core to it).

The paravirtualization of the NICs seemed to help things some, I haven't leveraged virtio yet for hard drive access, though I have doubts about that having much effect.

Now, maybe it IS that my hardware is underpowered, but I know that pfSense will run well in a small network (less than 8 clients or so) on a 500 MHZ Geode processor. I'm kind of amazed the virtual box adds that much overhead and that that overhead is apparently not mostly from virtualized NICs (if it were, going to virtio for the NICs should've solved the problem outright).

As far as real world performance, I'd say my pfsense VM still, despite everything, compares favorably to my old Netgear SOHO router, and it has allowed me to set up two lans on different subnets (one with restrictive firewall rules; that one's for guests that don't need to be able to talk to my personal desktop machine). Latency pinging out to the internet is about 10 ms higher on average than on my old router, but since I don't play much Quake anymore I can tolerate that ;D Ping latency goes quite high under load, but my aforementioned Netgear router would do that too, so that's clearly a different problem.

The last time I checked, the heavy CPU load only happened when pulling in data from the WAN. CPU usage was minimal when my Roku streamed music or video from the plex server, though the VM router might not have much direct involvement in that traffic (the Linux Mint host has a connection, through a physical interface, to the same lan that the Roku has a connection to)

Maybe I should just run it and not worry about it, but I don't really want to run the thing hard all the time if I can help it.

deagle

I have two ESX hosts and I can run two nodes in CARP, I don't wanna give up redundancy by going to a physical box. I wish we could figure this out, almost everyone I talk to who runs 2.x in ESX has this problem. I tried vmxnet3 NICs today with the latest driver from vmwaretools 9 and still 10x cpu in ESXTOP.

LogicalApex

I am also experiencing this issue on my ESXi box with pfSense 2.1 running in a VM with the latest open-vm-tools installed.

The ESXi host is a Dell R620 with Intel NICs.

I'm running ESXi 5.5. Will be upgrading to 5.5 U1 this weekend to see if that helps any.

kenshirothefist

Exactly the same here with pfSense 2.1.2 on top of ESXi 5.1u2 … Has this already been identified as a bug? Developers are aware of this issue?

LogicalApex

@kenshirothefist:

Exactly the same here with pfSense 2.1.2 on top of ESXi 5.1u2 … Has this already been identified as a bug? Developers are aware of this issue?

No one seems to acknowledge this. The base recommendation is to use Intel NICs, but in my case I'm using all Intel NICs and the problem persists. I'm also using HCL servers using the official OEM (Dell) install image.

This problem is still occurring on ESXi 5.5 U1.

If it matters, I'm using Intel I350 1G NICs.

kenshirothefist

Has anybody also noticed some increased latency when using pfSense in ESXi 5.x? I haven't had time to test it, but I'm really curios weather this hight ESXi cpu load is just "cpu load issue" or does it also affect firewall efficiency (especially when multiple TCP connections are being handled simultaneously)?

kenshirothefist

Bump. Any news on this? Anybody solved this?

biggsy

This keeps popping up. It might be helpful if people experiencing this problem posted a few standard things about their setup. Then we might be able to see whether there is something common between them:

What is the ESXi host machine and processor?
Which version of pfSense and whether 32 or 64-bit?
How many vCPUs have you allocated to the VM?
How much memory have you allocated to the VM?
Have you installed the pfSense packaged VM tools or the VMware-supplied tools?
Are you using the e1000 adapter type or something else?

kenshirothefist

What is the ESXi host machine and processor?

IBM, Intel Xeon X5650

Which version of pfSense and whether 32 or 64-bit?

2.1.3 64-bit

How many vCPUs have you allocated to the VM?

1 CPU @ 2.67 GHz

How much memory have you allocated to the VM?

512 MB

Have you installed the pfSense packaged VM tools or the VMware-supplied tools?

pfSense packaged VM tools

Are you using the e1000 adapter type or something else?

E1000

BTW: in my particular case there is very low and constant bandwidth (cca. 2 Mbps) but with thousands of active TCP connections (many small packets); currently I have only like 2% CPU load inside pfsense (cca. 50 Mhz), but cca. 1800 MHz Consumed Host CPU

biggsy

kenshirothefist,

Forgot to ask:

Any other pfSense packages running?

I assume you have seen the last post in this thread https://forum.pfsense.org/index.php?topic=41647.0. Anything like that going on in your system?

I should also say that I've never experienced this problem, even though I've run multiple 32 and 64-bit versions of pfSense on at least four different (HP) hardware platforms since ESXi 3.5 was released.

kenshirothefist

Any other pfSense packages running?

Open-VM-Tools, OpenVPN, pfBlocker, remote logging … however, even if I disable all these packages, cpu host usage still high

deagle

What is the ESXi host machine and processor?
Tried many builds of 5.1 and 5.5 with same result
Supermicro X8SIL
Intel(R) Xeon(R) CPU X3440 @ 2.53GHz (Lynnfield)
Which version of pfSense and whether 32 or 64-bit?
Tried 2.1.1 x64, then tried 2.1.2-3 x86
How many vCPUs have you allocated to the VM?
Tried 1, had to bump up to 2 because if this issue, 50Mbit throughput = 70-80% of one physical core
How much memory have you allocated to the VM?
Tried 512-2048 MB
Have you installed the pfSense packaged VM tools or the VMware-supplied tools?
Tried packaged tools in the past but since read not to use them. Then tried VMware-supplied, no difference
Are you using the e1000 adapter type or something else?
Tried both e1000 and vmxnet3 (w/VMware-supplied driver), no difference.

Packages - Avahi, OpenVPN export util, Cron, RRD Summary.
It also happens on fresh install.

Just to be clear, you have to watch esxtop to see this issue, it doesn't show up in the guest.

kenshirothefist

@biggsy, any news regarding this topic?

biggsy

I can't see anything common between these configs and haven't been able to reproduce it any way. Only have one machine to play with now though.

Have you guys checked that link about speed mismatch?