Pfsense 2.1 vmware cpu host high usage

kenshirothefist

What is the ESXi host machine and processor?

IBM, Intel Xeon X5650

Which version of pfSense and whether 32 or 64-bit?

2.1.3 64-bit

How many vCPUs have you allocated to the VM?

1 CPU @ 2.67 GHz

How much memory have you allocated to the VM?

512 MB

Have you installed the pfSense packaged VM tools or the VMware-supplied tools?

pfSense packaged VM tools

Are you using the e1000 adapter type or something else?

E1000

BTW: in my particular case there is very low and constant bandwidth (cca. 2 Mbps) but with thousands of active TCP connections (many small packets); currently I have only like 2% CPU load inside pfsense (cca. 50 Mhz), but cca. 1800 MHz Consumed Host CPU

biggsy

kenshirothefist,

Forgot to ask:

Any other pfSense packages running?

I assume you have seen the last post in this thread https://forum.pfsense.org/index.php?topic=41647.0. Anything like that going on in your system?

I should also say that I've never experienced this problem, even though I've run multiple 32 and 64-bit versions of pfSense on at least four different (HP) hardware platforms since ESXi 3.5 was released.

kenshirothefist

Any other pfSense packages running?

Open-VM-Tools, OpenVPN, pfBlocker, remote logging … however, even if I disable all these packages, cpu host usage still high

deagle

What is the ESXi host machine and processor?
Tried many builds of 5.1 and 5.5 with same result
Supermicro X8SIL
Intel(R) Xeon(R) CPU X3440 @ 2.53GHz (Lynnfield)
Which version of pfSense and whether 32 or 64-bit?
Tried 2.1.1 x64, then tried 2.1.2-3 x86
How many vCPUs have you allocated to the VM?
Tried 1, had to bump up to 2 because if this issue, 50Mbit throughput = 70-80% of one physical core
How much memory have you allocated to the VM?
Tried 512-2048 MB
Have you installed the pfSense packaged VM tools or the VMware-supplied tools?
Tried packaged tools in the past but since read not to use them. Then tried VMware-supplied, no difference
Are you using the e1000 adapter type or something else?
Tried both e1000 and vmxnet3 (w/VMware-supplied driver), no difference.

Packages - Avahi, OpenVPN export util, Cron, RRD Summary.
It also happens on fresh install.

Just to be clear, you have to watch esxtop to see this issue, it doesn't show up in the guest.

kenshirothefist

@biggsy, any news regarding this topic?

biggsy

I can't see anything common between these configs and haven't been able to reproduce it any way. Only have one machine to play with now though.

Have you guys checked that link about speed mismatch?

kenshirothefist

@biggsy:

Have you guys checked that link about speed mismatch?

I have auto negotiate and it negotiates at 1000 Full … Anyway, I have 20+ running VM's on this host and only this pfSense appliance is having these issues with high pCPU load, although pfSense is the only freeBSD-based VM (others are centos and ubuntu based).

biggsy

The worst I can do is about 93% CPU running a 120 Mbit/s download from AARNET (it's local).

That's with a single vCPU on a Xeon E3-1265L v2 @ 2.5 GHz inside a Gen8 MicroServer.

Idle the VM runs along at about 1.5% CPU :-[

deagle

That's the thing, it has no business doing 93% of one core at 120Mbit, virtualization overhead should be minimal like with other OSes.

I'm starting to think that people who "don't have" this problem aren't really seeing it.

kenshirothefist

Is it possible that this is related to VMware virtual machine monitor mode?

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1036775

For example:

datetime| vmx| MONITOR MODE: allowed modes : BT HV HWMMU
datetime| vmx| MONITOR MODE: user requested modes : BT HV HWMMU
datetime| vmx| MONITOR MODE: guestOS preferred modes: BT HWMMU HV
datetime| vmx| MONITOR MODE: filtered list : BT HWMMU HV

Where:

allowed modes – Refers to the mode that the underlying hardware is capable of.
user requested modes – Refers to the setting defined in the virtual machine configuration.
guestOS preferred modes – Refers to the default values for the selected guest operating system.
filtered list – Refers to the actual monitor modes acceptable for use by the Hypervisor, with the left-most mode being utilized.

I have "automatic" for my pfsense VM and it reads like this:

datetime| vmx| I120: MONITOR MODE: allowed modes : BT32 HV HWMMU
datetime| vmx| I120: MONITOR MODE: user requested modes : BT32 HV HWMMU
datetime| vmx| I120: MONITOR MODE: guestOS preferred modes: HWMMU HV BT32
datetime| vmx| I120: MONITOR MODE: filtered list : HWMMU HV BT32

Therefore it is using hardware MMU and hardware instruction set virtualization … can't change it right now, but can someone test with different settings and post results?

Also, is it possible that is related to using distributed vSwitch? Can someone test by using regular vSwitch vs. distributed vSwitch (again, I can't change my environment to regular vSwitch right now)?

deagle

Distributed vSwitch is just an abstraction for many standard vSwitches.

kenshirothefist

FYI: there is even more overhead when you go from 1 vCPU to 2 vCPU … I had 2.3 GHz CPU usage when my pfSense VM was configured with 1 vCPU (approaching to limit 2.67 GHz), then I reconfigured VM to use 2 vCPU and now I have 3.0 GHz CPU usage (probably from CPU threads trashing) ... this is really annoying ... and I really don't want to go back to physical ...

KOM

I've been following this thread but didn't think it was affecting me. Then I took a look. pfSense tells me it's using 2% CPU. VMware tells me it's using almost nothing. ESXTop tells me 20%…

My config:

Dell Powervault NX3000 (8 x L5520 @ 2.26 GHz)
ESXi 5.5U1
pfSense 2.1.3 i386
2 x vCPU, 2GB RAM
VM version 8 hardware
Intel E1000 vNICs

podilarius

I have 2 pfSense VMs running (one 2.1.3 [143MHz used] and the other 2.2-Alpha [95MHz used]) and both are running with very low CPU. Both as reported by ESX and pfSense.
Are you guys running powerd, I am?

Edit:
Meant to add my config:
ASUS Server
AMD Opteron 12 core processor
Intel NICs on 2.1.3
VMXNET3 on 2.2
1024MB of memory
VMTools package is installed
No other packages running.

biggsy

KOM and podilarius,

Under what sort of network load?

My CPU numbers are also low - until I start a heavy (120 Mb/s) download then they diverge very quickly. The ESXi/esxtop numbers go through the roof while pfSense sees little change.

KOM

I pumped up a few torrents to try and saturate my 90/90 link and could only manage about 10Mb/s. Even then, ESXTop showed pfsense taking from 50-103% of %USED.

I don't use powerd.

podilarius

I am about to load test my 2.2 so I will let you know.

podilarius

@podilarius:

I am about to load test my 2.2 so I will let you know.

Okay so pushing 500Mbits in and out of pfsense 2.2, I am getting 5987MHz and ESXTOP is showing 225-266% CPU usage for just pfsense 2.2. I hope that is out of 1000%.
The VM itself is showing 89-91% usage. I have a AMD 6234 at 2.4GHz per core and has 12 cores.
As you all know, if you are running 2.1 series and before, pf and i think ipfw are giant locked and will only use 1 processor.
To me it seems that 2.2 is giant locked to 2 CPUs. This could be because I have 2 nics involved, so I am not sure if it is one CPU per NIC or just locked to dual CPUs. I have asked in another thread with no answer.
I am trying to get to 10GB speed, but I seem locked to 1GBE, but alas that is another issue for another topic.

Gabri.91

Hi,
is this problem solved or still present?

I want to move my pfSense to ESXI purchasing new hardware, but now I'm not really sure about it..

MichelZ

I seem to have the same problem. 1.5 GhZ in ESX, ~14% on pfsense, about 1 MBit (!) traffic… :(
Happens to both VM's of a HA Pair. Using Intel NICs, E1000 vNIC