VMWare Pentest lab: Extremely high CPU on host

Supermule

Anyone solved this???

Has anyone tried without the VmTools package??

KalleL

I'm having the same problem running pfSense on ESXi 5.
I have also tried without vmTools package installed but the result is the same, CPU usage is extremely high.

Things that needs to be sorted out:

Does everybody running pfSense on ESXi 4/5 see this high CPU usage?
Is this problem related to FreeBSD or is it related to pfSense?

Supermule

I dont see it on the 1.2.3 version.

Pls test on your system to verify.

biggsy

Certainly no problem here.
ESXi 5.0.0 build 623860
2.0.1-RELEASE (i386) built on Mon Dec 12 18:24:17 EST 2011 with VMtools package installed
HP dc7900 SFF Core2 duo E7600 3.06 GHz with 8 GB and four Intel NICs (inc. 1 on board)

Supermule

test the performance graph instead in ESXi instead of the one in PFSense….

kaspro

Hi everyone,
I had the same issue running pfsense V2.0.1 on KVM.
I noticed that I had some systeme- time issues (I'm running on a mobile CPU).
I resolved it with the following options:
Go to the shell, and with vi edit the following files:

/etc/sysctl.conf: append: kern.timecounter.hardware=TSC ….at the end
/boot/loader.conf: append: kern.hz="100" ....at the end
...restart, so far - CPU usage OK ;)

Good luck,
Cheers,
Chris

Veni

@kaspro:

/etc/sysctl.conf: append: kern.timecounter.hardware=TSC ….at the end

/boot/loader.conf: append: kern.hz="100" ....at the end
...restart, so far - CPU usage OK ;)

Thanks for the tip :).

But no-go on my end ???. Still the same, running @ 1800-2400 MHz(according to ESXi) for only 20 Mbps traffic.
pfSense shows only 4% CPU usage.

It sounds to me that it must be close to the network part between the guest(pfSense higher than 1.2.3) and host.
Because my 1.2.3 guest that only has approx. 1,2 Mbps of traffic shows approx. 29 MHz according to ESXi.

I am running a Sony Ericsson USB connected UMTS cellular phone as a backup WAN. Have not excluded it as
the culprit. That is the reason why i have a pfSense 2.x running, because of the UMTS backup.
1.2.3 did not have support for these kind of connections so it takes care of a different part of the network.

kaspro

I would try "harder" to eliminate the VMWare CPU throttling effects… did you try something like this: http://communities.vmware.com/thread/87794

...reading:
Add in /boot/loader.conf:

Disable CPU frequency/voltage throttling control

hint.p4tcc.0.disabled=1
hint.acpi_throttle.0.disabled=1

Disable local APIC timers (FreeBSD 8+)

hint.apic.0.clock=0

Reduce interrupt rate (at the cost of slightly increases response time)

kern.hz=100

Saves 128 interrupts per second per core at the cost of reduced scheduling precision

hint.atrtc.0.clock=0

Add in /etc/rc.conf:

Turn off all CPU core clocks on idle

performance_cx_lowest="C2"
economy_cx_lowest="C2"

Disable background fsck at boot

background_fsck="NO"

also, are you getting the high CPU only on traffic or also when there is zero activity?
Did I get that right that you forward the USB- modem to the guest?

Cheers,
Chris

Veni

@kaspro:

I would try "harder" to eliminate the VMWare CPU throttling effects… did you try something like this: http://communities.vmware.com/thread/87794

...reading:
Add in /boot/loader.conf:

Disable CPU frequency/voltage throttling control

hint.p4tcc.0.disabled=1
hint.acpi_throttle.0.disabled=1

Disable local APIC timers (FreeBSD 8+)

hint.apic.0.clock=0

Reduce interrupt rate (at the cost of slightly increases response time)

kern.hz=100

Saves 128 interrupts per second per core at the cost of reduced scheduling precision

hint.atrtc.0.clock=0

Add in /etc/rc.conf:

Turn off all CPU core clocks on idle

performance_cx_lowest="C2"
economy_cx_lowest="C2"

Disable background fsck at boot

background_fsck="NO"

Will give it a try in the morning(in about ten hours).

@kaspro:

also, are you getting the high CPU only on traffic or also when there is zero activity?

Don't know. Will have to pull the plug to the primary, secondary and tertiary routes to get zero activity. Will give it a try in the morning.

@kaspro:

Did I get that right that you forward the USB- modem to the guest?

Correct. Using it as a tertiary WAN for my personal network part.

Veni

@kaspro:

Add in /boot/loader.conf:

Disable CPU frequency/voltage throttling control

hint.p4tcc.0.disabled=1
hint.acpi_throttle.0.disabled=1

Disable local APIC timers (FreeBSD 8+)

hint.apic.0.clock=0

Reduce interrupt rate (at the cost of slightly increases response time)

kern.hz=100

Saves 128 interrupts per second per core at the cost of reduced scheduling precision

hint.atrtc.0.clock=0

Add in /etc/rc.conf:

Turn off all CPU core clocks on idle

performance_cx_lowest="C2"
economy_cx_lowest="C2"

Disable background fsck at boot

background_fsck="NO"

No-go. Same result. Did not set background_fsck="NO".

@kaspro:

also, are you getting the high CPU only on traffic or also when there is zero activity?

pfSense 2.0.1 346-373 MHz when there is zero activity. About 4905 MHz when download client(Windows guest inside on the same host) only uses 1706 MHz. Pressed the disconnect button inside interfaces to disconnect the USB WWAN connection. pfSense 1.2.3 0 MHz when there is zero activity.

Both guests on same host. They share the same physical interfaces for primary WAN, secondary WAN and LAN. pfSense 2.0.1 also uses a physical interface for WLAN network for passthrough to Captive Portal.

Supermule

Imagine how expensice this would be if running in a cloud environment and you pay for CPU usage….

sleets

have the same problem on esxi 5.0.1 for both pfsense 2.0.1 and pfsense 2.1.

m0n0 and others is working great, just pfsense, And some times it use 100% cpu and loss response, the network also shutdown.

Supermule

Anybody that can test the same OS as pfsense running standalone in a VM to see if its the OS or specific to PFSense?

jimp

I just checked one of our ESX 5 boxes that houses not only our builder VMs but a batch of test pfSense VMs as well - at the moment, they're all idle.

FreeBSD 8.1 amd64 host - 81MHz
FreeBSD 8.1 i386 host - 86MHz
FreeBSD 8.3 amd64 host - 79MHz
FreeBSD 8.3 i386 host - 83MHz
pfSense 1.2.3 - 17MHz
pfSense 2.0.1 amd64 - 36MHz
pfSense 2.0.2 amd64 - 38MHz
pfSense 2.0.2 i386 - 51MHz
pfSense 2.1 amd64 - 41MHz
pfSense 2.1 i386 - 49MHz

The builders are running open-vm-tools-nox11, and at the moment the pfSense firewalls do not have tools installed.

So while the 1.2.3 VM is using less, it's not significantly less. I would still hesitate to call this a general issue. There must be something about the hardware (real or virtual)/config/etc bringing it out.

Supermule

Can people pls. make a list of what packages they run as well??

Veni

@Supermule:

Can people pls. make a list of what packages they run as well??

Packages:

AutoConfigBackup
Open-VM-Tools
Shellcmd
squid
squidGuard

Hardware:

Running a USB phone as tertiary route(only pfSense 2.0.1 is using it, not 1.2.3).
Supermicro X8DTi-LNF4 board with Intel Dual 82576 Dual-Port Gigabit Ethernet(all four NIC's are used by ESXi to share them among the two pfSense's inside the box)
2 x Xeon x5650
LSI 9280 24i4e with BBU and SafeStore.
Some serial ports and USB ports are used, but except for above, none of them are used by any pfSense.

iFloris

@Supermule:

Can people pls. make a list of what packages they run as well??

Packages:

Cron 0.1.5
Dashboard Widget: Snort 0.3.2
mailreport 1.2
mtr-nox11 0.82
NRPE v2 2.12_3 v2.1
ntop 4.1.0_3 v2.3
Open-VM-Tools-8.8.1 528969
OpenVPN Client Export Utility 0.24
RRD Summary 1.1
snort 2.9.2.3 pkg v. 2.5.1
Unbound 1.4.14_01

VMware version:

ESXi 5.0.0, 469512

Hardware:

P5K-E board
6 GB ram
Intel 82574L gigabit card
Intel 82576 dual-port gigabit card
Core 2 Duo E6750 @ 2.66Ghz
iScsi connection to Synology NAS

Veni

@Veni:

@Supermule:

Can people pls. make a list of what packages they run as well??

Packages:

AutoConfigBackup

Open-VM-Tools

Shellcmd

squid

squidGuard

Hardware:

Running a USB phone as tertiary route(only pfSense 2.0.1 is using it, not 1.2.3).

Supermicro X8DTi-LNF4 board with Intel Dual 82576 Dual-Port Gigabit Ethernet(all four NIC's are used by ESXi to share them among the two pfSense's inside the box)

2 x Xeon x5650

LSI 9280 24i4e with BBU and SafeStore.

Some serial ports and USB ports are used, but except for above, none of them are used by any pfSense.

I've now tried the following with these results:

pfSense 1.2.3 ~ 110 % off(pfSense CPU usage versus vCenter reported CPU usage)
pfSense 2.0.1 ~ 710 % off(pfSense CPU usage versus vCenter reported CPU usage)
pfSense 2.1.0 ~ 720 % off(pfSense CPU usage versus vCenter reported CPU usage)

All default installs, 32-bit. Only one LAN and one WAN. No packages.
Tried using the ATA VM drives instead of LSI Parallel SCSI on 2.0.1. No measurable difference.

I'm starting to wonder if it could be the pNIC's ???.
Anyone able to test with some other NIC's than 82576? Basically disabling/not using the 82576 inside ESXi.

Supermule

Very sure that its the OS….

Why should the nics use high CPU when no traffic and therefore no offloading occurs?

duanes

Running VMWare ESXi 5.0.0-469512-standard

pfSense 2.0.1-Release (amd64) 12/12/2011.

Packages:

Backup System 0.1.5
bandwidthd System 2.0.1.3
Cron Services 0.1.5
darkstat Network Management 3.0.714
Lightsquid Network Report 1.8.2 pkg v.2.32
mailreport Network Management 1.2
ntop Network Management 4.1.0_3 v2.3
Open-VM-Tools Services 8.7.0.3046 (build-313025)
OpenVPN Client Export Utility Security 0.24
pfBlocker Firewall 1.0.2
Sarg Network Report 2.3.2 pkg v.0.6.1
squid Network 2.7.9 pkg v.4.3.1
vnstat2 Network Management 1.10_2
widescreen Enhancements 0.2

Physical Hardware is a Dell PowerEdge 2950
1x Quad Core Xeon e5345 (2.33Ghz)
8gb RAM
Dual Ethernet Standard built-in Dell NICs (2x BroadCom BCM5708 Gig-Eth)
Running 3 Virtual machines, only one is pfSense.

pfSense Virtual Machine:
2x E1000 NICs
LSI Logic SAS.
1 GB RAM
20 GB Disk Space.
There are no resource allocation limits configured in VMWare.

pfSense has a couple of VPN clients connected and two gateways that are 2xT1 Each. 8 Routing policies to direct outbound traffic via gateways depending upon source and destination plus some route failover and load balancing. Traffic levels are almost max'd out on both gateways during the business day.

Both pfSense TOP and VMware Utilization show 15-25% and are very close to each other in readings.

Basically, mine works great.