Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    VMWare Pentest lab: Extremely high CPU on host

    Scheduled Pinned Locked Moved Virtualization
    85 Posts 29 Posters 70.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      Mattofsweden
      last edited by

      I'm seeing the same issues here on a DELL PowerEdge R310 Quad Core Xeon:
      Using ESXi 4.1 and pfSense 2.0, 2.0.1, old-2.1-dev in i386/amd64 flavors
      Using ESXi 5.0 and pfSense 2.0.1 and 2.1-dev in i386/amd64 flavors from feb/march/april.

      Same results on other host hardware also (Two DELL Servers with virtualized environment at home for testing purposes.)

      Have not tried the VMXNET due to others not seeing any performance gain, only been using virtualized E1000 so far.

      What I'm using a lot is VLANs, which might be a contributing culprit for some of us? Assigning VLANs directly in switch configuration in vSphere, or natively in pfSense has had "largely" the same results.

      I absolutely love pfSense, now that I've got a hang of it, and have deployed quite a few in different scenarios past few months. But, not to sound negative here, there gotta be something we can do about these high loads in virtualized environments. I had to switch over to bare-metal, on slightly aged HW, on our lab network which is a bit unsatisfying. I loose a bit of my redundancy (if one VM or host fails, just fire up the copy or using HA Sync).

      I suppose it's underlying FreeBSD issue?
      I don't really know how to set up something similar in any of the *BSD flavors, and honestly can't find the time to learn currently, but surely one of you guys could test a simple routing setup using FreeBSD/OpenBSD/NetBSD and see if there's the same performance issue? (Maybe with/without VLAN incl. trunking/non-native.)

      Regards,
      Mattias

      IT Teacher & Networking Consultant

      1 Reply Last reply Reply Quote 0
      • G
        goodspeedal
        last edited by

        Just let you know, there are one more case for reference.

        I have tested the pfsense with the follow spec

        1. DELL 9200
           i)Build-in LAN 82566DC Gigabit LAN Cards
           ii)3 Intel 82541 Gigabit LAN Cards
        2. VM under ESXi-5-U1
           i)Setting: only one pfsense VM with FreeBSD 64bit
           ii)2 e1000 virtual LAN Cards
           iii)1 vCPU, 1024MB RAM
        3. pfsense with "Open VM Tools package", "Snort" installed
           i) Assigned one LAN for each interfaces(WAN, LAN)

        Result:
        I have just started the machine to test the stability, not even use it. It will freeze after a day. The freeze will only in the VM level, not affect the ESXi.

        Please let me know if you need any information from my setting as well. Since this is only a test machine, i wanna to put the pfsense in the DELL R610 later. But the migration will be on held at the moment. Thanks for any fix for the issue.

        1 Reply Last reply Reply Quote 0
        • D
          dLockers
          last edited by

          Have you tried enabling vt-d and passing the intel nics directly to pfsense?

          1 Reply Last reply Reply Quote 0
          • S
            Supermule Banned
            last edited by

            So you want to risk frontend firewall with direct contact with the physical Nics on the server?

            Uninstall the vmtools package and reboot. Sed if it solves the issue…

            1.2.3 doesnt have any of this at all. Running in about 3% on the physical server.

            1 Reply Last reply Reply Quote 0
            • G
              goodspeedal
              last edited by

              @dLockers:

              Have you tried enabling vt-d and passing the intel nics directly to pfsense?

              Just checked with test system (DELL 9200) is not support pass-thr even the motherboard is enabled vt-d.
              But why can't i just use 2 virtual lan cards and connect each of them to a separate v-switch. And let other 2 real lan cards to connect the v-switches.
              It will be the same as setting as you suggested.

              1 Reply Last reply Reply Quote 0
              • S
                Supermule Banned
                last edited by

                Anyone solved this???

                Has anyone tried without the VmTools package??

                1 Reply Last reply Reply Quote 0
                • K
                  KalleL
                  last edited by

                  I'm having the same problem running pfSense on ESXi 5.
                  I have also tried without vmTools package installed but the result is the same, CPU usage is extremely high.

                  Things that needs to be sorted out:

                  Does everybody running pfSense on ESXi 4/5 see this high CPU usage?
                  Is this problem related to FreeBSD or is it related to pfSense?

                  1 Reply Last reply Reply Quote 0
                  • S
                    Supermule Banned
                    last edited by

                    I dont see it on the 1.2.3 version.

                    Pls test on your system to verify.

                    1 Reply Last reply Reply Quote 0
                    • B
                      biggsy
                      last edited by

                      Certainly no problem here. 
                      ESXi 5.0.0 build 623860
                      2.0.1-RELEASE (i386) built on Mon Dec 12 18:24:17 EST 2011 with VMtools package installed
                      HP dc7900 SFF Core2 duo E7600 3.06 GHz with 8 GB and four Intel NICs (inc. 1 on board)

                      1 Reply Last reply Reply Quote 0
                      • S
                        Supermule Banned
                        last edited by

                        test the performance graph instead in ESXi instead of the one in PFSense….

                        1 Reply Last reply Reply Quote 0
                        • K
                          kaspro
                          last edited by

                          Hi everyone,
                          I had the same issue running pfsense V2.0.1 on KVM.
                          I noticed that I had some systeme- time issues (I'm running on a mobile CPU).
                          I resolved it with the following options:
                          Go to the shell, and with vi edit the following files:

                          • /etc/sysctl.conf: append:      kern.timecounter.hardware=TSC    ….at the end
                          • /boot/loader.conf: append:      kern.hz="100"    ....at the end
                            ...restart, so far - CPU usage OK ;)

                          Good luck,
                          Cheers,
                          Chris

                          1 Reply Last reply Reply Quote 0
                          • V
                            Veni
                            last edited by

                            @kaspro:

                            • /etc/sysctl.conf: append:      kern.timecounter.hardware=TSC    ….at the end
                            • /boot/loader.conf: append:      kern.hz="100"    ....at the end
                              ...restart, so far - CPU usage OK ;)

                            Thanks for the tip :).

                            But no-go on my end ???. Still the same, running @ 1800-2400 MHz(according to ESXi) for only 20 Mbps traffic.
                            pfSense shows only 4% CPU usage.

                            It sounds to me that it must be close to the network part between the guest(pfSense higher than 1.2.3) and host.
                            Because my 1.2.3 guest that only has approx. 1,2 Mbps of traffic shows approx. 29 MHz according to ESXi.

                            I am running a Sony Ericsson USB connected UMTS cellular phone as a backup WAN. Have not excluded it as
                            the culprit. That is the reason why i have a pfSense 2.x running, because of the UMTS backup.
                            1.2.3 did not have support for these kind of connections so it takes care of a different part of the network.

                            1 Reply Last reply Reply Quote 0
                            • K
                              kaspro
                              last edited by

                              I would try "harder" to eliminate the VMWare CPU throttling effects… did you try something like this: http://communities.vmware.com/thread/87794

                              ...reading:
                              Add in /boot/loader.conf:

                              Disable CPU frequency/voltage throttling control

                              hint.p4tcc.0.disabled=1
                              hint.acpi_throttle.0.disabled=1

                              Disable local APIC timers (FreeBSD 8+)

                              hint.apic.0.clock=0

                              Reduce interrupt rate (at the cost of slightly increases response time)

                              kern.hz=100

                              Saves 128 interrupts per second per core at the cost of reduced scheduling precision

                              hint.atrtc.0.clock=0

                              Add in /etc/rc.conf:

                              Turn off all CPU core clocks on idle

                              performance_cx_lowest="C2"
                              economy_cx_lowest="C2"

                              Disable background fsck at boot

                              background_fsck="NO"

                              also, are you getting the high CPU only on traffic or also when there is zero activity?
                              Did I get that right that you forward the USB- modem to the guest?

                              Cheers,
                              Chris

                              1 Reply Last reply Reply Quote 0
                              • V
                                Veni
                                last edited by

                                @kaspro:

                                I would try "harder" to eliminate the VMWare CPU throttling effects… did you try something like this: http://communities.vmware.com/thread/87794

                                ...reading:
                                Add in /boot/loader.conf:

                                Disable CPU frequency/voltage throttling control

                                hint.p4tcc.0.disabled=1
                                hint.acpi_throttle.0.disabled=1

                                Disable local APIC timers (FreeBSD 8+)

                                hint.apic.0.clock=0

                                Reduce interrupt rate (at the cost of slightly increases response time)

                                kern.hz=100

                                Saves 128 interrupts per second per core at the cost of reduced scheduling precision

                                hint.atrtc.0.clock=0

                                Add in /etc/rc.conf:

                                Turn off all CPU core clocks on idle

                                performance_cx_lowest="C2"
                                economy_cx_lowest="C2"

                                Disable background fsck at boot

                                background_fsck="NO"

                                Will give it a try in the morning(in about ten hours).

                                @kaspro:

                                also, are you getting the high CPU only on traffic or also when there is zero activity?

                                Don't know. Will have to pull the plug to the primary, secondary and tertiary routes to get zero activity. Will give it a try in the morning.

                                @kaspro:

                                Did I get that right that you forward the USB- modem to the guest?

                                Correct. Using it as a tertiary WAN for my personal network part.

                                1 Reply Last reply Reply Quote 0
                                • V
                                  Veni
                                  last edited by

                                  @kaspro:

                                  Add in /boot/loader.conf:

                                  Disable CPU frequency/voltage throttling control

                                  hint.p4tcc.0.disabled=1
                                  hint.acpi_throttle.0.disabled=1

                                  Disable local APIC timers (FreeBSD 8+)

                                  hint.apic.0.clock=0

                                  Reduce interrupt rate (at the cost of slightly increases response time)

                                  kern.hz=100

                                  Saves 128 interrupts per second per core at the cost of reduced scheduling precision

                                  hint.atrtc.0.clock=0

                                  Add in /etc/rc.conf:

                                  Turn off all CPU core clocks on idle

                                  performance_cx_lowest="C2"
                                  economy_cx_lowest="C2"

                                  Disable background fsck at boot

                                  background_fsck="NO"

                                  No-go. Same result. Did not set background_fsck="NO".

                                  @kaspro:

                                  also, are you getting the high CPU only on traffic or also when there is zero activity?

                                  pfSense 2.0.1 346-373 MHz when there is zero activity. About 4905 MHz when download client(Windows guest inside on the same host) only uses 1706 MHz. Pressed the disconnect button inside interfaces to disconnect the USB WWAN connection. pfSense 1.2.3 0 MHz when there is zero activity.

                                  Both guests on same host. They share the same physical interfaces for primary WAN, secondary WAN and LAN. pfSense 2.0.1 also uses a physical interface for WLAN network for passthrough to Captive Portal.

                                  1 Reply Last reply Reply Quote 0
                                  • S
                                    Supermule Banned
                                    last edited by

                                    Imagine how expensice this would be if running in a cloud environment and you pay for CPU usage….

                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      sleets
                                      last edited by

                                      have the same problem on esxi 5.0.1 for both pfsense 2.0.1 and pfsense 2.1.

                                      m0n0 and others is working great, just pfsense,  And some times it use 100% cpu and loss response, the network also shutdown.

                                      1 Reply Last reply Reply Quote 0
                                      • S
                                        Supermule Banned
                                        last edited by

                                        Anybody that can test the same OS as pfsense running standalone in a VM to see if its the OS or specific to PFSense?

                                        1 Reply Last reply Reply Quote 0
                                        • jimpJ
                                          jimp Rebel Alliance Developer Netgate
                                          last edited by

                                          I just checked one of our ESX 5 boxes that houses not only our builder VMs but a batch of test pfSense VMs as well - at the moment, they're all idle.

                                          FreeBSD 8.1 amd64 host - 81MHz
                                          FreeBSD 8.1 i386 host - 86MHz
                                          FreeBSD 8.3 amd64 host - 79MHz
                                          FreeBSD 8.3 i386 host - 83MHz
                                          pfSense 1.2.3 - 17MHz
                                          pfSense 2.0.1 amd64 - 36MHz
                                          pfSense 2.0.2 amd64 - 38MHz
                                          pfSense 2.0.2 i386 - 51MHz
                                          pfSense 2.1 amd64 - 41MHz
                                          pfSense 2.1 i386 - 49MHz

                                          The builders are running open-vm-tools-nox11, and at the moment the pfSense firewalls do not have tools installed.

                                          So while the 1.2.3 VM is using less, it's not significantly less. I would still hesitate to call this a general issue. There must be something about the hardware (real or virtual)/config/etc bringing it out.

                                          Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                          Need help fast? Netgate Global Support!

                                          Do not Chat/PM for help!

                                          1 Reply Last reply Reply Quote 0
                                          • S
                                            Supermule Banned
                                            last edited by

                                            Can people pls. make a list of what packages they run as well??

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.