• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Pfsense high cpu usage KVM (Unraid)

Scheduled Pinned Locked Moved Virtualization
unraidhigh cpuvirtual nic
45 Posts 4 Posters 10.8k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • B
    BjornStevens
    last edited by Aug 31, 2019, 1:15 PM

    Hi, i am trying to figure out a nice set-up for all my virtual servers.
    Right now i have put all my VM's in a virtual network (vibr0) and added the pfsense to it as a firewall for all the VM's.

    The issue i am having right now is that the cpu usage is insane high when doing transfers/ speedtests over the firewall or even in the firewall terminal itself.

    Altough sometimes the speed i am supposed to get (250mbit/s download) is nearly reached, it comes with 100% cpu usage.

    I have done a check were i use speedtest-cli in the command line of the pfsense, and check in another window the cpu usage with top -S -H. This shows the following:53fc4209-4419-42aa-8c93-25538c7b5458-afbeelding.png
    The speed that i got with this test is 150mbit/s download.

    And according to unraid the cpu usage on the cores was around80% all used by pfsense VM.

    I tried:
    Switching virtual nic (i started with a virtual intel nic, but have the same results with a vmware network card (vmnetx3)).
    Shutting down all other vm's during the testing -> got me better results but still high cpu usage.

    Does anyone have any clue what might cause this or how to fix?
    fyi: i only have one physical nic in my server, which is bridged to my pfsense vm for the network connection. All the other VM's and the pfsense have a connection to vibr0, where IP's are set static.

    If anyone knows how to fix my issue or can help me i would really appriciate it.

    1 Reply Last reply Reply Quote 0
    • S
      stephenw10 Netgate Administrator
      last edited by Aug 31, 2019, 8:51 PM

      What CPU actually is it? What speed is it running at?

      If it was an older CPU stuck at, say, 800MHz you might see that sort of usage.

      Steve

      B 1 Reply Last reply Aug 31, 2019, 9:02 PM Reply Quote 0
      • B
        BjornStevens @stephenw10
        last edited by Aug 31, 2019, 9:02 PM

        @stephenw10 I am running on an (old-school) FX-8350. Stock speeds, water cooled running at 4ghz max (nearly always at maximum). I pass it trough 2 out of 8 cores, so i was thinking like 2*4000MHz would be enough.

        1 Reply Last reply Reply Quote 0
        • S
          stephenw10 Netgate Administrator
          last edited by Aug 31, 2019, 10:52 PM

          Hmm, yeah if that's what it's really getting it should be far more than what is needed for 250Mbps.

          What is the output of sysctl hw.clockrate or sysctl dev.cpu.0 ?

          Steve

          B 1 Reply Last reply Sep 1, 2019, 9:07 AM Reply Quote 0
          • B
            BjornStevens @stephenw10
            last edited by Sep 1, 2019, 9:07 AM

            @stephenw10 Heres the output:
            b5aacd21-ca29-4b6c-bbc6-7bceb21cfaca-afbeelding.png

            Default clock of an fx 8350 is 3.6Ghz. Just know that this is a Virtual Machine. Unraid config over here:
            514be666-7f25-4f3d-9ada-8361166d9694-afbeelding.png
            db78850f-b84b-444c-87ca-528685e525e3-afbeelding.png

            During a speedtest on the pfsense (speedtest-cli with 150mbit download) the clock rates are this on unraid (8 core cpu so 8 speeds):
            ebee771d-6297-453b-b54d-10ea38bba758-afbeelding.png

            1 Reply Last reply Reply Quote 0
            • B
              BjornStevens
              last edited by BjornStevens Sep 1, 2019, 9:16 AM Sep 1, 2019, 9:11 AM

              Also a little addon on how it looks in the pfsense WebGui when the firewall is at idle and when doing a speedtest:
              3cf15aa2-0e25-49c9-8fc9-885b86d42664-afbeelding.png
              22641d74-edf6-4785-bb28-4656f8a01811-afbeelding.png
              During a speedtest top -S -H:
              0b0e0f96-af34-4aed-bcf9-feddd7efbf1f-afbeelding.png

              1 Reply Last reply Reply Quote 0
              • B
                BjornStevens
                last edited by Sep 2, 2019, 6:38 PM

                From what i have found so far i think this has to do because i am using virtual nic and not a physical nic. Can someone confirm this?

                1 Reply Last reply Reply Quote 0
                • S
                  stephenw10 Netgate Administrator
                  last edited by Sep 2, 2019, 9:56 PM

                  It should not just of itself. There are many people running virtualised and not seeing that, including in KVM.

                  Something about Unraids setup perhaps? I've never run that personally.

                  Steve

                  1 Reply Last reply Reply Quote 0
                  • K
                    kiokoman LAYER 8
                    last edited by Sep 3, 2019, 10:32 AM

                    indeed , i'm using kvm on my ubuntu server and i don't have this. idk what unraid is so i can't be of any help

                    ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                    Please do not use chat/PM to ask for help
                    we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                    Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                    1 Reply Last reply Reply Quote 0
                    • B
                      BjornStevens
                      last edited by Sep 3, 2019, 10:37 AM

                      Maybe i should just try to reïnstall it. Shouldn't be that hard to do. Ill post more after some more testing.

                      B 1 Reply Last reply Sep 4, 2019, 5:15 PM Reply Quote 0
                      • B
                        BjornStevens @BjornStevens
                        last edited by Sep 4, 2019, 5:15 PM

                        A reïnstall made no change, the cpu usage went up on 1 of the cores. during this test i even gave it 8 Cpu core's (4.0ghz) and 4GB of RAM. Download speed was 150mbit. So i have no clue what the option is other than the virtual nic or something...
                        Sadly i dont have any other nics available to test with. Any suggestions on a step i might try out?

                        Thanks!

                        1 Reply Last reply Reply Quote 0
                        • S
                          stephenw10 Netgate Administrator
                          last edited by Sep 4, 2019, 9:43 PM

                          With vmx NICs you will need to add the following line to /boot/loader.conf.local to get multiple queue support:
                          hw.pci.honor_msi_blacklist=0

                          Reboot to apply that. Check the output of vmstat -i to be sure it's creating multiple queues.

                          Be sure all hardware offloading support is disabled in Sys > Adv > Networking.

                          Steve

                          B 1 Reply Last reply Sep 5, 2019, 3:16 PM Reply Quote 0
                          • B
                            BjornStevens @stephenw10
                            last edited by Sep 5, 2019, 3:16 PM

                            @stephenw10

                            Hi, Thanks for your reply,

                            I tried to find the /boot/loader.conf.local file but could only find a /boot/loader.conf
                            I tried adding it into there ( hw.pci.honor_msi_blacklist=0 ) but still no change.
                            It has done something because it moved up in the file.

                            During speedtest i get these results with vmstat -i:
                            ae70e044-cd64-426d-aa9b-831bafb1867b-afbeelding.png
                            And when using the top -S -H command still get the same results.

                            Any other suggestions?

                            Thanks!

                            1 Reply Last reply Reply Quote 0
                            • K
                              kiokoman LAYER 8
                              last edited by Sep 5, 2019, 5:57 PM

                              you need to create the file
                              /boot/loader.conf.local
                              if it's missing
                              copy inside
                              hw.pci.honor_msi_blacklist=0
                              save and reboot

                              ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                              Please do not use chat/PM to ask for help
                              we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                              Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                              1 Reply Last reply Reply Quote 0
                              • S
                                stephenw10 Netgate Administrator
                                last edited by Sep 6, 2019, 3:35 AM

                                Yup create the file if it doesn't exist. If you put it in loader.conf it may get overwritten.

                                However that will only do anything for vmx NICs. You have em NICs there currently.

                                Steve

                                B 1 Reply Last reply Sep 6, 2019, 8:50 AM Reply Quote 0
                                • B
                                  BjornStevens @stephenw10
                                  last edited by BjornStevens Sep 6, 2019, 8:55 AM Sep 6, 2019, 8:50 AM

                                  @stephenw10 Allright, will set them to VMXNET3, reboot, create the file with the line and inform if there are any changes.

                                  Thanks for the help @kiokoman & @stephenw10 !

                                  Creating config file:
                                  982b0dd4-2c9c-4ecf-9ac8-be2915f3b4be-afbeelding.png

                                  1 Reply Last reply Reply Quote 0
                                  • B
                                    BjornStevens
                                    last edited by BjornStevens Sep 6, 2019, 9:19 AM Sep 6, 2019, 9:02 AM

                                    Okay so further testing will come in later but for now i seem to reach my maximum provider speed on my linux server behind the firewall:
                                    30ee4f36-acce-4b80-9fbf-49be03d205a0-afbeelding.png

                                    BUT it did drop back down to 14.4Megabyte's per second and go up and down all the time:
                                    9e4f1f56-336e-4798-bf02-06b239e5bad7-afbeelding.png
                                    Cpu usage seems to have set a bit:
                                    e79a25f5-47e3-4adf-983d-fd919f94def1-afbeelding.png

                                    Using SMB protocol i get this from moving a file WAN to LAN:
                                    e45d4618-c85f-4ace-af3c-533642fda829-afbeelding.png

                                    It's 2 virtual cores are running at nearly full power (cpu 6/7) (cpu 4 is being used on the server side in the LAN network.):
                                    45cb9478-33ab-446f-b5ad-60bca539c805-afbeelding.png

                                    I don't know if this is just a performance bug but speeds seem to have increased, altough cpu usage is still high (compared to the hardware specifications of pfsense)

                                    Changing to a quad core (virtual processor) did not change much either, cpu usage stays high on 2 cores:
                                    016158cd-9055-41c1-9087-e3bc4ac87e56-afbeelding.png

                                    Wish i could put my finger on the issue.

                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      stephenw10 Netgate Administrator
                                      last edited by Sep 6, 2019, 1:52 PM

                                      I still only see one tx queue and one rx queue on each NIC. Does vmstat -i show more?

                                      I assume you created that file in /boot

                                      Steve

                                      B 1 Reply Last reply Sep 6, 2019, 2:00 PM Reply Quote 0
                                      • B
                                        BjornStevens @stephenw10
                                        last edited by Sep 6, 2019, 2:00 PM

                                        @stephenw10

                                        yep its placed under /boot/loader.conf.local
                                        9c8aefad-741a-439a-9981-93582a95b5a6-afbeelding.png

                                        vmstat -i during speedtest on server in lan side:
                                        a7a528cf-d4b7-4795-bc21-5c250ff4579f-afbeelding.png

                                        1 Reply Last reply Reply Quote 0
                                        • B
                                          BjornStevens
                                          last edited by Sep 6, 2019, 2:04 PM

                                          I actually don't know how to read the vmstat -i, but i hope you might know more @stephenw10

                                          1 Reply Last reply Reply Quote 0
                                          20 out of 45
                                          • First post
                                            20/45
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                            This community forum collects and processes your personal information.
                                            consent.not_received