Pfsense high cpu usage KVM (Unraid)

BjornStevens · Sep 1, 2019, 9:16 AM

Also a little addon on how it looks in the pfsense WebGui when the firewall is at idle and when doing a speedtest:

During a speedtest top -S -H:

BjornStevens · Sep 2, 2019, 6:38 PM

From what i have found so far i think this has to do because i am using virtual nic and not a physical nic. Can someone confirm this?

stephenw10 · Sep 2, 2019, 9:56 PM

It should not just of itself. There are many people running virtualised and not seeing that, including in KVM.

Something about Unraids setup perhaps? I've never run that personally.

Steve

kiokoman · Sep 3, 2019, 10:32 AM

indeed , i'm using kvm on my ubuntu server and i don't have this. idk what unraid is so i can't be of any help

BjornStevens · Sep 4, 2019, 5:15 PM

Maybe i should just try to reïnstall it. Shouldn't be that hard to do. Ill post more after some more testing.

BjornStevens · Sep 4, 2019, 5:15 PM

A reïnstall made no change, the cpu usage went up on 1 of the cores. during this test i even gave it 8 Cpu core's (4.0ghz) and 4GB of RAM. Download speed was 150mbit. So i have no clue what the option is other than the virtual nic or something...
Sadly i dont have any other nics available to test with. Any suggestions on a step i might try out?

Thanks!

stephenw10 · Sep 4, 2019, 9:43 PM

With vmx NICs you will need to add the following line to /boot/loader.conf.local to get multiple queue support:
hw.pci.honor_msi_blacklist=0

Reboot to apply that. Check the output of vmstat -i to be sure it's creating multiple queues.

Be sure all hardware offloading support is disabled in Sys > Adv > Networking.

Steve

BjornStevens · Sep 5, 2019, 3:16 PM

@stephenw10

Hi, Thanks for your reply,

I tried to find the /boot/loader.conf.local file but could only find a /boot/loader.conf
I tried adding it into there ( hw.pci.honor_msi_blacklist=0 ) but still no change.
It has done something because it moved up in the file.

During speedtest i get these results with vmstat -i:

And when using the top -S -H command still get the same results.

Any other suggestions?

Thanks!

kiokoman · Sep 5, 2019, 5:57 PM

you need to create the file
/boot/loader.conf.local
if it's missing
copy inside
hw.pci.honor_msi_blacklist=0
save and reboot

stephenw10 · Sep 6, 2019, 3:35 AM

Yup create the file if it doesn't exist. If you put it in loader.conf it may get overwritten.

However that will only do anything for vmx NICs. You have em NICs there currently.

Steve

BjornStevens · Sep 6, 2019, 8:55 AM

@stephenw10 Allright, will set them to VMXNET3, reboot, create the file with the line and inform if there are any changes.

Thanks for the help @kiokoman & @stephenw10 !

Creating config file:

BjornStevens · Sep 6, 2019, 9:19 AM

Okay so further testing will come in later but for now i seem to reach my maximum provider speed on my linux server behind the firewall:

BUT it did drop back down to 14.4Megabyte's per second and go up and down all the time:

Cpu usage seems to have set a bit:

Using SMB protocol i get this from moving a file WAN to LAN:

It's 2 virtual cores are running at nearly full power (cpu 6/7) (cpu 4 is being used on the server side in the LAN network.):

I don't know if this is just a performance bug but speeds seem to have increased, altough cpu usage is still high (compared to the hardware specifications of pfsense)

Changing to a quad core (virtual processor) did not change much either, cpu usage stays high on 2 cores:

Wish i could put my finger on the issue.

stephenw10 · Sep 6, 2019, 1:52 PM

I still only see one tx queue and one rx queue on each NIC. Does vmstat -i show more?

I assume you created that file in /boot

Steve

BjornStevens · Sep 6, 2019, 2:00 PM

@stephenw10

yep its placed under /boot/loader.conf.local

vmstat -i during speedtest on server in lan side:

BjornStevens · Sep 6, 2019, 2:04 PM

I actually don't know how to read the vmstat -i, but i hope you might know more @stephenw10

kiokoman · Sep 6, 2019, 2:10 PM

one queue

vmx0: tq0 (transmission queue 0)
vmx0: rq0 (receive queue 0)

with multiple queue you should see tq0 / tq1 etc etc

stephenw10 · Sep 6, 2019, 2:16 PM

Yeah, that. Though I don't have anything vmx to test again right now.
I think it probably is working as you are seeing the high numbered IRQs which MSI uses.
Try removing that line or commenting it out and rebooting. Do you see any change?

On other NICs you might see something like:

[2.4.4-RELEASE][root@5100.stevew.lan]/root: vmstat -i
interrupt                          total       rate
irq7: uart0                          432          0
irq16: sdhci_pci0                    536          0
cpu0:timer                      68688188       1001
cpu3:timer                       1069435         16
cpu2:timer                       1060293         15
cpu1:timer                       1086989         16
irq264: igb0:que 0                 68630          1
irq265: igb0:que 1                 68630          1
irq266: igb0:que 2                 68630          1
irq267: igb0:que 3                 68630          1
irq268: igb0:link                      3          0
irq269: igb1:que 0                 68630          1
irq270: igb1:que 1                 68630          1
irq271: igb1:que 2                 68630          1
irq272: igb1:que 3                 68630          1
irq273: igb1:link                      1          0
irq274: ahci0:ch0                   4473          0
irq290: xhci0                         85          0
irq291: ix0:q0                    216643          3
irq292: ix0:q1                     47933          1
irq293: ix0:q2                    325480          5
irq294: ix0:q3                    514752          7
irq295: ix0:link                       2          0
irq301: ix2:q0                     74629          1
irq302: ix2:q1                       507          0
irq303: ix2:q2                      1703          0
irq304: ix2:q3                     89446          1
irq305: ix2:link                       1          0
irq306: ix3:q0                     70295          1
irq307: ix3:q1                      4985          0
irq308: ix3:q2                    186433          3
irq309: ix3:q3                    413486          6
irq310: ix3:link                       1          0
Total                           74405771       1084

https://www.freebsd.org/cgi/man.cgi?query=vmx#MULTIPLE_QUEUES

Steve

kiokoman · Sep 6, 2019, 2:13 PM

try to add this on your loader.conf.local

hw.vmx.txnqueue="4"
hw.vmx.rxnqueue="4"

BjornStevens · Sep 6, 2019, 2:31 PM

@kiokoman & @stephenw10

I added the rule with
hw.vmx.txnqueue="4"
hw.vmx.rxnqueue="4"

I did not see any change whatsoever in vmstat -i:

and commenting out the first rule also did not change anything:

Edit:

Even when doing a download on a server in LAN and using top -S -H i have this outcome:

stephenw10 · Sep 6, 2019, 3:31 PM

You are seeing load on all CPUs there and none is at 100% so it's not CPU limited at that point.