Weird performance issue esxi6.5 multi-wan(pppoe) throughput multicore CPU
I am running several years of pfSense with satisfy in esxi with vmxnet3 interfaces. Few days before 1Gbit FTTH(pppoe) was integrated into the box. I used 4 vCpu. Then I measure the speed (only the new 1Gbit gateway was active) and I was shocked to the results. I've got 50Mbit downstream. After hours diggin in various forums I realised something is wrong with the CPUs. I decrease the number 1 core only, and miracle happens. I've 500Mbit! But the processor is burning when throttling... After this I re-add additional cores, but the throughput decreases as much as more CPUs have been added. I know pppoe is a single threaded thing in pfSense, but I cant belive such bottlenecks....
I tried hw offloading enable/disable, but not noticeable changes...
Can someone please give any advice how to solve this annoying issue?
Intel(R) Xeon(R) CPU E5506 @ 2.13GHz
3 WAN -1 DHCP -2 PPPoE
That is hard to explain. We will need more info.
top -aSHwhen you are testing to see how the load is distributed.
Do you have
hw.pci.honor_msi_blacklist=0set? Without that vmx only uses one queue per NIC.
Thanks for the suggestion I've added the line into the tunables section. Might a bit better. Which is the best multi core strategy in pfSense? 1 socket more cores or separate sockets?
No significant difference really.
There are some things that are still single threaded, like PPPoE. So fewer but faster cores can be help in some configurations.