Weird performance issue esxi6.5 multi-wan(pppoe) throughput multicore CPU

[Makkok]

Hello Guys,

I am running several years of pfSense with satisfy in esxi with vmxnet3 interfaces. Few days before 1Gbit FTTH(pppoe) was integrated into the box. I used 4 vCpu. Then I measure the speed (only the new 1Gbit gateway was active) and I was shocked to the results. I've got 50Mbit downstream. After hours diggin in various forums I realised something is wrong with the CPUs. I decrease the number 1 core only, and miracle happens. I've 500Mbit! But the processor is burning when throttling... After this I re-add additional cores, but the throughput decreases as much as more CPUs have been added. I know pppoe is a single threaded thing in pfSense, but I cant belive such bottlenecks....
I tried hw offloading enable/disable, but not noticeable changes...
Can someone please give any advice how to solve this annoying issue?

Thx!

Config:
Intel(R) Xeon(R) CPU E5506 @ 2.13GHz
8GB ram
2.4.4-RELEASE-p3 (amd64)
3 WAN -1 DHCP -2 PPPoE

[stephenw10]

That is hard to explain. We will need more info.

Try running top -aSH when you are testing to see how the load is distributed.

Do you have hw.pci.honor_msi_blacklist=0 set? Without that vmx only uses one queue per NIC.

Steve

[Makkok]

Thanks for the suggestion I've added the line into the tunables section. Might a bit better. Which is the best multi core strategy in pfSense? 1 socket more cores or separate sockets?

[stephenw10]

No significant difference really.

There are some things that are still single threaded, like PPPoE. So fewer but faster cores can be help in some configurations.

Steve