Weird performance issue esxi6.5 multi-wan(pppoe) throughput multicore CPU

  • Hello Guys,

    I am running several years of pfSense with satisfy in esxi with vmxnet3 interfaces. Few days before 1Gbit FTTH(pppoe) was integrated into the box. I used 4 vCpu. Then I measure the speed (only the new 1Gbit gateway was active) and I was shocked to the results. I've got 50Mbit downstream. After hours diggin in various forums I realised something is wrong with the CPUs. I decrease the number 1 core only, and miracle happens. I've 500Mbit! But the processor is burning when throttling... After this I re-add additional cores, but the throughput decreases as much as more CPUs have been added. I know pppoe is a single threaded thing in pfSense, but I cant belive such bottlenecks....
    I tried hw offloading enable/disable, but not noticeable changes...
    Can someone please give any advice how to solve this annoying issue?


    Intel(R) Xeon(R) CPU E5506 @ 2.13GHz
    8GB ram
    2.4.4-RELEASE-p3 (amd64)
    3 WAN -1 DHCP -2 PPPoE

  • Netgate Administrator

    That is hard to explain. We will need more info.

    Try running top -aSH when you are testing to see how the load is distributed.

    Do you have hw.pci.honor_msi_blacklist=0 set? Without that vmx only uses one queue per NIC.


  • Thanks for the suggestion I've added the line into the tunables section. Might a bit better. Which is the best multi core strategy in pfSense? 1 socket more cores or separate sockets?

  • Netgate Administrator

    No significant difference really.

    There are some things that are still single threaded, like PPPoE. So fewer but faster cores can be help in some configurations.


Log in to reply