VMware ESXi 6.0 tuning for 2.5GBit/s throughput?



  • Hello everyone,

    a week ago I got myself a cheap root server to experiment with.
    Currently I am working on setting up pfSense on ESXi 6.0 to do NAT for a few virtual machines on the server.
    I got IPv4 and IPv6 working the way I want but the network speed isn't that great.

    The server has an Intel Atom C2750 processor (8x2.4GHz) with 16GB RAM and 2.5Gbit/s NICs.

    I assigned the pfSense VM 1 vCPU and 512MB RAM. For network adapters I used VMXNET 3 and installed Open-VM-Tools from package list.
    As a client I set up a Debian 7 VM with 1 vCPU and 256MB RAM. Network adapter is VMXNET 3 and Open-VM-Tools are installed.

    For testing the speed I used a server in the same datacenter with an Intel Atom C2350 (2x1,7GHz) with 4GB RAM and 2.5Gbit/s NICs.
    I created a sparse file with the size of 10GB to test the download speed of the VM with a simple wget.

    The best speed I can get with the pfSense VM doing NAT at the moment is 108MB/s.

    100%[===================================>] 10,737,418,240  108M/s   in 95s
    
    2016-04-05 18:33:21 (108 MB/s) - `/dev/null' saved [10737418240/10737418240]
    

    The CPU load of the pfSense VM was at 100% when running the transfer. So I decided to give it more vCPUs, which surprisingly resulted in a much lower transfer speed of 72.5MB/s.

    100%[===================================>] 10,737,418,240 72.5M/s   in 2m 19s
    
    2016-04-05 19:11:22 (73.6 MB/s) - `/dev/null' saved [10737418240/10737418240]
    

    If I connect the Debian VM directly without using pfSense for NAT I get nearly wire speed at 280MB/s.

    100%[===================================>] 10,737,418,240  280M/s   in 37s
    
    2016-04-05 20:34:33 (279 MB/s) - `/dev/null' saved [10737418240/10737418240]
    

    Any ideas what I could try to achive higher speeds when using pfSense?



  • Have you test changing VMXNET3 for E1000 for example?
    I saw other posts where there are some tweaks to NIC and pfsense values to achieve high perf.

    If you already resolved, please share!

    thx



  • I think I did but I don't know for sure if I tried changing the type to E1000.
    This server is running Debian Linux now because I couldn't get it to work.

    I'll buy another server this week to test a few things.
    Do you have any other suggestions what I could try?



  • e1000 would be slower, and is only 1 Gb, don't bother.

    A C2758 will barely do 2.5 Gb on bare metal, it definitely won't in a VM with 1 vCPU. It might not in a VM with all 8 cores as vCPUs, but that would at least get closer.



  • @Crunk_Bass:

    The CPU load of the pfSense VM was at 100% when running the transfer. So I decided to give it more vCPUs, which surprisingly resulted in a much lower transfer speed of 72.5MB/s.

    Bad idea to give more vCpu than VM actually needs.
    Also, I had VERY inconsistent speed measurements when running test directly from pfS box, like ~5 mB\s on it and whole 40 mB\s on machine behind pfS.