Very slow throughput - please help!!!

  • Hello all.

    Firstly - I'm new to pfsense, so please any ignorance I may display from time to time….

    I'm running pfsense on a HP Proliant DL360 G7 server running ESXi v4.1 and the throughput/download speed and upload speeds are horribly slow.

    If I connect a PC directly to the cable modem I get speeds of ~20Mbps.
    When connected via pfsense - speeds drop to a fraction ~5.5Mbps.

    PfSense info:
    2.3.2-RELEASE (amd64)
    built on Tue Jul 19 12:44:43 CDT 2016
    FreeBSD 10.3-RELEASE-p5

    When I first noticed this I turned off all logging, OpenVPN and removed any additional NAT settings etc... PfSense is running bare bones.
    Turning everything else off on the server also makes no difference at all.
    CPU on both server and pfsense show little if any CPU usage when running speed tests.
    Both NIC's are set to auto negotiate.

    Will updating EXSi help?

    Anyone have an issue like this before?
    Really need some help with this.
    Any idea what I could try?

    Thanks in advance...

  • Hi weza,

    I'll try to help.

    I'm running pfSense 2.2.3 on esxi 5.1 (no exactly the latest versions) on a 1gigabit up/down link and i'm getting ~118 MBytes/s (950 mbps/s) on ASUS q87T and Intel i-5 4570S. So i'd say virtualization is not the problem.

    • Updating esxi?

    Yes definitely, that's the first thing you should try, esxi 4 has been released in 2009, if you're using the free license version you should definitely try to upgrade to 5.1 or 5.5.

    No need to destroy your current setup yet you will have to make some hardware changes. First problem, vmfs, esxi 4 uses 3.x vmfs version where esxi 5.x uses vmfs5 but can access vmfs3. So if you're using direct attached storage check if you have some room for adding another hdd that you will be able to use as vmfs5 local storage for test purposes (so that you don't change anything on your current vmfs3 local datastores).

    You can also use NFS storage that's quick to setup and won't require you to play with your current local datastores (hey you never know what might happen). I strongly suggest you go the NFS way, disconnect your DAS vmfs3 (not required if you are careful), install esxi 5.5 on a USB stick so that you don't have to upgrade anything.

    Your test setup should be just a USB stick with 5.1 or 5.5 plugged in your server, all the remaining storage disconnected, a NFS (a FreeNas NFS share with anonymous access is just fine) to store your pfSense VM. pfSense doesn't need fast storage but give your pfSense vm 512MB RAM just to be sure.

    That's the less destructive way to check both if the culprit to your problem is the esxi version and show if you're hardware is correctly detected with esxi 5.5 in case you want to keep esxi 5.x after the tests. Of course this means that all your vms will be offline for an hour or two.

    If you're not using the PCIe slots (you should have 2x PCIe through riser card by default, PCI-X riser cards are shipped as an option on request) in your server I would add a DUAL Intel NIC because I think your server is using HP NC382i (Broadcom BCM5709C chipset) for the 4x onboard NICS. Intel NICS chipsets seem to be favored by everyone these days.

    That's all for the hardware part.

    Once your server has an esxi 5.x running on it just create a VM with 512MB, 1vcpu with 4 cores and 2x vmxnet3 nics. pfSense works fine and rarely needa any tweaking unless you're using some exotic hardware (realtek nic chipset, etc..) and this is usually done at hypervisor level, not a problem inside the VM and you should be able to get maximum throughput from your WAN connection. Don't use PCI passthrough, keep it simple.

    I know it's a lot of work but it's a simple work that can be done and undone fast compared to the other options.

    Good luck

  • LAYER 8 Global Moderator

    5.1 or 5.5 - go to 6u2 which is the current.. freebsd 10.x was not supported until 5.5u2

    For what possible reason would you not go to 6 the current release when its FREE??

  • He's probably used to the C# client, no need to add confusion with the new web interface and anything newer than esxi 4 will probably do for testing purposes anyway. But you're right about esxi 6: his server hardware is still fully supported with esxi6 so if the tests are successful he might want to go to version 6. It's up to him to decide.

  • LAYER 8 Global Moderator

    the c# client works just fine with 6.  I use it all the time.. Not sure where you got the idea that they dropped it?  You can not edit some of the fancy enterprise licensed anyway features, but if he is running free doesn't really matter, etc.

    As for testing purposes.. The min version he should use would be 5.5u2 since this is the first version that esxi officially supports freebsd 10.x

  • Thanks heaps for your help.

    I installed ESXi 6 on a flash drive and ran it up - it showed no difference in throughput.
    So I guess I'll try pickup a dual intel pci card and try again - will have to order from china so may take a while to get here.

    I have a few single port Alloy-8169v3 cards (reltek chipset) hanging around - I might try one and see what happens.

    Will post my results.

    Thanks again for your help.

  • Hmm buying new hardware might be the answer but what if it's not the problem? Time, money wasted. Let's try something else.

    As you are behind a router, would you mind trying to use one of your other VMs like a Windows one, if you have one of course, and run the speed test from that VM? Just connect the virtual nic of your Windows VM to the WAN vSwitch so that it gets an IP from your ISP router (basically reproducing the tests you ran with the physical PC directly to the cable modem).

    Run the following tests:

    1st test

    1. Windows VM connected to WAN vSwitch
    2. Run speed tests against the Internet

    2nd test:

    1. Windows VM connected to WAN vSwitch (you can run this test with VM on LAN vSwitch as well)
    2. Physical PC connected to cable modem (or on the same switch where your LAN vSwitch is connected)
    3. Disable firewall on both Windows VM and physical
    4. Try to transfer a movie (or something big) between the Windows VM and physical PC or run any other speed test software you are usually using between the 2 hosts

    Removing pfSense from the equation and test from Windows to Windows will let you know if it is pfSense related or if it's hypervisor NIC drivers related.

    If you get the same poor results between the Windows systems with firewall disabled then you will probably have to order Intel NICs as a replacement for your broadcoms. If the results show that transfers between Windows VM and Windows physical PC are normal and you get the usual speed you can expect from a 1Gigabit network then it will be the time to see why pfSense is slowing down your network speed.

    Last question. When you have installed esxi 6 did you used the normal ISO or the HP custom ISO?

    Just in case here are the links:

    esxi 5.5 u2

    esxi 6.0.0

  • LAYER 8 Global Moderator

    what vnic are you using in esxi for your pfsense vmx3 or e1000?  How exactly do you have everything connected?  Are you running on the same vswitch as vmkern for your vms?

    Are you testing from vms, or other devices your routing through your pfsense vm..  I run pfsense as vm on old HP N40L..  I route multiple segments, multiple vms through it to the internet and the rest of my network.  Now while going from physical machine to another physical machine across pfsense does take a hit.  I see 150-200ish mbps between network segments that are routed/firewalled over the pfsense vm.  so this would be

    device - realswitch - esxi nic - vswitch - vmnic (lan) - pfsense - (wlan) vmnic - different vswitch - different esxi nic - realswitch - different device

    When I test from computer on real physical network I see my max internet of 80/12 mbps..  So even this old dated hardware can route like 10x the speed of your internet, so what your running shouldn't have any issues at all.

    Just curious why would you have to order a intel nic from china?  You can get one on amazon and have it 2 days, etc.  Here are the nics I have in my esxi box and how they are connected.  How do you have yours setup?

  • I had the same problem there is a setting or something in pfsense causing the problem because i used my old backup config file from and old machine and now speeds are great through the firewall !

Log in to reply