Slow throughput

rkelleyrtp · Feb 23, 2010, 4:53 PM

Can you please post a diagram of what your physical connectivity looks like? What sits between your PC testing machine on the WAN side and the LAN side? Any network switches? Are your testing boxes connected directly to the pfSense box?

jdejong · Feb 23, 2010, 4:59 PM

I will draw some up now. I have tried on the lan side to include a gb switch as well as direct connection to pfsense.

jdejong · Feb 23, 2010, 5:15 PM

Here you go. The pfSense box has a dual port Intel Pro/1000 64 bit PCI card (in a 32 bit slot). While I realize this will not yeild top numbers, it should do a heck of a lot better then it is. Also I am running these tests using part of my LAN as the WAN instead of putting this box in to my wan links (wanted to get the performance stuff licked before going any further).

Like I said before I have tried using a switch between the pfSense box and the laptop but that didn't make any difference in performance.

Thanks in advance.

rkelleyrtp · Feb 23, 2010, 6:34 PM

OK - the name of the game is isolation and elimination. Here is what I would do:

Setup base systems:
–-------------------------

Get three test machines, install your favorite OS (Linux, BSD, Win) on all 3
Using a pair of machines at a time, connect two test machines back-to-back in the same subnet to ensure all the NICs are working properly (system 1 <--> system 2, system-1 <--> system-3, system-2 <--> system-3)
Don't use any additional hardware (no network switches, etc)
Use a tool to measure download/upload performance (iPerf or use a web server on test machine 1 and download files to test machine 2)
Do this for all three machines until you have verified they all work properly
Get baseline traffic measured

Note: You mentioned your pfSense has two NICs. Use BOTH NICs in this test to ensure both ports are operating properly and at rated speeds

Test back-to-back with pfSense (see attached image):
–------------------------------------------------------------------

On machine one, perform a fresh install of pfSense 1.2.3-RELEASE (no additional packages, etc)
Configure the LAN and WAN ports on the pfSense box as necessary
Configure test box two with an ip address on the WAN side (don't change any OS stuff)
Configure test box three with an ip address on the LAN side (don't change any OS stuff)
Don't use any additional hardware (no network switches, use same cables, etc)
Use the same tool to measure download/upload performance

If you have problems with the back-to-back tests with pfSense, you have narrowed down the problem to the pfSense box:

Go into BIOS on pfSense box and disable any power-saving features (APIC, etc). Look for any adjustments to the PCI bus - re-run tests
Go into pfSense and disable any h/w offloading, r/x checksumming, h/w VLANs, etc - re-run tests
Find a completely different machine to run pfSense - re-run tests

Remember, the name of the game is isolation and elimination. Start with a good known and work from there. This could easily be a BIOS issue on your pfSense box and the PCI-X NIC, or it could be some incompatibility between your Dell network switch and the pfSense box (jumbo frames etc).

Let us know what you find...

jdejong · Feb 25, 2010, 11:05 PM

Worked out the issue. For some reason our current FW/Router was slowing down traffic. Our DHCP and Static zones are on seperate subnets, our existing FW is slow and was limiting the throughput. I put the pfsense box and my test server on the same subnet and voila everything worked fine. With a single dual port Pro/1000 card in a 64bit 66mhz slot i am getting about 260MBits/s with 8k window which is spot on. Your testing procedure made me realize what was happening so thanks for your help.

rkelleyrtp · Feb 25, 2010, 11:59 PM

Glad you got it sorted out!

calvinz · Apr 27, 2010, 3:53 PM

i'm having the same problem but still having problems tracing it out using the troubleshooting method (using iperf)

Here's the list of setups i've tried

Ubuntu -> Pfsense - 814Mb/s
Pfsense -> Ubuntu - 1Gb/s
Win2k8R2 -> Ubuntu - 727Mb/s
Win2k8R2 -> Pfsense - 460Mb/s
Win2k8R2 -> Win2k8R2 - 910Mb/s
Win2k8R2 -> FreeNAS - 502Mb/s
FreeNAS -> Pfsense - 293Mb/s
Pfsense -> FreeNAS - 400Mb/s

All of the devices above are equipped with gigabit ethernet interfaces with CAT6 cables.

calvinz · Apr 28, 2010, 11:22 AM

i have narrowed down the problem.. and it seems that the tcp window size on each of these machines are different.. i've tried to manually put in a fixed tcp window size on the windows machines but it doesn't commit to the changes no matter how :-\

rkelleyrtp · Apr 28, 2010, 12:19 PM

@calvinz:

i have narrowed down the problem.. and it seems that the tcp window size on each of these machines are different.. i've tried to manually put in a fixed tcp window size on the windows machines but it doesn't commit to the changes no matter how :-\

Have you tried the "DrTCP" utility from http://www.dslreports.com ?

calvinz · Apr 29, 2010, 6:59 AM

somehow the tweaks from DrTCP doesn't apply to Windows7/Windows Server 2008 R2.. :(

dreamslacker · May 7, 2010, 6:07 PM

Try running the following command in an elevated Command Prompt for Win7/ 2k8:

netsh int tcp set global rss=enabled autotuninglevel=experimental congestionprovider=ctcp netdma=enabled