Breaking the 6MB Barrier

dmoadab

Hi all,

I have 2 pipes coming into my Pfsense box (and old Barracuda box).
1 is 20/3 and the other is 75/5.
When I do speed tests, downloads, normal activity, all is well.
However, when trying to upload into a machines sitting behind the firewall from the outside world (2 server with 100mbps ports in 2 different DCs.), I cant seem to get passed 6megs.
This is happening when using Rsync, FTP and Openvpn.

I am open to any and all suggestions because after a month of trying all this, Im losing my mind.

Thank you!

timthetortoise

Are you saying that transfers stop after 6 MB or that you can't exceed 6 Mb/s?

dmoadab

I'm saying it tops off at 6Mb/s. :'(

dmoadab

Bump

chpalmer

Do you mean "while saturating your upload, your download is limited" ?

http://cable-dsl.navasgroup.com/#Asymmetry

SysIT

you only have 3Mb and 5Mb are the links bonded for upload, does pfsnse do bonding of links to combine the bandwidth?

what do the upload speeds show for your speed tests?

dmoadab

@SysIT:

you only have 3Mb and 5Mb are the links bonded for upload, does pfsense do bonding of links to combine the bandwidth?

what do the upload speeds show for your speed tests?

Ok, let me fill in some data here based on continued testing and a little recap.
Pfsense is sitting on a barracuda router (hardware) connected to 2 ISPs.
ISP1 has a 75/5Mb pipe on WAN.
ISP 2 has a 20/3Mb on Opt 1.
Machine 1 is sitting next to firewall with a 10/100 connection using Cat5 in Miami.
Server is sitting in a data center with a 100Mb connection in Los angeles.
There is a rule on Pfsense to allow a connection from Server 1 based on IP to Machine 1 on the Rsync port on WAN.
Server 1 runs rsync setup to connect to the external IP of WAN on pfsense.
The connection goes fine, starts at 17.74MB/s and then slowly (within 30 seconds) drops to 669.62kB/s and stays there.

While testing, I have tried this with an without a VPN connection.
I have also assumed Rsync was the issue so I tried a results Windows file system copy with similar results.

Speed test from Machine 1: http://www.speedtest.net/my-result/3114504233
Speed test from server: http://www.speedtest.net/my-result/3114488512

Hopefully that clarifies a bit and someone can help me find a solution.

Thanks much!

chpalmer

Can you answer the two questions above?

And did you read the link I provided?

dmoadab

I did indeed.
In terms of bonding, no, they are setup as failover.
In terms of symmetry, and this goes beyond my knowledge level, how is that affecting me when I'm essentially only downloading? Fro example, if I download something direct from http or ftp from a random place, I get the full 70+megs, but why am I getting the bottleneck on the upload (which is a download on my end)?

timthetortoise

Have you tested straightup disk throughput? dd if=/dev/zero of=~/test bs=512k count=1000 should give you a general idea of if your disk is fast enough to keep up. If disk is fine, check your CPU during transfers. If that's fine, check your memory. You're I/O-bound somewhere from the sound of it.

dmoadab

@timthetortoise:

Have you tested straightup disk throughput? dd if=/dev/zero of=~/test bs=512k count=1000 should give you a general idea of if your disk is fast enough to keep up. If disk is fine, check your CPU during transfers. If that's fine, check your memory. You're I/O-bound somewhere from the sound of it.

On the firewall?
If yes:
[2.1-RELEASE][root@local]/root(1): ~/test bs=512k count=1000
/root/test: Command not found.
[2.1-RELEASE][root@local]/root(2): dd if=/dev/zero of=~/test bs=512k count=1000
dd: ~/test: No such file or directory

timthetortoise

Oops, meant to say dd if=/dev/zero of=/root/test bs=512k count=1000
Do keep in mind that this will create a 512MB file, so if you don't have a lot of space you will want to alter your count argument.

dmoadab

@timthetortoise:

Oops, meant to say dd if=/dev/zero of=/root/test bs=512k count=1000
Do keep in mind that this will create a 512MB file, so if you don't have a lot of space you will want to alter your count argument.

\1000+0 records in
1000+0 records out
524288000 bytes transferred in 9.265605 secs (56584325 bytes/sec)

I ran this while doing a transfer.
Now, here's an interesting bit of info… I ran 3 rsyncs at the same time from the machine.
Each, hit the 760-780KB/s mark and sat there running around the same range.
This gave me about 24 Mb/s which now has me REAL confused because I think that would rule out any bandwidth or hardware issues.
It almost feels like something is limiting on a per "pipe" or connection basis.

dmoadab

Screen shot of system stats attached.

dmoadab

Bump. :-\

dmoadab

Buler…. Buler.... anyone?! :-\

stephenw10

@dmoadab:

Now, here's an interesting bit of info… I ran 3 rsyncs at the same time from the machine.
Each, hit the 760-780KB/s mark and sat there running around the same range.
This gave me about 24 Mb/s which now has me REAL confused because I think that would rule out any bandwidth or hardware issues.
It almost feels like something is limiting on a per "pipe" or connection basis.

Are you running any traffic shaping/QoS? Are you sure? Have you ever run traffic shaping?

Steve

dmoadab

@stephenw10:

Are you running any traffic shaping/QoS? Are you sure? Have you ever run traffic shaping?
Steve

Steve,

Just double checked traffic shaping, disabled (screen shot attached in case I'm wrong).
It may have been enabled at some point while playing around.
I'd be glad to delete anything related if I knew how. ???

Thank you for your help!

stephenw10

Just to be sure I'd look in the config.xml file and check you nothing in the <shaper>or <l7shaper>sections.

It's hard to imagine anything else that might limit you speed on a per connection basis.

Steve

Edit: reading through this again it looks like this could still be a limit on the remote machine. How did you test the connection speed with the 'Windows file system copy'?</l7shaper></shaper>

PertFlavus

This is probabably not entirely helpful but I had issues with encrypted file transfers being rate limited to 1mbps. Anything that went through openssl just couldn't pass that barrier. I could start up multiple instances, and they would both be 1mbps - but that I found was because openssl was single threaded and I had multiple cores to handle the transfer (I think..) Is your FTP transfer using SFTP?

mikeisfly

You didn't mention how the two sites were connected? Are you using a site to site VPN? If you are what kind are you using, IPSec or OpenVPN? In my experience using IPSec resulted in similar performance issues. When I switched to OpenVPN I saw almost not penalty in terms of encryption and decryption and was able to upload at almost full provisioned speeds. Never did figure out what the issue was could have been a CPU thing. I'm running a AMD Athlon X2 Dual Core Processor BE-2350.

dmoadab

When I say "Windows transfer" is when I have the 2 machines connected via openvpn. I literally open 2 explorer Windows and drag and drop.

I am not using Sftp.

I have tried this with both openvpn, and with allowing specific IPs connect to specific ports, all with the same results.

Here's my config file (attached), with some stuff asterisked out.

Thank you all again as this is driving me CRAZY!

config.txt

Supermule

How long is the ping between the 2 sites and do you have packetloss on the connection?

Use colasoft ping tool to monitor and post in here.

While you are monitoring ping, then load your connection with traffic. See if packetloss increases. pls. post results.

stephenw10

Have you in some way proved the connection to be good? Somehow removed pfSense from the link and tested?
Speedtest.net is not a good test because it uses multiple connections to maximise the throughput.
Try simply downloading a laarge file from a known good source. I have no idea where you are so I can't recommend one but I use the Thinkbroadband test files at http://www.thinkbroadband.com/download.html here in the UK. If it's still limited at your client machine you can then try downloading it directly to the pfSense box:

root@pfsense.fire.box]/root(2): fetch -o /dev/null http://download.thinkbroadband.com/50MB.zip
/dev/null                                     100% of   50 MB 1961 kBps 00m00s

I notice in your config file that you have some traffic shaping options:

 <ezshaper><step1><numberofconnections>1</numberofconnections></step1> 
		 <step3><enable>on</enable>
			<provider>Asterisk</provider>
			<connuploadspeed>%</connuploadspeed>
			<conndownloadspeed>%</conndownloadspeed>
			<connupload>30</connupload>
			<conndownload>30</conndownload>
			<download>300</download>
			<downloadspeed>Mb</downloadspeed>
			<conn0upload>300</conn0upload>
			<conn0uploadspeed>Mb</conn0uploadspeed></step3> 
		 <step4><step2><downloadscheduler>HFSC</downloadscheduler>
			<conn0uploadscheduler>HFSC</conn0uploadscheduler>
			<conn0upload>1</conn0upload>
			<conn0uploadspeed>Gb</conn0uploadspeed>
			<conn0download>1</conn0download>
			<conn0downloadspeed>Gb</conn0downloadspeed>
			<conn0interface>wan</conn0interface></step2></step4></ezshaper>

Are you running asterisk? Did you set these up intentionally? I'm no expert in traffic shaping (which seem like a bit of a black art!) and I can't see how this would be limiting you but still….

Steve