Breaking the 6MB Barrier

timthetortoise

Have you tested straightup disk throughput? dd if=/dev/zero of=~/test bs=512k count=1000 should give you a general idea of if your disk is fast enough to keep up. If disk is fine, check your CPU during transfers. If that's fine, check your memory. You're I/O-bound somewhere from the sound of it.

dmoadab

@timthetortoise:

Have you tested straightup disk throughput? dd if=/dev/zero of=~/test bs=512k count=1000 should give you a general idea of if your disk is fast enough to keep up. If disk is fine, check your CPU during transfers. If that's fine, check your memory. You're I/O-bound somewhere from the sound of it.

On the firewall?
If yes:
[2.1-RELEASE][root@local]/root(1): ~/test bs=512k count=1000
/root/test: Command not found.
[2.1-RELEASE][root@local]/root(2): dd if=/dev/zero of=~/test bs=512k count=1000
dd: ~/test: No such file or directory

timthetortoise

Oops, meant to say dd if=/dev/zero of=/root/test bs=512k count=1000
Do keep in mind that this will create a 512MB file, so if you don't have a lot of space you will want to alter your count argument.

dmoadab

@timthetortoise:

Oops, meant to say dd if=/dev/zero of=/root/test bs=512k count=1000
Do keep in mind that this will create a 512MB file, so if you don't have a lot of space you will want to alter your count argument.

\1000+0 records in
1000+0 records out
524288000 bytes transferred in 9.265605 secs (56584325 bytes/sec)

I ran this while doing a transfer.
Now, here's an interesting bit of info… I ran 3 rsyncs at the same time from the machine.
Each, hit the 760-780KB/s mark and sat there running around the same range.
This gave me about 24 Mb/s which now has me REAL confused because I think that would rule out any bandwidth or hardware issues.
It almost feels like something is limiting on a per "pipe" or connection basis.

dmoadab

Screen shot of system stats attached.

Capture.GIF_thumb

dmoadab

Bump. :-\

dmoadab

Buler…. Buler.... anyone?! :-\

stephenw10

@dmoadab:

Now, here's an interesting bit of info… I ran 3 rsyncs at the same time from the machine.
Each, hit the 760-780KB/s mark and sat there running around the same range.
This gave me about 24 Mb/s which now has me REAL confused because I think that would rule out any bandwidth or hardware issues.
It almost feels like something is limiting on a per "pipe" or connection basis.

Are you running any traffic shaping/QoS? Are you sure? Have you ever run traffic shaping?

Steve

dmoadab

@stephenw10:

Are you running any traffic shaping/QoS? Are you sure? Have you ever run traffic shaping?
Steve

Steve,

Just double checked traffic shaping, disabled (screen shot attached in case I'm wrong).
It may have been enabled at some point while playing around.
I'd be glad to delete anything related if I knew how. ???

Thank you for your help!

Capture.GIF_thumb

stephenw10

Just to be sure I'd look in the config.xml file and check you nothing in the <shaper>or <l7shaper>sections.

It's hard to imagine anything else that might limit you speed on a per connection basis.

Steve

Edit: reading through this again it looks like this could still be a limit on the remote machine. How did you test the connection speed with the 'Windows file system copy'?</l7shaper></shaper>

PertFlavus

This is probabably not entirely helpful but I had issues with encrypted file transfers being rate limited to 1mbps. Anything that went through openssl just couldn't pass that barrier. I could start up multiple instances, and they would both be 1mbps - but that I found was because openssl was single threaded and I had multiple cores to handle the transfer (I think..) Is your FTP transfer using SFTP?

mikeisfly

You didn't mention how the two sites were connected? Are you using a site to site VPN? If you are what kind are you using, IPSec or OpenVPN? In my experience using IPSec resulted in similar performance issues. When I switched to OpenVPN I saw almost not penalty in terms of encryption and decryption and was able to upload at almost full provisioned speeds. Never did figure out what the issue was could have been a CPU thing. I'm running a AMD Athlon X2 Dual Core Processor BE-2350.

dmoadab

When I say "Windows transfer" is when I have the 2 machines connected via openvpn. I literally open 2 explorer Windows and drag and drop.

I am not using Sftp.

I have tried this with both openvpn, and with allowing specific IPs connect to specific ports, all with the same results.

Here's my config file (attached), with some stuff asterisked out.

Thank you all again as this is driving me CRAZY!

config.txt

Supermule

How long is the ping between the 2 sites and do you have packetloss on the connection?

Use colasoft ping tool to monitor and post in here.

While you are monitoring ping, then load your connection with traffic. See if packetloss increases. pls. post results.

stephenw10

Have you in some way proved the connection to be good? Somehow removed pfSense from the link and tested?
Speedtest.net is not a good test because it uses multiple connections to maximise the throughput.
Try simply downloading a laarge file from a known good source. I have no idea where you are so I can't recommend one but I use the Thinkbroadband test files at http://www.thinkbroadband.com/download.html here in the UK. If it's still limited at your client machine you can then try downloading it directly to the pfSense box:

root@pfsense.fire.box]/root(2): fetch -o /dev/null http://download.thinkbroadband.com/50MB.zip
/dev/null                                     100% of   50 MB 1961 kBps 00m00s

I notice in your config file that you have some traffic shaping options:

 <ezshaper><step1><numberofconnections>1</numberofconnections></step1> 
		 <step3><enable>on</enable>
			<provider>Asterisk</provider>
			<connuploadspeed>%</connuploadspeed>
			<conndownloadspeed>%</conndownloadspeed>
			<connupload>30</connupload>
			<conndownload>30</conndownload>
			<download>300</download>
			<downloadspeed>Mb</downloadspeed>
			<conn0upload>300</conn0upload>
			<conn0uploadspeed>Mb</conn0uploadspeed></step3> 
		 <step4><step2><downloadscheduler>HFSC</downloadscheduler>
			<conn0uploadscheduler>HFSC</conn0uploadscheduler>
			<conn0upload>1</conn0upload>
			<conn0uploadspeed>Gb</conn0uploadspeed>
			<conn0download>1</conn0download>
			<conn0downloadspeed>Gb</conn0downloadspeed>
			<conn0interface>wan</conn0interface></step2></step4></ezshaper>

Are you running asterisk? Did you set these up intentionally? I'm no expert in traffic shaping (which seem like a bit of a black art!) and I can't see how this would be limiting you but still….

Steve