Hardware and Expectations..
-
Yes, I agree. You have to combine your 1Gbps connections is the correct way and test the resulting combination correctly. Otherwise you need to step up to 10G and that's a serious jump in expense! ;)
Your test figures are interesting. 130MBps is above the theoretical maximum throughput for a single Gigabit connection so you must be seeing some link aggregation advantage. :-\
Are you using jumbo frames?
See: http://www.freebsd.org/doc/handbook/network-aggregation.html Though that applies directly to FreeBSD it's consepts are pretty universal.
Steve
-
To your questions:
LACP is set for Static Link Aggregation on all the NICS | The switch trunks are configured accordingly to each incoming LACP Team.
Jumbo Packets is on in the switch(es)/ Router and NICs all at max bytes. NICS @ 9014 Bytes / Switch(s) have Jumbo Frame enabled.
Some transfer screenshots all using the same file: A 7.79GB ISO file of 12 monkeys.
Server 2: Areca 1222 array to desktop (Revo 3 X2) –(This one is always done copying before the transfer calculations are finished)
Server 1 to Workstation - (What's odd is that this is on a single Cat6 line)
Server1 to Server 2
I realize that this is Server 2008R2's own file copy system. If you have a preferred one let me know and I'll rerun the tests.
Again, thank you for the knowledge and help.
-
iperf is a commonly used speed test tool.
To test LACP, you need to test from multiple clients to the server.
LACP only helps out when you've got more than one client.
A single file copy won't use more than one network connection. The 125MB/s speed you're seeing is ideal for a single gigabit connection.
-
Well I ran Lan Speed test against the servers from several machines at once across the network. I then duplicated that same file transfer, only consecutively with 2,3,4 then 5 machines. It held up until the third and fourth machines, dropping by about 10M-20MB/s per each additional machine. 5th one pretty much took it down to an oscillating 10-60MB/s.
So you guys were spot on. I just find it hard to accept that I can barely push 15% of the total gigabit. Seems like there should be more available somewhere, I was hoping that PFSense could help push a few more MB/s out of the system. Stupid thing is that now I've been staring at 10GigE and Infiniband products.
Again, thanks for all the assistance, you guys have been very helpful.
-
I just find it hard to accept that I can barely push 15% of the total gigabit.
I think you may have a misunderstanding there.
Your test result, ~125MB/s, is very close to Gigabit wire speed.125MB/s is 1000Mb/s. 1B (byte) = 8b (bits).
The absolute maximum would be 1024Mbps but that doesn't allow for various overheads.For a single file transfer I think you're seeing the best you could get over Gigabit wiring. :)
If you're fully aware of that then I apologise for patronising you. ::)
It's hard to interpret your latest results since I'm not sure if all that traffic was over the same set of teamed NICs. If you used multiple servers and multiple clients the traffic would all be going through the switch but not necessarily over the same cables. Assuming it was all transfered from one server:
With 4 clients each was seeing ~105MB/s. That would be ~3.3Gbps. Seems pretty good to me!Steve
-
Ah, nope, you're not patronizing me, thanks for the correction on my "in head" math. For some reason I had 3 bits to a byte stuck in my noodle. Guess age is slowly dulling the blade. I realize now that I'm doing quite a bit better than I thought I was.
But I'm still on the hunt for more speed, just don't know where it'll take me. .
-
But I'm still on the hunt for more speed, just don't know where it'll take me. .
If you find some nice solutions in the future for better speed, please share. I have had Gigabit network for almost 10 years at home, and i'm searching for a suitable solution to increase it - just have not found any with great value that fits all machines and so forth.
-
Indeed ~125 MB/sec is 1 Gbps wire speed. Short of upgrading to 10 Gb, you won't ever get more than that between two given endpoints. With LACP bonding to your server machine you could get upwards of 1 Gbps to multiple clients in aggregate. There isn't anything that truly bonds NICs into one big pipe though, LACP and similar bonding is MAC-balanced, a single src/dst MAC pair cannot get > 1 Gbps.
You're doing as well as you possibly can on 1 Gb, you'll have to upgrade to 10G if you want more than that.
-
That PFS install is only temporary to that machine, I re-purposed that box for this "experiment". Once I've achieved what I'm after then I'll chuck PFS over to a much smaller dedicated machine. I've been lurking on these forums for a couple of months now and a guy over in the VM forum mentioned a small rack server that would suit the purpose well: http://www.supermicro.com/products/system/1u/5017/sys-5017p-tln4f.cfm
If you want something cheaper: 5015A-EHF-D525 / SYS-5017A-EF, atom based,more than suitable for a home pfSense box, unless you're doing >300 - 400mbps of traffic and plan on running IDS.
-
@Kr^PacMan:
If you find some nice solutions in the future for better speed, please share. I have had Gigabit network for almost 10 years at home, and i'm searching for a suitable solution to increase it - just have not found any with great value that fits all machines and so forth.
There are two emerging high speed links on the horizon. I know their pretty much limited 1-to-1 connections for communication at this point.
Thunderbolt up'd to 20Gb/s: http://www.fudzilla.com/home/item/31012-intel-doubles-thunderbolt-speed
Superspeed USB @10GB/s: http://www.tomshardware.com/news/USB-IF-IDF-SuperSpeed-Thunderbolt-Power,21963.html
Now if you could just build some Switches around this tech then you could have a serious contender for 10GigE. But then I'd imagine that the networking hardware companies are already aware of this and perhaps we'll start seeing 10GigE making a push into the consumer market in the near future. One can hope…... ::)
Of course then there is this: Researchers Create 3 Gb/s LiFi network with LEDs: http://www.tomshardware.com/news/VLC-LiFi-LED,21894.html
-
@Kr^PacMan:
If you find some nice solutions in the future for better speed, please share. I have had Gigabit network for almost 10 years at home, and i'm searching for a suitable solution to increase it - just have not found any with great value that fits all machines and so forth.
There are two emerging high speed links on the horizon. I know their pretty much limited 1-to-1 connections for communication at this point.
Thunderbolt up'd to 20Gb/s: http://www.fudzilla.com/home/item/31012-intel-doubles-thunderbolt-speed
Superspeed USB @10GB/s: http://www.tomshardware.com/news/USB-IF-IDF-SuperSpeed-Thunderbolt-Power,21963.html
Now if you could just build some Switches around this tech then you could have a serious contender for 10GigE. But then I'd imagine that the networking hardware companies are already aware of this and perhaps we'll start seeing 10GigE making a push into the consumer market in the near future. One can hope…... ::)
Of course then there is this: Researchers Create 3 Gb/s LiFi network with LEDs: http://www.tomshardware.com/news/VLC-LiFi-LED,21894.html
USB would be pretty pratical, but i think technology-wise USB is rather useless for network traffic. Since USB is not full-duplex it's heavily limited in my opinion. Thunderbolt is a very nice protocol but i think it's too locked (By Apple and Intel) to be used in a general enviroment.
I have looked at CX4, but havent found anything yet.
-
@Kr^PacMan:
USB would be pretty pratical, but i think technology-wise USB is rather useless for network traffic. Since USB is not full-duplex it's heavily limited in my opinion. Thunderbolt is a very nice protocol but i think it's too locked (By Apple and Intel) to be used in a general enviroment.
I have looked at CX4, but havent found anything yet.
Thunderbolt is really useful as it allows you to hook up pretty much anything that interfaces by PCI-e. For example, you could hook up a pair of 10Gbe ports to a Mac Mini or an Intel NUC. I don't see it, or USB, being used natively for network traffic though, they just weren't designed to do that.
-
Well I ran Lan Speed test against the servers from several machines at once across the network. I then duplicated that same file transfer, only consecutively with 2,3,4 then 5 machines. It held up until the third and fourth machines, dropping by about 10M-20MB/s per each additional machine. 5th one pretty much took it down to an oscillating 10-60MB/s.
So you guys were spot on. I just find it hard to accept that I can barely push 15% of the total gigabit. Seems like there should be more available somewhere, I was hoping that PFSense could help push a few more MB/s out of the system. Stupid thing is that now I've been staring at 10GigE and Infiniband products.
Again, thanks for all the assistance, you guys have been very helpful.
http://www.zdnet.com/netgear-launches-affordable-10-gigabit-switches-for-smes-7000013507/
-
Interesting. However from my point of view affordable is not £675, for a home setup at least. I think Ill wait for affordable to become more affordable. ;)
Steve
-
"The new 24-port XSM7224 enterprise switch has a maximum power consumption of 200W"
power-frugal my ass !!
my Netgear 48-port gigabit managed switch is around 73-75W max with all 48 ports being used at max capacity…
-
Power consumption for 10Gbe is considerably higher than 1Gbe, just as 1Gbe is much higher than 100Mbit.
I agree that 200W for a 24-port isn't great though. My Dell PowerConnect 8132F switches are rated at a max of 176W for either 32-ports of 10Gbe or 24-ports of 10Gbe and 2 ports of 40Gbe (which is how mine are configured).
EDIT: Just noticed that those Netgear switches are twisted pair. That will raise the power consumption considerably. In that case, 200W for 24 ports is pretty good.
