Sanity check for home network changes

jahonix

I'll be stacking an additional unit soon, giving me a total of four 10Gb SFP+ ports to tie the 3
machines to the server for large file transfers between them all.

How you do that?
~~Each switch has 2 SFP+ ports with one in each unit being used to stack them. Leaves 2 SFP+ port for connecting computers/servers.~~

Forget that, just found: Two dedicated rear stacking ports

BigDave

@johnpoz:

While I love the idea of 10ge network in the home.. What exactly is in your freenas that its going to be able to pump out data fast enough to justify a 10ge connection? You have some large amount of SSD storage? In a raid 0?

Last I looked even doing sfp and fiber is not cheap.. While I would love to do it myself just for the iperf test results and posting them on forums, etc.. ;) I just not as of yet justify the cost… But hey have fun with that.. Once you have it up and running would love to see the iperf test between box and your freenas, and a time copied of a say a 100GB file...

I have nothing special like striped SSDs, just six WD Red 3TB in RAIDz2 through a PCIe HBA card.
The bottle neck is the 1Gb LAN at the moment, but as solid state on PCIe comes down in price…
pausing here to dream of the future
Cheap is a relative term right? All the 10gig stuff is used equipment so it's not that bad.
I don't put a lot of stock in bench marks, but I'll post something you can drool over ;)
As always, thanks for the help. Y'all are great!

johnpoz

This is true, cheap is a relative term ;) And what your wife knows or allows you to spend on your hobbies/toys/tech/home network is also an issue….

Prob would of had a 10ge network long time ago if wouldn't get nagged forever on it cost how much ;) heheheeh

Got shit over the nest thermostat put in, only $250 -- but we already have a thermostat ;) etc...

Looking forward to those details and benchmarks to drool over...

David_W

I have two OM3 duplex fibres between the main home switch and the server room in a separate garage (with its own power feed, 42U rack, UPS and air conditioning). Both fibres are currently lit using 1000Base-SX and the two links are aggregated. 2Gbit/s duplex is sufficient for my purposes at the moment - I simply cannot justify the costs of switches with 10Gbit ports plus at least one pair of SFP+ modules to go with them. 48 port switches with 802.3at Power over Ethernet support are expensive enough, without throwing 10Gbit support into the mix. By the time 2Gbit/s duplex is inadequate for my usage, 10Gbit kit is likely to be much cheaper.

I wouldn't bother with fibre unless you need it for distance, galvanic isolation or noise immunity. In use, you can't tell the difference between data travelling over a copper or a fibre patch cable. If the server is close to a switch, 10GBase-T is an option - though make sure you meet the cabling specifications. Another option are dedicated SFP+ cables - they're cables with an SFP+ module on both ends.

If you have a sufficiently large switch, aggregating a couple of Gigabit ports is potentially worthwhile. Good quality one or two port Gigabit NICs aren't that expensive - even if you are buying decent server grade parts.

ZPrime

10GbaseT is actually more power hungry than a direct-attach TwinAX SFP+ (copper) cable.

The only reason to use 10GbaseT is if you don't have the clearance on the front of the device for twinax SFPs (they stick out a lot) or because you already ran the necessary Cat7 and have no other way to get 10Gb where you want it. Given the difficulties in CAT7 termination, it's often cheaper to use optical. :o

10GbaseT is a solution looking for a problem, in many cases. Most server rooms have either standardized on fiber already, or are just using twinax/direct connect since it's so much cheaper and eats less energy/generates less heat.

Guest

The switch was purchased for setting up 10Gb fiber between my home's FreeNAS server and
three desktop computers.

Ok perhaps we should or have to talk about DHCP here in this case at first.
We are talking here about how many devices? 6 I count in total. So why not setting up
a static IP address for them all and finish this thread as solved!

Connect the pfSense box over one GBit/s line to the switch and the Switch to the NAS
over 10 GBit/s with one DAC cable and at last the three desktop PCs get a static IP address
and all is running fine then for you.
pfSense 192.168.1.1/24
Switch 192.168.1.2/24
NAS 192.168.1.3/24
PC1 192.168.1.4/24
PC2 192.168.1.5/24
PC3 192.168.1.6/24

All is done or if you really need the DHCP server urgent, for what ever you think it must be there
you can go with the DHCP Realy function and using them both on the pfSense box and on the Switch.

BigDave

@johnpoz:

Looking forward to those details and benchmarks to drool over…

As promised, an iperf test…

Windows 7 desktop Samsung 840 Pro 256GB SSD with (Chelsio S310E-CR NIC) 50ft. OM3 LCLC MM fiber

iperf server and client connecting through the switch - Dell N2024 with two 10Gb SFP+ ports

FreeNAS box (intel X520-DA1) via Dell 1m SFP+ Copper Twinax


[root@freenas] ~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)

*********************************************************

C:\Users\Dave>iperf -c 192.168.1.13
------------------------------------------------------------
Client connecting to 192.168.1.13, TCP port 5001
TCP window size:  255 KByte (default)
------------------------------------------------------------
[160] local 192.168.1.17 port 52780 connected with 192.168.1.13 port 5001
[ ID] Interval       Transfer     Bandwidth
[160]  0.0-130.0 sec  10.8 GBytes   713 Mbits/sec
C:\Users\Dave>
------------------------------------------------------------

Not exactly drooling material I know, but we're working on it…

My desktop is gonna need a better NIC as the old Chelsio is PCI gen1 and a bit slow ;)

I have not purchased the second Dell switch yet, so I just have the one desktop connected at this point...

Realworld file transfer speed = 3.53 Gbps

BigDave

Ok perhaps we should or have to talk about DHCP here in this case at first.
We are talking here about how many devices? 6 I count in total. So why not setting up
a static IP address for them all and finish this thread as solved!

I have a total of 11 or 12 other CAT6 cables attached to the switch that go all over the house
for various things like printers, TVs, Bluray players, wireless AP and lots of spare ethernet jacks.
I found no way to mark the thread solved, what am I missing?

Connect the pfSense box over one GBit/s line to the switch

Yes

the Switch to the NAS over 10 GBit/s with one DAC cable

Yes sir.

and at last the three desktop PCs get a static IP address and all is running fine then for you.

After the second switch is stacked, this will be the way it gets done.

All is done or if you really need the DHCP server urgent, for what ever you think it must be there
you can go with the DHCP Realy function and using them both on the pfSense box and on the Switch.

I'm perfectly happy to let pfSense do the DHCP, no need to complicate anything.

johnpoz

"Not exactly drooling material I know, but we're working on it…"
"713 Mbits/sec"

Dude I get better than that from a virtual machine to my physical machine.. Using cheap nics, and old I5 dell workstation, with n40L hp running esxi where the vm sits.... Switch is a sg300.. Not sure how running 10g make any sense here if your not even showing over a gig with iperf..

How exactly are you seeing 3.3Gbps real world if your iperf doesn't even show gig??

Seems your iperf is old as well, might want to try out 3..

iperftestgig.png_thumb

BigDave

Ok, so I don't know how to use iperf, old or new >:(

'How exactly are you seeing 3.3Gbps real world if your iperf doesn't even show gig??"

realworld-speed.jpg_thumb

johnpoz

so why is iperf showing you crap then?

Windows file copy dialogs not known for their actuate transfer rates ;) But that is most likely the bottleneck of your disks… your running z2 right.. your in a 4+2 setup? So yeah that most likely max.. What are you writing it too on your pc? Maybe that is your limit to your SSD?

Now that you have 10gig your disks are too slow -- hehehee ;)

But yeah that is s nice fat number... Makes my 100+ look like crap ;) hehehe.. Then again mine is off simple drive pool running on a VM, with just gig cards..

If I had 10gig I would really be thinking of going SSD in some raid to be able to actually use that bandwidth...

robocopy.png_thumb

BigDave

@johnpoz:

so why is iperf showing you crap then?

Windows file copy dialogs not known for their actuate transfer rates ;)

I'll get to know iperf better, this is all new to me and I'm certain the TCP window size
(notice the different size between server and client) had something to do with the poor results.
My guess is once I learn more about iperf, the results will show somewhere north of 6Gb.
Right now, all I know is, I'm having fun AND it's much faster at transfering files AND those large
movie files (7 or 8GB) I watch on my iPad don't buffer anymore (that was a pleasant surprize). ;)
More to come.

johnpoz

I don't see how that could of been an issue, you plugging in your ipad? Its wifi even the latest pro model is only 866 PHY, so say 400mpbs over AC… So gig network would not of been a bottleneck..

BigDave

I finally learned to use iperf and posted a screen shot of the results from just now.
You may commence your drooling now ;D

iperf-jan20-2016.jpg_thumb

johnpoz

Oh yeah this has me drooling!!! F__K that would be sweet ;)

I may have to do this – heheeheh!!!

If you want the 3.1.1 version, I compile it all the time for windows both 32 and 64 bit.. Since nobody else seem to be doing it that I could find.. You can grab it here http://files.budman.pw/