Why does Gigabit throughput require such high end hardware?

razzfazz

@gonzopancho:

The datacenter will always be fiber (cross cabinet) or coax (in-cabinet).

Coax? Seriously? Color me skeptical…

Guest

OK, Twinax, which is just coax with two conductors instead of one.

Twinax is even good enough for 40Gbps to about 5m. How tall is your cabinet again?

Guest

@kejianshi:

Power isn't that big of an issue if you are not running lots of hardware in a data center or running off solar.

In either case, power is the biggest 'deal'.

Guest

@Supermule:

:D Thats why they put the new hub in Utah….Lots of sun all year round! :D

Only the sun can generate enough power unless they build a nuclear PP next to it #love #USA #freedom

It's actually not why the new NSA data center is in Utah. For that you can blame Senator Orrin Hatch, R-Utah, who is
the Chair of the Senate Finance Committee.

http://www.sltrib.com/sltrib/politics/56472070-90/utah-nsa-data-center.html.csp

Harvy66

@gonzopancho:

@Harvy66:

The max wattage on several 24 port 10Gb switches is in the 200-300 range, which is about the same as my *computer, my wife's *computer, my 24port 1Gb switch and firewall put together. Idle power isn't much better, typically only low double digit percentages less.

*Excluding monitors

Once we start seeing 10Gb fabbed in the 20nm or 15nm ranges, the power consumption should be dramatically reduced.

There is already talk about 2.5Gb and 5Gb standards. I guess these can easily done with current fabs and are able to get full rates at 100m over CAT5e and CAT6 respectively. 2.5Gb is supposed to be nearly identical power consumption as 1Gb.

2.5G is a backplane technology. The only IEEE standards activity in this area is a "study group", which is not a "working group".

The call for forming a working group has been voted down, repeatedly: http://www.ieee802.org/3/minutes/nov03/1103_CFI_2_5G_report.pdf,
but the parties concerned managed to pass a vote for a "study group" last November.

So good luck with that. Low-power 10GbaseT PHYs are already starting to appear, and many manufacturers are starting to "build in" 10Gbps on their SoC and motherboard designs. Once you have it on the desktop, it will start to appear in SMB and Enterprise. The datacenter will always be fiber (cross cabinet) or coax (in-cabinet).

Netgear sells an 8-port 10Gb switch that consumes 58.8W, max. How many computers did you have in your house again?

The low power 10Gb NICs are made on the 14nm process and are part of the SOC. All other non-SOC chipsets are still using 1-2 generation old fab processes, so 32nm or 45nm. Many years back, Intel said 10Gb won't be cheap or low power enough until 22nm. So until we start seeing 22nm chipsets or more SOCs using 22nm or smaller, we won't have 10Gb on general motherboards for cheap. Those new 40Gb NICs from Intel are on 28nm and are lower power than the x540 10Gb NICs that I assume are on 32nm. 4x faster than ~15% less power. And that's only 28nm. 22nm and 14nm are going to make 10Gb common.

Guest

… and you've completely ignored the PHY, which is where the real power consumption occurs.

10GBase-T components today require anywhere from 2 to 5 watts per port at each end of the cable –depending on the distance of the cable –while SFP+ requires approximately 0.7 watt, regardless of distance.

kejianshi

Sounds like we need to switch to optical only…

Supermule

Depends on where you get your power from….

You can easily get a sunpanel that can power your datacenter and thereby cut your spendings in half or more....

Optical has many weaknesses (IMHO).

Guest

@gonzopancho:

… and you've completely ignored the PHY, which is where the real power consumption occurs.

10GBase-T components today require anywhere from 2 to 5 watts per port at each end of the cable –depending on the distance of the cable –while SFP+ requires approximately 0.7 watt, regardless of distance.

For sure thats right but compared to the entire cost of the cards and SFP+ transceivers to the
throughput, it will then perhaps better to support more Mellanox ConnectX-3 Pro VPI adapters.
Then you will be able to get 10 GbE/40 GbE or 40GbE/56/GbE and this for a smaller budget.
Rangeley C2758 & ConnectX-3 VPI would be great to see some supprt for those cards in the future.

Guest

I don't see Melanox ConnectX-3 being significantly less expensive than Chelsio T5-based product.

For example:

Melanox ConnectX-3 dual-port 10G adapter $235:
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=1303&idcategory=6

Chelsio T520-SO-CR dual-port 10G adapter: $234:
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=2084&idcategory=6

Mellanox ConnectX-3 Dual-Port 40G Adapter $470:
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=1327&idcategory=6

Chelsio T580-SO-CR 40 Gigabit adapter: $399
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=2083&idcategory=6

And, of course:

Rangeley C2758 via Netgate $999:
http://store.netgate.com/Firewall/C2758.aspx

Chelsio T520-SO-CR via Netgate $245:
http://store.netgate.com/Chelsio/T520-SO-CR.aspx

Guest

@gonzopancho:

I don't see Melanox ConnectX-3 being significantly less expensive than Chelsio T5-based product.

For example:

Melanox ConnectX-3 dual-port 10G adapter $235:
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=1303&idcategory=6

Chelsio T520-SO-CR dual-port 10G adapter: $234:
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=2084&idcategory=6

Mellanox ConnectX-3 Dual-Port 40G Adapter $470:
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=1327&idcategory=6

Chelsio T580-SO-CR 40 Gigabit adapter: $399
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=2083&idcategory=6

Ahhh for sure I was not recognizing that Chelsio is also offering 40 GBit/s equipment!
So far and in short thats it, for sure you are right with the comparing, but before the European
pfSense Shop is opening I was thing to refer before about the new Intel Atom C2000 Boards
something, and I was finding up this thread here very exiting: A Rangeley Board Review and so I was triggered to research over the Mellanox cards and where finding this equipment right here: 12 Port Mellanox Switch &
Mellanox Dual Port 56 GBit/s card and this was bringing me to the point that it could perhaps being interesting to get support for.

And, of course:

Rangeley C2758 via Netgate $999:
http://store.netgate.com/Firewall/C2758.aspx

Chelsio T520-SO-CR via Netgate $245:
http://store.netgate.com/Chelsio/T520-SO-CR.aspx

For sure the "Rangeley" platform will be the right thing as I see it now also.
A little bit sad for the IPSec will not profit from AES-NI but ok I can life with that
for sure if the QuickAssist is speeding up then other things. Perhaps this is owed also
to the circumstances that I was having a looking eyes an a Supermicro platform at first!
Because there are boards with one PCIe slot for the Comtech AHA compression card I want
to use inside and adding much more RAM is also a point that was given, but wit three miniPCIe
slots for mSATA, Modem and WLAN and more GB lan ports it will be also very interesting for me
otherwise. For sure not so easy to come closer to a right decision these days.

So let me see what is going on if the pfSense shop in Europe is opened.

But for the detailed informations you draw spread over so many threads I will say thank you
at this time first, glad to here about those things more and more.

Guest

@BlueKobold:

@gonzopancho:

I don't see Melanox ConnectX-3 being significantly less expensive than Chelsio T5-based product.

For example:

Melanox ConnectX-3 dual-port 10G adapter $235:
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=1303&idcategory=6

Chelsio T520-SO-CR dual-port 10G adapter: $234:
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=2084&idcategory=6

Mellanox ConnectX-3 Dual-Port 40G Adapter $470:
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=1327&idcategory=6

Chelsio T580-SO-CR 40 Gigabit adapter: $399
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=2083&idcategory=6

Ahhh for sure I was not recognizing that Chelsio is also offering 40 GBit/s equipment!
So far and in short thats it, for sure you are right with the comparing, but before the European
pfSense Shop is opening I was thing to refer before about the new Intel Atom C2000 Boards
something, and I was finding up this thread here very exiting: A Rangeley Board Review and so I was triggered to research over the Mellanox cards and where finding this equipment right here: 12 Port Mellanox Switch &
Mellanox Dual Port 56 GBit/s card and this was bringing me to the point that it could perhaps being interesting to get support for.

And, of course:

Rangeley C2758 via Netgate $999:
http://store.netgate.com/Firewall/C2758.aspx

Chelsio T520-SO-CR via Netgate $245:
http://store.netgate.com/Chelsio/T520-SO-CR.aspx

For sure the "Rangeley" platform will be the right thing as I see it now also.
A little bit sad for the IPSec will not profit from AES-NI but ok I can life with that
for sure if the QuickAssist is speeding up then other things. Perhaps this is owed also
to the circumstances that I was having a looking eyes an a Supermicro platform at first!
Because there are boards with one PCIe slot for the Comtech AHA compression card I want
to use inside and adding much more RAM is also a point that was given, but wit three miniPCIe
slots for mSATA, Modem and WLAN and more GB lan ports it will be also very interesting for me
otherwise. For sure not so easy to come closer to a right decision these days.

So let me see what is going on if the pfSense shop in Europe is opened.

But for the detailed informations you draw spread over so many threads I will say thank you
at this time first, glad to here about those things more and more.

IPsec does profit from AES-NI, it's AES-CBC + HMAC-SHA1 that suffers. We're not done, either. For now, using AES-GCM with AES-NI will provide the largest gains. Again, we're not done here, but the current project is QAT.

I don't have as much interest in Mellanox as I do in Chelsio and the newer Intel 10G/40G parts.

Guest

I don't have as much interest in Mellanox as I do in Chelsio and the newer Intel 10G/40G parts.

For sure, after I was lead by your last comment that Chelsio is also having 40 Gbit/s adapters
it is really obsolete.

sujyo1

In MO… We have 10/1 MB ($40/m) now and cable co start to offer 100/20 MB ($110/m)...and here you guys talking GB!!$$$$

Supermule

Current offerings here in Denmark is 1Gbit/1Gbit for around 349DKK which is the equivalent of 49USD with current exchange rates….

Guest

@sujyo1:

In MO… We have 10/1 MB ($40/m) now and cable co start to offer 100/20 MB ($110/m)...and here you guys talking GB!!$$$$

You are talking about the WAN speed! And we where talking about the LAN speed!
This is quite different from each other, and the ConnectX 2 Series from Mellanox is supported
so it was only in my mind to find out if the new series will be perhaps also be supported, not
more and not less. And if we are talking about 40 GBit/s adapters this is more for the LAN
connection and not the WAN connection.

.and here you guys talking GB!!$$$$

pfSense is not only used in home networks and in corporate networks you have to deal
also with other throughput for the entire company to deliver a moderate speed to all
the clients, servers and SANs and there fore it would even be great, to be able not
to create a so called bottleneck inside your LAN.

edwardwong

@Supermule:

Current offerings here in Denmark is 1Gbit/1Gbit for around 349DKK which is the equivalent of 49USD with current exchange rates….

In HK 1G up/down is less than 30 USD/month, and there is vendor planning to work out a 10G residential plan.

Supermule

Thats freaking crazy! :D