Questions about 10 gbps nics
-
This could be a cheaper option
http://www.amazon.com/Comtech-AHA363PCIE0301G-Compression-Decompression-Accelerator/dp/B00NJ20I4Y -
Keljian,
I appreciate your permission to ping you privately, but as long as I'm not sharing trade secrets I'd just as soon have it be public. I learned a lot from reading other peoples' support threads, it seems selfish to hide mine from them.
I guess I need to restate my goals here with 10gbE. I'm trying to eliminate bottlenecks that significantly hurt. It may be that 10gbE doesn't help, I don't know.
Compression accelerator looks really interesting. I might have to look at that.
aes and hyperthreading: This was a really interesting read. The AES part is new to me, and the hyperthreading is something I've been puzzled about for awhile, not from not knowing what it was but rather why people are still so confused after all this time. It's two prefetch modules for a single core. Again, trying to eliminate bottlenecks.
It's too bad they didn't include atom c2000 processors in their benchmarks. I'm really curious. I guess I'll get some first-hand experience fairly soon anyway.
We seem to be on the same page right now.
Once I get to the point of max load on the VM hosts and NAS I don't care if there's network speed left over. I'm not chasing infinity here, I just want to kill bottlenecks. The 10gbps nics will be the last thing I install, guessing from where I sit now. I'm just trying to understand all the issues.
These e5 chips look incredibly interesting to me. I'm going to have to google some database performance benchmarks on them.
Thanks again for your time.
-
I don't know about your databases, but the ones I typically work with have lots of padding data, where a 20GB file will only have 1GB of actual data. Most of the time, using a fast compression is all that is needed. SQL2008, or was it 2008R2, and newer support compressed back-ups. They back up nearly as fast as an uncompressed backup.
-
Usually a 50gb backup for us translates to about 7 or 8 gb zipped. The ratio I think depends on how densely populated the rows are and for us a lot of columns have some sort of value in every row. The biggest zipped file I can recall was a little over 20 gb. We push our customers to prune their data but they hate doing it. Some have gone so far as to add disk storage several times to accommodate everything, which IMO is crazy.
Right now the backups are going to a 25/3 connection. Once I get my network together I'll upgrade to 200/45? Can't remember the upload speed right now, it's not as important. We never send back to the customer in a high pressure situation, it's always our direction.
Enterprise mssql will compress, but AFAIK the cheaper versions don't, and can't restore the zipped backups. For my home support I'm not forking over the price for a full enterprise database, that's real money. I can check again though, maybe they changed something. I'm getting the cheapest one I can get away with, our apps don't use the extra features anyway. But a lot of times our database winds up on an Enterprise server because that's what they're running.
Really in my scenario I think getting the backups, zipping/encrypting, transferring and decrypt/unzip is the lion's share of the bottleneck. The Internet speed is going to take care of part of that, the QuickAssist hardware on my end will take care of part of it, and hopefully I can get some sort of acceleration on their end.
After I get all that set up, I'll evaluate if adding 10gbE between the 'big three' makes a difference. The hardware will surely come with gigabit nics anyway, so I won't lose out by waiting.
Thanks.
-
Its worth looking at this board instead if you want to effectively use 10gbe http://www.supermicro.com/products/motherboard/Atom/X10/A1SRM-2758F.cfm
It has both a x4 and x8 PCIe slots. I use mine with a dual port 10gbe card and an additional i350 quad giving me a total of 8 1gbe NICS.
I work from home with large video and disk images (software development) and occasionally other developers and this gives me enough ports to firewall subnets effectively as well as 10gig throughput from my primary workstation (Macbook pro + Thunedrbolt>PCIe adapter) to the FreeNAS box. My 40TB FreeNAS archive can provide traditional platter based disk access at 500MB/s (10 * 4TB disks in RAIDZ2) which is fast enough to store and retrieve large images without inconvenience. The network itself is capable of much more (9.91gbps, 1MB/s).
This kind of stuff is far from plug and play and in my experience is an exercise is in balancing the bottlenecks between all of the components, i.e storage disks, server CPU, transports mechanism (CIFS vs NFS etc), switches, cables and network stack configuration (tune for latency or throughput). Plugging a 10gig card into a 4x slot is a compromise already but it won't be your biggest limiting factor. Theres also a huge difference with routing and bridging the interfaces obviously and introducing jumbo packets is likely a requirements due to the limited processor specs on those Atom boxes and that can cause further hassles.Edit: if you haven't bought your 10gig cards yet, Chelsio 5th gen cards are likely to see an increase in performance over Intels cheaper x520/x540 hardware)
-
10G: we ship Chelsio T5 on the c2758.
We may pick up the Intel x710.Everything else is crap.
We've enabled AES-NI/AES-GCM, more work to be done. Linux does 840Gb/s IPSec on c2758 platform. We do less, investigating.
We will enable QAT on this platform and faster THIS YEAR. C2758 should be good for 8Gb/s IPSec on c2758 with QAT.
We have hw coming this year that will do 6 x 10G with IPSec @ 60Gb/s with headroom.
Yes.I.Said.This.Year.
Many in this thread have zero clue. Half-duplex 10G? WTF, over?
-
OK and I see all of those require an 8-lane pcie-v3 slot. That's really what I needed to know. It would have been nice to get at least a single-lane 10gbE port into a switch or something when the time came, but I guess it is what it is.
Thanks.
-
@gonzopancho:
10G: we ship Chelsio T5 on the c2758.
We may pick up the Intel x710Many in this thread have zero clue. Half-duplex 10G? WTF, over?
I admit I made a mistake, the figures I quoted were based on the assumption that PCI-e was serial, it isn't. Would have been nice to have it corrected rather than being told I have no clue.. But whatever..
Per lane, PCI-e, in each direction (full duplex):
v1.x: 250 MB/s (2.5 GT/s)
v2.x: 500 MB/s (5 GT/s)
v3.0: 985 MB/s (8 GT/s)
v4.0: 1969 MB/s (16 GT/s) -
PCIe is serial it's just not over single communication medium like, say, 10base2 Ethernet.
I think Jim was pointing out that 10Gbit Ethernet is not half duplex unless presumably you've wired it very very wrong. ;) A simple misunderstanding.
In fact my earlier post was incorrect. I said 4 lanes of PCIe 2.0 would give you 16Gbps total but in fact that's in both directions. So a 4X slot could saturate a 10Gb Ethernet link in theory if nothing else throttles the data.Steve
-
OK for the record I'm pretty familiar with gigabit and lower ethernet, but not at certified network admin level. I've been "the guy" for years because I've typically worked at small companies and had an interest, while nobody else does.
10 gigabit is a whole different ballgame and that's why I started the thread. Everyone makes mistakes and I'm not holding any grudges or making judgements.
I have three use cases for 10gbps and it will be awhile before I implement any of them. They are:
-
A small number of hosts (probably 3) with 2-port 10gbps nics with direct interconnect.
-
The same number of hosts (maybe +1 in the case below) connected directly to a managed or smart switch which can handle routing and some sort of security directly.
-
My new but as yet unconfigured router which is a SuperMicro c2758 board with a single-port 10gbps nic to hook into the above switch.
All of the main systems will be VM hosts. Probably the router will be too, although the plan is to install that in several different ways to evaluate what Atom c2000 systems can do for other aspects of my network. So the NICs need to aware of virtualization optimizations.
I can see that the Chelsio nics would work for any of the three main systems.
For the router, if the 10gbps switch can handle VLANs and some fairly simple firewall rules between them, all I would need is to allow near-wire-speed gigabit VLAN traffic to hit the servers without the server-side nic or my router as the bottleneck.
I can see right off the bat that the board I have can't route at high speed between two NICs at 10gbps with the 4-lane pciev2 slot it has, and having it route the high speed traffic through a single port NIC is not reasonable. So really I'm just worried about high speed VPN performance plus routing with the 7 gigabit nics and a possible 10gbps nic.
So I'm still looking for a possible single-port NIC that can work with a 4-lane pciev2 slot which is good enough to do the job.
Thanks.
-
-
Interesting topic…. But what is QAT or where does it stand for?
-
Quick Assist Technology:
http://www.intel.com/content/www/us/en/io/quickassist-technology/quickassist-technology-developer.html -
I'm just a n00b but IMO if you're doing any sort of VPN without QAT hardware you're probably doing it wrong.
The software doesn't support it yet but it will, I'm guessing soon.
-
Ha! Well it depends if you need the throughput. I have an OpenVPN server running here at home to use for remote access and my hardware is way too old to support Quickassist. It's still fast enough to stream Dr Who to America though so that's fine (if you ask my sister!). Fast enough to secure my traffic when I'm using public wifi also.
Steve
-
I'm just a n00b but IMO if you're doing any sort of VPN without QAT hardware you're probably doing it wrong.
The software doesn't support it yet but it will, I'm guessing soon.
Aes-ni is more than enough for a good proportion of vpn use..
-
When did girls start watching Dr. Who?!!? I've never heard of such a thing.
Technically I don't "need" acceleration, but if you're buying hardware in anticipation of gigabit Internet and want a VPN which can even come close to that speed, you're going to need at least AES-NI.
I'm a bit too suspicious to put all my eggs in that one basket for encryption acceleration though, which is why I'm so excited about QAT. I also have a significant need for compression acceleration.
-
When did girls start watching Dr. Who?!!? I've never heard of such a thing.
When they started giving the role to actors like David Tennant and Matt Smith. ::)
Steve
-
PCIe is serial is just not over single communication medium like, say, 10base2 Ethernet.
I think Jim was pointing out that 10Gbit Ethernet is not half duplex unless presumably you've wired it very very wrong. ;) A simple misunderstanding.
In fact my earlier post was incorrect. I said 4 lanes of PCIe 2.0 would give you 16Gbps total but in fact that's in both directions. So a 4X slot could saturate a 10Gb Ethernet link in theory if nothing else throttles the data.Steve
I could be wrong, but I thought half duplex only worked with 10BaseT and 100BaseT networks. As soon as we got to 1000BaseT, if the connection isn't running in full duplex it, it isn't functioning at all.
Regardless, I find this thread to be a very interesting read.
-
Forget half duplex, what I was getting at was that you won't see the full bandwidth if you don't have the bandwidth over a PCI-e slot
-
I'm just a n00b but IMO if you're doing any sort of VPN without QAT hardware you're probably doing it wrong.
The software doesn't support it yet but it will, I'm guessing soon.
http://www.dumpaday.com/?attachment_id=58505
