InfiniBand instead of 10Gbit Ethernet
-
Hello forum,
as far I see 10Gbit NICs are still to expensive.
But I see many HCA/HBA InfiniBand cards on eBay so cheap, that I can pick up some for tests.
Giving example: http://www.aravision.com.tw/products/infiniband/pdf_qlogic/QLE7140.pdf
this card supports IP over InfiniBand or Qlogic QLE6140-SP PCI-Express card, which costs about price of good standard Gigabit NIC. But this is HBA = Host Based Adapter. Will it work as NIC = Network Interface Card? How do I use it? How do configure it? Is it even supported in pfSense? Can it be used with direct-connected hosts? -
You're going to need a few things going for you. While that card may do IP over Infiniband, I don't see BSD support under it's list of Operating Environments. I don't think it's in the BSD HCL either. If you can get drivers and get it to work, that's only half the problem.
Even if you can get the card itself to work, what do you plan to do with this fast interface? 'Cause you're going to need a switch, right? Cables too. I have a feeling that's going to get pretty expensive pretty quick. And you're going to need Infiniband NICs in each end device as well, as I don't believe I've seen convergent Infiniband/Ethernet switches (not sure that's possible at all,) so all of your Infiniband traffic is confined to your Infiniband network. I hope you're not expecting pfSense to act as a switch for 10Gbx2 worth of traffic from your internal Infiniband network to your internal Ethernet network (which is not 10Gb anyway?.) I just don't see what advantage it's going to get you.
Where, if you go with 10Gb Ethernet, you can mix & match all sorts of stuff. While the cards aren't super cheap, you can save a bit by going with unpopulated SFP cards and use SFP to SFP twinax cables, assuming your 10Gb capable switch has open SFP ports (and some switches are picky about the twinax cables you use caughJunipercaugh) So, if you have a switch with 4 10Gb ports, get a single port 10Gb card for your PFsense box and your servers, you can still connect other servers, end devices, or edge switches to your 1Gb ports as well. While not cheap, exactly, it's still probably more useful than a pure Infiniband network.
Unless this was all a wishful thinking, academic style question.
-
Ok, so what about use virtualization platform to expose pfSense as VM to network (I mean: Infiniband network)? Then it should work.
However, I read somewhere that there is a problem with bridging interface in Linux, which simply doesn't work with Infiniband IP interface, so I'm going to try luck with Open vSwitch.
http://openvswitch.org/features/
So, I understood there is no cheap InfiniBand card for FreeBSD. And this is the answer.I would like to use InfiniBand interface for DRBD or other HA solution between 2 nodes only. I'm new to IB, however I see it's cheaper than 10Gbit Ethernet. Replication requirements are huge in clustered environment.
I was asking because of being interesting where it can be also used.More info for interested: http://infiniband.sourceforge.net/NW/IPoIB/index.htm
-
Ok, so what about use virtualization platform to expose pfSense as VM to network (I mean: Infiniband network)? Then it should work.
However, I read somewhere that there is a problem with bridging interface in Linux, which simply doesn't work with Infiniband IP interface, so I'm going to try luck with Open vSwitch.
http://openvswitch.org/features/
So, I understood there is no cheap InfiniBand card for FreeBSD. And this is the answer.I would like to use InfiniBand interface for DRBD or other HA solution between 2 nodes only. I'm new to IB, however I see it's cheaper than 10Gbit Ethernet. Replication requirements are huge in clustered environment.
I was asking because of being interesting where it can be also used.More info for interested: http://infiniband.sourceforge.net/NW/IPoIB/index.htm
You'll have to find a virtualization platform that supports IP Over Infiniband as a network interface. I would presume it's possible that VMWare server, or other virtualization platform that can run as an application in Linux might work with that IP over Infiniband project. Otherwise, there seems to be an IP over Infiniband driver for ESX, possibly ESXi, but it might only work with certain cards, or even just cards from a single manufacturer.
If this is for a workplace solution, I'm sure that while it's cheaper in a place where 10Gb may be a cost concern, there's also certainly an amount of hourly cost to take in to account as well. For something that might work and would not be commodity technology from a support perspective (if you weren't there, could a consultant come in and work on it, if needed), there usually needs to be a very compelling reason to use -very- unproven technology on anything that's depended upon.
What I'm getting at is that while you certainly could purchase the equipment and try it out, I probably would imagine that the cost savings justification might get vastly overshadowed by the soft costs; unless you have unlimited time to throw at the idea.
-
There is some Infiniband support in FreeBSD v9.0
But that will have to wait untill the code is ported if that will even happen to v8.3Cheap Infiniband cards ?
I think this… http://www.ebay.com/itm/180651215177?ssPageName=STRK:MEWAX:IT&_trksid=p3984.m1423.l2649
is as cheap its gonna get for a mainstream card !!!You can try the mellanox or myrcom cards as well, but if your looking to go for multiple connections you have to look for a switch as well
-
However, I do have to comment on Cisco products…
If you want support for their products your not gonna get it without a service contract which isn't free.I stick with HP purely for the fact that no matter how old it is they will try an help you and resolve the issue without any questions.
Hence why i would rather spend 2x the $$ on HP rather than Cisco. -
You may want to look into CX4 - it works for us and its cheaper than 10gig fiber.
-
You may want to look into CX4 - it works for us and its cheaper than 10gig fiber.
To a certain point.
Fiber is becoming cheaper and cheaper as the processes are mature enough for mass production with very minimal rejects.
Also CX4 is limited in cable length where are with the proper SFP/XFP you can reach 20+ KM !!!
But there is a downfall…
Switch prices !
CX4 IB Swtiches are affordable now ( sometimes you get lucky you can find them going for $100-200 - Bought a TopSpin 120 for $125! which is the SAME as Cisco SFS7000P and a few other switches from HP and Voltair ) however comparative FC switches sit in the $2K plus range! And on top of that you still have to buy the SFP/XFP cables or fiber.I will stick with CX4 cards and cables, as for me 10/20G is plenty for streaming video and data between servers.
Now if only development would start on 2.2 which i hear it will be based on FreeBSD 9.0 or 9.1 which has support for OFED stack just need a bit of work for IPoIB :)
Once that happens my servers will have dedicated IB connection to the TopSpin 120 and a Myrcom 10GBe card (if Supported ) for my main switch.I already started buying a bunch of CX4 cables...
2x 1M, 4x 2M, and 4x 3M and 2x 15M GORE CX4 cables
Also on my todo list is a new PCIx card as the one that was in my DL360 ( pf box ) isn't responding :s -
If you guys want in on this there's 2 ways :
1. - Wait for 2.2
2. - Pony up and throw down some cash for someone to code in the OFED package into the existing releases.I posted up a Bounty a while back and no one else chimed in offering to chip in.
My offer still stands and even re-posted the bounty for V2.1
The main question is how fast (bad) do you want this feature ??
