Infiniband cards with pfSense 2.2

Wordo

Cards get recognized with Debian 8, but I'll ask for an older one, shouldn't be a problem.
Will come back soon … thanks!

mir

It seems X3 should be working on FreeBSD 10: https://community.mellanox.com/thread/1195

Forgot to add:
To compile the drivers I would install a full FreeBSD 10 on another server, could be a VM in VmWare or Virtualbox, and do the compiling on this server. Then transfer the binary drivers to your pfSense box. Just remember to grab the same kernel version (FreeBSD 10.1-RELEASE-p4)

Wordo

Thanks! I've never compiled a FreeBSD kernel, only on Linux.
Any good link how to do that?

Also, what happens when pfSense get updated to -p5, will the driver be useless after it?

mir

The important part is 10.1 patch level should have no influence on the driver.

For building just follow my link. Header files should be part of a full install.

Load: kldload your_module
Unload: kldunload your_module
List: kldstat

Wordo

Ok, compiled the modules and now it's recognozied by pfSense and I can create an Interface on mlxen0! :)

Howto and ready-to-go modules are here:

http://www.routerperformance.net/howtos/compile-infiniband-modules-for-pfsense-2-2/

stephenw10

What sort of performance are you seeing?

It's likely the drivers will continue to function across a minor kernel change but by no means guaranteed. Also something outside the kernel might break their operation in some way. I built some very mildly modified drivers for FreeBSD 8.1 and they continued (surprisingly) to function in 8.3. Of course I might have just been lucky. ;)

Steve

Wordo

@mir:

Your chances of success will greatly improve with X2 cards and older. X3 cards can be tricky even on Linux and Windows because you need a fairly new kernel and drivers supplied by Mellanox.

Just for the archive, tested a X2 card but with the default install the card was not detected.

Wordo

@stephenw10:

What sort of performance are you seeing?

It's likely the drivers will continue to function across a minor kernel change but by no means guaranteed. Also something outside the kernel might break their operation in some way. I built some very mildly modified drivers for FreeBSD 8.1 and they continued (surprisingly) to function in 8.3. Of course I might have just been lucky. ;)

Steve

I'm a bit stuck here.
Created OPT1 on both sides with mlxen0. IPs were 10.99.0.10 and 11. I can ping each other and everything is accepted by firewall.
But when I want to connect with TCP and don't get any answer. Also tried with pfctl -d and lowering MSS or set hight MTU :( Same with UDP when i try with iperf.

Any ideas?

mir

Do you run a subnet manager either in software or in hardware (switch)?

Wordo

No, they are direct attached, no opensm.

mir

Even if you directly attach you will still need a subnet manager. Without subnet manager you will have no route. https://software.intel.com/en-us/articles/understanding-the-infiniband-subnet-manager

Wordo

Hmpf .. isn't there a package for FreeBSD or will I have to compile by myself? Thought that direct attach doesn't need opensm

mir

did you not build ofed?
http://lists.freebsd.org/pipermail/freebsd-infiniband/2013-February/000028.html

stephenw10

Indeed is it not part of ofed?

Looks useful: http://really.zonky.org/?p=2927

Steve

Wordo

Sorry, my (foolish) fault :D Just compiled the Mellanox FreeBSD drivers.
Ok, I followed http://really.zonky.org/?p=2927 and made the world on my FreeBSD 10.1 build system.
Then I packed:

a usr/bin/opensm
a usr/bin/ibaddr
a usr/bin/ibnetdiscover
a usr/bin/ibping
a usr/bin/ibportstate
a usr/bin/ibroute
a usr/bin/ibsendtrap
a usr/bin/ibstat
a usr/bin/ibsysstat
a usr/bin/ibtracert
a usr/lib/libibcm.a
a usr/lib/libibcm.so
a usr/lib/libibcm.so.1
a usr/lib/libibcm_p.a
a usr/lib/libibcommon.a
a usr/lib/libibcommon.so
a usr/lib/libibcommon.so.1
a usr/lib/libibcommon_p.a
a usr/lib/libibmad.a
a usr/lib/libibmad.so
a usr/lib/libibmad.so.1
a usr/lib/libibmad_p.a
a usr/lib/libibsdp.a
a usr/lib/libibsdp.so
a usr/lib/libibsdp.so.1
a usr/lib/libibsdp_p.a
a usr/lib/libibumad.a
a usr/lib/libibumad.so
a usr/lib/libibumad.so.1
a usr/lib/libibumad_p.a
a usr/lib/libibverbs.a
a usr/lib/libibverbs.so
a usr/lib/libibverbs.so.1
a usr/lib/libibverbs_p.a
a usr/lib/libopensm.a
a usr/lib/libopensm.so
a usr/lib/libopensm.so.1
a usr/lib/libopensm_p.a
a usr/lib/libosmcomp.a
a usr/lib/libosmcomp.so
a usr/lib/libosmcomp.so.1
a usr/lib/libosmcomp_p.a
a usr/lib/libosmvendor.a
a usr/lib/libosmvendor.so
a usr/lib/libosmvendor.so.1
a usr/lib/libosmvendor_p.a

extracted on my pfSense and started opensm. But now I get:

Feb 13 08:43:25 443034 [2006400] 0x80 -> OpenSM 3.3.1
Entering DISCOVERING state

Feb 13 08:43:25 443142 [2006400] 0x02 -> osm_vendor_init: 1000 pending umads specified
Feb 13 08:43:25 443215 [2006400] 0x80 -> Entering DISCOVERING state
Feb 13 08:43:25 443238 [2006400] 0x02 -> osm_vendor_bind: Binding to port 0xf65214fffe63a411
Feb 13 08:43:25 445759 [2006400] 0x01 -> osm_vendor_bind: ERR 5426: Unable to register class 129 version 1
Feb 13 08:43:25 445765 [2006400] 0x01 -> osm_sm_mad_ctrl_bind: ERR 3118: Vendor specific bind failed
Feb 13 08:43:25 445768 [2006400] 0x01 -> osm_sm_bind: ERR 2E10: SM MAD Controller bind failed (IB_ERROR)

Error from osm_opensm_bind (0x2A)
Perhaps another instance of OpenSM is already running
Feb 13 08:43:25 445776 [2006400] 0x01 -> osm_sa_mad_ctrl_unbind: ERR 1A11: No previous bind
Exiting SM

There's a post around for missing files:

https://community.mellanox.com/thread/1364

Anyone know what files were missing (see list above)?

mir

Have you followed this: https://wiki.freebsd.org/InfiniBand

You might find some tips and tricks here: http://web.mit.edu/freebsd/head/contrib/ofed/management/

mir

btw. you have asserted that ps -ef |grep opensm gives zero hits?

Wordo

No hits, yes.

I've followed this http://really.zonky.org/?p=2927

Compared to the wiki I did not add to MYKERNEL:
options SDP
device mthca

Is this really essential?

Wondering if I could download the OFED stuff from the link you posted and compile it to a prefix like /usr/local/ofed/ and copy the stuff to pfSense? (like on Linux)

RobertFontaine

… Thread Resurrection...

I'm building a home lab and looking at QDR infiniband within my rack and ethernet coming in from the outside (wan/lan)

I have the occassional need for very high bandwidth between my storage and my compute nodes.

Am I correct in understanding that pfSense can be compiled with IP over IB and can include a subnet manager?
Can pfSense concurrently route RDMA and IP over IB? Does this question even make sense?

Would I be better off with a point to point connection between my storage node(s)/san and my compute nodes and/or adding a 4036 (4036E has a 10gb ethernet and subnet manager onboard but I'm poor) switch and effectively running both gigabit ethernet for ip traffic and QDR infiniband for NFS over RDMA?

Thanks,
Robert.

Guest

Hello,

in normal Infiniband is used to build separate SAN networks that are connected over Infiniband NICs
that are attached to Infiniband switches to deliver around ~10 GBit/s or 40 GBit/s or 56 GBit/s or
attached directly from card to card. And this extra or separate network will be then connected to
the rest entire network over 10 or 40 GBit/s in usual. So why pfSense as a firewall should be
invited in this game? And why for so much money? A normal Mellanox SRx3 VPI card is able
to get for the cost of ~1300 € (dual Port NIC) and a switch for around ~6500 € and then this
construct will be able to delivers 56 GBit/s at each port! But there is no need of the firewall or
a separate router that is doing anything in this "game".