Infiniband cards with pfSense 2.2
-
Ok, compiled the modules and now it's recognozied by pfSense and I can create an Interface on mlxen0! :)
Howto and ready-to-go modules are here:
http://www.routerperformance.net/howtos/compile-infiniband-modules-for-pfsense-2-2/
-
What sort of performance are you seeing?
It's likely the drivers will continue to function across a minor kernel change but by no means guaranteed. Also something outside the kernel might break their operation in some way. I built some very mildly modified drivers for FreeBSD 8.1 and they continued (surprisingly) to function in 8.3. Of course I might have just been lucky. ;)
Steve
-
@mir:
Your chances of success will greatly improve with X2 cards and older. X3 cards can be tricky even on Linux and Windows because you need a fairly new kernel and drivers supplied by Mellanox.
Just for the archive, tested a X2 card but with the default install the card was not detected.
-
What sort of performance are you seeing?
It's likely the drivers will continue to function across a minor kernel change but by no means guaranteed. Also something outside the kernel might break their operation in some way. I built some very mildly modified drivers for FreeBSD 8.1 and they continued (surprisingly) to function in 8.3. Of course I might have just been lucky. ;)
Steve
I'm a bit stuck here.
Created OPT1 on both sides with mlxen0. IPs were 10.99.0.10 and 11. I can ping each other and everything is accepted by firewall.
But when I want to connect with TCP and don't get any answer. Also tried with pfctl -d and lowering MSS or set hight MTU :( Same with UDP when i try with iperf.Any ideas?
-
Do you run a subnet manager either in software or in hardware (switch)?
-
No, they are direct attached, no opensm.
-
Even if you directly attach you will still need a subnet manager. Without subnet manager you will have no route. https://software.intel.com/en-us/articles/understanding-the-infiniband-subnet-manager
-
Hmpf .. isn't there a package for FreeBSD or will I have to compile by myself? Thought that direct attach doesn't need opensm
-
did you not build ofed?
http://lists.freebsd.org/pipermail/freebsd-infiniband/2013-February/000028.html -
Indeed is it not part of ofed?
Looks useful: http://really.zonky.org/?p=2927
Steve
-
Sorry, my (foolish) fault :D Just compiled the Mellanox FreeBSD drivers.
Ok, I followed http://really.zonky.org/?p=2927 and made the world on my FreeBSD 10.1 build system.
Then I packed:a usr/bin/opensm
a usr/bin/ibaddr
a usr/bin/ibnetdiscover
a usr/bin/ibping
a usr/bin/ibportstate
a usr/bin/ibroute
a usr/bin/ibsendtrap
a usr/bin/ibstat
a usr/bin/ibsysstat
a usr/bin/ibtracert
a usr/lib/libibcm.a
a usr/lib/libibcm.so
a usr/lib/libibcm.so.1
a usr/lib/libibcm_p.a
a usr/lib/libibcommon.a
a usr/lib/libibcommon.so
a usr/lib/libibcommon.so.1
a usr/lib/libibcommon_p.a
a usr/lib/libibmad.a
a usr/lib/libibmad.so
a usr/lib/libibmad.so.1
a usr/lib/libibmad_p.a
a usr/lib/libibsdp.a
a usr/lib/libibsdp.so
a usr/lib/libibsdp.so.1
a usr/lib/libibsdp_p.a
a usr/lib/libibumad.a
a usr/lib/libibumad.so
a usr/lib/libibumad.so.1
a usr/lib/libibumad_p.a
a usr/lib/libibverbs.a
a usr/lib/libibverbs.so
a usr/lib/libibverbs.so.1
a usr/lib/libibverbs_p.a
a usr/lib/libopensm.a
a usr/lib/libopensm.so
a usr/lib/libopensm.so.1
a usr/lib/libopensm_p.a
a usr/lib/libosmcomp.a
a usr/lib/libosmcomp.so
a usr/lib/libosmcomp.so.1
a usr/lib/libosmcomp_p.a
a usr/lib/libosmvendor.a
a usr/lib/libosmvendor.so
a usr/lib/libosmvendor.so.1
a usr/lib/libosmvendor_p.aextracted on my pfSense and started opensm. But now I get:
Feb 13 08:43:25 443034 [2006400] 0x80 -> OpenSM 3.3.1
Entering DISCOVERING stateFeb 13 08:43:25 443142 [2006400] 0x02 -> osm_vendor_init: 1000 pending umads specified
Feb 13 08:43:25 443215 [2006400] 0x80 -> Entering DISCOVERING state
Feb 13 08:43:25 443238 [2006400] 0x02 -> osm_vendor_bind: Binding to port 0xf65214fffe63a411
Feb 13 08:43:25 445759 [2006400] 0x01 -> osm_vendor_bind: ERR 5426: Unable to register class 129 version 1
Feb 13 08:43:25 445765 [2006400] 0x01 -> osm_sm_mad_ctrl_bind: ERR 3118: Vendor specific bind failed
Feb 13 08:43:25 445768 [2006400] 0x01 -> osm_sm_bind: ERR 2E10: SM MAD Controller bind failed (IB_ERROR)Error from osm_opensm_bind (0x2A)
Perhaps another instance of OpenSM is already running
Feb 13 08:43:25 445776 [2006400] 0x01 -> osm_sa_mad_ctrl_unbind: ERR 1A11: No previous bind
Exiting SMThere's a post around for missing files:
https://community.mellanox.com/thread/1364
Anyone know what files were missing (see list above)?
-
Have you followed this: https://wiki.freebsd.org/InfiniBand
You might find some tips and tricks here: http://web.mit.edu/freebsd/head/contrib/ofed/management/
-
btw. you have asserted that ps -ef |grep opensm gives zero hits?
-
No hits, yes.
I've followed this http://really.zonky.org/?p=2927
Compared to the wiki I did not add to MYKERNEL:
options SDP
device mthcaIs this really essential?
Wondering if I could download the OFED stuff from the link you posted and compile it to a prefix like /usr/local/ofed/ and copy the stuff to pfSense? (like on Linux)
-
… Thread Resurrection...
I'm building a home lab and looking at QDR infiniband within my rack and ethernet coming in from the outside (wan/lan)
I have the occassional need for very high bandwidth between my storage and my compute nodes.
Am I correct in understanding that pfSense can be compiled with IP over IB and can include a subnet manager?
Can pfSense concurrently route RDMA and IP over IB? Does this question even make sense?Would I be better off with a point to point connection between my storage node(s)/san and my compute nodes and/or adding a 4036 (4036E has a 10gb ethernet and subnet manager onboard but I'm poor) switch and effectively running both gigabit ethernet for ip traffic and QDR infiniband for NFS over RDMA?
Thanks,
Robert. -
Hello,
in normal Infiniband is used to build separate SAN networks that are connected over Infiniband NICs
that are attached to Infiniband switches to deliver around ~10 GBit/s or 40 GBit/s or 56 GBit/s or
attached directly from card to card. And this extra or separate network will be then connected to
the rest entire network over 10 or 40 GBit/s in usual. So why pfSense as a firewall should be
invited in this game? And why for so much money? A normal Mellanox SRx3 VPI card is able
to get for the cost of ~1300 € (dual Port NIC) and a switch for around ~6500 € and then this
construct will be able to delivers 56 GBit/s at each port! But there is no need of the firewall or
a separate router that is doing anything in this "game". -
@BlueKobold:
Hello,
in normal Infiniband is used to build separate SAN networks that are connected over Infiniband NICs
that are attached to Infiniband switches to deliver around ~10 GBit/s or 40 GBit/s or 56 GBit/s or
attached directly from card to card. And this extra or separate network will be then connected to
the rest entire network over 10 or 40 GBit/s in usual. So why pfSense as a firewall should be
invited in this game? And why for so much money? A normal Mellanox SRx3 VPI card is able
to get for the cost of ~1300 € (dual Port NIC) and a switch for around ~6500 € and then this
construct will be able to delivers 56 GBit/s at each port! But there is no need of the firewall or
a separate router that is doing anything in this "game".I am building a small compute network (single rack)
Kvm/Centos / dual xeon / 4 xeon phi - compute nodes
Kvm/Solaris / zfs mirrored / striped ssds - data server (San - napp-it)Remote VPN
Development VMs
LAMP, misc servers smtp, ftp, etc…Qdr infiniband - cheap, high bandwidth, low latency
Ib/nfs/rdmaIf pfsense can support ipoib and bridge wan to lan
then I don't have to buy a bridging switch in the short term. I will likely vm pfsense as well.A Voltaire 4036 is relatively inexpensive.
This is being done as a home basement project, low budget, incremental build out.... Data mining, machine learning, parallel programming, networking lab/sandbox, kaggle contests
Matlab, pysci, R, OpenMP, openacc, c++, Fortran, etc.
a few remote developers, analysts, VPN / ssh.
