Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Infiniband cards with pfSense 2.2

    Scheduled Pinned Locked Moved Hardware
    23 Posts 5 Posters 6.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S Offline
      stephenw10 Netgate Administrator
      last edited by

      What sort of performance are you seeing?

      It's likely the drivers will continue to function across a minor kernel change but by no means guaranteed. Also something outside the kernel might break their operation in some way. I built some very mildly modified drivers for FreeBSD 8.1 and they continued (surprisingly) to function in 8.3. Of course I might have just been lucky.  ;)

      Steve

      1 Reply Last reply Reply Quote 0
      • W Offline
        Wordo
        last edited by

        @mir:

        Your chances of success will greatly improve with X2 cards and older. X3 cards can be tricky even on Linux and Windows because you need a fairly new kernel and drivers supplied by Mellanox.

        Just for the archive, tested a X2 card but with the default install the card was not detected.

        1 Reply Last reply Reply Quote 0
        • W Offline
          Wordo
          last edited by

          @stephenw10:

          What sort of performance are you seeing?

          It's likely the drivers will continue to function across a minor kernel change but by no means guaranteed. Also something outside the kernel might break their operation in some way. I built some very mildly modified drivers for FreeBSD 8.1 and they continued (surprisingly) to function in 8.3. Of course I might have just been lucky.  ;)

          Steve

          I'm a bit stuck here.
          Created OPT1 on both sides with mlxen0. IPs were 10.99.0.10 and 11. I can ping each other and everything is accepted by firewall.
          But when I want to connect with TCP and don't get any answer. Also tried with pfctl -d and lowering MSS or set hight MTU :( Same with UDP when i try with iperf.

          Any ideas?

          1 Reply Last reply Reply Quote 0
          • M Offline
            mir
            last edited by

            Do you run a subnet manager either in software or in hardware (switch)?

            1 Reply Last reply Reply Quote 0
            • W Offline
              Wordo
              last edited by

              No, they are direct attached, no opensm.

              1 Reply Last reply Reply Quote 0
              • M Offline
                mir
                last edited by

                Even if you directly attach you will still need a subnet manager. Without subnet manager you will have no route. https://software.intel.com/en-us/articles/understanding-the-infiniband-subnet-manager

                1 Reply Last reply Reply Quote 0
                • W Offline
                  Wordo
                  last edited by

                  Hmpf .. isn't there a package for FreeBSD or will I have to compile by myself? Thought that direct attach doesn't need opensm

                  1 Reply Last reply Reply Quote 0
                  • M Offline
                    mir
                    last edited by

                    did you not build ofed?
                    http://lists.freebsd.org/pipermail/freebsd-infiniband/2013-February/000028.html

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S Offline
                      stephenw10 Netgate Administrator
                      last edited by

                      Indeed is it not part of ofed?

                      Looks useful: http://really.zonky.org/?p=2927

                      Steve

                      1 Reply Last reply Reply Quote 0
                      • W Offline
                        Wordo
                        last edited by

                        Sorry, my (foolish) fault :D Just compiled the Mellanox FreeBSD drivers.
                        Ok, I followed http://really.zonky.org/?p=2927 and made the world on my FreeBSD 10.1 build system.
                        Then I packed:

                        a usr/bin/opensm
                        a usr/bin/ibaddr
                        a usr/bin/ibnetdiscover
                        a usr/bin/ibping
                        a usr/bin/ibportstate
                        a usr/bin/ibroute
                        a usr/bin/ibsendtrap
                        a usr/bin/ibstat
                        a usr/bin/ibsysstat
                        a usr/bin/ibtracert
                        a usr/lib/libibcm.a
                        a usr/lib/libibcm.so
                        a usr/lib/libibcm.so.1
                        a usr/lib/libibcm_p.a
                        a usr/lib/libibcommon.a
                        a usr/lib/libibcommon.so
                        a usr/lib/libibcommon.so.1
                        a usr/lib/libibcommon_p.a
                        a usr/lib/libibmad.a
                        a usr/lib/libibmad.so
                        a usr/lib/libibmad.so.1
                        a usr/lib/libibmad_p.a
                        a usr/lib/libibsdp.a
                        a usr/lib/libibsdp.so
                        a usr/lib/libibsdp.so.1
                        a usr/lib/libibsdp_p.a
                        a usr/lib/libibumad.a
                        a usr/lib/libibumad.so
                        a usr/lib/libibumad.so.1
                        a usr/lib/libibumad_p.a
                        a usr/lib/libibverbs.a
                        a usr/lib/libibverbs.so
                        a usr/lib/libibverbs.so.1
                        a usr/lib/libibverbs_p.a
                        a usr/lib/libopensm.a
                        a usr/lib/libopensm.so
                        a usr/lib/libopensm.so.1
                        a usr/lib/libopensm_p.a
                        a usr/lib/libosmcomp.a
                        a usr/lib/libosmcomp.so
                        a usr/lib/libosmcomp.so.1
                        a usr/lib/libosmcomp_p.a
                        a usr/lib/libosmvendor.a
                        a usr/lib/libosmvendor.so
                        a usr/lib/libosmvendor.so.1
                        a usr/lib/libosmvendor_p.a

                        extracted on my pfSense and started opensm. But now I get:

                        Feb 13 08:43:25 443034 [2006400] 0x80 -> OpenSM 3.3.1
                        Entering DISCOVERING state

                        Feb 13 08:43:25 443142 [2006400] 0x02 -> osm_vendor_init: 1000 pending umads specified
                        Feb 13 08:43:25 443215 [2006400] 0x80 -> Entering DISCOVERING state
                        Feb 13 08:43:25 443238 [2006400] 0x02 -> osm_vendor_bind: Binding to port 0xf65214fffe63a411
                        Feb 13 08:43:25 445759 [2006400] 0x01 -> osm_vendor_bind: ERR 5426: Unable to register class 129 version 1
                        Feb 13 08:43:25 445765 [2006400] 0x01 -> osm_sm_mad_ctrl_bind: ERR 3118: Vendor specific bind failed
                        Feb 13 08:43:25 445768 [2006400] 0x01 -> osm_sm_bind: ERR 2E10: SM MAD Controller bind failed (IB_ERROR)

                        Error from osm_opensm_bind (0x2A)
                        Perhaps another instance of OpenSM is already running
                        Feb 13 08:43:25 445776 [2006400] 0x01 -> osm_sa_mad_ctrl_unbind: ERR 1A11: No previous bind
                        Exiting SM

                        There's a post around for missing files:

                        https://community.mellanox.com/thread/1364

                        Anyone know what files were missing (see list above)?

                        1 Reply Last reply Reply Quote 0
                        • M Offline
                          mir
                          last edited by

                          Have you followed this: https://wiki.freebsd.org/InfiniBand

                          You might find some tips and tricks here: http://web.mit.edu/freebsd/head/contrib/ofed/management/

                          1 Reply Last reply Reply Quote 0
                          • M Offline
                            mir
                            last edited by

                            btw. you have asserted that ps -ef |grep opensm gives zero hits?

                            1 Reply Last reply Reply Quote 0
                            • W Offline
                              Wordo
                              last edited by

                              No hits, yes.

                              I've followed this http://really.zonky.org/?p=2927

                              Compared to the wiki I did not add to MYKERNEL:
                              options        SDP 
                              device        mthca

                              Is this really essential?

                              Wondering if I could download the OFED stuff from the link you posted and compile it to a prefix like /usr/local/ofed/ and copy the stuff to pfSense? (like on Linux)

                              1 Reply Last reply Reply Quote 0
                              • R Offline
                                RobertFontaine
                                last edited by

                                … Thread Resurrection...

                                I'm building a home lab and looking at QDR infiniband within my rack and ethernet coming in from the outside (wan/lan)

                                I have the occassional need for very high bandwidth between my storage and my compute nodes.

                                Am I correct in understanding that pfSense can be compiled with IP over IB and can include a subnet manager?
                                Can pfSense concurrently route RDMA and IP over IB?  Does this question even make sense?

                                Would I be better off with a point to point connection between my storage node(s)/san and my compute nodes and/or adding a 4036 (4036E has a 10gb ethernet and subnet manager onboard but I'm poor) switch and effectively running both gigabit ethernet for ip traffic and QDR infiniband for NFS over RDMA?

                                Thanks,
                                Robert.

                                1 Reply Last reply Reply Quote 0
                                • ? This user is from outside of this forum
                                  Guest
                                  last edited by

                                  Hello,

                                  in normal Infiniband is used to build separate SAN networks that are connected over Infiniband NICs
                                  that are attached to Infiniband switches to deliver around ~10 GBit/s or 40 GBit/s or 56 GBit/s or
                                  attached directly from card to card. And this extra or separate network will be then connected to
                                  the rest entire network over 10 or 40 GBit/s in usual. So why pfSense as a firewall should be
                                  invited in this game? And why for so much money? A normal Mellanox SRx3 VPI card is able
                                  to get for the cost of ~1300 € (dual Port NIC) and a switch for around ~6500 € and then this
                                  construct will be able to delivers 56 GBit/s at each port! But there is no need of the firewall or
                                  a separate router that is doing anything in this "game".

                                  1 Reply Last reply Reply Quote 0
                                  • R Offline
                                    RobertFontaine
                                    last edited by

                                    @BlueKobold:

                                    Hello,

                                    in normal Infiniband is used to build separate SAN networks that are connected over Infiniband NICs
                                    that are attached to Infiniband switches to deliver around ~10 GBit/s or 40 GBit/s or 56 GBit/s or
                                    attached directly from card to card. And this extra or separate network will be then connected to
                                    the rest entire network over 10 or 40 GBit/s in usual. So why pfSense as a firewall should be
                                    invited in this game? And why for so much money? A normal Mellanox SRx3 VPI card is able
                                    to get for the cost of ~1300 € (dual Port NIC) and a switch for around ~6500 € and then this
                                    construct will be able to delivers 56 GBit/s at each port! But there is no need of the firewall or
                                    a separate router that is doing anything in this "game".

                                    I am building a small compute network (single rack) 
                                    Kvm/Centos / dual  xeon  / 4 xeon phi -    compute nodes
                                    Kvm/Solaris / zfs  mirrored / striped ssds -  data server (San -  napp-it)

                                    Remote VPN
                                    Development VMs
                                    LAMP,  misc servers smtp, ftp, etc…

                                    Qdr infiniband -  cheap,  high bandwidth,  low latency
                                    Ib/nfs/rdma

                                    If pfsense can support ipoib and bridge wan to lan
                                    then I don't have to buy a bridging switch in the short term.  I  will likely vm pfsense as well.

                                    A Voltaire 4036 is relatively  inexpensive.

                                    This is being done as a home basement  project,  low budget,  incremental build out....  Data mining,  machine learning,  parallel programming,  networking lab/sandbox, kaggle contests

                                    Matlab,  pysci,  R, OpenMP,  openacc,  c++,  Fortran,  etc.

                                    a few remote developers,  analysts,  VPN / ssh.

                                    1 Reply Last reply Reply Quote 0
                                    • First post
                                      Last post
                                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.