Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Infiniband cards with pfSense 2.2

    Scheduled Pinned Locked Moved Hardware
    23 Posts 5 Posters 6.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator
      last edited by

      What sort of performance are you seeing?

      It's likely the drivers will continue to function across a minor kernel change but by no means guaranteed. Also something outside the kernel might break their operation in some way. I built some very mildly modified drivers for FreeBSD 8.1 and they continued (surprisingly) to function in 8.3. Of course I might have just been lucky.  ;)

      Steve

      1 Reply Last reply Reply Quote 0
      • W
        Wordo
        last edited by

        @mir:

        Your chances of success will greatly improve with X2 cards and older. X3 cards can be tricky even on Linux and Windows because you need a fairly new kernel and drivers supplied by Mellanox.

        Just for the archive, tested a X2 card but with the default install the card was not detected.

        1 Reply Last reply Reply Quote 0
        • W
          Wordo
          last edited by

          @stephenw10:

          What sort of performance are you seeing?

          It's likely the drivers will continue to function across a minor kernel change but by no means guaranteed. Also something outside the kernel might break their operation in some way. I built some very mildly modified drivers for FreeBSD 8.1 and they continued (surprisingly) to function in 8.3. Of course I might have just been lucky.  ;)

          Steve

          I'm a bit stuck here.
          Created OPT1 on both sides with mlxen0. IPs were 10.99.0.10 and 11. I can ping each other and everything is accepted by firewall.
          But when I want to connect with TCP and don't get any answer. Also tried with pfctl -d and lowering MSS or set hight MTU :( Same with UDP when i try with iperf.

          Any ideas?

          1 Reply Last reply Reply Quote 0
          • M
            mir
            last edited by

            Do you run a subnet manager either in software or in hardware (switch)?

            1 Reply Last reply Reply Quote 0
            • W
              Wordo
              last edited by

              No, they are direct attached, no opensm.

              1 Reply Last reply Reply Quote 0
              • M
                mir
                last edited by

                Even if you directly attach you will still need a subnet manager. Without subnet manager you will have no route. https://software.intel.com/en-us/articles/understanding-the-infiniband-subnet-manager

                1 Reply Last reply Reply Quote 0
                • W
                  Wordo
                  last edited by

                  Hmpf .. isn't there a package for FreeBSD or will I have to compile by myself? Thought that direct attach doesn't need opensm

                  1 Reply Last reply Reply Quote 0
                  • M
                    mir
                    last edited by

                    did you not build ofed?
                    http://lists.freebsd.org/pipermail/freebsd-infiniband/2013-February/000028.html

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      Indeed is it not part of ofed?

                      Looks useful: http://really.zonky.org/?p=2927

                      Steve

                      1 Reply Last reply Reply Quote 0
                      • W
                        Wordo
                        last edited by

                        Sorry, my (foolish) fault :D Just compiled the Mellanox FreeBSD drivers.
                        Ok, I followed http://really.zonky.org/?p=2927 and made the world on my FreeBSD 10.1 build system.
                        Then I packed:

                        a usr/bin/opensm
                        a usr/bin/ibaddr
                        a usr/bin/ibnetdiscover
                        a usr/bin/ibping
                        a usr/bin/ibportstate
                        a usr/bin/ibroute
                        a usr/bin/ibsendtrap
                        a usr/bin/ibstat
                        a usr/bin/ibsysstat
                        a usr/bin/ibtracert
                        a usr/lib/libibcm.a
                        a usr/lib/libibcm.so
                        a usr/lib/libibcm.so.1
                        a usr/lib/libibcm_p.a
                        a usr/lib/libibcommon.a
                        a usr/lib/libibcommon.so
                        a usr/lib/libibcommon.so.1
                        a usr/lib/libibcommon_p.a
                        a usr/lib/libibmad.a
                        a usr/lib/libibmad.so
                        a usr/lib/libibmad.so.1
                        a usr/lib/libibmad_p.a
                        a usr/lib/libibsdp.a
                        a usr/lib/libibsdp.so
                        a usr/lib/libibsdp.so.1
                        a usr/lib/libibsdp_p.a
                        a usr/lib/libibumad.a
                        a usr/lib/libibumad.so
                        a usr/lib/libibumad.so.1
                        a usr/lib/libibumad_p.a
                        a usr/lib/libibverbs.a
                        a usr/lib/libibverbs.so
                        a usr/lib/libibverbs.so.1
                        a usr/lib/libibverbs_p.a
                        a usr/lib/libopensm.a
                        a usr/lib/libopensm.so
                        a usr/lib/libopensm.so.1
                        a usr/lib/libopensm_p.a
                        a usr/lib/libosmcomp.a
                        a usr/lib/libosmcomp.so
                        a usr/lib/libosmcomp.so.1
                        a usr/lib/libosmcomp_p.a
                        a usr/lib/libosmvendor.a
                        a usr/lib/libosmvendor.so
                        a usr/lib/libosmvendor.so.1
                        a usr/lib/libosmvendor_p.a

                        extracted on my pfSense and started opensm. But now I get:

                        Feb 13 08:43:25 443034 [2006400] 0x80 -> OpenSM 3.3.1
                        Entering DISCOVERING state

                        Feb 13 08:43:25 443142 [2006400] 0x02 -> osm_vendor_init: 1000 pending umads specified
                        Feb 13 08:43:25 443215 [2006400] 0x80 -> Entering DISCOVERING state
                        Feb 13 08:43:25 443238 [2006400] 0x02 -> osm_vendor_bind: Binding to port 0xf65214fffe63a411
                        Feb 13 08:43:25 445759 [2006400] 0x01 -> osm_vendor_bind: ERR 5426: Unable to register class 129 version 1
                        Feb 13 08:43:25 445765 [2006400] 0x01 -> osm_sm_mad_ctrl_bind: ERR 3118: Vendor specific bind failed
                        Feb 13 08:43:25 445768 [2006400] 0x01 -> osm_sm_bind: ERR 2E10: SM MAD Controller bind failed (IB_ERROR)

                        Error from osm_opensm_bind (0x2A)
                        Perhaps another instance of OpenSM is already running
                        Feb 13 08:43:25 445776 [2006400] 0x01 -> osm_sa_mad_ctrl_unbind: ERR 1A11: No previous bind
                        Exiting SM

                        There's a post around for missing files:

                        https://community.mellanox.com/thread/1364

                        Anyone know what files were missing (see list above)?

                        1 Reply Last reply Reply Quote 0
                        • M
                          mir
                          last edited by

                          Have you followed this: https://wiki.freebsd.org/InfiniBand

                          You might find some tips and tricks here: http://web.mit.edu/freebsd/head/contrib/ofed/management/

                          1 Reply Last reply Reply Quote 0
                          • M
                            mir
                            last edited by

                            btw. you have asserted that ps -ef |grep opensm gives zero hits?

                            1 Reply Last reply Reply Quote 0
                            • W
                              Wordo
                              last edited by

                              No hits, yes.

                              I've followed this http://really.zonky.org/?p=2927

                              Compared to the wiki I did not add to MYKERNEL:
                              options        SDP 
                              device        mthca

                              Is this really essential?

                              Wondering if I could download the OFED stuff from the link you posted and compile it to a prefix like /usr/local/ofed/ and copy the stuff to pfSense? (like on Linux)

                              1 Reply Last reply Reply Quote 0
                              • R
                                RobertFontaine
                                last edited by

                                … Thread Resurrection...

                                I'm building a home lab and looking at QDR infiniband within my rack and ethernet coming in from the outside (wan/lan)

                                I have the occassional need for very high bandwidth between my storage and my compute nodes.

                                Am I correct in understanding that pfSense can be compiled with IP over IB and can include a subnet manager?
                                Can pfSense concurrently route RDMA and IP over IB?  Does this question even make sense?

                                Would I be better off with a point to point connection between my storage node(s)/san and my compute nodes and/or adding a 4036 (4036E has a 10gb ethernet and subnet manager onboard but I'm poor) switch and effectively running both gigabit ethernet for ip traffic and QDR infiniband for NFS over RDMA?

                                Thanks,
                                Robert.

                                1 Reply Last reply Reply Quote 0
                                • ?
                                  Guest
                                  last edited by

                                  Hello,

                                  in normal Infiniband is used to build separate SAN networks that are connected over Infiniband NICs
                                  that are attached to Infiniband switches to deliver around ~10 GBit/s or 40 GBit/s or 56 GBit/s or
                                  attached directly from card to card. And this extra or separate network will be then connected to
                                  the rest entire network over 10 or 40 GBit/s in usual. So why pfSense as a firewall should be
                                  invited in this game? And why for so much money? A normal Mellanox SRx3 VPI card is able
                                  to get for the cost of ~1300 € (dual Port NIC) and a switch for around ~6500 € and then this
                                  construct will be able to delivers 56 GBit/s at each port! But there is no need of the firewall or
                                  a separate router that is doing anything in this "game".

                                  1 Reply Last reply Reply Quote 0
                                  • R
                                    RobertFontaine
                                    last edited by

                                    @BlueKobold:

                                    Hello,

                                    in normal Infiniband is used to build separate SAN networks that are connected over Infiniband NICs
                                    that are attached to Infiniband switches to deliver around ~10 GBit/s or 40 GBit/s or 56 GBit/s or
                                    attached directly from card to card. And this extra or separate network will be then connected to
                                    the rest entire network over 10 or 40 GBit/s in usual. So why pfSense as a firewall should be
                                    invited in this game? And why for so much money? A normal Mellanox SRx3 VPI card is able
                                    to get for the cost of ~1300 € (dual Port NIC) and a switch for around ~6500 € and then this
                                    construct will be able to delivers 56 GBit/s at each port! But there is no need of the firewall or
                                    a separate router that is doing anything in this "game".

                                    I am building a small compute network (single rack) 
                                    Kvm/Centos / dual  xeon  / 4 xeon phi -    compute nodes
                                    Kvm/Solaris / zfs  mirrored / striped ssds -  data server (San -  napp-it)

                                    Remote VPN
                                    Development VMs
                                    LAMP,  misc servers smtp, ftp, etc…

                                    Qdr infiniband -  cheap,  high bandwidth,  low latency
                                    Ib/nfs/rdma

                                    If pfsense can support ipoib and bridge wan to lan
                                    then I don't have to buy a bridging switch in the short term.  I  will likely vm pfsense as well.

                                    A Voltaire 4036 is relatively  inexpensive.

                                    This is being done as a home basement  project,  low budget,  incremental build out....  Data mining,  machine learning,  parallel programming,  networking lab/sandbox, kaggle contests

                                    Matlab,  pysci,  R, OpenMP,  openacc,  c++,  Fortran,  etc.

                                    a few remote developers,  analysts,  VPN / ssh.

                                    1 Reply Last reply Reply Quote 0
                                    • First post
                                      Last post
                                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.