Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Infiniband cards with pfSense 2.2

    Scheduled Pinned Locked Moved Hardware
    23 Posts 5 Posters 6.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • W Offline
      Wordo
      last edited by

      Hey guys,

      searched around the forums for Infiniband compability, only with answers "wait for version 2.2". Now there's 2.2 out and I've plugged in a Connect X3 card from Mellanox, did a fresh install but it won't be recognized. Nothing to see in dmesg, and also pciconf is very quiet :(

      There's a FreeBSD driver available, but I need Kernelfiles to compile them by myself.
      Any ideas how to do that?

      Thanks!

      1 Reply Last reply Reply Quote 0
      • M Offline
        mir
        last edited by

        Your chances of success will greatly improve with X2 cards and older. X3 cards can be tricky even on Linux and Windows because you need a fairly new kernel and drivers supplied by Mellanox.

        1 Reply Last reply Reply Quote 0
        • W Offline
          Wordo
          last edited by

          Cards get recognized with Debian 8, but I'll ask for an older one, shouldn't be a problem.
          Will come back soon … thanks!

          1 Reply Last reply Reply Quote 0
          • M Offline
            mir
            last edited by

            It seems X3 should be working on FreeBSD 10: https://community.mellanox.com/thread/1195

            Forgot to add:
            To compile the drivers I would install a full FreeBSD 10 on another server, could be a VM in VmWare or Virtualbox, and do the compiling on this server. Then transfer the binary drivers to your pfSense box. Just remember to grab the same kernel version (FreeBSD 10.1-RELEASE-p4)

            1 Reply Last reply Reply Quote 0
            • W Offline
              Wordo
              last edited by

              Thanks! I've never compiled a FreeBSD kernel, only on Linux.
              Any good link how to do that?

              Also, what happens when pfSense get updated to -p5, will the driver be useless after it?

              1 Reply Last reply Reply Quote 0
              • M Offline
                mir
                last edited by

                The important part is 10.1 patch level should have no influence on the driver.

                For building just follow my link. Header files should be part of a full install.

                Load: kldload your_module
                Unload: kldunload your_module
                List: kldstat

                1 Reply Last reply Reply Quote 0
                • W Offline
                  Wordo
                  last edited by

                  Ok, compiled the modules and now it's recognozied by pfSense and I can create an Interface on mlxen0! :)

                  Howto and ready-to-go modules are here:

                  http://www.routerperformance.net/howtos/compile-infiniband-modules-for-pfsense-2-2/

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S Offline
                    stephenw10 Netgate Administrator
                    last edited by

                    What sort of performance are you seeing?

                    It's likely the drivers will continue to function across a minor kernel change but by no means guaranteed. Also something outside the kernel might break their operation in some way. I built some very mildly modified drivers for FreeBSD 8.1 and they continued (surprisingly) to function in 8.3. Of course I might have just been lucky.  ;)

                    Steve

                    1 Reply Last reply Reply Quote 0
                    • W Offline
                      Wordo
                      last edited by

                      @mir:

                      Your chances of success will greatly improve with X2 cards and older. X3 cards can be tricky even on Linux and Windows because you need a fairly new kernel and drivers supplied by Mellanox.

                      Just for the archive, tested a X2 card but with the default install the card was not detected.

                      1 Reply Last reply Reply Quote 0
                      • W Offline
                        Wordo
                        last edited by

                        @stephenw10:

                        What sort of performance are you seeing?

                        It's likely the drivers will continue to function across a minor kernel change but by no means guaranteed. Also something outside the kernel might break their operation in some way. I built some very mildly modified drivers for FreeBSD 8.1 and they continued (surprisingly) to function in 8.3. Of course I might have just been lucky.  ;)

                        Steve

                        I'm a bit stuck here.
                        Created OPT1 on both sides with mlxen0. IPs were 10.99.0.10 and 11. I can ping each other and everything is accepted by firewall.
                        But when I want to connect with TCP and don't get any answer. Also tried with pfctl -d and lowering MSS or set hight MTU :( Same with UDP when i try with iperf.

                        Any ideas?

                        1 Reply Last reply Reply Quote 0
                        • M Offline
                          mir
                          last edited by

                          Do you run a subnet manager either in software or in hardware (switch)?

                          1 Reply Last reply Reply Quote 0
                          • W Offline
                            Wordo
                            last edited by

                            No, they are direct attached, no opensm.

                            1 Reply Last reply Reply Quote 0
                            • M Offline
                              mir
                              last edited by

                              Even if you directly attach you will still need a subnet manager. Without subnet manager you will have no route. https://software.intel.com/en-us/articles/understanding-the-infiniband-subnet-manager

                              1 Reply Last reply Reply Quote 0
                              • W Offline
                                Wordo
                                last edited by

                                Hmpf .. isn't there a package for FreeBSD or will I have to compile by myself? Thought that direct attach doesn't need opensm

                                1 Reply Last reply Reply Quote 0
                                • M Offline
                                  mir
                                  last edited by

                                  did you not build ofed?
                                  http://lists.freebsd.org/pipermail/freebsd-infiniband/2013-February/000028.html

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S Offline
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    Indeed is it not part of ofed?

                                    Looks useful: http://really.zonky.org/?p=2927

                                    Steve

                                    1 Reply Last reply Reply Quote 0
                                    • W Offline
                                      Wordo
                                      last edited by

                                      Sorry, my (foolish) fault :D Just compiled the Mellanox FreeBSD drivers.
                                      Ok, I followed http://really.zonky.org/?p=2927 and made the world on my FreeBSD 10.1 build system.
                                      Then I packed:

                                      a usr/bin/opensm
                                      a usr/bin/ibaddr
                                      a usr/bin/ibnetdiscover
                                      a usr/bin/ibping
                                      a usr/bin/ibportstate
                                      a usr/bin/ibroute
                                      a usr/bin/ibsendtrap
                                      a usr/bin/ibstat
                                      a usr/bin/ibsysstat
                                      a usr/bin/ibtracert
                                      a usr/lib/libibcm.a
                                      a usr/lib/libibcm.so
                                      a usr/lib/libibcm.so.1
                                      a usr/lib/libibcm_p.a
                                      a usr/lib/libibcommon.a
                                      a usr/lib/libibcommon.so
                                      a usr/lib/libibcommon.so.1
                                      a usr/lib/libibcommon_p.a
                                      a usr/lib/libibmad.a
                                      a usr/lib/libibmad.so
                                      a usr/lib/libibmad.so.1
                                      a usr/lib/libibmad_p.a
                                      a usr/lib/libibsdp.a
                                      a usr/lib/libibsdp.so
                                      a usr/lib/libibsdp.so.1
                                      a usr/lib/libibsdp_p.a
                                      a usr/lib/libibumad.a
                                      a usr/lib/libibumad.so
                                      a usr/lib/libibumad.so.1
                                      a usr/lib/libibumad_p.a
                                      a usr/lib/libibverbs.a
                                      a usr/lib/libibverbs.so
                                      a usr/lib/libibverbs.so.1
                                      a usr/lib/libibverbs_p.a
                                      a usr/lib/libopensm.a
                                      a usr/lib/libopensm.so
                                      a usr/lib/libopensm.so.1
                                      a usr/lib/libopensm_p.a
                                      a usr/lib/libosmcomp.a
                                      a usr/lib/libosmcomp.so
                                      a usr/lib/libosmcomp.so.1
                                      a usr/lib/libosmcomp_p.a
                                      a usr/lib/libosmvendor.a
                                      a usr/lib/libosmvendor.so
                                      a usr/lib/libosmvendor.so.1
                                      a usr/lib/libosmvendor_p.a

                                      extracted on my pfSense and started opensm. But now I get:

                                      Feb 13 08:43:25 443034 [2006400] 0x80 -> OpenSM 3.3.1
                                      Entering DISCOVERING state

                                      Feb 13 08:43:25 443142 [2006400] 0x02 -> osm_vendor_init: 1000 pending umads specified
                                      Feb 13 08:43:25 443215 [2006400] 0x80 -> Entering DISCOVERING state
                                      Feb 13 08:43:25 443238 [2006400] 0x02 -> osm_vendor_bind: Binding to port 0xf65214fffe63a411
                                      Feb 13 08:43:25 445759 [2006400] 0x01 -> osm_vendor_bind: ERR 5426: Unable to register class 129 version 1
                                      Feb 13 08:43:25 445765 [2006400] 0x01 -> osm_sm_mad_ctrl_bind: ERR 3118: Vendor specific bind failed
                                      Feb 13 08:43:25 445768 [2006400] 0x01 -> osm_sm_bind: ERR 2E10: SM MAD Controller bind failed (IB_ERROR)

                                      Error from osm_opensm_bind (0x2A)
                                      Perhaps another instance of OpenSM is already running
                                      Feb 13 08:43:25 445776 [2006400] 0x01 -> osm_sa_mad_ctrl_unbind: ERR 1A11: No previous bind
                                      Exiting SM

                                      There's a post around for missing files:

                                      https://community.mellanox.com/thread/1364

                                      Anyone know what files were missing (see list above)?

                                      1 Reply Last reply Reply Quote 0
                                      • M Offline
                                        mir
                                        last edited by

                                        Have you followed this: https://wiki.freebsd.org/InfiniBand

                                        You might find some tips and tricks here: http://web.mit.edu/freebsd/head/contrib/ofed/management/

                                        1 Reply Last reply Reply Quote 0
                                        • M Offline
                                          mir
                                          last edited by

                                          btw. you have asserted that ps -ef |grep opensm gives zero hits?

                                          1 Reply Last reply Reply Quote 0
                                          • W Offline
                                            Wordo
                                            last edited by

                                            No hits, yes.

                                            I've followed this http://really.zonky.org/?p=2927

                                            Compared to the wiki I did not add to MYKERNEL:
                                            options        SDP 
                                            device        mthca

                                            Is this really essential?

                                            Wondering if I could download the OFED stuff from the link you posted and compile it to a prefix like /usr/local/ofed/ and copy the stuff to pfSense? (like on Linux)

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.