Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Infiniband cards with pfSense 2.2

    Hardware
    5
    23
    6.0k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • W
      Wordo
      last edited by

      Hey guys,

      searched around the forums for Infiniband compability, only with answers "wait for version 2.2". Now there's 2.2 out and I've plugged in a Connect X3 card from Mellanox, did a fresh install but it won't be recognized. Nothing to see in dmesg, and also pciconf is very quiet :(

      There's a FreeBSD driver available, but I need Kernelfiles to compile them by myself.
      Any ideas how to do that?

      Thanks!

      1 Reply Last reply Reply Quote 0
      • M
        mir
        last edited by

        Your chances of success will greatly improve with X2 cards and older. X3 cards can be tricky even on Linux and Windows because you need a fairly new kernel and drivers supplied by Mellanox.

        1 Reply Last reply Reply Quote 0
        • W
          Wordo
          last edited by

          Cards get recognized with Debian 8, but I'll ask for an older one, shouldn't be a problem.
          Will come back soon … thanks!

          1 Reply Last reply Reply Quote 0
          • M
            mir
            last edited by

            It seems X3 should be working on FreeBSD 10: https://community.mellanox.com/thread/1195

            Forgot to add:
            To compile the drivers I would install a full FreeBSD 10 on another server, could be a VM in VmWare or Virtualbox, and do the compiling on this server. Then transfer the binary drivers to your pfSense box. Just remember to grab the same kernel version (FreeBSD 10.1-RELEASE-p4)

            1 Reply Last reply Reply Quote 0
            • W
              Wordo
              last edited by

              Thanks! I've never compiled a FreeBSD kernel, only on Linux.
              Any good link how to do that?

              Also, what happens when pfSense get updated to -p5, will the driver be useless after it?

              1 Reply Last reply Reply Quote 0
              • M
                mir
                last edited by

                The important part is 10.1 patch level should have no influence on the driver.

                For building just follow my link. Header files should be part of a full install.

                Load: kldload your_module
                Unload: kldunload your_module
                List: kldstat

                1 Reply Last reply Reply Quote 0
                • W
                  Wordo
                  last edited by

                  Ok, compiled the modules and now it's recognozied by pfSense and I can create an Interface on mlxen0! :)

                  Howto and ready-to-go modules are here:

                  http://www.routerperformance.net/howtos/compile-infiniband-modules-for-pfsense-2-2/

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    What sort of performance are you seeing?

                    It's likely the drivers will continue to function across a minor kernel change but by no means guaranteed. Also something outside the kernel might break their operation in some way. I built some very mildly modified drivers for FreeBSD 8.1 and they continued (surprisingly) to function in 8.3. Of course I might have just been lucky.  ;)

                    Steve

                    1 Reply Last reply Reply Quote 0
                    • W
                      Wordo
                      last edited by

                      @mir:

                      Your chances of success will greatly improve with X2 cards and older. X3 cards can be tricky even on Linux and Windows because you need a fairly new kernel and drivers supplied by Mellanox.

                      Just for the archive, tested a X2 card but with the default install the card was not detected.

                      1 Reply Last reply Reply Quote 0
                      • W
                        Wordo
                        last edited by

                        @stephenw10:

                        What sort of performance are you seeing?

                        It's likely the drivers will continue to function across a minor kernel change but by no means guaranteed. Also something outside the kernel might break their operation in some way. I built some very mildly modified drivers for FreeBSD 8.1 and they continued (surprisingly) to function in 8.3. Of course I might have just been lucky.  ;)

                        Steve

                        I'm a bit stuck here.
                        Created OPT1 on both sides with mlxen0. IPs were 10.99.0.10 and 11. I can ping each other and everything is accepted by firewall.
                        But when I want to connect with TCP and don't get any answer. Also tried with pfctl -d and lowering MSS or set hight MTU :( Same with UDP when i try with iperf.

                        Any ideas?

                        1 Reply Last reply Reply Quote 0
                        • M
                          mir
                          last edited by

                          Do you run a subnet manager either in software or in hardware (switch)?

                          1 Reply Last reply Reply Quote 0
                          • W
                            Wordo
                            last edited by

                            No, they are direct attached, no opensm.

                            1 Reply Last reply Reply Quote 0
                            • M
                              mir
                              last edited by

                              Even if you directly attach you will still need a subnet manager. Without subnet manager you will have no route. https://software.intel.com/en-us/articles/understanding-the-infiniband-subnet-manager

                              1 Reply Last reply Reply Quote 0
                              • W
                                Wordo
                                last edited by

                                Hmpf .. isn't there a package for FreeBSD or will I have to compile by myself? Thought that direct attach doesn't need opensm

                                1 Reply Last reply Reply Quote 0
                                • M
                                  mir
                                  last edited by

                                  did you not build ofed?
                                  http://lists.freebsd.org/pipermail/freebsd-infiniband/2013-February/000028.html

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    Indeed is it not part of ofed?

                                    Looks useful: http://really.zonky.org/?p=2927

                                    Steve

                                    1 Reply Last reply Reply Quote 0
                                    • W
                                      Wordo
                                      last edited by

                                      Sorry, my (foolish) fault :D Just compiled the Mellanox FreeBSD drivers.
                                      Ok, I followed http://really.zonky.org/?p=2927 and made the world on my FreeBSD 10.1 build system.
                                      Then I packed:

                                      a usr/bin/opensm
                                      a usr/bin/ibaddr
                                      a usr/bin/ibnetdiscover
                                      a usr/bin/ibping
                                      a usr/bin/ibportstate
                                      a usr/bin/ibroute
                                      a usr/bin/ibsendtrap
                                      a usr/bin/ibstat
                                      a usr/bin/ibsysstat
                                      a usr/bin/ibtracert
                                      a usr/lib/libibcm.a
                                      a usr/lib/libibcm.so
                                      a usr/lib/libibcm.so.1
                                      a usr/lib/libibcm_p.a
                                      a usr/lib/libibcommon.a
                                      a usr/lib/libibcommon.so
                                      a usr/lib/libibcommon.so.1
                                      a usr/lib/libibcommon_p.a
                                      a usr/lib/libibmad.a
                                      a usr/lib/libibmad.so
                                      a usr/lib/libibmad.so.1
                                      a usr/lib/libibmad_p.a
                                      a usr/lib/libibsdp.a
                                      a usr/lib/libibsdp.so
                                      a usr/lib/libibsdp.so.1
                                      a usr/lib/libibsdp_p.a
                                      a usr/lib/libibumad.a
                                      a usr/lib/libibumad.so
                                      a usr/lib/libibumad.so.1
                                      a usr/lib/libibumad_p.a
                                      a usr/lib/libibverbs.a
                                      a usr/lib/libibverbs.so
                                      a usr/lib/libibverbs.so.1
                                      a usr/lib/libibverbs_p.a
                                      a usr/lib/libopensm.a
                                      a usr/lib/libopensm.so
                                      a usr/lib/libopensm.so.1
                                      a usr/lib/libopensm_p.a
                                      a usr/lib/libosmcomp.a
                                      a usr/lib/libosmcomp.so
                                      a usr/lib/libosmcomp.so.1
                                      a usr/lib/libosmcomp_p.a
                                      a usr/lib/libosmvendor.a
                                      a usr/lib/libosmvendor.so
                                      a usr/lib/libosmvendor.so.1
                                      a usr/lib/libosmvendor_p.a

                                      extracted on my pfSense and started opensm. But now I get:

                                      Feb 13 08:43:25 443034 [2006400] 0x80 -> OpenSM 3.3.1
                                      Entering DISCOVERING state

                                      Feb 13 08:43:25 443142 [2006400] 0x02 -> osm_vendor_init: 1000 pending umads specified
                                      Feb 13 08:43:25 443215 [2006400] 0x80 -> Entering DISCOVERING state
                                      Feb 13 08:43:25 443238 [2006400] 0x02 -> osm_vendor_bind: Binding to port 0xf65214fffe63a411
                                      Feb 13 08:43:25 445759 [2006400] 0x01 -> osm_vendor_bind: ERR 5426: Unable to register class 129 version 1
                                      Feb 13 08:43:25 445765 [2006400] 0x01 -> osm_sm_mad_ctrl_bind: ERR 3118: Vendor specific bind failed
                                      Feb 13 08:43:25 445768 [2006400] 0x01 -> osm_sm_bind: ERR 2E10: SM MAD Controller bind failed (IB_ERROR)

                                      Error from osm_opensm_bind (0x2A)
                                      Perhaps another instance of OpenSM is already running
                                      Feb 13 08:43:25 445776 [2006400] 0x01 -> osm_sa_mad_ctrl_unbind: ERR 1A11: No previous bind
                                      Exiting SM

                                      There's a post around for missing files:

                                      https://community.mellanox.com/thread/1364

                                      Anyone know what files were missing (see list above)?

                                      1 Reply Last reply Reply Quote 0
                                      • M
                                        mir
                                        last edited by

                                        Have you followed this: https://wiki.freebsd.org/InfiniBand

                                        You might find some tips and tricks here: http://web.mit.edu/freebsd/head/contrib/ofed/management/

                                        1 Reply Last reply Reply Quote 0
                                        • M
                                          mir
                                          last edited by

                                          btw. you have asserted that ps -ef |grep opensm gives zero hits?

                                          1 Reply Last reply Reply Quote 0
                                          • W
                                            Wordo
                                            last edited by

                                            No hits, yes.

                                            I've followed this http://really.zonky.org/?p=2927

                                            Compared to the wiki I did not add to MYKERNEL:
                                            options        SDP 
                                            device        mthca

                                            Is this really essential?

                                            Wondering if I could download the OFED stuff from the link you posted and compile it to a prefix like /usr/local/ofed/ and copy the stuff to pfSense? (like on Linux)

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.