Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    HowTo: Mellanox Connectx-2 10gb SFP+

    Scheduled Pinned Locked Moved Hardware
    8 Posts 5 Posters 8.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      nasomi
      last edited by

      I managed to pull this off with some help all over the internet, but I want to compile it all in one place, especially for those who are non-freebsd/linux/not-windows savvy.

      As for the "why" the reason is simple. I had a stack of these cards, they're in all my servers and worked fine natively. They're also $10-14 on ebay, which is pretty damned cheap.

      This will require the use of another computer or virtual machine running freebsd. or i'll upload the files necessary and you can download them and do what you need.

      This is written for freebsd 10.3 which is used for pfsense 2.3.4, but the instructions should remain fairly consistent in future versions.


      This guide assumes a fresh install of pfsense 2.3.4 on a computer, and you have a different router and LAN already in place to help you configure this one.


      1. Spin up a FreeBSD vm or install FreeBSD on a computer, you'll need 10.3 for this case. You can download a pre-made vhd here:
      https://download.freebsd.org/ftp/releases/VM-IMAGES/10.3-RELEASE/amd64/Latest/FreeBSD-10.3-RELEASE-amd64.vhd.xz

      2. Unzip that, and drop it in a location to spin up your vm, or find the 10.3 installer.

      3. When you boot up, it will give you a menu on how to boot. Just hit enter, and you'll be in.

      4. Assuming all went well, you're now at a command prompt. Type "ifconfig" and get your adapter name. For a hyper-v vm it will say something like hn0 or hn1. For a realtek network card it will say re0 or re1.

      5. Once you have that type "dhclient hn0" or whatever your network card's name is. That should get you your ip.

      6. Once in, type "pkg install git" and it'll say no package manager installed, press enter to install. Press enter, it'll work it's magic, just use all default prompts hitting enter.

      7. Once that's done, type again, "pkg install git" and it should say it's already installed, but if it isn't, it'll install it.

      8. Next you need to clone the repo. If you're not git friendly, like me, here you go:

      git clone https://github.com/freebsd/freebsd.git /usr/src
      

      ^This is approximately 3gb at the time of this writing, it's going to take a bit of time depending on your internet speed.

      9. Once complete you need to change to 10.3. You do that with the following command:

      cd /usr/src && git checkout releng/10.3
      

      It'll do more stuff and take a few minutes.

      10. Now you've successfully downloaded the source. time to go to the place with the we want. This will take you there:

      cd /usr/src/sys/modules/
      

      11. From here you need to run a makefile to compile the network drivers.

      make && make install && kldload mlxen
      

      12. After that finishes:

      cd /boot/kernel/
      ls ml*
      

      That should show a handful of files. Time to copy them to their new home.

      13. Connect a network cable from your LAN to the WAN port on the pfsense box. On the pfsense box console hit 1 to configure interfaces. The one that says (up) is going to be your WAN interface. Hit n for setup vlan's now. Enter your WAN interface name you just saw. When it prompts you to create a LAN interface just hit enter for none. Now you're back at the console with your WAN IP showing(actually inside your LAN now, but eh.

      14. Go back to your FreeBSD box. Assuming your ip of your pfsense box is 10.0.0.10, enter the following command:

      scp ml*.ko root@10.0.0.1:/boot/kernel/
      

      It will prompt you for a password, the default is "pfsense". This will copy the drivers you just made to the pfsense box in the proper location.

      15. Go to your pfsense box and hit 8 to enter shell. Lets make sure the files are there.

      ls /boot/kernel/
      ls ml*
      

      Assuming they're there, we're looking good.

      16. Load the drivers.

      kldload mlxen
      

      17. Now we want to edit the kernal so we don't need to do this every time we reboot. This may be messy.

      vi /boot/loader.conf
      

      This loads loader.conf into vi, a text editor. Vi is as user friendly as a rabid porcupine, and I'm being generous here.

      18. Arrow down to the end of the last line, mine is "hw.usb.no_pf="1". Over the last " hit enter, you're on the next line. Hit a and you should be able to type. Add the following line:

      mlxen_load="YES"
      

      19. Now to a) save changes and b) quit vi. Hit escape a few times, then press : and pray that : appears at hte bottom. Type wq after : so it says :wq This means write and quit. Hit enter, and assuming you were granted a small mirricle, you're back at the command line.

      20. Type exit to get back into the world of pfsense.

      21. Press 1 to assign interfaces. Assuming all went well, you should have mlxen0 as an available interface. That's your new interface for your LAN.

      22. Have a beer. You dun good.

      1 Reply Last reply Reply Quote 0
      • A
        arisythila
        last edited by

        Tried this…  Didn't work directly how it was suppose too.  There were a few spots where ktrace dependencies were not in place.

        Finally, after taking care of that.

        install -o root -g wheel -m 555  wlan_wep.ko /boot/kernel
        ===> wlan_xauth (install)
        install -o root -g wheel -m 555  wlan_xauth.ko /boot/kernel
        ===> wpi (install)
        install -o root -g wheel -m 555  if_wpi.ko /boot/kernel
        ===> wpifw (install)
        install -o root -g wheel -m 555  wpifw.ko /boot/kernel
        ===> x86bios (install)
        install -o root -g wheel -m 555  x86bios.ko /boot/kernel
        ===> xl (install)
        install -o root -g wheel -m 555  if_xl.ko /boot/kernel
        ===> zfs (install)
        install -o root -g wheel -m 555  zfs.ko /boot/kernel
        ===> zlib (install)
        install -o root -g wheel -m 555  zlib.ko /boot/kernel
        ===> iscsi (install)
        install -o root -g wheel -m 555  iscsi.ko /boot/kernel
        ===> iscsi_initiator (install)
        install -o root -g wheel -m 555  iscsi_initiator.ko /boot/kernel
        kldxref /boot/kernel
        kldload: can't load mlxen: No such file or directory

        I get this error after Step 11.

        Thoughts?

        Thanks,

        Michael

        1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          It looks like you're building all the modules there which is unnecessary.

          That last error implies he mlxen module is not present so it either failed to build or failed to copy.
          Is it in the source directory?

          Steve

          1 Reply Last reply Reply Quote 0
          • F
            fwcheck
            last edited by

            Hi,

            thanks for this solution, it is still working for 2.4.4-(p3). You have to use

            cd /usr/src && git checkout releng/11.2
            

            for step 9. There is no mlxen-Module output. You also need

            linuxkpi.ko
            

            for the driver to work.
            dmesg showed

            mlx4_core0: <mlx4_core> mem 0xdf800000-0xdf8fffff,0xd9000000-0xd97fffff irq 48 at device 0.0 numa-domain 0 on pci5
            mlx4_core: Mellanox ConnectX core driver v3.4.1 (October 2017)
            mlx4_core: Initializing mlx4_core
            mlx4_core0: Unable to determine PCI device chain minimum BW
            

            Cause of this message is, that the device default mode is infiniband not ethernet. To fix this you can use

            sysctl sys.device.mlx4_core0.mlx4_port1=eth
            

            depending on your card you may also need this for port2.

            mlx4_en mlx4_core0: Activating port:1
            mlxen0: Ethernet address: 00:01:02:03:04:05
            mlx4_en: mlx4_core0: Port 1: Using 32 TX rings
            mlx4_en: mlx4_core0: Port 1: Using 16 RX rings
            mlxen0: link state changed to DOWN
            mlx4_en: mlxen0: Using 32 TX rings
            mlx4_en: mlxen0: Using 16 RX rings
            mlx4_en: mlxen0: Initializing port
            

            There should be a tool called mlxconfig to set this permanent.

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              You can just copy the modules from a FreeBSD 11 ISO. No need to build anything.

              On my card I did not need that sysctl, it established as an Ethernet device anyway.
              Tested with this:

              none5@pci0:2:0:0:       class=0x020000 card=0x675015b3 chip=0x675015b3 rev=0xb0 hdr=0x00
                  vendor     = 'Mellanox Technologies'
                  device     = 'MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s]'
                  class      = network
                  subclass   = ethernet
              

              Steve

              H 1 Reply Last reply Reply Quote 0
              • H
                Hammer8
                last edited by Hammer8

                Hi, can someone provide the commands and steps needed to copy the files so one can use a mellanox card with pfsense? Thank you.

                1 Reply Last reply Reply Quote 0
                • H
                  Hammer8 @stephenw10
                  last edited by Hammer8

                  @stephenw10 said in HowTo: Mellanox Connectx-2 10gb SFP+:

                  You can just copy the modules from a FreeBSD 11 ISO. No need to build anything.

                  On my card I did not need that sysctl, it established as an Ethernet device anyway.
                  Tested with this:

                  none5@pci0:2:0:0:       class=0x020000 card=0x675015b3 chip=0x675015b3 rev=0xb0 hdr=0x00
                      vendor     = 'Mellanox Technologies'
                      device     = 'MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s]'
                      class      = network
                      subclass   = ethernet
                  

                  Steve

                  hi, what command to I use to get that info? also, does anyone know if/when pfsense is upgraded that the added drivers will be retained? Thank you.

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    To see that use pciconf -lv

                    1 Reply Last reply Reply Quote 0
                    • First post
                      Last post
                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.