HowTo: Mellanox Connectx-2 10gb SFP+

nasomi

I managed to pull this off with some help all over the internet, but I want to compile it all in one place, especially for those who are non-freebsd/linux/not-windows savvy.

As for the "why" the reason is simple. I had a stack of these cards, they're in all my servers and worked fine natively. They're also $10-14 on ebay, which is pretty damned cheap.

This will require the use of another computer or virtual machine running freebsd. or i'll upload the files necessary and you can download them and do what you need.

This is written for freebsd 10.3 which is used for pfsense 2.3.4, but the instructions should remain fairly consistent in future versions.

This guide assumes a fresh install of pfsense 2.3.4 on a computer, and you have a different router and LAN already in place to help you configure this one.

1. Spin up a FreeBSD vm or install FreeBSD on a computer, you'll need 10.3 for this case. You can download a pre-made vhd here:
https://download.freebsd.org/ftp/releases/VM-IMAGES/10.3-RELEASE/amd64/Latest/FreeBSD-10.3-RELEASE-amd64.vhd.xz

2. Unzip that, and drop it in a location to spin up your vm, or find the 10.3 installer.

3. When you boot up, it will give you a menu on how to boot. Just hit enter, and you'll be in.

4. Assuming all went well, you're now at a command prompt. Type "ifconfig" and get your adapter name. For a hyper-v vm it will say something like hn0 or hn1. For a realtek network card it will say re0 or re1.

5. Once you have that type "dhclient hn0" or whatever your network card's name is. That should get you your ip.

6. Once in, type "pkg install git" and it'll say no package manager installed, press enter to install. Press enter, it'll work it's magic, just use all default prompts hitting enter.

7. Once that's done, type again, "pkg install git" and it should say it's already installed, but if it isn't, it'll install it.

8. Next you need to clone the repo. If you're not git friendly, like me, here you go:

git clone https://github.com/freebsd/freebsd.git /usr/src

^This is approximately 3gb at the time of this writing, it's going to take a bit of time depending on your internet speed.

9. Once complete you need to change to 10.3. You do that with the following command:

cd /usr/src && git checkout releng/10.3

It'll do more stuff and take a few minutes.

10. Now you've successfully downloaded the source. time to go to the place with the we want. This will take you there:

cd /usr/src/sys/modules/

11. From here you need to run a makefile to compile the network drivers.

make && make install && kldload mlxen

12. After that finishes:

cd /boot/kernel/
ls ml*

That should show a handful of files. Time to copy them to their new home.

13. Connect a network cable from your LAN to the WAN port on the pfsense box. On the pfsense box console hit 1 to configure interfaces. The one that says (up) is going to be your WAN interface. Hit n for setup vlan's now. Enter your WAN interface name you just saw. When it prompts you to create a LAN interface just hit enter for none. Now you're back at the console with your WAN IP showing(actually inside your LAN now, but eh.

14. Go back to your FreeBSD box. Assuming your ip of your pfsense box is 10.0.0.10, enter the following command:

scp ml*.ko root@10.0.0.1:/boot/kernel/

It will prompt you for a password, the default is "pfsense". This will copy the drivers you just made to the pfsense box in the proper location.

15. Go to your pfsense box and hit 8 to enter shell. Lets make sure the files are there.

ls /boot/kernel/
ls ml*

Assuming they're there, we're looking good.

16. Load the drivers.

kldload mlxen

17. Now we want to edit the kernal so we don't need to do this every time we reboot. This may be messy.

vi /boot/loader.conf

This loads loader.conf into vi, a text editor. Vi is as user friendly as a rabid porcupine, and I'm being generous here.

18. Arrow down to the end of the last line, mine is "hw.usb.no_pf="1". Over the last " hit enter, you're on the next line. Hit a and you should be able to type. Add the following line:

mlxen_load="YES"

19. Now to a) save changes and b) quit vi. Hit escape a few times, then press : and pray that : appears at hte bottom. Type wq after : so it says :wq This means write and quit. Hit enter, and assuming you were granted a small mirricle, you're back at the command line.

20. Type exit to get back into the world of pfsense.

21. Press 1 to assign interfaces. Assuming all went well, you should have mlxen0 as an available interface. That's your new interface for your LAN.

22. Have a beer. You dun good.

arisythila

Tried this… Didn't work directly how it was suppose too. There were a few spots where ktrace dependencies were not in place.

Finally, after taking care of that.

install -o root -g wheel -m 555 wlan_wep.ko /boot/kernel
===> wlan_xauth (install)
install -o root -g wheel -m 555 wlan_xauth.ko /boot/kernel
===> wpi (install)
install -o root -g wheel -m 555 if_wpi.ko /boot/kernel
===> wpifw (install)
install -o root -g wheel -m 555 wpifw.ko /boot/kernel
===> x86bios (install)
install -o root -g wheel -m 555 x86bios.ko /boot/kernel
===> xl (install)
install -o root -g wheel -m 555 if_xl.ko /boot/kernel
===> zfs (install)
install -o root -g wheel -m 555 zfs.ko /boot/kernel
===> zlib (install)
install -o root -g wheel -m 555 zlib.ko /boot/kernel
===> iscsi (install)
install -o root -g wheel -m 555 iscsi.ko /boot/kernel
===> iscsi_initiator (install)
install -o root -g wheel -m 555 iscsi_initiator.ko /boot/kernel
kldxref /boot/kernel
kldload: can't load mlxen: No such file or directory

I get this error after Step 11.

Thoughts?

Thanks,

Michael

stephenw10

It looks like you're building all the modules there which is unnecessary.

That last error implies he mlxen module is not present so it either failed to build or failed to copy.
Is it in the source directory?

Steve

fwcheck

Hi,

thanks for this solution, it is still working for 2.4.4-(p3). You have to use

cd /usr/src && git checkout releng/11.2

for step 9. There is no mlxen-Module output. You also need

linuxkpi.ko

for the driver to work.
dmesg showed

mlx4_core0: <mlx4_core> mem 0xdf800000-0xdf8fffff,0xd9000000-0xd97fffff irq 48 at device 0.0 numa-domain 0 on pci5
mlx4_core: Mellanox ConnectX core driver v3.4.1 (October 2017)
mlx4_core: Initializing mlx4_core
mlx4_core0: Unable to determine PCI device chain minimum BW

Cause of this message is, that the device default mode is infiniband not ethernet. To fix this you can use

sysctl sys.device.mlx4_core0.mlx4_port1=eth

depending on your card you may also need this for port2.

mlx4_en mlx4_core0: Activating port:1
mlxen0: Ethernet address: 00:01:02:03:04:05
mlx4_en: mlx4_core0: Port 1: Using 32 TX rings
mlx4_en: mlx4_core0: Port 1: Using 16 RX rings
mlxen0: link state changed to DOWN
mlx4_en: mlxen0: Using 32 TX rings
mlx4_en: mlxen0: Using 16 RX rings
mlx4_en: mlxen0: Initializing port

There should be a tool called mlxconfig to set this permanent.

stephenw10

You can just copy the modules from a FreeBSD 11 ISO. No need to build anything.

On my card I did not need that sysctl, it established as an Ethernet device anyway.
Tested with this:

none5@pci0:2:0:0:       class=0x020000 card=0x675015b3 chip=0x675015b3 rev=0xb0 hdr=0x00
    vendor     = 'Mellanox Technologies'
    device     = 'MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s]'
    class      = network
    subclass   = ethernet

Steve

Hammer8

Hi, can someone provide the commands and steps needed to copy the files so one can use a mellanox card with pfsense? Thank you.

Hammer8

@stephenw10 said in HowTo: Mellanox Connectx-2 10gb SFP+:

You can just copy the modules from a FreeBSD 11 ISO. No need to build anything.

On my card I did not need that sysctl, it established as an Ethernet device anyway.
Tested with this:
none5@pci0:2:0:0:       class=0x020000 card=0x675015b3 chip=0x675015b3 rev=0xb0 hdr=0x00
    vendor     = 'Mellanox Technologies'
    device     = 'MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s]'
    class      = network
    subclass   = ethernet
Steve

hi, what command to I use to get that info? also, does anyone know if/when pfsense is upgraded that the added drivers will be retained? Thank you.

stephenw10

To see that use pciconf -lv