HowTo: Mellanox Connectx-2 10gb SFP+

[nasomi]

I managed to pull this off with some help all over the internet, but I want to compile it all in one place, especially for those who are non-freebsd/linux/not-windows savvy.

As for the "why" the reason is simple. I had a stack of these cards, they're in all my servers and worked fine natively. They're also $10-14 on ebay, which is pretty damned cheap.

This will require the use of another computer or virtual machine running freebsd. or i'll upload the files necessary and you can download them and do what you need.

This is written for freebsd 10.3 which is used for pfsense 2.3.4, but the instructions should remain fairly consistent in future versions.

---

This guide assumes a fresh install of pfsense 2.3.4 on a computer, and you have a different router and LAN already in place to help you configure this one.

---

1. Spin up a FreeBSD vm or install FreeBSD on a computer, you'll need 10.3 for this case. You can download a pre-made vhd here:
https://download.freebsd.org/ftp/releases/VM-IMAGES/10.3-RELEASE/amd64/Latest/FreeBSD-10.3-RELEASE-amd64.vhd.xz

2. Unzip that, and drop it in a location to spin up your vm, or find the 10.3 installer.

3. When you boot up, it will give you a menu on how to boot. Just hit enter, and you'll be in.

4. Assuming all went well, you're now at a command prompt. Type "ifconfig" and get your adapter name. For a hyper-v vm it will say something like hn0 or hn1. For a realtek network card it will say re0 or re1.

5. Once you have that type "dhclient hn0" or whatever your network card's name is. That should get you your ip.

6. Once in, type "pkg install git" and it'll say no package manager installed, press enter to install. Press enter, it'll work it's magic, just use all default prompts hitting enter.

7. Once that's done, type again, "pkg install git" and it should say it's already installed, but if it isn't, it'll install it.

8. Next you need to clone the repo. If you're not git friendly, like me, here you go:

git clone https://github.com/freebsd/freebsd.git /usr/src

^This is approximately 3gb at the time of this writing, it's going to take a bit of time depending on your internet speed.

9. Once complete you need to change to 10.3. You do that with the following command:

cd /usr/src && git checkout releng/10.3

It'll do more stuff and take a few minutes.

10. Now you've successfully downloaded the source. time to go to the place with the we want. This will take you there:

cd /usr/src/sys/modules/

11. From here you need to run a makefile to compile the network drivers.

make && make install && kldload mlxen

12. After that finishes:

cd /boot/kernel/
ls ml*

That should show a handful of files. Time to copy them to their new home.

13. Connect a network cable from your LAN to the WAN port on the pfsense box. On the pfsense box console hit 1 to configure interfaces. The one that says (up) is going to be your WAN interface. Hit n for setup vlan's now. Enter your WAN interface name you just saw. When it prompts you to create a LAN interface just hit enter for none. Now you're back at the console with your WAN IP showing(actually inside your LAN now, but eh.

14. Go back to your FreeBSD box. Assuming your ip of your pfsense box is 10.0.0.10, enter the following command:

scp ml*.ko root@10.0.0.1:/boot/kernel/

It will prompt you for a password, the default is "pfsense". This will copy the drivers you just made to the pfsense box in the proper location.

15. Go to your pfsense box and hit 8 to enter shell. Lets make sure the files are there.

ls /boot/kernel/
ls ml*

Assuming they're there, we're looking good.

16. Load the drivers.

kldload mlxen

17. Now we want to edit the kernal so we don't need to do this every time we reboot. This may be messy.

vi /boot/loader.conf

This loads loader.conf into vi, a text editor. Vi is as user friendly as a rabid porcupine, and I'm being generous here.

18. Arrow down to the end of the last line, mine is "hw.usb.no_pf="1". Over the last " hit enter, you're on the next line. Hit a and you should be able to type. Add the following line:

mlxen_load="YES"

19. Now to a) save changes and b) quit vi. Hit escape a few times, then press : and pray that : appears at hte bottom. Type wq after : so it says :wq This means write and quit. Hit enter, and assuming you were granted a small mirricle, you're back at the command line.

20. Type exit to get back into the world of pfsense.

21. Press 1 to assign interfaces. Assuming all went well, you should have mlxen0 as an available interface. That's your new interface for your LAN.

22. Have a beer. You dun good.

[arisythila]

Tried this…  Didn't work directly how it was suppose too.  There were a few spots where ktrace dependencies were not in place.

Finally, after taking care of that.

install -o root -g wheel -m 555  wlan_wep.ko /boot/kernel
===> wlan_xauth (install)
install -o root -g wheel -m 555  wlan_xauth.ko /boot/kernel
===> wpi (install)
install -o root -g wheel -m 555  if_wpi.ko /boot/kernel
===> wpifw (install)
install -o root -g wheel -m 555  wpifw.ko /boot/kernel
===> x86bios (install)
install -o root -g wheel -m 555  x86bios.ko /boot/kernel
===> xl (install)
install -o root -g wheel -m 555  if_xl.ko /boot/kernel
===> zfs (install)
install -o root -g wheel -m 555  zfs.ko /boot/kernel
===> zlib (install)
install -o root -g wheel -m 555  zlib.ko /boot/kernel
===> iscsi (install)
install -o root -g wheel -m 555  iscsi.ko /boot/kernel
===> iscsi_initiator (install)
install -o root -g wheel -m 555  iscsi_initiator.ko /boot/kernel
kldxref /boot/kernel
kldload: can't load mlxen: No such file or directory

I get this error after Step 11.

Thoughts?

Thanks,

Michael

[stephenw10]

It looks like you're building all the modules there which is unnecessary.

That last error implies he mlxen module is not present so it either failed to build or failed to copy.
Is it in the source directory?

Steve