mellanox cards change from Infiniband to Ethernet

UserCo

Hello,

I have a few Mellanox connect x3 cards. which according to the FreeBSD Doc are well supported since 2017 or so. However when I boot up PFsense I don't see them as available interfaces. When I run

sysctl -a | grep mlx4_core

I see the cards but the mode appears to be ib.

then I then try to change it to eth with the command I found below

sysctl sys.device.mlx4_core0.mlx4_port1=eth

I see a lot of logs in the serial output which are so fast that I cant read any. after a few minutes ca. 2 the server then crashes (or I don't know as I cannot see the logs: is there a way to see the logs after the next reboot?) and reboots. after that the mode of the card is still ib

how can I change the mode of the cards to be ETH

Any help is appreciated

Thanks

Popolou

@UserCo In my experience, i had to use mlxconfig to make the change persistent before setting the driver to use it in ETH mode.

UserCo

Thanks for the answer @Popolou . How do I do this in pfsense? can you elaborate further. what do you mean by making the change persistent before setting the driver?

Thanks

Popolou

@UserCo Nvidia/Mellanox have a FreeBSD package you can either build on another (test) system with the card installed or copy the modules (if you can find them) on to your router pc. There are a suite of utilities but you want mlxconfig. With the card installed in that test system, you can then configure the card to ethernet mode.

Their site has some pretty comprehensive instructions on how to do this and the switches required.

stephenw10

How does it appear in the boot log?

How does it appear in the output of pciconf -lv ?

UserCo

@stephenw10

pciconf -lv shows

mlx4_core1@pci0:94:0:0: class=0x028000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1003 subvendor=0x15b3 subdevice=0x0050
    vendor     = 'Mellanox Technologies'
    device     = 'MT27500 Family [ConnectX-3]'
    class      = network

or the full output attached
see the boot log attached
pciconf.txt
bootlog.txt

UserCo

Weird thing just happened.

if I plug in all the ports of all the mellanox cards to my switch and boot the switch so it is operational and then boot the PFsense box, it starts with no errors and shows the interfaces even in the web gui.

when i do the

sysctl -a | grep mlx4_core

it show auto (eth) this time and i can set it to

sysctl sys.device.mlx4_core0.mlx4_port1=eth

manually whiteout issues for each network interface.

I can even reboot without issues as long as all cables stay connected. but once I disconnect one of the cables an reboot then the disconnected interface is missing and when I then plug in the disconnected interface again the firewall crashes like described in my other forum post https://forum.netgate.com/topic/185482/pfsense-crashes-after-plugging-in-cable-mellanox-connect-x3-card

stephenw10

Hmm, well that's a newer NIC than the one I tested:

none5@pci0:2:0:0:       class=0x020000 card=0x675015b3 chip=0x675015b3 rev=0xb0 hdr=0x00
    vendor     = 'Mellanox Technologies'
    device     = 'MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s]'
    class      = network
    subclass   = ethernet

But I have to say that I would avoid Mellanox in pfSense if possible. That NIC has always behaved strangely for me. It does work but it throws errors seemingly at random and doesnt give anywhere near the expected throughput. YMMV!

UserCo

[SOLVED]

Thank you all for all the tips and recommendations. I have now solved the issue by doing the following:

As pfSense was crashing when I plugged a cable into one of the ports, and the ports were all in auto (IB) mode as I discovered, I figured it might be an issue related to the NIC changing its configuration from IB to ETH, and FreeBSD trying to load the driver or something (I'm not sure).

However, I saw this amazing video on changing the mode of the NIC permanently with the MFT tool. As the configuration gets saved to the NIC itself, you can use any other Linux or FreeBSD installation, use MFT to change the values, and then reinstall pfSense. After that, the NIC is correctly in the ETH mode, and the driver loads correctly without issues. (I see no issues displayed in dmesg as far as I can tell.)

Video: https://www.youtube.com/watch?v=D1qN7Qg3bSg

Now, I just need to make sure the cards run stable and reliably, but from what I see now, I am confident.

Thanks again to everyone who helped.