mellanox cards change from Infiniband to Ethernet
-
Hello,
I have a few Mellanox connect x3 cards. which according to the FreeBSD Doc are well supported since 2017 or so. However when I boot up PFsense I don't see them as available interfaces. When I run
sysctl -a | grep mlx4_core
I see the cards but the mode appears to be ib.
then I then try to change it to eth with the command I found below
sysctl sys.device.mlx4_core0.mlx4_port1=eth
I see a lot of logs in the serial output which are so fast that I cant read any. after a few minutes ca. 2 the server then crashes (or I don't know as I cannot see the logs: is there a way to see the logs after the next reboot?) and reboots. after that the mode of the card is still ib
how can I change the mode of the cards to be ETH
Any help is appreciated
Thanks
-
-
@UserCo In my experience, i had to use mlxconfig to make the change persistent before setting the driver to use it in ETH mode.
-
Thanks for the answer @Popolou . How do I do this in pfsense? can you elaborate further. what do you mean by making the change persistent before setting the driver?
Thanks
-
@UserCo Nvidia/Mellanox have a FreeBSD package you can either build on another (test) system with the card installed or copy the modules (if you can find them) on to your router pc. There are a suite of utilities but you want mlxconfig. With the card installed in that test system, you can then configure the card to ethernet mode.
Their site has some pretty comprehensive instructions on how to do this and the switches required.
-
How does it appear in the boot log?
How does it appear in the output of
pciconf -lv
? -
pciconf -lv shows
mlx4_core1@pci0:94:0:0: class=0x028000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1003 subvendor=0x15b3 subdevice=0x0050 vendor = 'Mellanox Technologies' device = 'MT27500 Family [ConnectX-3]' class = network
or the full output attached
see the boot log attached
pciconf.txt
bootlog.txt -
Weird thing just happened.
if I plug in all the ports of all the mellanox cards to my switch and boot the switch so it is operational and then boot the PFsense box, it starts with no errors and shows the interfaces even in the web gui.
when i do the
sysctl -a | grep mlx4_core
it show auto (eth) this time and i can set it to
sysctl sys.device.mlx4_core0.mlx4_port1=eth
manually whiteout issues for each network interface.
I can even reboot without issues as long as all cables stay connected. but once I disconnect one of the cables an reboot then the disconnected interface is missing and when I then plug in the disconnected interface again the firewall crashes like described in my other forum post https://forum.netgate.com/topic/185482/pfsense-crashes-after-plugging-in-cable-mellanox-connect-x3-card
-
Hmm, well that's a newer NIC than the one I tested:
none5@pci0:2:0:0: class=0x020000 card=0x675015b3 chip=0x675015b3 rev=0xb0 hdr=0x00 vendor = 'Mellanox Technologies' device = 'MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s]' class = network subclass = ethernet
But I have to say that I would avoid Mellanox in pfSense if possible. That NIC has always behaved strangely for me. It does work but it throws errors seemingly at random and doesnt give anywhere near the expected throughput. YMMV!
-
[SOLVED]
Thank you all for all the tips and recommendations. I have now solved the issue by doing the following:
As pfSense was crashing when I plugged a cable into one of the ports, and the ports were all in auto (IB) mode as I discovered, I figured it might be an issue related to the NIC changing its configuration from IB to ETH, and FreeBSD trying to load the driver or something (I'm not sure).
However, I saw this amazing video on changing the mode of the NIC permanently with the MFT tool. As the configuration gets saved to the NIC itself, you can use any other Linux or FreeBSD installation, use MFT to change the values, and then reinstall pfSense. After that, the NIC is correctly in the ETH mode, and the driver loads correctly without issues. (I see no issues displayed in dmesg as far as I can tell.)
Video: https://www.youtube.com/watch?v=D1qN7Qg3bSg
Now, I just need to make sure the cards run stable and reliably, but from what I see now, I am confident.
Thanks again to everyone who helped.