My Mellanox ConnectX-3 shows 10gbe then 1000BaseT

N0_Klu3

Hi all, I have a Mellanox MCX312B-XCCT CX312B ConnectX-3.

When I first started pfSense and set it up it was showing 10Gbe. After a few hours and finishing the setup its now showing 1000BaseT...
I have it setup with 2 VLAN's on it and its connected via a Unifi DAC UC-DAC-SFP+.

I'm not sure if its after adding the VLANs that it dropped link speed, or what caused it. But now even restarting it will not show as 10Gbe.
Can anyone help me diagnose why this might not run at full 10Gbe anymore when it did for a short period of time?

I tried Ubiquiti UF-MM-10G U Fibre Multi-Mode 10G Module with a 0.5M OM4 50/125 LC-LC DLX fibre cable but it doesnt even show lights or start at all. (Think it didnt like the Unifi SPF+ modules).

Does anyone have any SFP+ modules that will work with this Mellanox and a Unifi Switch US-48-500W?

Thanks all

stephenw10

What does the output of ifconfig -vvma show?

N0_Klu3

@stephenw10

mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: LAN
	options=ed00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	ether e4:1d:2d:07:a6:00
	inet6 fe80::e61d:2dff:fe07:a600%mlxen0 prefixlen 64 scopeid 0x9
	inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
	media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
	status: active
	supported media:
		media autoselect
		media 40Gbase-CR4 mediaopt full-duplex
		media 10Gbase-CX4 mediaopt full-duplex
		media 10Gbase-SR mediaopt full-duplex
		media 1000baseT mediaopt full-duplex
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 1X Copper Passive (Copper pigtail)
	vendor: UbiquitiNetworks PN: UC-DAC-SFP+ SN: H 21038003673 DATE: 2021-03-03
mlxen1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	ether e4:1d:2d:07:a6:01
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
		media 40Gbase-CR4 mediaopt full-duplex
		media 10Gbase-CX4 mediaopt full-duplex
		media 10Gbase-SR mediaopt full-duplex
		media 1000baseT mediaopt full-duplex
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
mlxen0.107: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: IOT
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=680703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	ether e4:1d:2d:07:a6:00
	inet6 fe80::e61d:2dff:fe07:a600%mlxen0.107 prefixlen 64 scopeid 0xb
	inet 192.168.107.1 netmask 0xffffff00 broadcast 192.168.107.255
	groups: vlan
	vlan: 107 vlanpcp: 0 parent interface: mlxen0
	media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
	status: active
	supported media:
		media autoselect
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
mlxen0.200: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: GUEST
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=680703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	ether e4:1d:2d:07:a6:00
	inet6 fe80::e61d:2dff:fe07:a600%mlxen0.200 prefixlen 64 scopeid 0xc
	inet 192.168.200.1 netmask 0xffffff00 broadcast 192.168.200.255
	groups: vlan
	vlan: 200 vlanpcp: 0 parent interface: mlxen0
	media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
	status: active
	supported media:
		media autoselect
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Do you need all the rest too?

stephenw10

DAC cables don't usually present any media options but the Mellanox NICs are unusual.

Have you tried forcing the link speed to '10Gbase-SR' rather than autoselect in the interface config?

N0_Klu3

@stephenw10

Ok so I went into my switch and forced 10gbe there.

This pushed it to pfSense too.
I set pfSense to 10Gbase-SR but nothing changed until I did the switch.

N0_Klu3

@stephenw10

Thanks for your help.
Did I do it right by doing the switch?
Is there any negatives to doing it this way?

Doing the switch via pfSense didnt do anything on its own.

stephenw10

Yes. Generally DAC cables do not offer multiple link speeds in pfSense as I said so it can usually only operate as autoselect. Selecting it at the switch is probably better though I would expect it to require the same setting at both ends.

Steve

N0_Klu3

@stephenw10
Thanks makes a bit of sense.

Would it be better to use fibre SFP+?
I did try the unifi 10gbe SFP+ modules with a fibre cable and it wouldnt even light up at all.

Do you know if there are any specifically compatible SFP+ Modules that would work with both Unifi and Mellanox?

stephenw10

SFP modules and fibre usually work more consistently in my experience and present more options in the NIC.

That allows you to get modules tested to be compatible at each end. It's possible to get custom DAC cables where each end is programmed for the device it's connected to but waaaay more expensive!

Steve