Intel adapter, ixl driver and Mellanox cable

proxymoxy

Hi, I am trying to get a link up between a Microtik switch and a Supermicro server used as a router/firewall running pfsense.

The hardware in question is a Mellanox cable, MFA2P10-AXXX 25GbE SFP28 AOC, a Supermicro server that according to pciconf -lv is a 'Ethernet Connection X722 for 10GbE SFP+' and uses the ixl driver. On the other end is a Mikrotik CRS354-48G-4S+2Q+RM switch. The cable is connected to a SFP+ port on the mikrotik switch and on it the speed is set to 10Gb and full duplex.

No matter what settings I try then I never get a link between the two. If i set the cable to connect between two mikrotik switches then I get a link right away so the cable is fine.

What I read on the interwebz is that intel hardware like the X722 is really picky about what transceivers it talks to but this is on a Supermicro motherboard so I thought they perhaps might reprogram the firmware to be a bit more acceptable to other transceivers then intels. Also I read that the X722 only supports single speed transceivers but the cable is double speed (25Gb/10Gb), is this right about the X722 chipset?

When trying to set the speed on the X722 then I always get some kind of error and trying with sysctl dev.ixl.2.supported_speeds=4 (which I think is for 10Gb) then I get an error saying it is read only. I will try to set it in /boot/loader.conf.local later today, restart it and see if that works.

I am not that familiar with FreeBSD/pfSense so I might be missing something, but what can I do to get this fiber working between these two machines? Is this perhaps a dead quest?

stephenw10

Does it even recognise the module? What speeds does it offer?

Try running ifconfig -vvvm ixl0 at the CLI (assuming it's ixl0).

Steve

proxymoxy

@stephenw10 thanks for your reply.

Yes it does recognize the module, here is the output

ifconfig -vvvm ixl3
ixl3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: Office2
	options=e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether ac:1f:6b:79:85:eb
	inet6 fe80::ae1f:6bff:fe79:85eb%ixl3 prefixlen 64 scopeid 0x8
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
		media 10GBase-SFI
		media 1000Base-KX
		media 10Gbase-LR
		media 10Gbase-SR
		media 10Gbase-Twinax
		media 1000baseLX
		media 1000baseSX
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 100G AOC or 25GAUI C2M AOC (No separable connector)
	vendor: Mellanox PN: MFA2P10-A030 SN: MT1948FT02028 DATE: 2019-11-25
	module temperature: 46.56 C Voltage: 3.25 Volts
	RX: 0.80 mW (-0.94 dBm) TX: 0.72 mW (-1.38 dBm)

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 23 00 00 00 00 00 08 00 00 06 FF 00 00 00 
	00 00 1E 00 4D 65 6C 6C 61 6E 6F 78 20 20 20 20 
	20 20 20 20 18 00 02 C9 4D 46 41 32 50 31 30 2D 
	41 30 33 30 20 20 20 20 41 34 20 20 00 00 00 65 
	18 3A 67 00 4D 54 31 39 34 38 46 54 30 32 30 32 
	38 20 20 20 31 39 31 31 32 35 20 20 68 F8 08 01 
	39 34 32 46 4D 41 32 34 32 4A 30 31 33 37 20 20 
	00 00 00 00 00 00 00 00 00 00 01 00 20 00 00 00

stephenw10

Hmm, well that looks good. Interesting set of media options presented.

Note it shows zero incoming signal strength.

First thing I would try is setting the media type to a fixed value to match the other end.

proxymoxy

@stephenw10 the media types is one of the problems. With the ixl driver I can't set any media types, they cannot be changed it seems. If I try

sysctl dev.ixl.2.supported_speeds: 4

which I think is supposed to be 10Gb then I get

sysctl: oid 'dev.ixl.2.supported_speeds' is read only

The opposite end is set to 10Gb full duplex.
I tried to use /boot/loader.conf.local to set the values and I get

rc.bootup: The command '/sbin/ifconfig 'ixl2' media '10GBase-SFI'' returned exit code '1', the output was 'ifconfig: SIOCSIFMEDIA (media): Operation not supported by device'
ixl2: Media change is not supported.

stephenw10

If they show as available mediaopts in ifconfig -m they should be selectable. They should appear in the drop-down in the GUI. Though that error seems to imply they are not.

To restrict what speeds it can negotiate to in autoselect you should be able to use the advertise_speed sysctl. Though I don't have an ixl NIC up right now to test that with.

proxymoxy

@stephenw10 yes I can set the advertise_speed, but that means autonegotiation. This cable is set as 25Gb but can run as 10Gb if it disables CDR and it can only do that if you manually set the speed. So it seems like a catch-22 instance to me

stephenw10

So your options at thew switch do not include autoselect?

I've never used an 'optical' DAC device like that so I'm unsure how they usually appear. With a regular DAC cable though it's common to not present any media types except autoselect. And in those cases the NIC will usually only link at the native port speed, usually 10G. That might work fine here if both ends are 10G and support autoselect.

proxymoxy

@stephenw10 yes the switch can do autoselect. I have tried that and got no link between the two. The other end is a microtik switch on a sfp+ port (so max speed is 10Gb).

stephenw10

Hmm, well you have any other DAC cables or SFP modules and fiber you can try?

proxymoxy

@stephenw10 unfortunately not between these two machines. One is in a locked cabin on the ground floor, the second is in a server room on the second floor. This cable was layed out some time ago by electricians that support this building and can not be moved easily.
I think I have tried this with a 10Gb DAC (copper) cable and got a link, I don't remember exactly. Will try that tomorrow.