Intel adapter, ixl driver and Mellanox cable

stephenw10

Does it even recognise the module? What speeds does it offer?

Try running ifconfig -vvvm ixl0 at the CLI (assuming it's ixl0).

Steve

proxymoxy

@stephenw10 thanks for your reply.

Yes it does recognize the module, here is the output

ifconfig -vvvm ixl3
ixl3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: Office2
	options=e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether ac:1f:6b:79:85:eb
	inet6 fe80::ae1f:6bff:fe79:85eb%ixl3 prefixlen 64 scopeid 0x8
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
		media 10GBase-SFI
		media 1000Base-KX
		media 10Gbase-LR
		media 10Gbase-SR
		media 10Gbase-Twinax
		media 1000baseLX
		media 1000baseSX
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 100G AOC or 25GAUI C2M AOC (No separable connector)
	vendor: Mellanox PN: MFA2P10-A030 SN: MT1948FT02028 DATE: 2019-11-25
	module temperature: 46.56 C Voltage: 3.25 Volts
	RX: 0.80 mW (-0.94 dBm) TX: 0.72 mW (-1.38 dBm)

	SFF8472 DUMP (0xA0 0..127 range):
	03 04 23 00 00 00 00 00 08 00 00 06 FF 00 00 00 
	00 00 1E 00 4D 65 6C 6C 61 6E 6F 78 20 20 20 20 
	20 20 20 20 18 00 02 C9 4D 46 41 32 50 31 30 2D 
	41 30 33 30 20 20 20 20 41 34 20 20 00 00 00 65 
	18 3A 67 00 4D 54 31 39 34 38 46 54 30 32 30 32 
	38 20 20 20 31 39 31 31 32 35 20 20 68 F8 08 01 
	39 34 32 46 4D 41 32 34 32 4A 30 31 33 37 20 20 
	00 00 00 00 00 00 00 00 00 00 01 00 20 00 00 00

stephenw10

Hmm, well that looks good. Interesting set of media options presented.

Note it shows zero incoming signal strength.

First thing I would try is setting the media type to a fixed value to match the other end.

proxymoxy

@stephenw10 the media types is one of the problems. With the ixl driver I can't set any media types, they cannot be changed it seems. If I try

sysctl dev.ixl.2.supported_speeds: 4

which I think is supposed to be 10Gb then I get

sysctl: oid 'dev.ixl.2.supported_speeds' is read only

The opposite end is set to 10Gb full duplex.
I tried to use /boot/loader.conf.local to set the values and I get

rc.bootup: The command '/sbin/ifconfig 'ixl2' media '10GBase-SFI'' returned exit code '1', the output was 'ifconfig: SIOCSIFMEDIA (media): Operation not supported by device'
ixl2: Media change is not supported.

stephenw10

If they show as available mediaopts in ifconfig -m they should be selectable. They should appear in the drop-down in the GUI. Though that error seems to imply they are not.

To restrict what speeds it can negotiate to in autoselect you should be able to use the advertise_speed sysctl. Though I don't have an ixl NIC up right now to test that with.

proxymoxy

@stephenw10 yes I can set the advertise_speed, but that means autonegotiation. This cable is set as 25Gb but can run as 10Gb if it disables CDR and it can only do that if you manually set the speed. So it seems like a catch-22 instance to me

stephenw10

So your options at thew switch do not include autoselect?

I've never used an 'optical' DAC device like that so I'm unsure how they usually appear. With a regular DAC cable though it's common to not present any media types except autoselect. And in those cases the NIC will usually only link at the native port speed, usually 10G. That might work fine here if both ends are 10G and support autoselect.

proxymoxy

@stephenw10 yes the switch can do autoselect. I have tried that and got no link between the two. The other end is a microtik switch on a sfp+ port (so max speed is 10Gb).

stephenw10

Hmm, well you have any other DAC cables or SFP modules and fiber you can try?

proxymoxy

@stephenw10 unfortunately not between these two machines. One is in a locked cabin on the ground floor, the second is in a server room on the second floor. This cable was layed out some time ago by electricians that support this building and can not be moved easily.
I think I have tried this with a 10Gb DAC (copper) cable and got a link, I don't remember exactly. Will try that tomorrow.