SFP+ DAC "media: Ethernet Unknown <full-duplex,rxpause,txpause>"

kklouzal

I'm using a Direct Attach Cable between a Chelsio T422-CR adapter and a Ubiquity Unifi US-16-XG switch.

On PfSense, ifconfig shows the Chelsio adapter as unable to determine an appropriate media type and link speed which appears to be causing issues establishing a LAGG connection.

The DAC being used is listed here https://www.provantage.com/accortec-tapcable1m-acc~7ACCBE4X.htm

Can anyone give me guidance here?
Thank you.

NogBadTheBad

You might be better asking here:-

https://service.chelsio.com/support/support.html

I wonder if the card supports the DAC, does the Ubiquity switch see the DAC ?

stephenw10

What do you see from ifconfig cxl0 -vv? Assuming cxl0 is the problem interface. cxgbe0 on the T4?

Steve

kklouzal

cxgbe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 98:be:94:12:d5:e0
	hwaddr 98:be:94:12:d5:e0
	inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
	inet6 2600:8800:9d00:17d:9abe:94ff:fe12:d5e0 prefixlen 64
	inet6 fe80::1:1%cxgbe0 prefixlen 64 scopeid 0x1
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet Unknown <full-duplex,rxpause,txpause>
	status: active
	plugged: SFP/SFP+/SFP28 Unknown (LC)
	vendor: OEM PN: 81.8800-OC3SFPS SN: ACCT0999 DATE: 2018-11-02
	module temperature: 256.00 C Voltage: 6.55 Volts
	RX: 6.55 mW (8.16 dBm) TX: 6.55 mW (8.16 dBm)

From that output it appears as if the DAC isn't being recognized. "plugged: SFP/SFP+/SFP28 Unknown (LC)"

I'm not sure if it's related but, assuming PCI4 relates to the Chelsio adapter and no other system hardware then these system log entries show some driver issues.

Sep 4 09:20:08	kernel		pci4: <network, ethernet> at device 0.7 (no driver attached)
Sep 4 09:20:08	kernel		pci4: <serial bus, Fibre Channel> at device 0.6 (no driver attached)
Sep 4 09:20:08	kernel		pci4: <mass storage, SCSI> at device 0.5 (no driver attached)
Sep 4 09:20:08	kernel		t4nex0: PCIe gen2 x8, 4 ports, 42 MSI-X interrupts, 109 eq, 41 iq
Sep 4 09:20:08	kernel		cxgbe3: 8 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)
Sep 4 09:20:08	kernel		cxgbe3: Ethernet address: 98:be:94:12:d5:f8
Sep 4 09:20:08	kernel		cxgbe3: <port 3> on t4nex0
Sep 4 09:20:08	kernel		cxgbe2: 8 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)
Sep 4 09:20:08	kernel		cxgbe2: Ethernet address: 98:be:94:12:d5:f0
Sep 4 09:20:08	kernel		cxgbe2: <port 2> on t4nex0
Sep 4 09:20:08	kernel		cxgbe1: 8 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)
Sep 4 09:20:08	kernel		cxgbe1: Ethernet address: 98:be:94:12:d5:e8
Sep 4 09:20:08	kernel		cxgbe1: <port 1> on t4nex0
Sep 4 09:20:08	kernel		cxgbe0: 8 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)
Sep 4 09:20:08	kernel		cxgbe0: Ethernet address: 98:be:94:12:d5:e0
Sep 4 09:20:08	kernel		cxgbe0: <port 0> on t4nex0
Sep 4 09:20:08	kernel		t4nex0: <Chelsio T422-CR> mem 0xdf080000-0xdf0bffff,0xde800000-0xdeffffff,0xdf2c4000-0xdf2c5fff irq 22 at device 0.4 on pci4
Sep 4 09:20:08	kernel		pci4: <network, ethernet> at device 0.3 (no driver attached)
Sep 4 09:20:08	kernel		pci4: <network, ethernet> at device 0.2 (no driver attached)
Sep 4 09:20:08	kernel		pci4: <ACPI PCI bus> on pcib4
Sep 4 09:20:08	kernel		pcib4: <ACPI PCI-PCI bridge> mem 0xde1a0000-0xde1bffff irq 20 at device 3.0 on pci0

stephenw10

Does it recognise other DACs? Or other SFP modules?

You might try a different setting for hw.cxgbe.autoneg.

Steve

kklouzal

I only have a handfull of these specific DACs. I was thinking my next step would be to purchase a pair that Ubiquiti sells in their store https://store.ui.com/collections/accessories/products/unifi-direct-attach-copper-cable-10-gbps-1-meter

Originally I meant to purchase a set of Chelsio DACs https://www.chelsio.com/wp-content/uploads/resources/tapcable-1m-pb.pdf and I thought that's what I did but on closer inspection I don't believe these are actually what I acquired.

I'll go ahead and set hw.cxgbe.autoneg="1" in loader.conf and loader.conf.local after I get home from work in the morning and see if that makes a difference.

On a side note, whenever I try to query a hw.cxgbe.* sysctl from the command line I always get the output sysctl: unknown oid. Is there something extra I need to do to query their values? I would like to check that they were set properly after rebooting.

stephenw10

I see output there even on a device that doesn't have a Chelsio card:

[2.5.0-DEVELOPMENT][admin@apu.stevew.lan]/root: sysctl hw.cxgbe
hw.cxgbe.lazy_tx_credit_flush: 1
hw.cxgbe.starve_fl: 0
hw.cxgbe.nm_holdoff_tmr_idx: 2
hw.cxgbe.nm_rx_nframes: 64
hw.cxgbe.nm_rx_ndesc: 256
hw.cxgbe.nm_black_hole: 0

You won't see those values there are they are loader tunables:
https://www.freebsd.org/cgi/man.cgi?query=cxl&apropos=0&sektion=4&manpath=FreeBSD+11-current&format=html

Steve

kklouzal

After setting hw.cxgbe.autoneg="1" and rebooting the firewall we are still unable to identify the DAC. I'm thinking this cable is unsupported. Might there be anything else to try?

stephenw10

That seems likely. There is no config option to allow SFP modules I can see for cxgbe. They do have a lot of options though. Updating the card firmware for example might be an option.
Using a different DAC cable is probably a better option though.

Steve

kklouzal

dev.t4nex.0.firmware_version: 1.19.1.0 I'm not sure if this is the latest available but I have hw.cxgbe.fw_install="2" so it should force install whatever firmware is present in the driver being used, that being said it is most likely the latest available for the Chelsio driver provided by FreeBSD 11.2-RELEASE-p10.

I'll go ahead and order two of the Ubiquiti DACs and see what happens.

Thank you for all your help I really appreciate it.

hydrian

@kklouzal said in SFP+ DAC "media: Ethernet Unknown <full-duplex,rxpause,txpause>":

hw.cxgbe.autoneg="1"

What ever became of this? I'm hitting the same situation with a Chelsio 422-CR. I have the Ubiquiti UDC-3 (3 meter) and USW-16-150 switch.

stephenw10

So 1G uplink ports and you need to force the cxl ports to match?

That can be an problem with DAC cables.

Try running ifconfig -vvvvvm cxl and see what it offers.

Steve

hydrian

@stephenw10 ,

I forced that switch to 1Gb/Full-duplex. This is what I got from the ifconfig -vvvvm cxgbe1

stephenw10

Hmm, what NIC is that? Is it not 10G?

Seems like it's autoselect only which is quite common with a DAC attached. In which case I would expect it to only link with the switch also set to auto-negotiate. I assume you tried that first though?

hydrian

@stephenw10 ,
I'm connecting the ports with a Ubiquiti UDC-3 DAC. It is a passive DAC that is certified to run both 1Gb and 10Gb with Ubiquiti equipment. I figured that was my best chance of getting a working DAC. I'm using the SPF+ ports on the T422-CR to the SPF ports of the USW-16-150.

I tried getting it to connect with hw.cxgbe.autoneg = 1, 0, and -1. None of them worked. I've also force the USW-16-150 sfp ports to 1000Gb/Full. So there should be no autoneg needed.

stephenw10

With that DAC cable there is no option to set a fixed speed in the Chelsio NIC. It can only be autoselect.
That is quite common and in those situations it will often only link at 10G.

Using fibre modules instead of DAC will probably allow it if you can test that.

Steve