SFP+ DAC "media: Ethernet Unknown <full-duplex,rxpause,txpause>"
-
I'm using a Direct Attach Cable between a Chelsio T422-CR adapter and a Ubiquity Unifi US-16-XG switch.
On PfSense, ifconfig shows the Chelsio adapter as unable to determine an appropriate media type and link speed which appears to be causing issues establishing a LAGG connection.
The DAC being used is listed here https://www.provantage.com/accortec-tapcable1m-acc~7ACCBE4X.htm
Can anyone give me guidance here?
Thank you. -
You might be better asking here:-
https://service.chelsio.com/support/support.html
I wonder if the card supports the DAC, does the Ubiquity switch see the DAC ?
-
What do you see from
ifconfig cxl0 -vv
? Assuming cxl0 is the problem interface. cxgbe0 on the T4?Steve
-
cxgbe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether 98:be:94:12:d5:e0 hwaddr 98:be:94:12:d5:e0 inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255 inet6 2600:8800:9d00:17d:9abe:94ff:fe12:d5e0 prefixlen 64 inet6 fe80::1:1%cxgbe0 prefixlen 64 scopeid 0x1 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet Unknown <full-duplex,rxpause,txpause> status: active plugged: SFP/SFP+/SFP28 Unknown (LC) vendor: OEM PN: 81.8800-OC3SFPS SN: ACCT0999 DATE: 2018-11-02 module temperature: 256.00 C Voltage: 6.55 Volts RX: 6.55 mW (8.16 dBm) TX: 6.55 mW (8.16 dBm)
From that output it appears as if the DAC isn't being recognized. "plugged: SFP/SFP+/SFP28 Unknown (LC)"
I'm not sure if it's related but, assuming PCI4 relates to the Chelsio adapter and no other system hardware then these system log entries show some driver issues.
Sep 4 09:20:08 kernel pci4: <network, ethernet> at device 0.7 (no driver attached) Sep 4 09:20:08 kernel pci4: <serial bus, Fibre Channel> at device 0.6 (no driver attached) Sep 4 09:20:08 kernel pci4: <mass storage, SCSI> at device 0.5 (no driver attached) Sep 4 09:20:08 kernel t4nex0: PCIe gen2 x8, 4 ports, 42 MSI-X interrupts, 109 eq, 41 iq Sep 4 09:20:08 kernel cxgbe3: 8 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE) Sep 4 09:20:08 kernel cxgbe3: Ethernet address: 98:be:94:12:d5:f8 Sep 4 09:20:08 kernel cxgbe3: <port 3> on t4nex0 Sep 4 09:20:08 kernel cxgbe2: 8 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE) Sep 4 09:20:08 kernel cxgbe2: Ethernet address: 98:be:94:12:d5:f0 Sep 4 09:20:08 kernel cxgbe2: <port 2> on t4nex0 Sep 4 09:20:08 kernel cxgbe1: 8 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE) Sep 4 09:20:08 kernel cxgbe1: Ethernet address: 98:be:94:12:d5:e8 Sep 4 09:20:08 kernel cxgbe1: <port 1> on t4nex0 Sep 4 09:20:08 kernel cxgbe0: 8 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE) Sep 4 09:20:08 kernel cxgbe0: Ethernet address: 98:be:94:12:d5:e0 Sep 4 09:20:08 kernel cxgbe0: <port 0> on t4nex0 Sep 4 09:20:08 kernel t4nex0: <Chelsio T422-CR> mem 0xdf080000-0xdf0bffff,0xde800000-0xdeffffff,0xdf2c4000-0xdf2c5fff irq 22 at device 0.4 on pci4 Sep 4 09:20:08 kernel pci4: <network, ethernet> at device 0.3 (no driver attached) Sep 4 09:20:08 kernel pci4: <network, ethernet> at device 0.2 (no driver attached) Sep 4 09:20:08 kernel pci4: <ACPI PCI bus> on pcib4 Sep 4 09:20:08 kernel pcib4: <ACPI PCI-PCI bridge> mem 0xde1a0000-0xde1bffff irq 20 at device 3.0 on pci0
-
Does it recognise other DACs? Or other SFP modules?
You might try a different setting for
hw.cxgbe.autoneg
.Steve
-
I only have a handfull of these specific DACs. I was thinking my next step would be to purchase a pair that Ubiquiti sells in their store https://store.ui.com/collections/accessories/products/unifi-direct-attach-copper-cable-10-gbps-1-meter
Originally I meant to purchase a set of Chelsio DACs https://www.chelsio.com/wp-content/uploads/resources/tapcable-1m-pb.pdf and I thought that's what I did but on closer inspection I don't believe these are actually what I acquired.
I'll go ahead and set
hw.cxgbe.autoneg="1"
inloader.conf
andloader.conf.local
after I get home from work in the morning and see if that makes a difference.On a side note, whenever I try to query a
hw.cxgbe.*
sysctl from the command line I always get the outputsysctl: unknown oid
. Is there something extra I need to do to query their values? I would like to check that they were set properly after rebooting. -
I see output there even on a device that doesn't have a Chelsio card:
[2.5.0-DEVELOPMENT][admin@apu.stevew.lan]/root: sysctl hw.cxgbe hw.cxgbe.lazy_tx_credit_flush: 1 hw.cxgbe.starve_fl: 0 hw.cxgbe.nm_holdoff_tmr_idx: 2 hw.cxgbe.nm_rx_nframes: 64 hw.cxgbe.nm_rx_ndesc: 256 hw.cxgbe.nm_black_hole: 0
You won't see those values there are they are loader tunables:
https://www.freebsd.org/cgi/man.cgi?query=cxl&apropos=0&sektion=4&manpath=FreeBSD+11-current&format=htmlSteve
-
After setting
hw.cxgbe.autoneg="1"
and rebooting the firewall we are still unable to identify the DAC. I'm thinking this cable is unsupported. Might there be anything else to try? -
That seems likely. There is no config option to allow SFP modules I can see for cxgbe. They do have a lot of options though. Updating the card firmware for example might be an option.
Using a different DAC cable is probably a better option though.Steve
-
dev.t4nex.0.firmware_version: 1.19.1.0
I'm not sure if this is the latest available but I havehw.cxgbe.fw_install="2"
so it should force install whatever firmware is present in the driver being used, that being said it is most likely the latest available for the Chelsio driver provided by FreeBSD 11.2-RELEASE-p10.I'll go ahead and order two of the Ubiquiti DACs and see what happens.
Thank you for all your help I really appreciate it.
-
@kklouzal said in SFP+ DAC "media: Ethernet Unknown <full-duplex,rxpause,txpause>":
hw.cxgbe.autoneg="1"
What ever became of this? I'm hitting the same situation with a Chelsio 422-CR. I have the Ubiquiti UDC-3 (3 meter) and USW-16-150 switch.
-
So 1G uplink ports and you need to force the cxl ports to match?
That can be an problem with DAC cables.
Try running
ifconfig -vvvvvm cxl
and see what it offers.Steve
-
I forced that switch to 1Gb/Full-duplex. This is what I got from the
ifconfig -vvvvm cxgbe1
-
Hmm, what NIC is that? Is it not 10G?
Seems like it's autoselect only which is quite common with a DAC attached. In which case I would expect it to only link with the switch also set to auto-negotiate. I assume you tried that first though?
-
@stephenw10 ,
I'm connecting the ports with a Ubiquiti UDC-3 DAC. It is a passive DAC that is certified to run both 1Gb and 10Gb with Ubiquiti equipment. I figured that was my best chance of getting a working DAC. I'm using the SPF+ ports on the T422-CR to the SPF ports of the USW-16-150.I tried getting it to connect with hw.cxgbe.autoneg = 1, 0, and -1. None of them worked. I've also force the USW-16-150 sfp ports to 1000Gb/Full. So there should be no autoneg needed.
-
With that DAC cable there is no option to set a fixed speed in the Chelsio NIC. It can only be autoselect.
That is quite common and in those situations it will often only link at 10G.Using fibre modules instead of DAC will probably allow it if you can test that.
Steve