Chelsio T520-CR problem

sites

Installed a new T520 in a Dell R420. The interfaces are there but the only time they light up is when looped to each other. Connect it to a switch (Netgear, Mikrotik, Arista) and, nothing. Tried several different SFP+ modules, even picked up the Chelsio that was listed on their site for this card & the interfaces just will not activate. But again, connect it to itself and they light up so I don't think the card is defective. I've even tried a few different firmware versions on this card. Same situation when I tried Opnsense.

What am I missing?

stephenw10

Linking at 10G to the switches? I assume they link at 10G to each other?

If those are 1G switches you will almost certainly need to set the link speed.

Steve

sites

@stephenw10 10G switches.

stephenw10

Does ifconfig -vvvm cxl0 report signal in and out?

sites

@stephenw10

       media: Ethernet 10Gbase-LR <full-duplex,rxpause,txpause>
       status: no carrier
       supported media:
               media 10Gbase-LR mediaopt full-duplex,rxpause,txpause
               media 10Gbase-LR mediaopt full-duplex,rxpause
               media 10Gbase-LR mediaopt full-duplex,txpause
               media 10Gbase-LR mediaopt full-duplex
               media 1000baseLX mediaopt full-duplex,rxpause,txpause
               media 1000baseLX mediaopt full-duplex,rxpause
               media 1000baseLX mediaopt full-duplex,txpause
               media 1000baseLX mediaopt full-duplex
       nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
       plugged: SFP/SFP+/SFP28 10G Base-LR (LC)
       vendor: Chelsio PN: SM10G-LR SN: WT101K82436 DATE: 2020-08-13
       module temperature: 49.00 C Voltage: 3.18 Volts
       RX: 0.65 mW (-1.82 dBm) TX: 0.56 mW (-2.47 dBm)

stephenw10

So not set to auto-select there? Is flowcontrol enabled at the switch end?

sites

@stephenw10 Auto-select is on, flow control is off. We can see light at the switch but the T520 just won't talk.

stephenw10

Ok try disabling flow control then since it doesn't appear to be negotiating.

sites

@stephenw10 Flow control is already disabled.

To be specific, auto-negotiation is off on the switch. It's enabled in the firewall interface. I've tried manually selecting the speed but same results.

stephenw10

<full-duplex,rxpause,txpause> this indicates flow-control is enabled.

sites

@stephenw10 added hw.cxgbe.pause_settings="0" to /boot/loader.conf but same results

stephenw10

Can you not set it in the interface config since it's a mediaopt in cxl?

Can you change the switch side settings?

Do you have anything besides the switch you can test linking it with?

Steve

sites

@stephenw10 One of the first changes I made was to explicitly set the speed and duplex mode on these interfaces directly in the GUI. We can change anything, and have been trying. We have the Arista, a Mikrotik, and a Juniper. We have another Dell server with T440 cards that links right up with all of these switches.

stephenw10

Mmm, to a 10G switch I expect it to link up in the default mode. You usually only need to set a fixed speed/duplex to connect at 1G.

But does it not offer you all the medioopt types in the webgui? That's where I would expect to set full-duplex without flow control.

Steve

sites

@stephenw10 Yes, there's a list of 8 options plus auto-detection. One of those options is "10Gbase-LR full-duplex" with no tx/rx pause. I've tried using this setting.

stephenw10

Do the switches show any error?

Does it link directly to the Dell server maybe?

sites

@stephenw10 Had to put this to the side for a bit, but I'll return to troubleshooting at a later date.