Chelsio T580-SO-CR

Grimson

Ether I am crazy or there is a conspiracy going on

You are crazy. Driver updates and deprecation is largely done by the FreeBSD developers, not the pfSense developers. If at all the pfSense devs add additional drivers.

stephenw10

There is no conspiracy to break NICs I assure you. But feel free to review the source code:
https://github.com/pfsense/FreeBSD-src/tree/RELENG_2_4_4/sys/dev/cxgbe

What is the actual PCI ID of that card? Is it listed here?:
https://github.com/pfsense/FreeBSD-src/blob/021c0c7c1ad07dad2227fb7529e176648cc53716/sys/dev/cxgbe/t4_main.c#L648

Yes, if there are no errors you can see there a good test here would be to try the adapter in FreeBSD 11.2. If it works there and not in pfSense then it could be a change we've accidentally introduced and we can look into that.

Steve

xlameee

@stephenw10
Sep 30 13:40:57 kernel t5nex0: <Chelsio T580-LP-SO-CR> mem 0xd9d00000-0xd9d7ffff,0xd8000000-0xd8ffffff,0xd9ffa000-0xd9ffbfff irq 48 at device 0.4 numa-domain 0 on pci2
Sep 30 13:40:57 kernel t5nex0: fw init failed: 5.
Sep 30 13:40:57 kernel t5nex0: error during attach, adapter is now in recovery mode.

stephenw10

Hmm, so some firmware issue. The t5 firmware modules seem to be loaded:

[2.4.4-RELEASE][root@4220.stevew.lan]/root: kldstat -v | grep t5
		112 pci/t5iov
		118 t5nex/cxl
		115 pci/t5nex
		124 t5fw_cfg_fw

What firmware version is on the card? Has it been updated?

Testing in FreeBSD 11.2 is probably the next step though just to be sure it works as expected there.

Steve

tman222

I'm not sure whether this is helpful, but I have the T520-SO-CR version of the card and also experienced some issues after the upgrade from 2.4.3_1 to 2.4.4, though nothing nearly as severe as what the OP is going through. When 2.4.4 came up I lost one of the network card tunables I had set. I had disabled Flow Control on the card, and while the tunable was still set in advanced settings, the card showed flow control as enabled after the system booted. Oddly enough, going into advanced settings and opening the flow control tunable and saving it again would then disable flow control. I have not found a way to get around this, including adding the flow control options into loader.conf.local. The system will boot with flow control enabled until I manually disable it. Could it be that something with the driver did change in FreeBSD?

stephenw10

It might have been. I'm not aware if any changes there though. Even if there were I would still expect it to conform to the sysctl settings.

What values are you actually setting? hw.cxgbe.pause_settings?

Steve

tman222

Hi @stephenw10 - actually I'm setting this tunable (covers both ports on the card):

dev.cxl.0.pause_settings=0
dev.cxl.1.pause_settings=0

This worked perfectly in 2.4.3_1 and prior. In 2.4.4 even with these values set under System Tunables and within loader.conf.local, flow control is enabled when the system fresh boots. Maybe something changed?

stephenw10

Well I would certainly try that variable I posted that should cover both ports. I wasn't aware that had changed but...

Steve

Derelict

According to the 11.2 cxl manual page nothing has changed there.

How are you determining flow control is still enabled after boot?

tman222

Thanks @stephenw10 and @Derelict.

Two ways:

I first saw on the interfaces widget in the dashboard that besides just "10Gbase-SR <full-duplex>", "RX Pause", "TX Pause" were now also included between the < >.
I confirmed this by via sysctl from the shell and saw that dev.cxl.x.pause_settings was set equal to 3 for both ports.

I'll try hw.cxgbe.pause_settings=0 next to see if that might make a difference. Thanks again for all your help.

xlameee

@stephenw10 here is what I get

stephenw10

Ok so nothing different. Are you able to test that card in FreeBSD 11.2?

Steve

xlameee

@tman222 I had that problem after I upgraded to 2.4.2_1 and then the same problem with 2.4.4

stephenw10

So can you retest with a version you didn't have a problem in?

The Chelsio driver can update the firmware on the card if it finds it's older than the version it has. That may have happened and it may be that that's causing the problem. If it is it would probably be a problem in any OS.

Do you actually see cxlX devices in pciconf -lv ?

Steve

xlameee

OK my other server is working
R320 exactly the same configuration and firmware and T580 firmware is the same and it is working perfectly just pfsense come up with the bug see https://forum.netgate.com/topic/136203/bug-v2-4-4

xlameee

@stephenw10 yes I can I thing

stephenw10

Ok so that is comparing your two systems?

Both are running 2.4.4 and both have the same Chelsio card in?

In that case I would try swapping the cards between the systems and see if the fault follows the card.

Otherwise can we see the complete output of pciconf -lv from both systems?

Steve

xlameee

@stephenw10 I did still no change :(

stephenw10

So you swapped the cards and the issue stayed with the same system?

Then if the same card works fine in pfSense 2.4.4 on the other system there is a difference with the system setup or something in the config that is different. The pciconf output between the two systems looks identical?

Steve

xlameee

@stephenw10 Hello

I just compare both firmware in lifecycle controller seems the stupid dell have mind of it own the lifecycle controller had upgraded itself to a newer version

FIRMWARE:

BIOS : v2.5.1
Lifecycle Controller : v1.1.1.18

HARDWARE I USED

CPU: E5-2450L 1.8GHz 8-core
RAM : 24GB ECC Registered
Network adapter: Chelsio T580-SO-CR
SAS Controller: PERC H310 mini
HDD: Intel SSD 545s Series 256GB
IDRAC: IDRAC7 Enterprise with dedicated card

Some tips:
iDRAC7 (if) make sure you disable auto update option
Lifecycle Controller: Disabled - Just in-case

I will tested to see if it works as is without tuning it and I will post it here so if someone wants to build system like mine won't go trough hell like me

Thank you