Chelsio T580-SO-CR

tman222

Thanks @stephenw10 and @Derelict.

Two ways:

I first saw on the interfaces widget in the dashboard that besides just "10Gbase-SR <full-duplex>", "RX Pause", "TX Pause" were now also included between the < >.
I confirmed this by via sysctl from the shell and saw that dev.cxl.x.pause_settings was set equal to 3 for both ports.

I'll try hw.cxgbe.pause_settings=0 next to see if that might make a difference. Thanks again for all your help.

xlameee

@stephenw10 here is what I get

stephenw10

Ok so nothing different. Are you able to test that card in FreeBSD 11.2?

Steve

xlameee

@tman222 I had that problem after I upgraded to 2.4.2_1 and then the same problem with 2.4.4

stephenw10

So can you retest with a version you didn't have a problem in?

The Chelsio driver can update the firmware on the card if it finds it's older than the version it has. That may have happened and it may be that that's causing the problem. If it is it would probably be a problem in any OS.

Do you actually see cxlX devices in pciconf -lv ?

Steve

xlameee

OK my other server is working
R320 exactly the same configuration and firmware and T580 firmware is the same and it is working perfectly just pfsense come up with the bug see https://forum.netgate.com/topic/136203/bug-v2-4-4

xlameee

@stephenw10 yes I can I thing

stephenw10

Ok so that is comparing your two systems?

Both are running 2.4.4 and both have the same Chelsio card in?

In that case I would try swapping the cards between the systems and see if the fault follows the card.

Otherwise can we see the complete output of pciconf -lv from both systems?

Steve

xlameee

@stephenw10 I did still no change :(

stephenw10

So you swapped the cards and the issue stayed with the same system?

Then if the same card works fine in pfSense 2.4.4 on the other system there is a difference with the system setup or something in the config that is different. The pciconf output between the two systems looks identical?

Steve

xlameee

@stephenw10 Hello

I just compare both firmware in lifecycle controller seems the stupid dell have mind of it own the lifecycle controller had upgraded itself to a newer version

FIRMWARE:

BIOS : v2.5.1
Lifecycle Controller : v1.1.1.18

HARDWARE I USED

CPU: E5-2450L 1.8GHz 8-core
RAM : 24GB ECC Registered
Network adapter: Chelsio T580-SO-CR
SAS Controller: PERC H310 mini
HDD: Intel SSD 545s Series 256GB
IDRAC: IDRAC7 Enterprise with dedicated card

Some tips:
iDRAC7 (if) make sure you disable auto update option
Lifecycle Controller: Disabled - Just in-case

I will tested to see if it works as is without tuning it and I will post it here so if someone wants to build system like mine won't go trough hell like me

Thank you

xlameee

UPDATE

Everything is working perfectly.
The total price of this configuration is less then $700 including R320, Chelsio T580-SO-CR and Intel 545 SSD
I will lower it to $600 because I just found seller on eBay selling Chelsio T580-LP-CR for less then $200. All recommendations on the net was, that T580-LP-CR is the best choice for pfSense.
I've got stuck for day on configuring NEXUS 3132Q-V, but was worth it. When I am done building the HA Cluster I will have the chance to test the speed and submit it to this post.

Thank you all

stephenw10

So just to confirm the cause of the issue here was the "lifecycle controller"? And only in the updated firmware version?

Glad it wasn't a conspiracy anyway.

Steve

xlameee

@stephenw10 Yes that was the problem stupid new auto firmware update on the idrac7 option
and before that to old version of the BIOS v1.5.2 upgraded to v2.5.1 and downgraded to LC v1.1.1.18 and all seems to work fine

Yes, sorry about that I was 24h straight trying to find the solution I am glad you are not a conspirator :)

Thank you

xlameee

Is that a bug or brand new Chelsio T580-SO-CR one of the ports just failed

"cxl1: Interface stopped DISTRIBUTING, possible flapping "

thank you

stephenw10

It's not a bug as far as I know. What is it connected to? Does it show the link as down? Does it comes back up if you disconnect and reconnect the link?

Steve

xlameee

@stephenw10 the link is up does not come down but I am getting this error on the vga console and I can't make sense of it.

stephenw10

If it part of a lagg group? Usually that indicates it's been removed from the group because it was showing the link go up and down too rapidly to be actual disconnects.
Is there nothing logged in the system log when that happens?
Are you actually seeing it lose connectivity.

Steve

Derelict

What is it connected to? What is in the logs there?

pglover19

xlameee… Can you provide me a link to the LC v1.1.1.18 software. I am trying to setup a DELL R320 server for pfSense and having some issues.

Thanks..