Chelsio T580-SO-CR



  • Hello

    I just build pfsense capable system with DELL R320 and Chelsio T580-SO-CR.
    The pfsense installed with no problem at all. My question is should I make any firmware or driver changes to trunk the vlans trough this adapter

    Thank you


  • Netgate Administrator

    I would not expect any changes to be required.

    If you've not seeing problems run as it is.

    Steve



  • @stephenw10 Ok is chelsio driver or firmware have some setting for a vlan or trunking because i kind of crash the server firmware and I will have to fix it before I can test the vlans
    if anyone know any guide how to configure or what can be configure on the driver

    Just note to anyone who is going to install pfsense on DELL R320 :
    1st make sure you get the server with BIOS 1.5.2 and Lifecycle Controller 1.1.5
    DO NOT UPGRADE THE FIRMWARE ON THE SERVER pfSense will not work with the latest firmware. Remove any PERC controller before you install pfSense R320 have embedded controller that will work fine with pfsense. If you get R320 with latest firmware roll them back in a proper way not like me :) 1st roll back Lifecycle controller to legacy version then IDRAC7 (if any) then BIOS. Unplug the power cords hold the power on/off key for at least 1 min and then power it on.
    I know is off the topic but I learn from my mistake because there was no info on internet how to do it property and I believe I've damage the main board.

    I hope someone can share with me how to configure or what I can configure on the driver for this card

    Thank you


  • Netgate Administrator

    I'm unsure what the situation was then. As far as I know there is no special setting required for that card. It should be supported by cxgbe(4).
    I expect the NICs to appear as cxl0 and cxl1, did that not happen?

    Steve



  • Did happen until I upgraded to pfsense 2.4.4 no longer supported or I don't know !!!!
    any suggestions


  • Netgate Administrator

    The card was working in 2.4.3 and is not in 2.4.4?

    Check the boot log for errors from the card/driver.

    Steve



  • @stephenw10 here it go 2.4.3 all interfaces working perfectly when upgrade to 2.4.3_1 or 2.4.4 that's it I feel like I am giving you hardware information to frucked up on the next version so you can sell your own hardware. No one had used this hardware before because I was looking all over this forum and then I start talking for this particular adapter well next not major version is no longer plug-n-play How about that ? Ether I am crazy or there is a conspiracy going on
    Yes I will check the log for errors if I find it !!!
    I do not have much of experience with freebsd that's why was looking for something that is plug-n-play and match my network gear

    Thank you



  • @xlameee said in Chelsio T580-SO-CR:

    Ether I am crazy or there is a conspiracy going on

    You are crazy. Driver updates and deprecation is largely done by the FreeBSD developers, not the pfSense developers. If at all the pfSense devs add additional drivers.


  • Netgate Administrator

    There is no conspiracy to break NICs I assure you. 😉 But feel free to review the source code:
    https://github.com/pfsense/FreeBSD-src/tree/RELENG_2_4_4/sys/dev/cxgbe

    What is the actual PCI ID of that card? Is it listed here?:
    https://github.com/pfsense/FreeBSD-src/blob/021c0c7c1ad07dad2227fb7529e176648cc53716/sys/dev/cxgbe/t4_main.c#L648

    Yes, if there are no errors you can see there a good test here would be to try the adapter in FreeBSD 11.2. If it works there and not in pfSense then it could be a change we've accidentally introduced and we can look into that.

    Steve



  • @stephenw10
    Sep 30 13:40:57 kernel t5nex0: <Chelsio T580-LP-SO-CR> mem 0xd9d00000-0xd9d7ffff,0xd8000000-0xd8ffffff,0xd9ffa000-0xd9ffbfff irq 48 at device 0.4 numa-domain 0 on pci2
    Sep 30 13:40:57 kernel t5nex0: fw init failed: 5.
    Sep 30 13:40:57 kernel t5nex0: error during attach, adapter is now in recovery mode.


  • Netgate Administrator

    Hmm, so some firmware issue. The t5 firmware modules seem to be loaded:

    [2.4.4-RELEASE][root@4220.stevew.lan]/root: kldstat -v | grep t5
    		112 pci/t5iov
    		118 t5nex/cxl
    		115 pci/t5nex
    		124 t5fw_cfg_fw
    

    What firmware version is on the card? Has it been updated?

    Testing in FreeBSD 11.2 is probably the next step though just to be sure it works as expected there.

    Steve



  • I'm not sure whether this is helpful, but I have the T520-SO-CR version of the card and also experienced some issues after the upgrade from 2.4.3_1 to 2.4.4, though nothing nearly as severe as what the OP is going through. When 2.4.4 came up I lost one of the network card tunables I had set. I had disabled Flow Control on the card, and while the tunable was still set in advanced settings, the card showed flow control as enabled after the system booted. Oddly enough, going into advanced settings and opening the flow control tunable and saving it again would then disable flow control. I have not found a way to get around this, including adding the flow control options into loader.conf.local. The system will boot with flow control enabled until I manually disable it. Could it be that something with the driver did change in FreeBSD?


  • Netgate Administrator

    It might have been. I'm not aware if any changes there though. Even if there were I would still expect it to conform to the sysctl settings.

    What values are you actually setting? hw.cxgbe.pause_settings?

    Steve



  • Hi @stephenw10 - actually I'm setting this tunable (covers both ports on the card):

    dev.cxl.0.pause_settings=0
    dev.cxl.1.pause_settings=0

    This worked perfectly in 2.4.3_1 and prior. In 2.4.4 even with these values set under System Tunables and within loader.conf.local, flow control is enabled when the system fresh boots. Maybe something changed?


  • Netgate Administrator

    Well I would certainly try that variable I posted that should cover both ports. I wasn't aware that had changed but...

    Steve


  • Netgate

    According to the 11.2 cxl manual page nothing has changed there.

    How are you determining flow control is still enabled after boot?



  • Thanks @stephenw10 and @Derelict.

    Two ways:

    1. I first saw on the interfaces widget in the dashboard that besides just "10Gbase-SR <full-duplex>", "RX Pause", "TX Pause" were now also included between the < >.
    2. I confirmed this by via sysctl from the shell and saw that dev.cxl.x.pause_settings was set equal to 3 for both ports.

    I'll try hw.cxgbe.pause_settings=0 next to see if that might make a difference. Thanks again for all your help.



  • @stephenw10 here is what I get
    0_1538385768270_2018-10-01_4-21-59.jpg


  • Netgate Administrator

    Ok so nothing different. Are you able to test that card in FreeBSD 11.2?

    Steve



  • @tman222 I had that problem after I upgraded to 2.4.2_1 and then the same problem with 2.4.4


  • Netgate Administrator

    So can you retest with a version you didn't have a problem in?

    The Chelsio driver can update the firmware on the card if it finds it's older than the version it has. That may have happened and it may be that that's causing the problem. If it is it would probably be a problem in any OS.

    Do you actually see cxlX devices in pciconf -lv ?

    Steve



  • OK my other server is working
    R320 exactly the same configuration and firmware and T580 firmware is the same and it is working perfectly just pfsense come up with the bug see https://forum.netgate.com/topic/136203/bug-v2-4-4

    3_1538409964397_chelsio-version.jpg 2_1538409964397_2018-10-01_6-20-23.jpg 1_1538409964397_2018-10-01_6-04-53.jpg 0_1538409964392_2018-10-01_6-01-47.jpg



  • @stephenw10 yes I can I thing

    0_1538411091800_2018-10-01_11-12-38.jpg

    1_1538411091800_2018-10-01_11-13-14.jpg


  • Netgate Administrator

    Ok so that is comparing your two systems?

    Both are running 2.4.4 and both have the same Chelsio card in?

    In that case I would try swapping the cards between the systems and see if the fault follows the card.

    Otherwise can we see the complete output of pciconf -lv from both systems?

    Steve



  • @stephenw10 I did still no change :(


  • Netgate Administrator

    So you swapped the cards and the issue stayed with the same system?

    Then if the same card works fine in pfSense 2.4.4 on the other system there is a difference with the system setup or something in the config that is different. The pciconf output between the two systems looks identical?

    Steve



  • @stephenw10 Hello

    I just compare both firmware in lifecycle controller seems the stupid dell have mind of it own the lifecycle controller had upgraded itself to a newer version

    FIRMWARE:

    BIOS : v2.5.1
    Lifecycle Controller : v1.1.1.18

    HARDWARE I USED

    CPU: E5-2450L 1.8GHz 8-core
    RAM : 24GB ECC Registered
    Network adapter: Chelsio T580-SO-CR
    SAS Controller: PERC H310 mini
    HDD: Intel SSD 545s Series 256GB
    IDRAC: IDRAC7 Enterprise with dedicated card

    Some tips:
    iDRAC7 (if) make sure you disable auto update option
    Lifecycle Controller: Disabled - Just in-case

    I will tested to see if it works as is without tuning it and I will post it here so if someone wants to build system like mine won't go trough hell like me

    Thank you



  • UPDATE

    Everything is working perfectly.
    The total price of this configuration is less then $700 including R320, Chelsio T580-SO-CR and Intel 545 SSD
    I will lower it to $600 because I just found seller on eBay selling Chelsio T580-LP-CR for less then $200. All recommendations on the net was, that T580-LP-CR is the best choice for pfSense.
    I've got stuck for day on configuring NEXUS 3132Q-V, but was worth it. When I am done building the HA Cluster I will have the chance to test the speed and submit it to this post.

    Thank you all


  • Netgate Administrator

    So just to confirm the cause of the issue here was the "lifecycle controller"? And only in the updated firmware version?

    Glad it wasn't a conspiracy anyway. 😉

    Steve



  • @stephenw10 Yes that was the problem stupid new auto firmware update on the idrac7 option
    and before that to old version of the BIOS v1.5.2 upgraded to v2.5.1 and downgraded to LC v1.1.1.18 and all seems to work fine

    Yes, sorry about that I was 24h straight trying to find the solution I am glad you are not a conspirator :)

    Thank you



  • Is that a bug or brand new Chelsio T580-SO-CR one of the ports just failed

    "cxl1: Interface stopped DISTRIBUTING, possible flapping "

    thank you


  • Netgate Administrator

    It's not a bug as far as I know. What is it connected to? Does it show the link as down? Does it comes back up if you disconnect and reconnect the link?

    Steve



  • @stephenw10 the link is up does not come down but I am getting this error on the vga console and I can't make sense of it.


  • Netgate Administrator

    If it part of a lagg group? Usually that indicates it's been removed from the group because it was showing the link go up and down too rapidly to be actual disconnects.
    Is there nothing logged in the system log when that happens?
    Are you actually seeing it lose connectivity.

    Steve


  • Netgate

    What is it connected to? What is in the logs there?



  • xlameee… Can you provide me a link to the LC v1.1.1.18 software. I am trying to setup a DELL R320 server for pfSense and having some issues.

    Thanks..



  • @pglover19 said in Chelsio T580-SO-CR:

    xlameee… Can you provide me a link to the LC v1.1.1.18 software. I am trying to setup a DELL R320 server for pfSense and having some issues.

    Thanks..

    One more thing.. How do you install the LC without an OS installed? I can't seem to install it from the LifeCycle Controller GUI upon boot up.



  • xlameee… Can you please answer my questions about the LC version for the Dell R320.