Chelsio cxgb driver not working?



  • Hello folks,
    I was attempting to use a Chelsio 10GB CX4 nic with pfsense 1.2.2.  According to the FreeBSD hardware list (http://www.freebsd.org/releases/7.0R/hardware.html#ETHERNET) this driver should be present in 1.2.2, but I never see it load.  No interface ever shows up, and I don't know where to go from here.

    The card I'm using is a T3 series Chelsio 10GB CX4 NIC model S310E-CXA.



  • pfSense 1.2.2 is based on FreeBSD 6.x hence there isn't much point looking at the FreeBSD 7.0 hardware release notes for supported hardware for pfSense 1.2.2. You will have to use a reasonably recent pfSense 1.2.3 build if you want a kernel based on FreeBSD 7.x.



  • Sorry, looks as if pfSense 1.2.3 is now available for download. I suggest you try that.

    By the way, the version of FreeBSD used is displayed in the first few lines of the startup output, e.g.

    $ dmesg
    Copyright © 1992-2009 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 7.2-RELEASE-p4 #0: Mon Nov 30 15:01:49 EST 2009
        sullrich@FreeBSD_7.2_pfSense_1.2.3_snaps.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense.7

    This is a from a FreeBSD 7.2 based build of pfSense.



  • wallabybob,

    Thanks.  I guess the site was updated to point to FreeBSD 7.0 when 1.2.3 was released - I didn't even check my running copy (1.2.2) to verify it was not 7.0.

    I just downloaded 1.2.3 and will give it a try again with the Chelsio cards.

    Thanks!



  • Well, the cxgb driver is supported with pfsense 1.2.3, but it doesn't appear to support VLANs, which is a requirement for my setup with this network card.

    These cards are super expensive so putting two in each firewall (I have two firewalls) would be cost-prohibitive.

    I also tried 5 different models of the newer Intel 10Gb cards and none of those were detected - I didn't expect them to be since the 7.0 hardware list shows the older style Intel PRO 10gb adapters being supported.  I have some of those, but they are notoriously poor performers.

    Back to 1Gb NICs  :-\



  • Does an ifconfig show any vlan capabilities? At one time, there was an internal list of VLAN capable interfaces that would need occasional updating. I thought that behavior had changed, but I'm not sure.



  • You're right, it does appear to support it:

    cxgb0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 9000
            options=5bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,lro>ether 00:07:43:05:06:06
            media: Ethernet 10Gbase-CX4 <full-duplex>(autoselect)
            status: no carrier

    And here's what a normal Broadcom or Intel interface looks like:

    bce3: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
            options=1bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4>ether 00:24:e8:78:ef:30
            media: Ethernet autoselect (none)
            status: no carrier

    How does one go about telling pfsense it's OK to do VLANs?  This would be huge if we could do this….</rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4></broadcast,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,lro></broadcast,simplex,multicast>



  • Found it.  http://forum.pfsense.org/index.php/topic,738.msg4853.html#msg4853

    I will give it a try now and post my results back here.



  • @stevemitchell:

    How does one go about telling pfsense it's OK to do VLANs?  This would be huge if we could do this….

    Edit /etc/inc/globals.inc, look for the line beginning "vlan_long_frame" => and add "cxgb" to the list of interface name strings and save the file. You MIGHT need to reboot.

    If you are planning to have quality of service on the cxgb interfaces you may need to add cxgb to the list of ALTQ capable interfaces in /etc/inc/interfaces.inc.

    I noticed igb is NOT in either of those lists. I suspect it should be in both lists but I haven't I checked against the igb man page and I don't know for certain that igb is built into the pfSense kernel.



  • Thanks.  Tried that and it worked to assign VLANs, but there's another more serious issue.

    Even without trying VLANs, which I didn't do because the 10Gb nic was useless to me without that, loading the cxgb driver and bringing up the interface throws a bunch of errors about firmware missing, draining tx and a bunch of other items.

    It assigns IP addresses to the interface and brings it up, but it never actually works correctly.  I never get link/carrier no matter what switch I plug it into.  Looks like there are missing firmware packages or something similar on the disk that the cxgb drivers require.

    I'm going to see if I can get it up on the 'net so I can get the errors off of the console and do a couple of searches, but this makes me feel less happy about using these interfaces in a production environment.  Especially if whatever I hack gets toasted when I upgrade.



  • Here are the errors.  It looks based on my Google searches that all I would need would be the firmware, which can't be included in FreeBSD due to licensing.  However, I have no idea where I would put the firmware package on the file system. Continuing to research…

    firmware_get: failed to load firmware image cxgb_t3fw
    cxgbc0: Could not find firmware image cxgb_t3fw
    draining slow intr
    draining ext intr
    draining tick task



  • It appears that either the driver was compiled weird, or that the necessary files were left out of the distribution.  I can find next to no people who use this card on FreeBSD 7.x so that makes me a bit scared to try it in production for a pretty big implementation.  We use these cards in CentOS systems everywhere with phenomenal performance over any other 10Gb NIC we've ever tested, but it appears the stars aren't aligned for us to use it here.

    I'm going to go back to the 1Gb Intel NICs we had prior to this and continue my build out.



  • This problem most likely due to pfsense having not included various cxgb  components. please contact support@chelsio.com and they can help get past this.



  • I have contacted them.  Still waiting to hear back…



  • I think "options firmware" is missing in the kernel config.



  • Sorry - not a FreeBSD guru - how would one go about adding that to pfsense?  I assume that it's something that the developers would have to change at compile time?


Log in to reply