Chelsio cxgb driver not working?

wallabybob

Sorry, looks as if pfSense 1.2.3 is now available for download. I suggest you try that.

By the way, the version of FreeBSD used is displayed in the first few lines of the startup output, e.g.

$ dmesg
Copyright 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.2-RELEASE-p4 #0: Mon Nov 30 15:01:49 EST 2009
sullrich@FreeBSD_7.2_pfSense_1.2.3_snaps.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense.7

This is a from a FreeBSD 7.2 based build of pfSense.

stevemitchell

wallabybob,

Thanks. I guess the site was updated to point to FreeBSD 7.0 when 1.2.3 was released - I didn't even check my running copy (1.2.2) to verify it was not 7.0.

I just downloaded 1.2.3 and will give it a try again with the Chelsio cards.

Thanks!

stevemitchell

Well, the cxgb driver is supported with pfsense 1.2.3, but it doesn't appear to support VLANs, which is a requirement for my setup with this network card.

These cards are super expensive so putting two in each firewall (I have two firewalls) would be cost-prohibitive.

I also tried 5 different models of the newer Intel 10Gb cards and none of those were detected - I didn't expect them to be since the 7.0 hardware list shows the older style Intel PRO 10gb adapters being supported. I have some of those, but they are notoriously poor performers.

Back to 1Gb NICs :-\

dotdash

Does an ifconfig show any vlan capabilities? At one time, there was an internal list of VLAN capable interfaces that would need occasional updating. I thought that behavior had changed, but I'm not sure.

stevemitchell

You're right, it does appear to support it:

cxgb0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 9000
options=5bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,lro>ether 00:07:43:05:06:06
media: Ethernet 10Gbase-CX4 <full-duplex>(autoselect)
status: no carrier

And here's what a normal Broadcom or Intel interface looks like:

bce3: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
options=1bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4>ether 00:24:e8:78:ef:30
media: Ethernet autoselect (none)
status: no carrier

How does one go about telling pfsense it's OK to do VLANs? This would be huge if we could do this….</rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4></broadcast,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,lro></broadcast,simplex,multicast>

stevemitchell

Found it. http://forum.pfsense.org/index.php/topic,738.msg4853.html#msg4853

I will give it a try now and post my results back here.

wallabybob

@stevemitchell:

How does one go about telling pfsense it's OK to do VLANs? This would be huge if we could do this….

Edit /etc/inc/globals.inc, look for the line beginning "vlan_long_frame" => and add "cxgb" to the list of interface name strings and save the file. You MIGHT need to reboot.

If you are planning to have quality of service on the cxgb interfaces you may need to add cxgb to the list of ALTQ capable interfaces in /etc/inc/interfaces.inc.

I noticed igb is NOT in either of those lists. I suspect it should be in both lists but I haven't I checked against the igb man page and I don't know for certain that igb is built into the pfSense kernel.

stevemitchell

Thanks. Tried that and it worked to assign VLANs, but there's another more serious issue.

Even without trying VLANs, which I didn't do because the 10Gb nic was useless to me without that, loading the cxgb driver and bringing up the interface throws a bunch of errors about firmware missing, draining tx and a bunch of other items.

It assigns IP addresses to the interface and brings it up, but it never actually works correctly. I never get link/carrier no matter what switch I plug it into. Looks like there are missing firmware packages or something similar on the disk that the cxgb drivers require.

I'm going to see if I can get it up on the 'net so I can get the errors off of the console and do a couple of searches, but this makes me feel less happy about using these interfaces in a production environment. Especially if whatever I hack gets toasted when I upgrade.

stevemitchell

Here are the errors. It looks based on my Google searches that all I would need would be the firmware, which can't be included in FreeBSD due to licensing. However, I have no idea where I would put the firmware package on the file system. Continuing to research…

firmware_get: failed to load firmware image cxgb_t3fw
cxgbc0: Could not find firmware image cxgb_t3fw
draining slow intr
draining ext intr
draining tick task

stevemitchell

It appears that either the driver was compiled weird, or that the necessary files were left out of the distribution. I can find next to no people who use this card on FreeBSD 7.x so that makes me a bit scared to try it in production for a pretty big implementation. We use these cards in CentOS systems everywhere with phenomenal performance over any other 10Gb NIC we've ever tested, but it appears the stars aren't aligned for us to use it here.

I'm going to go back to the 1Gb Intel NICs we had prior to this and continue my build out.

kartik

This problem most likely due to pfsense having not included various cxgb components. please contact support@chelsio.com and they can help get past this.

stevemitchell

I have contacted them. Still waiting to hear back…

lsf

I think "options firmware" is missing in the kernel config.

stevemitchell

Sorry - not a FreeBSD guru - how would one go about adding that to pfsense? I assume that it's something that the developers would have to change at compile time?