Chelsio T540-CR Not showing up

tsexton

Good afternoon,
I'm building a new Pf-sense box with a Chelsio T540-CR which was recommended to me and supported by the FreeBSD OS and i believe the Pf-sense Hardware list. The card does not post during boot so im left to believe i need to enable some sort of driver for this. I'm new to FreeBSD but not to UNIX/Linux. Can someone point me to a walk though on how to enable this card on PF-Sense?

stephenw10

Does pfSense boot just without seeing the card?

Does the card appear in the output of pciconf -lv?

Steve

tsexton

No it does not show up with pciconf -lv
the Intel quad gigabit card shows up as well as the 2 Intel I350 ports
along with every other thing in the HP DL120 Server. I thought the card would have been recognized at boot.. from what i read. Maybe the card is just bad..

stephenw10

Yes, it could be. Or there's some low level incompatibility. It's not a driver issue if it doesn't appear there at all. Even as noneX. I would test it in something else if you can.

Steve

tsexton

So the New Card came in yesterday and i put it in the server. and its now recognized. the first boot said it updated the driver. but the sfp+ modules are not coming up. i have 4 Chelsio Avago AFBR-703SDZ which are supposed to be compatible with the T540-CR. so I'm more than a bit stumped now.

stephenw10

Ok. The links won't come up until the NIC itself is up. They may not link at 1G unless forced.

Do the NICs see the SFP modules? Try running ifconfig -vvvm cxl0, assuming the NICs are seen as cxl. It should show the module info.

Steve

tsexton

this is the output..

cxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=3ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,TXRTLMT,HWRXTSTMP>
capabilities=3ecc7bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,TOE4,TOE6,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,TXRTLMT,HWRXTSTMP>
ether 00:07:43:30:ab:b0
inet6 fe80::207:43ff:fe30:abb0%cxl0 prefixlen 64 scopeid 0x1
media: Ethernet 10Gbase-SR <full-duplex,rxpause,txpause>
status: no carrier
supported media:
media 10Gbase-SR mediaopt full-duplex,rxpause,txpause
media 10Gbase-SR mediaopt full-duplex,rxpause
media 10Gbase-SR mediaopt full-duplex,txpause
media 10Gbase-SR mediaopt full-duplex
media 1000baseSX mediaopt full-duplex,rxpause,txpause
media 1000baseSX mediaopt full-duplex,rxpause
media 1000baseSX mediaopt full-duplex,txpause
media 1000baseSX mediaopt full-duplex
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
vendor: AVAGO PN: AFBR-703SDZ SN: AA1202AFZLK DATE: 2012-01-12
module temperature: 36.53 C Voltage: 3.26 Volts
RX: 0.00 mW (-40.00 dBm) TX: 0.57 mW (-2.42 dBm)

    SFF8472 DUMP (0xA0 0..127 range):
    03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00 
    08 03 00 1E 41 56 41 47 4F 20 20 20 20 20 20 20 
    20 20 20 20 00 00 17 6A 41 46 42 52 2D 37 30 33 
    53 44 5A 20 20 20 20 20 47 32 2E 33 03 52 00 A5 
    00 1A 00 00 41 41 31 32 30 32 41 46 5A 4C 4B 20 
    20 20 20 20 31 32 30 31 31 32 20 20 68 F0 03 3B 
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

It definitely sees the card and the sfp modules..

stephenw10

Ok great, ht eNIC shows, the driver is attached and it sees the SFP+ module. But it's showing:

RX: 0.00 mW (-40.00 dBm)

It's not seeing any incoming signal on the fiber. Is it connected? What is it connected to?

tsexton

I played around with it.. and it turns out that the card interfaces comes up the opposite way that i would figure instead of left to right it came up right to left. so what i thought was cxl0 was actually cxl3 I always thought the first interface was generally the one closest to the PCIe slot but not in this case.

Anyway I got it up and was able to get it to route traffic. but i followed this guide
https://protectli.com/kb/how-to-enable-lan-bridge-with-pfsense/] to create a Bridge of the 4 10G Ports and 2 1G ports from my Intel card. but it seems that unless i directly assign the 10G cards a IP they do not provide link for traffic routing.

currently i have a bridge created with
WAN= igb0
WAN1= igb1
LAN = bridge-0 -->cxl3 cxl2 cxl1 cxl0 em0 em1
DMZ0 = em2
DMZ1 = em3

WAN goes directly to my ISP 1G Fiber
WAN1 = is currently not in use
DMZ0 will be sand-boxed and routed only to WAN
DMZ1 will be sand-boxed as well and routed to WAN1

am i missing something? should i set LAN = cxl3 and then bridge LAN with the other 4 ports I dont understand why i need to assign them IP's for them to provide a routable interface. Maybe if i assign the Switch port a IP? i just don't know.

some Output
CLX3 shows Active

[2.6.0-RELEASE][admin@gateway.local]/root: ifconfig cxl3
cxl3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: 10G3
options=3a800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE,RXCSUM_IPV6,TXRTLMT,HWRXTSTMP>
ether 00:07:43:30:ab:c8
inet6 fe80::207:43ff:fe30:abc8%cxl3 prefixlen 64 scopeid 0x4
media: Ethernet 10Gbase-SR <full-duplex,rxpause,txpause>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

I'm missing something stupid. and i just cant figure out what.

tsexton

So turned out i had a bad SFP Module. Replaced that and im good to go.
Thanks for your help and pointing me in the right direction.. You got me pointed in the right direction.