NIC Broadcom 57412 10Gb not recognized



  • Hi,

    We're running pfSense 2.4.4 on a Dell Poweredge R640 with 2 Broadcom 57412, one with two 1Gb and two 10gb ports and the other with two 10gb ports. None of the four 10Gb ports are recognized.

    I checked with "pciconf -lv", the ports appear as none@ :

    none46@pci0:25:0:0: class=0x020000 card=0x412014e4 chip=0x16d614e4 rev=0x01 hdr=0x00
    vendor = 'Broadcom Limited'
    device = 'BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller'
    class = network
    subclass = ethernet
    none47@pci0:25:0:1: class=0x020000 card=0x412014e4 chip=0x16d614e4 rev=0x01 hdr=0x00
    vendor = 'Broadcom Limited'
    device = 'BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller'
    class = network
    subclass = ethernet
    ...

    /var/log/dmesg shows the following :

    pci13: <dasp, performance counters> at device 14.0 (no driver attached)
    pci13: <dasp, performance counters> at device 15.0 (no driver attached)
    pci13: <dasp, performance counters> at device 18.0 (no driver attached)
    pci13: <dasp, performance counters> at device 18.1 (no driver attached)

    Is there some way to load the driver for this NIC ?

    Thanks a lot for your help.

    Regards,
    Alain


  • Netgate Administrator

    I would have expected those to be supported by the bnxt(4) driver.

    https://github.com/pfsense/FreeBSD-src/blob/b00c407ba5d1eb17263979f8182003ee0288d88b/sys/dev/bnxt/if_bnxt.c#L70

    That chip shown in your output is listed:
    https://github.com/pfsense/FreeBSD-src/blob/b00c407ba5d1eb17263979f8182003ee0288d88b/sys/dev/bnxt/bnxt.h#L67

    I would, therefore, expect to see some log lines at boot showing the driver trying and failing to attach.

    Steve



  • Re: NIC Broadcom 57412 10Gb not recognized
    Hello,
    I have exactly the same problem with a brand new Dell Poweredge R440 with two BCM57412 (10G SFP+), one BCM57416 (10G Base-T) and two onboard BCM5720 1Gb ports. Only the 1Gb ports are reconized on pfsense 2.4.4, none of the 10Gb ports.

    pciconf is same as Alain :
    [2.4.4-RELEASE][root@pfSense.localdomain]/root: pciconf -lv
    [...]
    bge0@pci0:4:0:0: class=0x020000 card=0x001f1028 chip=0x165f14e4 rev=0x00 hdr=0x00
    vendor = 'Broadcom Limited'
    device = 'NetXtreme BCM5720 Gigabit Ethernet PCIe'
    class = network
    subclass = ethernet
    bge1@pci0:4:0:1: class=0x020000 card=0x001f1028 chip=0x165f14e4 rev=0x00 hdr=0x00
    vendor = 'Broadcom Limited'
    device = 'NetXtreme BCM5720 Gigabit Ethernet PCIe'
    class = network
    subclass = ethernet
    none74@pci0:59:0:0: class=0x020000 card=0x412114e4 chip=0x16d614e4 rev=0x01 hdr=0x00
    vendor = 'Broadcom Limited'
    device = 'BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller'
    class = network
    subclass = ethernet
    none75@pci0:59:0:1: class=0x020000 card=0x412114e4 chip=0x16d614e4 rev=0x01 hdr=0x00
    vendor = 'Broadcom Limited'
    device = 'BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller'
    class = network
    subclass = ethernet
    none88@pci0:94:0:0: class=0x020000 card=0x1fea1028 chip=0x16d814e4 rev=0x01 hdr=0x00
    vendor = 'Broadcom Limited'
    device = 'BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller'
    class = network
    subclass = ethernet
    none89@pci0:94:0:1: class=0x020000 card=0x1fea1028 chip=0x16d814e4 rev=0x01 hdr=0x00
    vendor = 'Broadcom Limited'
    device = 'BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller'
    class = network
    subclass = ethernet
    none156@pci0:175:0:0: class=0x020000 card=0x412114e4 chip=0x16d614e4 rev=0x01 hdr=0x00
    vendor = 'Broadcom Limited'
    device = 'BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller'
    class = network
    subclass = ethernet
    none157@pci0:175:0:1: class=0x020000 card=0x412114e4 chip=0x16d614e4 rev=0x01 hdr=0x00
    vendor = 'Broadcom Limited'
    device = 'BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller'
    class = network
    subclass = ethernet

    Dmesg don't seem to show some particular driver error, just :
    pci7: <network, ethernet> at device 0.0 (no driver attached)
    pci7: <network, ethernet> at device 0.1 (no driver attached)

    (full dmesg here : https://pastebin.com/SsryZ08C)

    I updated the cards firmware to 20.08.04.04, booted in UEFI or BIOS and try several settings via the cards bios, but without any success so far.

    I would be very glad if I could provide some help with this issue.

    Edouard



  • Well, I manage to get the 10Gb interfaces working, but not yet in a satisfactory way.

    I installed FreeBSD 11.2 on one server and like pfSense the 10Gb interfaces didn't show up after install. But the bnxt driver module was present in /boot/kernel and manual loading via 'kldload if_bnxt.ko' get all things up as intended.

    Back to pfSense 2.4.4, I couln't find the bnxt module to load. However copying /boot/kernel/if_bnxt.ko from FreeBSD to pfSense permit me to load it and get interfaces up. Finaly putting 'if_bnxt_load="YES"' in /boot/loader.conf.local made the module load on boot.

    That sure is a crude workaround, as I would prefer not having to manualy add a kernel module. I looked for a bnxt kernel driver or module if pfsense without success.

    Is there a way to force the bnxt driver shipped with pfSense to load ? If not, where is the recommanded location to place if_bnxt.ko for it to persist upon future upgrades ?

    Regards,

    Edouard


  • Banned

    @edouard-moreau said in NIC Broadcom 57412 10Gb not recognized:

    If not, where is the recommanded location to place if_bnxt.ko for it to persist upon future upgrades ?

    Further upgrades will likely bump the FreeBSD version and as such will require a new matching kernel module anyway. So your best bet is to open a request on https://redmine.pfsense.org/ to have it included by default.



  • @grimson You're right. But this driver is supposed to already be part of pfSense, as Steve stated.


  • Banned

    @edouard-moreau said in NIC Broadcom 57412 10Gb not recognized:

    @grimson You're right. But this driver is supposed to already be part of pfSense, as Steve stated.

    Then open a bug report on redmine.


  • Netgate Administrator

    Whilst the driver is in the codebase it does not look like it's being built:

    [2.4.5-DEVELOPMENT][admin@5100.stevew.lan]/root: kldstat -v | grep bnxt
    [2.4.5-DEVELOPMENT][admin@5100.stevew.lan]/root: 
    

    So, yes, open that as a feature request on redmine to get it added.

    Steve



  • Feature request created : https://redmine.pfsense.org/issues/9155



  • @Edouard-Moreau Can you please send me a link for the if_bnxt.ko driver? I having the same problem with Broadcom 57412 Dual Port 10GbE SFP+.

    [2.4.4-RELEASE][admin@pfSense.localdomain]/root: dmesg | grep eth
    pci6: <network, ethernet> at device 0.0 (no driver attached)
    pci6: <network, ethernet> at device 0.1 (no driver attached)
    pci6: <network, ethernet> at device 0.0 (no driver attached)
    pci6: <network, ethernet> at device 0.1 (no driver attached)


  • Netgate Administrator

    Get it from the FreeBSD 11.2 ISO image.

    Comment on the feature request if it works for you.

    Steve



  • @stephenw10 I simply copied the file via scp and ran kldload ./if_bnxt.ko ; is that all I need to do? Do you mind sending instructions?

    I do see it now with kldstat :

    /root: kldstat -v | grep bnxt
    3 1 0xffffffff83223000 15b38 if_bnxt.ko (./if_bnxt.ko)
    659 pci/bnxt

    just reboot?



  • @rivest1000 Oh I guess I don't need a reboot :) :

    bnxt0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether b0:26:28:24:eb:e0
    hwaddr b0:26:28:24:eb:e0
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active
    bnxt1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether b0:26:28:24:eb:e1
    hwaddr b0:26:28:24:eb:e1
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active


  • Netgate Administrator

    Yup, you should just be able to kldload it like that. Put the kernel module in /boot/modules and then add the line if_bnxt_load="YES" to /boot/loader.conf.local so that it loads correctly at boot.

    Steve



  • @stephenw10
    I'm glad that you were able to get your nic working, but please comment on the feature request at https://redmine.pfsense.org/issues/9155. It would be much more confortable not to have to use the workaround every major update.



  • @stephenw10 Is there still an issue with pinging IPs on these interfaces? I got the to come up, but no ping after configuring same IPs that were on the 1G nics.



  • @stephenw10 I can get the nic to ping for about 5 minutes and then dies. Cannot ping out or in. It also doesn't seem to work with a tagged Vlan at all :(. Is this a driver issue?



  • @stephenw10 I believe I figured it out. It was not pinging when I made a vlan tag. I had to run "ifconfig bnxt0 promisc". how can I force it to stay promisc on reboot? and is it safe to leave promisc on in production environment?


  • Netgate Administrator

    Usually that's no problem. You can use a shellcmd to put the NIC promiscuous mode.

    You should probably try disabling any hw vlan tagging on the NIC too. That may be the cause there.

    Steve


Log in to reply