Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    NIC Broadcom 57412 10Gb not recognized

    Scheduled Pinned Locked Moved
    Hardware
    6
    20
    5.5k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      Admin-loma
      last edited by

      Hi,

      We're running pfSense 2.4.4 on a Dell Poweredge R640 with 2 Broadcom 57412, one with two 1Gb and two 10gb ports and the other with two 10gb ports. None of the four 10Gb ports are recognized.

      I checked with "pciconf -lv", the ports appear as none@ :

      none46@pci0:25:0:0: class=0x020000 card=0x412014e4 chip=0x16d614e4 rev=0x01 hdr=0x00
      vendor = 'Broadcom Limited'
      device = 'BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller'
      class = network
      subclass = ethernet
      none47@pci0:25:0:1: class=0x020000 card=0x412014e4 chip=0x16d614e4 rev=0x01 hdr=0x00
      vendor = 'Broadcom Limited'
      device = 'BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller'
      class = network
      subclass = ethernet
      ...

      /var/log/dmesg shows the following :

      pci13: <dasp, performance counters> at device 14.0 (no driver attached)
      pci13: <dasp, performance counters> at device 15.0 (no driver attached)
      pci13: <dasp, performance counters> at device 18.0 (no driver attached)
      pci13: <dasp, performance counters> at device 18.1 (no driver attached)

      Is there some way to load the driver for this NIC ?

      Thanks a lot for your help.

      Regards,
      Alain

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        I would have expected those to be supported by the bnxt(4) driver.

        https://github.com/pfsense/FreeBSD-src/blob/b00c407ba5d1eb17263979f8182003ee0288d88b/sys/dev/bnxt/if_bnxt.c#L70

        That chip shown in your output is listed:
        https://github.com/pfsense/FreeBSD-src/blob/b00c407ba5d1eb17263979f8182003ee0288d88b/sys/dev/bnxt/bnxt.h#L67

        I would, therefore, expect to see some log lines at boot showing the driver trying and failing to attach.

        Steve

        E 1 Reply Last reply Reply Quote 0
        • E
          Edouard Moreau @stephenw10
          last edited by

          Re: NIC Broadcom 57412 10Gb not recognized
          Hello,
          I have exactly the same problem with a brand new Dell Poweredge R440 with two BCM57412 (10G SFP+), one BCM57416 (10G Base-T) and two onboard BCM5720 1Gb ports. Only the 1Gb ports are reconized on pfsense 2.4.4, none of the 10Gb ports.

          pciconf is same as Alain :
          [2.4.4-RELEASE][root@pfSense.localdomain]/root: pciconf -lv
          [...]
          bge0@pci0:4:0:0: class=0x020000 card=0x001f1028 chip=0x165f14e4 rev=0x00 hdr=0x00
          vendor = 'Broadcom Limited'
          device = 'NetXtreme BCM5720 Gigabit Ethernet PCIe'
          class = network
          subclass = ethernet
          bge1@pci0:4:0:1: class=0x020000 card=0x001f1028 chip=0x165f14e4 rev=0x00 hdr=0x00
          vendor = 'Broadcom Limited'
          device = 'NetXtreme BCM5720 Gigabit Ethernet PCIe'
          class = network
          subclass = ethernet
          none74@pci0:59:0:0: class=0x020000 card=0x412114e4 chip=0x16d614e4 rev=0x01 hdr=0x00
          vendor = 'Broadcom Limited'
          device = 'BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller'
          class = network
          subclass = ethernet
          none75@pci0:59:0:1: class=0x020000 card=0x412114e4 chip=0x16d614e4 rev=0x01 hdr=0x00
          vendor = 'Broadcom Limited'
          device = 'BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller'
          class = network
          subclass = ethernet
          none88@pci0:94:0:0: class=0x020000 card=0x1fea1028 chip=0x16d814e4 rev=0x01 hdr=0x00
          vendor = 'Broadcom Limited'
          device = 'BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller'
          class = network
          subclass = ethernet
          none89@pci0:94:0:1: class=0x020000 card=0x1fea1028 chip=0x16d814e4 rev=0x01 hdr=0x00
          vendor = 'Broadcom Limited'
          device = 'BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller'
          class = network
          subclass = ethernet
          none156@pci0:175:0:0: class=0x020000 card=0x412114e4 chip=0x16d614e4 rev=0x01 hdr=0x00
          vendor = 'Broadcom Limited'
          device = 'BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller'
          class = network
          subclass = ethernet
          none157@pci0:175:0:1: class=0x020000 card=0x412114e4 chip=0x16d614e4 rev=0x01 hdr=0x00
          vendor = 'Broadcom Limited'
          device = 'BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller'
          class = network
          subclass = ethernet

          Dmesg don't seem to show some particular driver error, just :
          pci7: <network, ethernet> at device 0.0 (no driver attached)
          pci7: <network, ethernet> at device 0.1 (no driver attached)

          (full dmesg here : https://pastebin.com/SsryZ08C)

          I updated the cards firmware to 20.08.04.04, booted in UEFI or BIOS and try several settings via the cards bios, but without any success so far.

          I would be very glad if I could provide some help with this issue.

          Edouard

          E 1 Reply Last reply Reply Quote 0
          • E
            Edouard Moreau @Edouard Moreau
            last edited by Edouard Moreau

            Well, I manage to get the 10Gb interfaces working, but not yet in a satisfactory way.

            I installed FreeBSD 11.2 on one server and like pfSense the 10Gb interfaces didn't show up after install. But the bnxt driver module was present in /boot/kernel and manual loading via 'kldload if_bnxt.ko' get all things up as intended.

            Back to pfSense 2.4.4, I couln't find the bnxt module to load. However copying /boot/kernel/if_bnxt.ko from FreeBSD to pfSense permit me to load it and get interfaces up. Finaly putting 'if_bnxt_load="YES"' in /boot/loader.conf.local made the module load on boot.

            That sure is a crude workaround, as I would prefer not having to manualy add a kernel module. I looked for a bnxt kernel driver or module if pfsense without success.

            Is there a way to force the bnxt driver shipped with pfSense to load ? If not, where is the recommanded location to place if_bnxt.ko for it to persist upon future upgrades ?

            Regards,

            Edouard

            GrimsonG R 2 Replies Last reply Reply Quote 0
            • GrimsonG
              Grimson Banned @Edouard Moreau
              last edited by

              @edouard-moreau said in NIC Broadcom 57412 10Gb not recognized:

              If not, where is the recommanded location to place if_bnxt.ko for it to persist upon future upgrades ?

              Further upgrades will likely bump the FreeBSD version and as such will require a new matching kernel module anyway. So your best bet is to open a request on https://redmine.pfsense.org/ to have it included by default.

              E 1 Reply Last reply Reply Quote 0
              • E
                Edouard Moreau @Grimson
                last edited by

                @grimson You're right. But this driver is supposed to already be part of pfSense, as Steve stated.

                GrimsonG 1 Reply Last reply Reply Quote 0
                • GrimsonG
                  Grimson Banned @Edouard Moreau
                  last edited by

                  @edouard-moreau said in NIC Broadcom 57412 10Gb not recognized:

                  @grimson You're right. But this driver is supposed to already be part of pfSense, as Steve stated.

                  Then open a bug report on redmine.

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by stephenw10

                    Whilst the driver is in the codebase it does not look like it's being built:

                    [2.4.5-DEVELOPMENT][admin@5100.stevew.lan]/root: kldstat -v | grep bnxt
[2.4.5-DEVELOPMENT][admin@5100.stevew.lan]/root:

                    So, yes, open that as a feature request on redmine to get it added.

                    Steve

                    1 Reply Last reply Reply Quote 0
                    • E
                      Edouard Moreau
                      last edited by

                      Feature request created : https://redmine.pfsense.org/issues/9155

                      1 Reply Last reply Reply Quote 1
                      • R
                        rivest1000 @Edouard Moreau
                        last edited by

                        @Edouard-Moreau Can you please send me a link for the if_bnxt.ko driver? I having the same problem with Broadcom 57412 Dual Port 10GbE SFP+.

                        [2.4.4-RELEASE][admin@pfSense.localdomain]/root: dmesg | grep eth
                        pci6: <network, ethernet> at device 0.0 (no driver attached)
                        pci6: <network, ethernet> at device 0.1 (no driver attached)
                        pci6: <network, ethernet> at device 0.0 (no driver attached)
                        pci6: <network, ethernet> at device 0.1 (no driver attached)

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by stephenw10

                          Get it from the FreeBSD 11.2 ISO image.

                          Comment on the feature request if it works for you.

                          Steve

                          R 1 Reply Last reply Reply Quote 0
                          • R
                            rivest1000 @stephenw10
                            last edited by

                            @stephenw10 I simply copied the file via scp and ran kldload ./if_bnxt.ko ; is that all I need to do? Do you mind sending instructions?

                            I do see it now with kldstat :

                            /root: kldstat -v | grep bnxt
                            3 1 0xffffffff83223000 15b38 if_bnxt.ko (./if_bnxt.ko)
                            659 pci/bnxt

                            just reboot?

                            R 1 Reply Last reply Reply Quote 0
                            • R
                              rivest1000 @rivest1000
                              last edited by

                              @rivest1000 Oh I guess I don't need a reboot :) :

                              bnxt0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
                              options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
                              ether b0:26:28:24:eb:e0
                              hwaddr b0:26:28:24:eb:e0
                              nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                              media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
                              status: active
                              bnxt1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
                              options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
                              ether b0:26:28:24:eb:e1
                              hwaddr b0:26:28:24:eb:e1
                              nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                              media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
                              status: active

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                Yup, you should just be able to kldload it like that. Put the kernel module in /boot/modules and then add the line if_bnxt_load="YES" to /boot/loader.conf.local so that it loads correctly at boot.

                                Steve

                                E R 4 Replies Last reply Reply Quote 0
                                • E
                                  Edouard Moreau @stephenw10
                                  last edited by

                                  @stephenw10
                                  I'm glad that you were able to get your nic working, but please comment on the feature request at https://redmine.pfsense.org/issues/9155. It would be much more confortable not to have to use the workaround every major update.

                                  1 Reply Last reply Reply Quote 0
                                  • R
                                    rivest1000 @stephenw10
                                    last edited by

                                    @stephenw10 Is there still an issue with pinging IPs on these interfaces? I got the to come up, but no ping after configuring same IPs that were on the 1G nics.

                                    1 Reply Last reply Reply Quote 0
                                    • R
                                      rivest1000 @stephenw10
                                      last edited by

                                      @stephenw10 I can get the nic to ping for about 5 minutes and then dies. Cannot ping out or in. It also doesn't seem to work with a tagged Vlan at all :(. Is this a driver issue?

                                      1 Reply Last reply Reply Quote 0
                                      • R
                                        rivest1000 @stephenw10
                                        last edited by

                                        @stephenw10 I believe I figured it out. It was not pinging when I made a vlan tag. I had to run "ifconfig bnxt0 promisc". how can I force it to stay promisc on reboot? and is it safe to leave promisc on in production environment?

                                        1 Reply Last reply Reply Quote 0
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          Usually that's no problem. You can use a shellcmd to put the NIC promiscuous mode.

                                          You should probably try disabling any hw vlan tagging on the NIC too. That may be the cause there.

                                          Steve

                                          1 Reply Last reply Reply Quote 0
                                          • viktor_gV
                                            viktor_g Netgate
                                            last edited by

                                            now in 2.5:
                                            https://redmine.pfsense.org/issues/9155#note-7

                                            1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post