Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    bnxt NICs not passing VLAN traffic

    Scheduled Pinned Locked Moved Hardware
    16 Posts 2 Posters 1.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator
      last edited by

      Does it carry untagged traffic though?

      What drivers are the NICs using not drives in the system. Like ix or ixl for example.

      What does the output of ifconfig -vvvma show?

      Steve

      1 Reply Last reply Reply Quote 0
      • F
        fealiaga
        last edited by stephenw10

        @stephenw10 said in Help with this:

        ifconfig -vvvma

        the command:

        bnxt0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: TRUNK
        options=e120bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media 1000baseT
                media 10Gbase-T
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bnxt1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=e120bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:01
        inet6 fe80::be97:e1ff:fed8:c601%bnxt1 prefixlen 64 scopeid 0x2
        inet 146.155.7.186 netmask 0xffffffc0 broadcast 146.155.7.191
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media 1000baseT
                media 10Gbase-T
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
        capabilities=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 70:b5:e8:d1:39:84
        inet6 fe80::72b5:e8ff:fed1:3984%bge0 prefixlen 64 scopeid 0x3
        inet 146.155.5.189 netmask 0xffffffc0 broadcast 146.155.5.191
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        supported media:
                media autoselect mediaopt flowcontrol
                media autoselect
                media 1000baseT mediaopt full-duplex,master
                media 1000baseT mediaopt full-duplex
                media 1000baseT mediaopt master
                media 1000baseT
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        capabilities=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 70:b5:e8:d1:39:85
        media: Ethernet autoselect
        supported media:
                media autoselect mediaopt flowcontrol
                media autoselect
                media 1000baseT mediaopt full-duplex,master
                media 1000baseT mediaopt full-duplex
                media 1000baseT mediaopt master
                media 1000baseT
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
        groups: enc
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
        groups: pfsync
bnxt0.11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: ICE
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0.11 prefixlen 64 scopeid 0x9
        inet 146.155.11.247 netmask 0xffffff00 broadcast 146.155.11.255
        groups: vlan
        vlan: 11 vlanpcp: 0 parent interface: bnxt0
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bnxt0.15: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Red15
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0.15 prefixlen 64 scopeid 0xa
        inet 146.155.15.252 netmask 0xffffff00 broadcast 146.155.15.255
        groups: vlan
        vlan: 15 vlanpcp: 0 parent interface: bnxt0
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bnxt0.20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: R20
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0.20 prefixlen 64 scopeid 0xb
        inet 146.155.20.120 netmask 0xffffff00 broadcast 146.155.20.255
        groups: vlan
        vlan: 20 vlanpcp: 0 parent interface: bnxt0
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bnxt0.71: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Red71
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0.71 prefixlen 64 scopeid 0xc
        inet 146.155.7.5 netmask 0xffffffc0 broadcast 146.155.7.63
        groups: vlan
        vlan: 71 vlanpcp: 0 parent interface: bnxt0
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bnxt0.4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Red4
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0.4 prefixlen 64 scopeid 0xd
        inet 146.155.4.231 netmask 0xffffff00 broadcast 146.155.4.255
        groups: vlan
        vlan: 4 vlanpcp: 0 parent interface: bnxt0
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          OK, so Broadcom NICs. Less common but I'd expect them to work.

          Ok first thing I would do is run a packet capture on bnxt0 directly to be sure tagged traffic is arriving and/or leaving.

          There are a bunch of VLAN specific hardware off-loading options there. So if you see tagged traffic missing in the pcap the next thing I would do is start disabling those.

          Steve

          1 Reply Last reply Reply Quote 0
          • F
            fealiaga
            last edited by

            Thank you very much, today I made a tcpdump to the door and at the time of doing it the arp tables were completed but at the end of that process the vlan fell again. after that I created a bridge on the pfsense trunk interface and the arp tables were populated again

            without the bridge configuration the tables cannot be complete and the traffic does not leave the pfsense

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by stephenw10

              Hmm, well obviously you should not need to do that!
              It is probably putting the NIC in promiscuous mode. You should be able to see that in the ifconfig output. Doing so allows it to see all traffic on the wire.
              This looks like an offload option getting on the way. There are quite a lot of things you can set:
              https://www.freebsd.org/cgi/man.cgi?query=bnxt#SYSCTL_VARIABLES

              Steve

              1 Reply Last reply Reply Quote 0
              • F
                fealiaga
                last edited by

                Sorry but i am new in this, can you tell me how a can config the promiscuous mode

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  I would expect to be able to do it with:

                  ifconfig bnxt0 promisc

                  I don't have any of those NICs to test with though.

                  1 Reply Last reply Reply Quote 0
                  • F
                    fealiaga
                    last edited by

                    thank you very much remove the bridge (vlans disabled) and enable promiscuous mode on the interface and they got back up

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by stephenw10

                      Ah here we go!
                      https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236983

                      Seems like it may be still partially broken in 12 stable then.

                      You shouldn't need to enable promiscuous mode. But you probably do need to disable VLAN GW Filtering.

                      Steve

                      1 Reply Last reply Reply Quote 1
                      • F
                        fealiaga
                        last edited by

                        Maybe it is the best way to configure it, but the promiscuous mode works, you know how to leave the configuration saved, so that it starts in promiscuous mode if the server is restarted

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by stephenw10

                          Add it as a shellcmd:
                          https://docs.netgate.com/pfsense/en/latest/development/boot-commands.html#shellcmd-option

                          There's no way to set that in the config otherwise.

                          But I would try disabling VLAN_GW_Filter instead. You don't really want the NIC running in promiscuous mode unless you have to.

                          Steve

                          1 Reply Last reply Reply Quote 0
                          • F
                            fealiaga
                            last edited by

                            How can I disable the VLAN_GW_Filter, sorry for the basic questions and thank you very much for your help.

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by 

                              ifconfig bnxt0 -vlanhwfilter

                              Is what I expect. Again I can't test that.

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post
                              Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.