Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    bnxt NICs not passing VLAN traffic

    Scheduled Pinned Locked Moved Hardware
    16 Posts 2 Posters 1.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F
      fealiaga
      last edited by

      The parent NIC is UP, but doesn´t carry tagged traffic

      The drives are the default of the server

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Does it carry untagged traffic though?

        What drivers are the NICs using not drives in the system. Like ix or ixl for example.

        What does the output of ifconfig -vvvma show?

        Steve

        1 Reply Last reply Reply Quote 0
        • F
          fealiaga
          last edited by stephenw10

          @stephenw10 said in Help with this:

          ifconfig -vvvma

          the command:

          bnxt0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: TRUNK
        options=e120bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media 1000baseT
                media 10Gbase-T
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bnxt1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=e120bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:01
        inet6 fe80::be97:e1ff:fed8:c601%bnxt1 prefixlen 64 scopeid 0x2
        inet 146.155.7.186 netmask 0xffffffc0 broadcast 146.155.7.191
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media 1000baseT
                media 10Gbase-T
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
        capabilities=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 70:b5:e8:d1:39:84
        inet6 fe80::72b5:e8ff:fed1:3984%bge0 prefixlen 64 scopeid 0x3
        inet 146.155.5.189 netmask 0xffffffc0 broadcast 146.155.5.191
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        supported media:
                media autoselect mediaopt flowcontrol
                media autoselect
                media 1000baseT mediaopt full-duplex,master
                media 1000baseT mediaopt full-duplex
                media 1000baseT mediaopt master
                media 1000baseT
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        capabilities=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 70:b5:e8:d1:39:85
        media: Ethernet autoselect
        supported media:
                media autoselect mediaopt flowcontrol
                media autoselect
                media 1000baseT mediaopt full-duplex,master
                media 1000baseT mediaopt full-duplex
                media 1000baseT mediaopt master
                media 1000baseT
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
        groups: enc
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
        groups: pfsync
bnxt0.11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: ICE
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0.11 prefixlen 64 scopeid 0x9
        inet 146.155.11.247 netmask 0xffffff00 broadcast 146.155.11.255
        groups: vlan
        vlan: 11 vlanpcp: 0 parent interface: bnxt0
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bnxt0.15: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Red15
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0.15 prefixlen 64 scopeid 0xa
        inet 146.155.15.252 netmask 0xffffff00 broadcast 146.155.15.255
        groups: vlan
        vlan: 15 vlanpcp: 0 parent interface: bnxt0
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bnxt0.20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: R20
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0.20 prefixlen 64 scopeid 0xb
        inet 146.155.20.120 netmask 0xffffff00 broadcast 146.155.20.255
        groups: vlan
        vlan: 20 vlanpcp: 0 parent interface: bnxt0
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bnxt0.71: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Red71
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0.71 prefixlen 64 scopeid 0xc
        inet 146.155.7.5 netmask 0xffffffc0 broadcast 146.155.7.63
        groups: vlan
        vlan: 71 vlanpcp: 0 parent interface: bnxt0
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bnxt0.4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Red4
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0.4 prefixlen 64 scopeid 0xd
        inet 146.155.4.231 netmask 0xffffff00 broadcast 146.155.4.255
        groups: vlan
        vlan: 4 vlanpcp: 0 parent interface: bnxt0
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            OK, so Broadcom NICs. Less common but I'd expect them to work.

            Ok first thing I would do is run a packet capture on bnxt0 directly to be sure tagged traffic is arriving and/or leaving.

            There are a bunch of VLAN specific hardware off-loading options there. So if you see tagged traffic missing in the pcap the next thing I would do is start disabling those.

            Steve

            1 Reply Last reply Reply Quote 0
            • F
              fealiaga
              last edited by

              Thank you very much, today I made a tcpdump to the door and at the time of doing it the arp tables were completed but at the end of that process the vlan fell again. after that I created a bridge on the pfsense trunk interface and the arp tables were populated again

              without the bridge configuration the tables cannot be complete and the traffic does not leave the pfsense

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by stephenw10

                Hmm, well obviously you should not need to do that!
                It is probably putting the NIC in promiscuous mode. You should be able to see that in the ifconfig output. Doing so allows it to see all traffic on the wire.
                This looks like an offload option getting on the way. There are quite a lot of things you can set:
                https://www.freebsd.org/cgi/man.cgi?query=bnxt#SYSCTL_VARIABLES

                Steve

                1 Reply Last reply Reply Quote 0
                • F
                  fealiaga
                  last edited by

                  Sorry but i am new in this, can you tell me how a can config the promiscuous mode

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    I would expect to be able to do it with:

                    ifconfig bnxt0 promisc

                    I don't have any of those NICs to test with though.

                    1 Reply Last reply Reply Quote 0
                    • F
                      fealiaga
                      last edited by

                      thank you very much remove the bridge (vlans disabled) and enable promiscuous mode on the interface and they got back up

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by stephenw10

                        Ah here we go!
                        https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236983

                        Seems like it may be still partially broken in 12 stable then.

                        You shouldn't need to enable promiscuous mode. But you probably do need to disable VLAN GW Filtering.

                        Steve

                        1 Reply Last reply Reply Quote 1
                        • F
                          fealiaga
                          last edited by

                          Maybe it is the best way to configure it, but the promiscuous mode works, you know how to leave the configuration saved, so that it starts in promiscuous mode if the server is restarted

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by stephenw10

                            Add it as a shellcmd:
                            https://docs.netgate.com/pfsense/en/latest/development/boot-commands.html#shellcmd-option

                            There's no way to set that in the config otherwise.

                            But I would try disabling VLAN_GW_Filter instead. You don't really want the NIC running in promiscuous mode unless you have to.

                            Steve

                            1 Reply Last reply Reply Quote 0
                            • F
                              fealiaga
                              last edited by

                              How can I disable the VLAN_GW_Filter, sorry for the basic questions and thank you very much for your help.

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by 

                                ifconfig bnxt0 -vlanhwfilter

                                Is what I expect. Again I can't test that.

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post
                                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.