bnxt NICs not passing VLAN traffic

fealiaga

Hi, i need your help to know why i can't get vlans to work in my pfSense, my configuration is as follows:

The pfsense is installed on a DELL server with two 10 GB and two GB NICs, the version of pfsense is 2.5.2

WAM ---> NIC to 10GB
LAN ---> NIC to 1 GB
TRUNK ---> 10 GB NIC

And at the other end I have a CISCO NEXUS, which is my router, with a 10 GB SFP module.

the cable link (cat6a) is carried out without problems but the vlans cannot reach each other's gateways.

But if I change the trunk link to 1GB, the vlans lift without problems.

stephenw10

So the parent NIC doesn't link at 10G at all? Or just doesn't carry tagged traffic?

What drivers are those NICs using?

Steve

fealiaga

The parent NIC is UP, but doesn´t carry tagged traffic

The drives are the default of the server

stephenw10

Does it carry untagged traffic though?

What drivers are the NICs using not drives in the system. Like ix or ixl for example.

What does the output of ifconfig -vvvma show?

Steve

fealiaga

@stephenw10 said in Help with this:

ifconfig -vvvma

the command:

bnxt0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: TRUNK
        options=e120bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media 1000baseT
                media 10Gbase-T
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bnxt1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=e120bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:01
        inet6 fe80::be97:e1ff:fed8:c601%bnxt1 prefixlen 64 scopeid 0x2
        inet 146.155.7.186 netmask 0xffffffc0 broadcast 146.155.7.191
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media 1000baseT
                media 10Gbase-T
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
        capabilities=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 70:b5:e8:d1:39:84
        inet6 fe80::72b5:e8ff:fed1:3984%bge0 prefixlen 64 scopeid 0x3
        inet 146.155.5.189 netmask 0xffffffc0 broadcast 146.155.5.191
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        supported media:
                media autoselect mediaopt flowcontrol
                media autoselect
                media 1000baseT mediaopt full-duplex,master
                media 1000baseT mediaopt full-duplex
                media 1000baseT mediaopt master
                media 1000baseT
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        capabilities=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether 70:b5:e8:d1:39:85
        media: Ethernet autoselect
        supported media:
                media autoselect mediaopt flowcontrol
                media autoselect
                media 1000baseT mediaopt full-duplex,master
                media 1000baseT mediaopt full-duplex
                media 1000baseT mediaopt master
                media 1000baseT
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
        groups: enc
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
        groups: pfsync
bnxt0.11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: ICE
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0.11 prefixlen 64 scopeid 0x9
        inet 146.155.11.247 netmask 0xffffff00 broadcast 146.155.11.255
        groups: vlan
        vlan: 11 vlanpcp: 0 parent interface: bnxt0
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bnxt0.15: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Red15
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0.15 prefixlen 64 scopeid 0xa
        inet 146.155.15.252 netmask 0xffffff00 broadcast 146.155.15.255
        groups: vlan
        vlan: 15 vlanpcp: 0 parent interface: bnxt0
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bnxt0.20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: R20
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0.20 prefixlen 64 scopeid 0xb
        inet 146.155.20.120 netmask 0xffffff00 broadcast 146.155.20.255
        groups: vlan
        vlan: 20 vlanpcp: 0 parent interface: bnxt0
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bnxt0.71: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Red71
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0.71 prefixlen 64 scopeid 0xc
        inet 146.155.7.5 netmask 0xffffffc0 broadcast 146.155.7.63
        groups: vlan
        vlan: 71 vlanpcp: 0 parent interface: bnxt0
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bnxt0.4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Red4
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether bc:97:e1:d8:c6:00
        inet6 fe80::be97:e1ff:fed8:c600%bnxt0.4 prefixlen 64 scopeid 0xd
        inet 146.155.4.231 netmask 0xffffff00 broadcast 146.155.4.255
        groups: vlan
        vlan: 4 vlanpcp: 0 parent interface: bnxt0
        media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

stephenw10

OK, so Broadcom NICs. Less common but I'd expect them to work.

Ok first thing I would do is run a packet capture on bnxt0 directly to be sure tagged traffic is arriving and/or leaving.

There are a bunch of VLAN specific hardware off-loading options there. So if you see tagged traffic missing in the pcap the next thing I would do is start disabling those.

Steve

fealiaga

Thank you very much, today I made a tcpdump to the door and at the time of doing it the arp tables were completed but at the end of that process the vlan fell again. after that I created a bridge on the pfsense trunk interface and the arp tables were populated again

without the bridge configuration the tables cannot be complete and the traffic does not leave the pfsense

stephenw10

Hmm, well obviously you should not need to do that!
It is probably putting the NIC in promiscuous mode. You should be able to see that in the ifconfig output. Doing so allows it to see all traffic on the wire.
This looks like an offload option getting on the way. There are quite a lot of things you can set:
https://www.freebsd.org/cgi/man.cgi?query=bnxt#SYSCTL_VARIABLES

Steve

fealiaga

Sorry but i am new in this, can you tell me how a can config the promiscuous mode

stephenw10

I would expect to be able to do it with:

ifconfig bnxt0 promisc

I don't have any of those NICs to test with though.

fealiaga

thank you very much remove the bridge (vlans disabled) and enable promiscuous mode on the interface and they got back up

stephenw10

Ah here we go!
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236983

Seems like it may be still partially broken in 12 stable then.

You shouldn't need to enable promiscuous mode. But you probably do need to disable VLAN GW Filtering.

Steve

fealiaga

Maybe it is the best way to configure it, but the promiscuous mode works, you know how to leave the configuration saved, so that it starts in promiscuous mode if the server is restarted

stephenw10

Add it as a shellcmd:
https://docs.netgate.com/pfsense/en/latest/development/boot-commands.html#shellcmd-option

There's no way to set that in the config otherwise.

But I would try disabling VLAN_GW_Filter instead. You don't really want the NIC running in promiscuous mode unless you have to.

Steve

fealiaga

How can I disable the VLAN_GW_Filter, sorry for the basic questions and thank you very much for your help.

stephenw10

ifconfig bnxt0 -vlanhwfilter

Is what I expect. Again I can't test that.