bnxt NICs not passing VLAN traffic
-
Hi, i need your help to know why i can't get vlans to work in my pfSense, my configuration is as follows:
The pfsense is installed on a DELL server with two 10 GB and two GB NICs, the version of pfsense is 2.5.2
WAM ---> NIC to 10GB
LAN ---> NIC to 1 GB
TRUNK ---> 10 GB NICAnd at the other end I have a CISCO NEXUS, which is my router, with a 10 GB SFP module.
the cable link (cat6a) is carried out without problems but the vlans cannot reach each other's gateways.
But if I change the trunk link to 1GB, the vlans lift without problems.
-
So the parent NIC doesn't link at 10G at all? Or just doesn't carry tagged traffic?
What drivers are those NICs using?
Steve
-
The parent NIC is UP, but doesn´t carry tagged traffic
The drives are the default of the server
-
Does it carry untagged traffic though?
What drivers are the NICs using not drives in the system. Like ix or ixl for example.
What does the output of
ifconfig -vvvma
show?Steve
-
@stephenw10 said in Help with this:
ifconfig -vvvma
the command:
bnxt0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: TRUNK options=e120bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether bc:97:e1:d8:c6:00 inet6 fe80::be97:e1ff:fed8:c600%bnxt0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>) status: active supported media: media 1000baseT media 10Gbase-T media autoselect nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> bnxt1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: WAN options=e120bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=f507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6> ether bc:97:e1:d8:c6:01 inet6 fe80::be97:e1ff:fed8:c601%bnxt1 prefixlen 64 scopeid 0x2 inet 146.155.7.186 netmask 0xffffffc0 broadcast 146.155.7.191 media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>) status: active supported media: media 1000baseT media 10Gbase-T media autoselect nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: LAN options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE> capabilities=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE> ether 70:b5:e8:d1:39:84 inet6 fe80::72b5:e8ff:fed1:3984%bge0 prefixlen 64 scopeid 0x3 inet 146.155.5.189 netmask 0xffffffc0 broadcast 146.155.5.191 media: Ethernet autoselect (1000baseT <full-duplex>) status: active supported media: media autoselect mediaopt flowcontrol media autoselect media 1000baseT mediaopt full-duplex,master media 1000baseT mediaopt full-duplex media 1000baseT mediaopt master media 1000baseT media 100baseTX mediaopt full-duplex media 100baseTX media 10baseT/UTP mediaopt full-duplex media 10baseT/UTP nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> bge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE> capabilities=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE> ether 70:b5:e8:d1:39:85 media: Ethernet autoselect supported media: media autoselect mediaopt flowcontrol media autoselect media 1000baseT mediaopt full-duplex,master media 1000baseT mediaopt full-duplex media 1000baseT mediaopt master media 1000baseT media 100baseTX mediaopt full-duplex media 100baseTX media 10baseT/UTP mediaopt full-duplex media 10baseT/UTP nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> enc0: flags=0<> metric 0 mtu 1536 groups: enc nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> pflog0: flags=100<PROMISC> metric 0 mtu 33160 groups: pflog pfsync0: flags=0<> metric 0 mtu 1500 groups: pfsync bnxt0.11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: ICE options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether bc:97:e1:d8:c6:00 inet6 fe80::be97:e1ff:fed8:c600%bnxt0.11 prefixlen 64 scopeid 0x9 inet 146.155.11.247 netmask 0xffffff00 broadcast 146.155.11.255 groups: vlan vlan: 11 vlanpcp: 0 parent interface: bnxt0 media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>) status: active supported media: media autoselect nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> bnxt0.15: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: Red15 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether bc:97:e1:d8:c6:00 inet6 fe80::be97:e1ff:fed8:c600%bnxt0.15 prefixlen 64 scopeid 0xa inet 146.155.15.252 netmask 0xffffff00 broadcast 146.155.15.255 groups: vlan vlan: 15 vlanpcp: 0 parent interface: bnxt0 media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>) status: active supported media: media autoselect nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> bnxt0.20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: R20 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether bc:97:e1:d8:c6:00 inet6 fe80::be97:e1ff:fed8:c600%bnxt0.20 prefixlen 64 scopeid 0xb inet 146.155.20.120 netmask 0xffffff00 broadcast 146.155.20.255 groups: vlan vlan: 20 vlanpcp: 0 parent interface: bnxt0 media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>) status: active supported media: media autoselect nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> bnxt0.71: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: Red71 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether bc:97:e1:d8:c6:00 inet6 fe80::be97:e1ff:fed8:c600%bnxt0.71 prefixlen 64 scopeid 0xc inet 146.155.7.5 netmask 0xffffffc0 broadcast 146.155.7.63 groups: vlan vlan: 71 vlanpcp: 0 parent interface: bnxt0 media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>) status: active supported media: media autoselect nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> bnxt0.4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: Red4 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether bc:97:e1:d8:c6:00 inet6 fe80::be97:e1ff:fed8:c600%bnxt0.4 prefixlen 64 scopeid 0xd inet 146.155.4.231 netmask 0xffffff00 broadcast 146.155.4.255 groups: vlan vlan: 4 vlanpcp: 0 parent interface: bnxt0 media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>) status: active supported media: media autoselect nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
-
OK, so Broadcom NICs. Less common but I'd expect them to work.
Ok first thing I would do is run a packet capture on bnxt0 directly to be sure tagged traffic is arriving and/or leaving.
There are a bunch of VLAN specific hardware off-loading options there. So if you see tagged traffic missing in the pcap the next thing I would do is start disabling those.
Steve
-
Thank you very much, today I made a tcpdump to the door and at the time of doing it the arp tables were completed but at the end of that process the vlan fell again. after that I created a bridge on the pfsense trunk interface and the arp tables were populated again
without the bridge configuration the tables cannot be complete and the traffic does not leave the pfsense
-
Hmm, well obviously you should not need to do that!
It is probably putting the NIC in promiscuous mode. You should be able to see that in the ifconfig output. Doing so allows it to see all traffic on the wire.
This looks like an offload option getting on the way. There are quite a lot of things you can set:
https://www.freebsd.org/cgi/man.cgi?query=bnxt#SYSCTL_VARIABLESSteve
-
Sorry but i am new in this, can you tell me how a can config the promiscuous mode
-
I would expect to be able to do it with:
ifconfig bnxt0 promisc
I don't have any of those NICs to test with though.
-
thank you very much remove the bridge (vlans disabled) and enable promiscuous mode on the interface and they got back up
-
Ah here we go!
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236983Seems like it may be still partially broken in 12 stable then.
You shouldn't need to enable promiscuous mode. But you probably do need to disable VLAN GW Filtering.
Steve
-
Maybe it is the best way to configure it, but the promiscuous mode works, you know how to leave the configuration saved, so that it starts in promiscuous mode if the server is restarted
-
Add it as a shellcmd:
https://docs.netgate.com/pfsense/en/latest/development/boot-commands.html#shellcmd-optionThere's no way to set that in the config otherwise.
But I would try disabling VLAN_GW_Filter instead. You don't really want the NIC running in promiscuous mode unless you have to.
Steve
-
How can I disable the VLAN_GW_Filter, sorry for the basic questions and thank you very much for your help.
-
ifconfig bnxt0 -vlanhwfilter
Is what I expect. Again I can't test that.