VLAN interface on WAN interface not tagging frames

gravyface · Aug 9, 2019, 5:25 PM

Synopsis:

Can't access management VLAN 10 that's configured on Microtik from pfSense (ARP entry incomplete on pfSense, suspect VLAN tagging issue).

Can access management VLAN 10 that's configured on Microtik via StarTech ASIX USB NIC with VLAN 10 configured on it from PC via same cable (unplug pfSense > WAN > ethernet cable, plug into USB NIC on PC).

Hardware:

Microtik wAP LTE Kit (Latest RouterOS; single NIC with built-in LTE modem) patched into WAN (igb0) of
ALIX 2d4 running pfSense 2.4.4-RELEASE-p1

Microtik is configured as a LTE bridge to ether1; config here if you know/care about RouterOS:

aug/07/2019 17:32:03 by RouterOS 6.45.2
software id = E3Q7-3NTE

model = RouterBOARD wAP R-2nD
serial number = xxxxxxx46
/interface lte
set [ find ] mac-address=xx:xx:xx:xx:xx:31 mtu=1480 name=lte1
/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik
/interface vlan
add interface=ether1 name=mgmt vlan-id=10
/interface lte apn
set [ find default=yes ] apn=inet.someisp.com passthrough-interface=ether1 \
    passthrough-mac=xx:xx:xx:xx:E7:9A
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip address
add address=192.168.88.1/24 interface=mgmt network=192.168.88.0
/system clock
set time-zone-name=America/New_York

pfSense:

I have configured VLAN 10 on the WAN interface (igb0) by following these steps:

Interfaces > Assignments then VLANs tab.

Add VLAN:
Interface: igb0 (wan)
VLAN tag: 10
Priority: (blank)
Description: Microtik mgmt

back to Interfaces > Assignments
Add available network ports: "VLAN 10 on igb0 - wan (Microtik mgmt)"

Edit Interfaces/LTEMGMT (igb0.10)
Enable Interface
IPv4 configuration type: Static IPv4
Static IPv4 configuration:
192.168.88.254/24

Under Reserved Networks, block private/bogon unchecked.

Cannot ping from pfSense > Diagnostics and Diagnostics > ARP Table shows incomplete entry for 192.168.88.1, which means it's still waiting for an ARP reply.

Packet capture on LTEMGMT interface (host 192.168.88.1, full, promiscuous mode) with another tab open and running a Diagnostics > Ping (10 packets, source interface LTEMGMT, destination host 192.168.88.1) just shows the ARP Request who-has responses). Ethernet frames do not show VLAN in the header information in WireShark.

Thanks in advance.

Derelict · Aug 11, 2019, 5:42 PM

Whatever you are connecting there needs to expect that traffic with a VLAN tag of 10.

If that was the case it would be working. Cannot speak to whether or not that MikroTik configuration is correct.

JKnott · Aug 11, 2019, 6:41 PM

@Derelict said in VLAN interface on WAN interface not tagging frames:

Whatever you are connecting there needs to expect that traffic with a VLAN tag of 10.

While I don't know about his situation, business connections over fibre will often use VLANs even 2 levels of it (Q-in-Q).

gravyface · Aug 12, 2019, 12:28 PM

@Derelict as stated above, if I simply take the cable from the pfSense WAN port and plug it into a USB 3.0 NIC on my PC that has VLAN 10 tagged on it, it works fine.

NogBadTheBad · Aug 12, 2019, 12:37 PM

@gravyface said in VLAN interface on WAN interface not tagging frames:

Can't access management VLAN 10 that's configured on Microtik from pfSense (ARP entry incomplete on pfSense, suspect VLAN tagging issue).
Can access management VLAN 10 that's configured on Microtik via StarTech ASIX USB NIC with VLAN 10 configured on it from PC via same cable (unplug pfSense > WAN > ethernet cable, plug into USB NIC on PC).

Do a packetcapture on the WAN interface, open it up in Wireshark and create a column for vlan and use the following, that will tell you if pfSense is tagging or not.

gravyface · Aug 12, 2019, 12:42 PM

@NogBadTheBad I did; it is not. But nice tip re: adding a column for VLAN ID.

I210-AT Intel NICs on that ALIX do support VLANs too.

NogBadTheBad · Aug 12, 2019, 12:45 PM

@gravyface said in VLAN interface on WAN interface not tagging frames:

I210-AT

No worries, it was worth a try

Derelict · Aug 12, 2019, 3:17 PM

I have certainly never heard of an igb interface not supporting VLAN tags.

Post Interfaces > Assignments and, for good measure, the output of ifconfig -vma.

gravyface · Aug 12, 2019, 4:19 PM

@Derelict

[2.4.4-RELEASE][root@pfSense.localdomain]/root: ifconfig -vma
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6500bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS                                                                                                                                                             UM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=753fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN                                                                                                                                                             _HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NET                                                                                                                                                             MAP,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:1f:29:bc:e7:9a
        hwaddr 00:0d:b9:52:3b:e8
        inet6 fe80::20d:b9ff:fe52:3be8%igb0 prefixlen 64 scopeid 0x1
        inet xx.xx.xx.220 netmask 0xfffffff8 broadcast xx.xx.xx.223
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        supported media:
                media autoselect
                media 1000baseT
                media 1000baseT mediaopt full-duplex
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS                                                                                                                                                             UM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=753fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN                                                                                                                                                             _HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NET                                                                                                                                                             MAP,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:52:3b:e9
        hwaddr 00:0d:b9:52:3b:e9
        inet 10.171.1.1 netmask 0xffffff00 broadcast 10.171.1.255
        inet6 fe80::1:1%igb1 prefixlen 64 scopeid 0x2
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        supported media:
                media autoselect
                media 1000baseT
                media 1000baseT mediaopt full-duplex
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP
igb2: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS                                                                                                                                                             UM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=753fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN                                                                                                                                                             _HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NET                                                                                                                                                             MAP,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:52:3b:ea
        hwaddr 00:0d:b9:52:3b:ea
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: no carrier
        supported media:
                media autoselect
                media 1000baseT
                media 1000baseT mediaopt full-duplex
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP
enc0: flags=41<UP,RUNNING> metric 0 mtu 1536
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: enc
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
        groups: pfsync
        syncpeer: 224.0.0.240 maxupd: 128 defer: on
        syncok: 1
igb0.10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:1f:29:bc:e7:9a
        inet6 fe80::20d:b9ff:fe52:3be8%igb0.10 prefixlen 64 scopeid 0x8
        inet 192.168.88.254 netmask 0xffffff00 broadcast 192.168.88.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        supported media:
                media autoselect
        vlan: 10 vlanpcp: 0 parent interface: igb0
        groups: vlan

gravyface · Aug 12, 2019, 4:30 PM

Something is definitely unstable with this Microtik: I've attempted to remove the MAC address restriction from the passthrough options and it's now unresponsive.

Derelict · Aug 12, 2019, 4:45 PM

I have never, ever, seen an igb port (or any port) not tag in that case. I would look elsewhere for the problem.

You will not see VLAN tags capturing on LTEMGMT there. You will have to capture on WAN.

If you don't want to trust pfSense's tcpdump/packet capture, capture on a mirror port on a switch between igb0 and the mikrotik/wan.

gravyface · Aug 12, 2019, 9:04 PM

I think the problem may be due to the fact that parent/child interfaces share the same MAC address. I have passthrough enabled on the lte1/ether1 interfaces, which is locked to the MAC address of pfSense's WAN interface, but on the same physical interface, igb0.10 shares the same MAC. Might be throwing off the Mikrotik.

Derelict · Aug 12, 2019, 9:06 PM

That is 100% expected for VLANs.

gravyface · Aug 12, 2019, 9:23 PM

@Derelict I changed it via ifconfig and it didn't make a difference anyways.

gravyface · Aug 12, 2019, 9:27 PM

Wondering if I'd have better luck getting the Sierra Wireless MC7700 running on the ALIX and ditch the Microtik (which honestly feels kind of Fisher Price to me).

gravyface · Aug 13, 2019, 5:33 PM

@Derelict Ok, found a Microtik post on the parameters around the passthrough and it will reject traffic from a device with the same MAC as the passthrough device. As a workaround, you can create another VLAN interface on Microtik (I created VLAN 11) and did likewise on the pfSense.