pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue

NRgia

@stephenw10
Now for pfSense 22.05
First I set the interfaces to Legacy mode, disabled Suricata and uninstalled it.
Rebooted pfSense machine.
ifconfig shows like this after the reboot:

[22.05-RELEASE][root@Entaro.Blueshift]/root: ifconfig
ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER>
        ether ac:1f:6b:45:fa:88
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER>
        ether ac:1f:6b:45:fa:89
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
        ether ac:1f:6b:45:fa:8a
        inet6 fe80::ae1f:6bff:fe45:fa8a%ix2 prefixlen 64 scopeid 0x3
        inet 172.18.0.12 netmask 0xfffe0000 broadcast 172.19.255.255
        media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=8138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER>
        ether ac:1f:6b:45:fa:8b
        inet6 fe80::ae1f:6bff:fe45:fa8b%ix3 prefixlen 64 scopeid 0x4
        inet *********** netmask 0xfffff800 broadcast 92.83.255.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
        groups: enc
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pfsync0: flags=0<> metric 0 mtu 1500
        groups: pfsync
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
ix2.20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: IoT
        ether ac:1f:6b:45:fa:8a
        inet6 fe80::ae1f:6bff:fe45:fa8a%ix2.20 prefixlen 64 scopeid 0x9
        inet 192.168.10.1 netmask 0xffffffc0 broadcast 192.168.10.63
        groups: vlan
        vlan: 20 vlanpcp: 0 parent interface: ix2
        media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix2.30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: GuestNetwork
        ether ac:1f:6b:45:fa:8a
        inet6 fe80::ae1f:6bff:fe45:fa8a%ix2.30 prefixlen 64 scopeid 0xa
        inet 192.168.20.1 netmask 0xffffffc0 broadcast 192.168.20.63
        groups: vlan
        vlan: 30 vlanpcp: 0 parent interface: ix2
        media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::ae1f:6bff:fe45:fa88%ovpns1 prefixlen 64 scopeid 0xb
        inet 10.0.8.1 --> 10.0.8.2 netmask 0xffffff00
        groups: tun openvpn
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 49792

My packages list

This is the tcpdump from pfSense machine on ix2 interface with suricata uninstalled:
tcpdump_from_pfsense_22.05_ix2.txt

Setting the interface to legacy and uninstalling Suricata did not help

NRgia

@stephenw10
Last test
Test parameters:

1) pfSense 22.05
2) Suricata uninstalled.
3) Connected to mirror port which mirrors pfSense LAN side port
4) Did a tcpdump on connected device wired to the mirrored port.

tcpdump_port_mirroring_pfsense_22.05.txt
wireshark_pfsense_22.05.pcap

I hope I did the tests with port mirroring right.

NRgia

@stephenw10
I also deleted this line:

System tunables, mtag

but it gets restored after reboot.

NRgia

@stephenw10 Any feedback about the above. Are the tests done wrong? Can I fix this somehow? Or what is your suggestion?

stephenw10

Ok, so:

No difference with Netmap/Inline mode enabled in 22.01.

The port mirror doesn't show the VLAN tags as you say so either tcpdump/wireshark isn't showing them or the mirror port is somehow setup to strip them.
I would test the pcap host you're using by connecting it to a port you know has tagged traffic on and make sure you can see it there.

Steve

johnpoz

@stephenw10 wireshark did have some issues with npcap and vlans. I haven't looked into it in a while, since I don't normally have to capture traffic other than on pfsense. And then just open in wireshark.

But yeah your test of making sure sees the vlan tags good test

edit: yeah if your using npcap on windows you might have issues with seeing tags.

https://npcap.com/guide/npcap-users-guide.html

/vlan_support (deprecated, ignored)

Support 802.1Q VLAN tag when capturing and sending data (currently unsupported). This feature was disabled in 2016 to prevent a crash and has not been re-enabled.

I haven't had to do any vlan capture stuff on wireshark in windows for a while - like I said haven't looked it for a while. But yeah there my be some problems there.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Ok, so:

No difference with Netmap/Inline mode enabled in 22.01.

The port mirror doesn't show the VLAN tags as you say so either tcpdump/wireshark isn't showing them or the mirror port is somehow setup to strip them.
I would test the pcap host you're using by connecting it to a port you know has tagged traffic on and make sure you can see it there.

Steve

So in order for you guys to have visibility I connected a host to port 5 on the switch ( I can connect it directly to pfSense LAN interface, if you guys say it's good test).

In port 5 is where my AP gets connected usually, and port 5 is a member of:

VLAN 1 Untagged
VLAN 20 Tagged with 20
VLAN 30 Tagged with 30
as you can see in the screenshot
https://imgur.com/a/9t44Qbp

This is a tcpdump from pfsense 22.01. I can see some traffic from other VLANs but I don't see any VLAN tags. Am I doing something wrong?

tcpdump_port_with_multiple_vlans_pfsense_22.01.txt

I can upgrade to pfSense 22.05, but first, I try to understand why I cannot provide something for you guys to investigate?

Should I upgrade to 22.05 now and provide a tcpdump, or I'm not doing this right?

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Ok, so:

No difference with Netmap/Inline mode enabled in 22.01.

Neither on 22.05, except the NETMAP tag which is removed, after the interfaces are set to Legacy mode, from Suricata's GUI. Even uninstalling the package didn't make any difference.

NRgia

@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@stephenw10 wireshark did have some issues with npcap and vlans. I haven't looked into it in a while, since I don't normally have to capture traffic other than on pfsense. And then just open in wireshark.

But yeah your test of making sure sees the vlan tags good test

edit: yeah if your using npcap on windows you might have issues with seeing tags.

https://npcap.com/guide/npcap-users-guide.html

/vlan_support (deprecated, ignored)
Support 802.1Q VLAN tag when capturing and sending data (currently unsupported). This feature was disabled in 2016 to prevent a crash and has not been re-enabled. 
I haven't had to do any vlan capture stuff on wireshark in windows for a while - like I said haven't looked it for a while. But yeah there my be some problems there.

Thank you @johnpoz I did not know about the Wireshark issues. The idea is that the host has a Windows 10 OS.
I found this : " If your machine is not plugged into a switched network or a dual-speed hub, or it is plugged into a switched network but the port is set up to have all traffic replicated to it, the problem might be that the network interface on that you're capturing doesn't support "promiscuous" mode, or because your OS can't put the interface into promiscuous mode. Normally, network interfaces supply to the host only:

packets sent to one of that host's link-layer addresses;
broadcast packets;
multicast packets sent to a multicast address that the host has configured the interface to accept."

from here : https://www.tcpdump.org/faq.html#q13
So maybe Windows it's not the best OS to pcap something.

NRgia

@stephenw10 @johnpoz
Ok I repeated the test using Manjaro this time.
Now I got vlan tags.

Connected to the same port.

Here you go:
tcpdump_pfsense_22.01.txt

I think you will need a pcap from pfSense 22.05 also.

johnpoz

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

I think you will need a pcap from pfSense 22.05 also.

How would the version of pfsense have anything to do with what the switch is sending out?

Span/Mirror port could come into play here.

What I would do is sniff with pfsense, then replace pfsense connected to the same port on the switch with your laptop or pc doing sniff - do you still zero those zero tags?

stephenw10

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Here you go:
tcpdump_pfsense_22.01.txt
I think you will need a pcap from pfSense 22.05 also.

OK, so where exactly was that captured? On the mirror port? pfSense LAN port directly?

Looks like probably the latter since there are no replies. Try pcaping like that on the mirror port with successful traffic passing. We should be able to see all the traffic on the link between pfSense and the switch.

stephenw10

One other test you might try here is to actually set a priority tag other than 0 in the switch and see where that is applied to traffic in the pcap. I expect it to be on the VLAN 20/30tag but if it appears on the VLAN0 tag that would implicate the switch.

And just to review (since we've been at this some days now!) we initially though the switch was tagging the packets with VLAN0 but ruled that out when you ran a test with the AP connect to pfSense directly and still saw VLAN0 tags. Can you just reconfirm that? If I've misunderstood that result the switch still seems like the mist likely suspect.

Steve

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Here you go:
tcpdump_pfsense_22.01.txt
I think you will need a pcap from pfSense 22.05 also.

OK, so where exactly was that captured? On the mirror port? pfSense LAN port directly?

Looks like probably the latter since there are no replies. Try pcaping like that on the mirror port with successful traffic passing. We should be able to see all the traffic on the link between pfSense and the switch.

It wasn't the mirror port. It was the port where the AP was connected, and had access to Native LAN, VLAN20 and VLAN30. (you can forget about VLAN30, there is no traffic, is just a guest VLAN)

The problem was, the Host OS was Windows, and the interface couldn't be set into promisc mode to capture vlan tags. It worked from a host with a Linux OS

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

One other test you might try here is to actually set a priority tag other than 0 in the switch and see where that is applied to traffic in the pcap. I expect it to be on the VLAN 20/30tag but if it appears on the VLAN0 tag that would implicate the switch.

I looked in the switch menus, I don't think my switch can assign a priority. Only by port in QoS menu. I saw that only in pfSense (in my network config)

And just to review (since we've been at this some days now!) we initially though the switch was tagging the packets with VLAN0 but ruled that out when you ran a test with the AP connect to pfSense directly and still saw VLAN0 tags. Can you just reconfirm that? If I've misunderstood that result the switch still seems like the mist likely suspect.

Steve

You are correct, I confirm.

Also I use a laptop with Manjaro now, so I can plug in any port and the pcap will catch vlan tags also. Maybe to make sure our result were not botched by Windows OS, I can repeat them.

Please always specify the pfSense version on which you want me to run them.

Thank you, and waiting for your input

stephenw10

The traffic coming into the pfSense LAN port should not vary between pfSense versions. What might be different is the way the driver is changing what tcpdump in pfSense sees. Though it's unlikely. So I'd stick with 22.01 until we know we have a good capture from the mirror port showing the traffic both ways with all the tags.

Your switch does have priority tagging. You showed it here: https://imgur.com/GdPzhEn
Set that to 802.1p mode and set a priority level on one of the ports. You should be able to see that in the pcap.

Steve

johnpoz

Exactly pfsense isn't setting vlan 0 on the incoming traffic.

Your trick now is to figure out where they are coming from.. This has nothing to do with pfsense other than old version via a problem was letting it work, now with 22.05 is acting how it should on such traffic.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

The traffic coming into the pfSense LAN port should not vary between pfSense versions. What might be different is the way the driver is changing what tcpdump in pfSense sees. Though it's unlikely. So I'd stick with 22.01 until we know we have a good capture from the mirror port showing the traffic both ways with all the tags.

Your switch does have priority tagging. You showed it here: https://imgur.com/GdPzhEn
Set that to 802.1p mode and set a priority level on one of the ports. You should be able to see that in the pcap.

Steve

You' re correct, but if I switch from Port based to 802.1p mode, I cannot set any priorities, the list will disappear.

I uploaded 2 more screenshots as an example:

https://imgur.com/a/f81vKbB

NRgia

@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Exactly pfsense isn't setting vlan 0 on the incoming traffic.

Your trick now is to figure out where they are coming from.. This has nothing to do with pfsense other than old version via a problem was letting it work, now with 22.05 is acting how it should on such traffic.

Got it, let's hope we find something :)

johnpoz

@nrgia delete those priorities

Why do you think you need to set priorities, and why can you not set them in 802.1p?

You have all the ports set for low priority - that is pointless to do anyway. Now if you actually had some ports set differently, etc. but all the same makes the whole setup useless..