pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue

stephenw10

Hmm, no vlan0 traffic at all there...
Does that mean the earlier driver is filtering it before the pcap can see it?
Or that the later driver is somehow adding it?

Are you able to create a mirror port on the switch so we can see what's actually on the wire?

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Hmm, no vlan0 traffic at all there...
Does that mean the earlier driver is filtering it before the pcap can see it?

I don't know what to respond to this. I will let you draw any conclusions.

Or that the later driver is somehow adding it?

Are you able to create a mirror port on the switch so we can see what's actually on the wire?

I'm still on 22.01, I changed from Inline to Legacy Mode, and then disabled Suricata.
The NETMAP tag is not there anymore.

The test with the mirror port you want it done on 22.01 or 22.05?
And which port do you want me to mirror the pfSense LAN side?
And then what information do you want me to record, and how?
For example I will connect a device to that mirror port, and then? A tcpdump from pfsense, from the device, which interface?

Also do you want a full pcap file taken with wireshark, or with tcpdump like before?

stephenw10

You might rerun the pcap in 22.01 in Inline mode to be sure it looks the same. Confirm it's not netmap adding the vlan0 tags somehow.

The test with the mirror port you want it done on 22.01 or 22.05?

Both. Once it's configured you can use it to see what's on the wire in both situations.

And which port do you want me to mirror the pfSense LAN side?

Yes, the port linked to the pfSense LAN would be most useful there I think.

And then what information do you want me to record, and how?

Connect a client to the mirror port and run a pcap on that client. You will see everything that the pfSense LAN sees.

That will confirm which of those two suppositions is correct.
If you do see vlan0 tagged traffic there the we will know the older driver in 22.01 is actually stripping those tags allowing it to work.

Steve

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

You might rerun the pcap in 22.01 in Inline mode to be sure it looks the same. Confirm it's not netmap adding the vlan0 tags somehow.

This is the the with Suricata enabled in inline mode:

tcp_dump_pfsense_22.01_Suricata_Inline_mode.txt

Same device 192.168.10.57 with MAC 08:c5:e1:97:fa:ab but there is other traffic also.

Next test, the mirror port. I will report shortly.

For the mirror port test, do you want me to leave Suricata enabled and the interface in legacy mode? Or disabled, to exclude it from interfering?

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

You might rerun the pcap in 22.01 in Inline mode to be sure it looks the same. Confirm it's not netmap adding the vlan0 tags somehow.

Yes, the port linked to the pfSense LAN would be most useful there I think.

Test parameters:

pfSense 22.01
Suricata enabled on ix2 and ix3, in inline mode
Connected to mirror port which mirrors pfSense LAN side port
Did a tcpdump on connected device wired to the mirrored port.

This is the result:

tcpdump_port_mirroring_pfsense_22.01.txt

I cannot see any VLANs tags. I can only see them if I do a tcpdump from pfSense. Does this mean the switch strips the tags, and then add the tags only to tagged ports? (Which is correct behavior?)

Now I will upgrade to 22.05 if there are no tests to run on 22.01.

Please let me know if on 22.05 you want me to let Suricata run or should I set the interfaces to legacy and disable/uninstall Suricata?

NRgia

@stephenw10
Now for pfSense 22.05
First I set the interfaces to Legacy mode, disabled Suricata and uninstalled it.
Rebooted pfSense machine.
ifconfig shows like this after the reboot:

[22.05-RELEASE][root@Entaro.Blueshift]/root: ifconfig
ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER>
        ether ac:1f:6b:45:fa:88
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER>
        ether ac:1f:6b:45:fa:89
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
        ether ac:1f:6b:45:fa:8a
        inet6 fe80::ae1f:6bff:fe45:fa8a%ix2 prefixlen 64 scopeid 0x3
        inet 172.18.0.12 netmask 0xfffe0000 broadcast 172.19.255.255
        media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=8138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER>
        ether ac:1f:6b:45:fa:8b
        inet6 fe80::ae1f:6bff:fe45:fa8b%ix3 prefixlen 64 scopeid 0x4
        inet *********** netmask 0xfffff800 broadcast 92.83.255.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
        groups: enc
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pfsync0: flags=0<> metric 0 mtu 1500
        groups: pfsync
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
ix2.20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: IoT
        ether ac:1f:6b:45:fa:8a
        inet6 fe80::ae1f:6bff:fe45:fa8a%ix2.20 prefixlen 64 scopeid 0x9
        inet 192.168.10.1 netmask 0xffffffc0 broadcast 192.168.10.63
        groups: vlan
        vlan: 20 vlanpcp: 0 parent interface: ix2
        media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix2.30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: GuestNetwork
        ether ac:1f:6b:45:fa:8a
        inet6 fe80::ae1f:6bff:fe45:fa8a%ix2.30 prefixlen 64 scopeid 0xa
        inet 192.168.20.1 netmask 0xffffffc0 broadcast 192.168.20.63
        groups: vlan
        vlan: 30 vlanpcp: 0 parent interface: ix2
        media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ovpns1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::ae1f:6bff:fe45:fa88%ovpns1 prefixlen 64 scopeid 0xb
        inet 10.0.8.1 --> 10.0.8.2 netmask 0xffffff00
        groups: tun openvpn
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 49792

My packages list

This is the tcpdump from pfSense machine on ix2 interface with suricata uninstalled:
tcpdump_from_pfsense_22.05_ix2.txt

Setting the interface to legacy and uninstalling Suricata did not help

NRgia

@stephenw10
Last test
Test parameters:

1) pfSense 22.05
2) Suricata uninstalled.
3) Connected to mirror port which mirrors pfSense LAN side port
4) Did a tcpdump on connected device wired to the mirrored port.

tcpdump_port_mirroring_pfsense_22.05.txt
wireshark_pfsense_22.05.pcap

I hope I did the tests with port mirroring right.

NRgia

@stephenw10
I also deleted this line:

System tunables, mtag

but it gets restored after reboot.

NRgia

@stephenw10 Any feedback about the above. Are the tests done wrong? Can I fix this somehow? Or what is your suggestion?

stephenw10

Ok, so:

No difference with Netmap/Inline mode enabled in 22.01.

The port mirror doesn't show the VLAN tags as you say so either tcpdump/wireshark isn't showing them or the mirror port is somehow setup to strip them.
I would test the pcap host you're using by connecting it to a port you know has tagged traffic on and make sure you can see it there.

Steve

johnpoz

@stephenw10 wireshark did have some issues with npcap and vlans. I haven't looked into it in a while, since I don't normally have to capture traffic other than on pfsense. And then just open in wireshark.

But yeah your test of making sure sees the vlan tags good test

edit: yeah if your using npcap on windows you might have issues with seeing tags.

https://npcap.com/guide/npcap-users-guide.html

/vlan_support (deprecated, ignored)

Support 802.1Q VLAN tag when capturing and sending data (currently unsupported). This feature was disabled in 2016 to prevent a crash and has not been re-enabled.

I haven't had to do any vlan capture stuff on wireshark in windows for a while - like I said haven't looked it for a while. But yeah there my be some problems there.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Ok, so:

No difference with Netmap/Inline mode enabled in 22.01.

The port mirror doesn't show the VLAN tags as you say so either tcpdump/wireshark isn't showing them or the mirror port is somehow setup to strip them.
I would test the pcap host you're using by connecting it to a port you know has tagged traffic on and make sure you can see it there.

Steve

So in order for you guys to have visibility I connected a host to port 5 on the switch ( I can connect it directly to pfSense LAN interface, if you guys say it's good test).

In port 5 is where my AP gets connected usually, and port 5 is a member of:

VLAN 1 Untagged
VLAN 20 Tagged with 20
VLAN 30 Tagged with 30
as you can see in the screenshot
https://imgur.com/a/9t44Qbp

This is a tcpdump from pfsense 22.01. I can see some traffic from other VLANs but I don't see any VLAN tags. Am I doing something wrong?

tcpdump_port_with_multiple_vlans_pfsense_22.01.txt

I can upgrade to pfSense 22.05, but first, I try to understand why I cannot provide something for you guys to investigate?

Should I upgrade to 22.05 now and provide a tcpdump, or I'm not doing this right?

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Ok, so:

No difference with Netmap/Inline mode enabled in 22.01.

Neither on 22.05, except the NETMAP tag which is removed, after the interfaces are set to Legacy mode, from Suricata's GUI. Even uninstalling the package didn't make any difference.

NRgia

@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@stephenw10 wireshark did have some issues with npcap and vlans. I haven't looked into it in a while, since I don't normally have to capture traffic other than on pfsense. And then just open in wireshark.

But yeah your test of making sure sees the vlan tags good test

edit: yeah if your using npcap on windows you might have issues with seeing tags.

https://npcap.com/guide/npcap-users-guide.html

/vlan_support (deprecated, ignored)
Support 802.1Q VLAN tag when capturing and sending data (currently unsupported). This feature was disabled in 2016 to prevent a crash and has not been re-enabled. 
I haven't had to do any vlan capture stuff on wireshark in windows for a while - like I said haven't looked it for a while. But yeah there my be some problems there.

Thank you @johnpoz I did not know about the Wireshark issues. The idea is that the host has a Windows 10 OS.
I found this : " If your machine is not plugged into a switched network or a dual-speed hub, or it is plugged into a switched network but the port is set up to have all traffic replicated to it, the problem might be that the network interface on that you're capturing doesn't support "promiscuous" mode, or because your OS can't put the interface into promiscuous mode. Normally, network interfaces supply to the host only:

packets sent to one of that host's link-layer addresses;
broadcast packets;
multicast packets sent to a multicast address that the host has configured the interface to accept."

from here : https://www.tcpdump.org/faq.html#q13
So maybe Windows it's not the best OS to pcap something.

NRgia

@stephenw10 @johnpoz
Ok I repeated the test using Manjaro this time.
Now I got vlan tags.

Connected to the same port.

Here you go:
tcpdump_pfsense_22.01.txt

I think you will need a pcap from pfSense 22.05 also.

johnpoz

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

I think you will need a pcap from pfSense 22.05 also.

How would the version of pfsense have anything to do with what the switch is sending out?

Span/Mirror port could come into play here.

What I would do is sniff with pfsense, then replace pfsense connected to the same port on the switch with your laptop or pc doing sniff - do you still zero those zero tags?

stephenw10

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Here you go:
tcpdump_pfsense_22.01.txt
I think you will need a pcap from pfSense 22.05 also.

OK, so where exactly was that captured? On the mirror port? pfSense LAN port directly?

Looks like probably the latter since there are no replies. Try pcaping like that on the mirror port with successful traffic passing. We should be able to see all the traffic on the link between pfSense and the switch.

stephenw10

One other test you might try here is to actually set a priority tag other than 0 in the switch and see where that is applied to traffic in the pcap. I expect it to be on the VLAN 20/30tag but if it appears on the VLAN0 tag that would implicate the switch.

And just to review (since we've been at this some days now!) we initially though the switch was tagging the packets with VLAN0 but ruled that out when you ran a test with the AP connect to pfSense directly and still saw VLAN0 tags. Can you just reconfirm that? If I've misunderstood that result the switch still seems like the mist likely suspect.

Steve

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Here you go:
tcpdump_pfsense_22.01.txt
I think you will need a pcap from pfSense 22.05 also.

OK, so where exactly was that captured? On the mirror port? pfSense LAN port directly?

Looks like probably the latter since there are no replies. Try pcaping like that on the mirror port with successful traffic passing. We should be able to see all the traffic on the link between pfSense and the switch.

It wasn't the mirror port. It was the port where the AP was connected, and had access to Native LAN, VLAN20 and VLAN30. (you can forget about VLAN30, there is no traffic, is just a guest VLAN)

The problem was, the Host OS was Windows, and the interface couldn't be set into promisc mode to capture vlan tags. It worked from a host with a Linux OS

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

One other test you might try here is to actually set a priority tag other than 0 in the switch and see where that is applied to traffic in the pcap. I expect it to be on the VLAN 20/30tag but if it appears on the VLAN0 tag that would implicate the switch.

I looked in the switch menus, I don't think my switch can assign a priority. Only by port in QoS menu. I saw that only in pfSense (in my network config)

And just to review (since we've been at this some days now!) we initially though the switch was tagging the packets with VLAN0 but ruled that out when you ran a test with the AP connect to pfSense directly and still saw VLAN0 tags. Can you just reconfirm that? If I've misunderstood that result the switch still seems like the mist likely suspect.

Steve

You are correct, I confirm.

Also I use a laptop with Manjaro now, so I can plug in any port and the pcap will catch vlan tags also. Maybe to make sure our result were not botched by Windows OS, I can repeat them.

Please always specify the pfSense version on which you want me to run them.

Thank you, and waiting for your input