pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

If you can make one VLAN work 100 will also work, that's not the issue.

There are two tests you can do.

Connect one of the APs directly to ix2. Run a pcap in pfSense on ix2 as you did before. Try to connect to the SSID using the VLAN and see what appears.
If you see nothing try running it without the 'vlan' option in case the traffic is arriving untagged.

With the switch comnected to ix2 add a port as untagged in VLAN 20 with the pvid set to 20 and connect a host to it. Does it pull a lease from VLAN 20.
If not run a pcap in pfSense again to see what's happening.
You can also pcap on the wired host to see what that sees.

Steve

Ok I'll start with the first one...

NRgia

@stephenw10

This test is like this:

Connect AP directly to pfsense LAN port
Connect a laptop wireless to Native LAN(via non tagged SSID) in order to access pfSense
Start tcpdump on ix2
Try to connect a mobile phone to VLAN 20 SSID

tcpdump -e -i ix2 vlan
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ix2, link-type EN10MB (Ethernet), capture size 262144 bytes
22:07:10.378711 cc:f4:11:c5:bc:81 (oui Unknown) > 33:33:00:0c:00:0c (oui Unknown), ethertype 802.1Q (0x8100), length 108: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv6, fe80::cef4:11ff:fec5:bc81.10101 > ff02::c:c.10101: UDP, length 38
22:07:10.378716 cc:f4:11:c5:bc:81 (oui Unknown) > 33:33:00:00:0c:0c (oui Unknown), ethertype 802.1Q (0x8100), length 108: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv6, fe80::cef4:11ff:fec5:bc81.10101 > ff05::c0c.10101: UDP, length 38
22:07:12.677777 dc:f5:05:4d:ec:1a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 358: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from dc:f5:05:4d:ec:1a (oui Unknown), length 308
22:07:14.649775 dc:f5:05:70:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 358: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from dc:f5:05:70:fa:8a (oui Unknown), length 308
22:07:14.678753 08:c5:e1:97:fa:ab (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, LLC, dsap Null (0x00) Individual, ssap Null (0x00) Response, ctrl 0xaf: Unnumbered, xid, Flags [Response], length 46: 01 02
22:07:14.830058 08:c5:e1:97:fa:ab (oui Unknown) > 33:33:ff:47:dd:e1 (oui Unknown), ethertype 802.1Q (0x8100), length 86: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv6, :: > ff02::1:ff47:dde1: ICMP6, neighbor solicitation, who has fe80::a715:fd20:be47:dde1, length 24
22:07:14.840986 08:c5:e1:97:fa:ab (oui Unknown) > 33:33:00:00:00:16 (oui Unknown), ethertype 802.1Q (0x8100), length 98: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv6, :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
22:07:14.997613 08:c5:e1:97:fa:ab (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 348: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 08:c5:e1:97:fa:ab (oui Unknown), length 298
22:07:15.393940 08:c5:e1:97:fa:ab (oui Unknown) > 33:33:00:00:00:16 (oui Unknown), ethertype 802.1Q (0x8100), length 98: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv6, :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
22:07:15.585986 08:c5:e1:97:fa:ab (oui Unknown) > 33:33:00:00:00:02 (oui Unknown), ethertype 802.1Q (0x8100), length 78: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv6, fe80::a715:fd20:be47:dde1 > ff02::2: ICMP6, router solicitation, length 16
22:07:15.585989 08:c5:e1:97:fa:ab (oui Unknown) > 33:33:00:00:00:16 (oui Unknown), ethertype 802.1Q (0x8100), length 98: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv6, fe80::a715:fd20:be47:dde1 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
22:07:15.717065 08:c5:e1:97:fa:ab (oui Unknown) > 33:33:00:00:00:16 (oui Unknown), ethertype 802.1Q (0x8100), length 98: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv6, fe80::a715:fd20:be47:dde1 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
22:07:16.145724 08:c5:e1:97:fa:ab (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 348: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 08:c5:e1:97:fa:ab (oui Unknown), length 298
22:07:18.122369 08:c5:e1:97:fa:ab (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 348: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 08:c5:e1:97:fa:ab (oui Unknown), length 298
22:07:19.424819 08:c5:e1:97:fa:ab (oui Unknown) > 33:33:00:00:00:02 (oui Unknown), ethertype 802.1Q (0x8100), length 78: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv6, fe80::a715:fd20:be47:dde1 > ff02::2: ICMP6, router solicitation, length 16
22:07:22.104277 08:c5:e1:97:fa:ab (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 348: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 08:c5:e1:97:fa:ab (oui Unknown), length 298
22:07:26.599913 08:c5:e1:97:fa:ab (oui Unknown) > 33:33:00:00:00:02 (oui Unknown), ethertype 802.1Q (0x8100), length 78: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv6, fe80::a715:fd20:be47:dde1 > ff02::2: ICMP6, router solicitation, length 16
22:07:30.353091 08:c5:e1:97:fa:ab (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 348: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 08:c5:e1:97:fa:ab (oui Unknown), length 298
22:07:32.190446 dc:f5:05:3d:18:2d (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 358: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from dc:f5:05:3d:18:2d (oui Unknown), length 308

stephenw10

Ok, interesting. No outbound traffic there but there's no reason that should be any different.
We still see all the VLAN0 tagged traffic arriving which implies the APs are tagging it.

I would bet if you run the second test with only the switch it will work fine.

What sort of QoS settings do you have on the APs?

Steve

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

would bet if you run the second test with only the switch it will work fine.

In a moment, I had to do some tricks for the tests :). Don't worry I cooperate :) Thank you for staying this long with me.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Ok, interesting. No outbound traffic there but there's no reason that should be any different.
We still see all the VLAN0 tagged traffic arriving which implies the APs are tagging it.

I would bet if you run the second test with only the switch it will work fine.

What sort of QoS settings do you have on the APs?

Steve

I don't think Ubiquity does that on AP. AFAIK you must purchase a router or a switch to achieve that, but I can look in the Unifi Network Controller .
I don't see nothing that I can turn on the AP regarding QoS.

NRgia

@stephenw10

Last test as you instructed:

Added port 7 to VLAN 20 group and marked the port Untagged
Set PVID 20 to port 7
Started tcpdump on pfsense for ix2
Connected the cable to a desktop and to port 7

The desktop MAC is b4:2e:99:c7:b4:26

tcpdump.txt - sorry was too big

No lease, no internet

Maybe I should've removed port 7 from VLAN1 group also?
https://imgur.com/c5wOVI1

stephenw10

Hmm, well that's weird!

As long as you changed the PVID on port 7 to 20 then it would only be one way traffic anyway. I shouldn't prevent traffic VLAN20 working.

Were you able to take a pcap on the connected desktop?

It's hard to imagine what could be tagging that. It seems very unlikely the AP and switch would be doing so independently.

Steve

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Hmm, well that's weird!

As long as you changed the PVID on port 7 to 20 then it would only be one way traffic anyway. I shouldn't prevent traffic VLAN20 working.

Were you able to take a pcap on the connected desktop?

It's hard to imagine what could be tagging that. It seems very unlikely the AP and switch would be doing so independently.

Steve

I thought you're off for today :)

I started to revert back to 22.01, in order to get things running again.

If you need pictures with the switch I can provide. I don't have any reason not to follow your suggestions. I'm the first who wants to solve this.

I tried to dump the desktop, but the interface was not getting any ip just a local one with 169...something...so I provided the ix2 one, dumped from pfsense.

I can reinstall 22.05 tomorrow if you are still willing, and still have ideas :)

stephenw10

Hmm, do you have Snort or Suricata in in-line mode on ix2 by any chance?

In 22.05 you have netmap enabled on ix2 (and ix3) and not in 22.01.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Hmm, do you have Snort or Suricata in in-line mode on ix2 by any chance?

In 22.05 you have netmap enabled on ix2 (and ix3) and not in 22.01.

The configuration is the same on both.
On 22.01 I have Suricata running in Inline mode yes, on ix2 and ix3.
On 22.05 Suricata it is installed but disabled on all interfaces for testing purposes.
Also pfblockerNG installed but disabled.
Besides Avahi and NUT I don't have anything else enabled as packages.

stephenw10

Hmm, did you reboot since disabling it? Is ix2 still showing netmap enabled?
There are known issues with netmap and VLANs and that's definitely a difference between your setup and mine. I'm trying to replicate it now...

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Hmm, did you reboot since disabling it? Is ix2 still showing netmap enabled?
There are known issues with netmap and VLANs and that's definitely a difference between your setup and mine. I'm trying to replicate it now...

I did reboot after I set PROMISC tag to see if it works. So at least one reboot.

stephenw10

Do you still see 'netmap' listed as an option on ix2 though?
Try setting the interface to legacy mode.

I can't replicate exactly what you're seeing but with Snort in-line enabled it cannot pass vlan traffic.

Just disabling the interface in Snort removes the netmap setting and allows vlan tagged traffic to pass here.

Steve

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Do you still see 'netmap' listed as an option on ix2 though?
Try setting the interface to legacy mode.

I can't replicate exactly what you're seeing but with Snort in-line enabled it cannot pass vlan traffic.

Just disabling the interface in Snort removes the netmap setting and allows vlan tagged traffic to pass here.

Steve

After I disabled Suricata I did not saw Netmap on the interfaces. I paid close attention to tags, I was looking for "PROMISC". :)
But to exclude Suricata I can uninstall it altogether.
For now I reverted back to 22.01, but I can upgrade again tomorrow, and uninstall Suricata after the upgrade to 22.05. That is the easy part.

If you have other ideas you can post them, and I will try them in order and report back.

First I will try to set the interfaces to legacy, second I will uninstall the package.

stephenw10

That should be a good test. When I enable netmap here I see no VLAN tagged traffic arrive at all.

Do you have anything else running on the patent interface that might be setting it in promisc mode?

Steve

stephenw10

It would be good to repeat the pcap on ix2 with the VLAN working in 22.01. That will show us if the traffic is still tagged VLAN0 in your setup but the driver in 22.01 just allows it.

Steve

NRgia

This post is deleted!

NRgia

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@stephenw10
Besides Suricata, I do not run anything that should enable promisc mode. I don't think pfblocker enables, promisc mode.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

It would be good to repeat the pcap on ix2 with the VLAN working in 22.01. That will show us if the traffic is still tagged VLAN0 in your setup but the driver in 22.01 just allows it.

Steve

Run a test on pfSense 22.01 as instructed.

started tcpdump on ix2
Connected a mobile device to VLAN 20 with IP 192.168.10.57 with MAC 08:c5:e1:97:fa:ab

tcpdump_pfsense_22.01.txt

Any other tests on pfSense 22.01, or should I upgrade to 22.05 and continue with Suricata testing, like you proposed yesterday?

Thank you

stephenw10

Hmm, no vlan0 traffic at all there...
Does that mean the earlier driver is filtering it before the pcap can see it?
Or that the later driver is somehow adding it?

Are you able to create a mirror port on the switch so we can see what's actually on the wire?