pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue

stephenw10

The QoS and PVID tabs?

NRgia

@johnpoz
So it's like this
On port 5 it is connected a Unifi AP - VLAN aware
On port 15 is pfsense (LAN side)

stephenw10

The more I look into this the more it looks like an incorrect QoS setting being applied.

NRgia

@stephenw10

PVID:
https://imgur.com/1hOGcjW

QOS page 1

https://imgur.com/GdPzhEn

QOS page 2

https://imgur.com/jS8Px5Y

QOS page 3

https://imgur.com/ef3h9kF

If this don't work I can hook up a laptop with Manjaro if you tell me what to do

stephenw10

Hmm, well you wouldn't expect it to be doing anything with those settings but try setting QoS to 802.1p mode with no port selected and see if that changes anything in pcaps.

It pretty much has to be the switch doing that since it's just passing the tagged traffic.

NRgia

@stephenw10

I will do as you say, but I remind you that with pfsense 22.01 worked, I did not touch the switches.

So, first change the setting and then to do a dump from where? pfsense or hook up a laptop to vlan2.20 port ?

stephenw10

I'd repeat the previous dump where we could see the double tagged traffic arriving from the the device at .56.

I'm suggesting that this was working in 22.01 at earlier because the driver was incorrectly stripping the tags and now after the fix it is not. FreeBSD now drops the traffic because that's what it's supposed top do with VLAN0.
The last snapshot that worked was built just before that fix was added. On the same day.

NRgia

@stephenw10

19:54:15.069333 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:20.067967 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:25.067447 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:30.256858 dc:f5:05:3d:18:2d (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, LLC, dsap Null (0x00) Individual, ssap Null (0x00) Response, ctrl 0xaf: Unnumbered, xid, Flags [Response], length 46: 01 02
19:54:32.205770 dc:f5:05:3d:18:2d (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 358: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from dc:f5:05:3d:18:2d (oui Unknown), length 308
19:54:36.198452 dc:f5:05:3d:18:2d (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 358: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from dc:f5:05:3d:18:2d (oui Unknown), length 308
19:54:44.184506 dc:f5:05:3d:18:2d (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 358: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from dc:f5:05:3d:18:2d (oui Unknown), length 308
19:54:45.079594 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:46.068199 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:47.067681 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:49.068646 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:49.743799 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype 802.1Q (0x8100), length 86: vlan 20, p 0, ethertype IPv4, 192.168.10.1.mdns > 224.0.0.251.mdns: 0 PTR (QM)? _googlezone._tcp.local. (40)
19:54:49.743972 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype 802.1Q (0x8100), length 123: vlan 20, p 0, ethertype IPv4, 192.168.10.1.mdns > 224.0.0.251.mdns: 0 SRV (QM)? ee41442d-2c14-cc09-fde8-2be16f84be32._googlezone._tcp.local. (77)
19:54:49.744264 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype 802.1Q (0x8100), length 256: vlan 20, p 0, ethertype IPv4, 192.168.10.1.mdns > 224.0.0.251.mdns: 0*- [0q] 4/0/0 PTR ee41442d-2c14-cc09-fde8-2be16f84be32._googlezone._tcp.local., (Cache flush) A 172.18.0.14, (Cache flush) SRV ee41442d-2c14-cc09-fde8-2be16f84be32.local.:10001 1100 0, (Cache flush) TXT "id=3CABD325728E72997BA6735F95651E36" "UDS" (210)
19:54:52.068645 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:55.068665 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:59.070888 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:55:00.157354 dc:f5:05:3d:18:2d (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 358: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from dc:f5:05:3d:18:2d (oui Unknown), length 308
19:55:03.070243 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:55:06.797226 dc:f5:05:4d:ec:1a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, LLC, dsap Null (0x00) Individual, ssap Null (0x00) Response, ctrl 0xaf: Unnumbered, xid, Flags [Response], length 46: 01 02
19:55:08.072959 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:55:08.751278 dc:f5:05:4d:ec:1a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 358: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from dc:f5:05:4d:ec:1a (oui Unknown), length 308
19:55:10.253923 cc:f4:11:c5:bc:81 (oui Unknown) > 33:33:00:0c:00:0c (oui Unknown), ethertype 802.1Q (0x8100), length 108: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv6, fe80::cef4:11ff:fec5:bc81.10101 > ff02::c:c.10101: UDP, length 38
19:55:10.253930 cc:f4:11:c5:bc:81 (oui Unknown) > 33:33:00:00:0c:0c (oui Unknown), ethertype 802.1Q (0x8100), length 108: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv6, fe80::cef4:11ff:fec5:bc81.10101 > ff05::c0c.10101: UDP, length 38
^C
29 packets captured
424 packets received by filter
0 packets dropped by kernel

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

I'm suggesting that this was working in 22.01 at earlier because the driver was incorrectly stripping the tags and now after the fix it is not. FreeBSD now drops the traffic because that's what it's supposed top do with VLAN0.
The last snapshot that worked was built just before that fix was added. On the same day.

At least I'm not crazy. :)

stephenw10

Ok, no difference.

Ok let's try to verify it is the switch doing this. Can you connect one of the access points to ix2 directly?
Otherwise lets get a laptop on one of the ports and see what's arriving at the other end.

It looks very likely to be the switch. There's probably some combination of QoS settings that will allow it to work. Enabling it on an unused port. Enabling it on the ports we need (pfSense doesn't care about the priority tag).

Steve

NRgia

@stephenw10
Yes I can, and I can validate by having Internet on My phone.
For example both WLANS are having the same tags as our VLANS.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Ok, no difference.

Ok let's try to verify it is the switch doing this. Can you connect one of the access points to ix2 directly?
Otherwise lets get a laptop on one of the ports and see what's arriving at the other end.

It looks very likely to be the switch. There's probably some combination of QoS settings that will allow it to work. Enabling it on an unused port. Enabling it on the ports we need (pfSense doesn't care about the priority tag).

Steve

I did like you suggested:
pfSense->Unifi AP->Mobile Device
The only SSID that gets internet is the one without VLANS same with the switch.

So it's not the switch. I excluded all the switches.

stephenw10

Ok, did you get a pcap from it?

You might have to run it without 'vlan' in case traffic is arriving untagged entirely.

stephenw10

Another test you could do would be to add a port on the switch as untagged in VLAN 20 with the pvid set to 20 and connect a host to it. Does it pull a lease from VLAN 20.

Steve

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Ok, did you get a pcap from it?

You might have to run it without 'vlan' in case traffic is arriving untagged entirely.

No I was connected only on my mobile device, but I can do the following:

Connect with a laptop via Wi-fi
Or directly to pfsense via cable

How do you prefer?

stephenw10

A pcap on ix2 in pfSense as before would show if it's still seeing the off double tagged traffic arrive. If it is that implies the AP is tagging it and the switch is just passing that.

Testing the switch without the AP would also confirm that.

NRgia

@stephenw10 Yes but I want the same port to have both 20 and 30. Can I do that as you instructed? I mean let's take an AP, you will have multiple SSIDs but connected to different VLANs

NRgia

@stephenw10
The tests done previously were done via wire, not wireless, the last test was done after you suggested to change the QOS setting.

I'm not sure I follow what you suggest to do now.

bingo600

Suggestion

Do you know how to make a "Monitor/Span Port" on the Switch ?
If yes ... You said you had a Manjaro Linux where you could run WireShark.

Make a Monitor port on the switch , that monitors the port connected to pfSense , both RX & TX.

Connect wireshark to the Monitor port , and make a packet trace that way.

/Bingo

stephenw10

If you can make one VLAN work 100 will also work, that's not the issue.

There are two tests you can do.

Connect one of the APs directly to ix2. Run a pcap in pfSense on ix2 as you did before. Try to connect to the SSID using the VLAN and see what appears.
If you see nothing try running it without the 'vlan' option in case the traffic is arriving untagged.

With the switch comnected to ix2 add a port as untagged in VLAN 20 with the pvid set to 20 and connect a host to it. Does it pull a lease from VLAN 20.
If not run a pcap in pfSense again to see what's happening.
You can also pcap on the wired host to see what that sees.

Steve