pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue

stephenw10

Anything is possible! You should never use VLAN1 (or 0 IMO) because some switches do weird things with that.
That double tagged traffic looks wrong and it's hard to see how pfSense could be causing it.

Can we see the switch config? Does it have any QinQ or Priority tagging options?

I would have to guess that something in incorrectly applying priority tags to already VLAN tagged traffic.

johnpoz

@stephenw10 isn't there a whole other thread going on for quite some time about vlan 0? I have just stayed out of that one - no experience ever using a vlan 0.

But vlan 1 should pretty much never be tagged. That is just the default vlan ID uses use for their default untagged default network.

But yeah that double tag thing doesn't look right to me.

edit: Just me and my ocd I think - but why would you tag 192.168.10 with an ID of 20 and 192.168.20 with 30?, wouldn't 10 be better ;) just so you know hey 192.168.10 - that is vlan 10, I do that with my vlan 4 and 6, they are 192.168.4 and 192.168.6 networks ;)

NRgia

@stephenw10
You can, sure, my native VLAN is untagged with vlan 1 on the switch. It worked before so I did not bother.

https://imgur.com/a/hHtfPQ8

NRgia

@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@stephenw10 isn't there a whole other thread going on for quite some time about vlan 0? I have just stayed out of that one - no experience ever using a vlan 0.

But vlan 1 should pretty much never be tagged. That is just the default vlan ID uses use for their default untagged default network.

But yeah that double tag thing doesn't look right to me.

edit: Just me and my ocd I think - but why would you tag 192.168.10 with an ID of 20, wouldn't 10 be better ;) just so you know hey 192.168.10 - that is vlan 10, I do that with my vlan 4 and 6, they are 192.168.4 and 192.168.6 networks ;)

Watched Tom Lawrence once, and he tagged them that way :) If it matters I can rename them, if you think it matters

VLAN 1 is not tagged in my case. It's only in the switch. All the ports on Group VLAN 1 are untagged.

johnpoz

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

if you think it matters

no doesn't matter - just odd, it is common practice to use an ID that somehow relates to the IP range is all.. But the vlan ID has zero to do with the IP space used on the vlan..

NRgia

@johnpoz I will rename them, I know it's not logic to follow when debugging.
On your primary switch what do you have for native 1 or 0 ?

stephenw10

Can we see the other VLAN config tabs? What is that switch? What firmware version?

But I would still get a laptop on to it and take some pcaps there to see what's happening.

Steve

johnpoz

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

primary switch what do you have for native 1 or 0 ?

My default is 9 ;) common practice to move away from 1 in the enterprise. Have never seen 0 to be honest.. Its more of a special use ID, have never ever seen 0 used on a switch as the default vlan. Every switch that I can remember has always been 1 as the default vlan.

Notice - doesn't allow you to set 0, its 1-4094

NRgia

@stephenw10
The Model is GS116Ev2 firmware version 2.6.0.48

VLAN1
https://imgur.com/Js7iYjc

VLAN20
https://imgur.com/keYmhMB

VLAN30
https://imgur.com/gW0qBhc

johnpoz

@nrgia why do you not have any untagged ports in your 10 or 20 vlans? Do you have no devices actually plugged into this switch on those vlans, and only other switches or AP?

stephenw10

The QoS and PVID tabs?

NRgia

@johnpoz
So it's like this
On port 5 it is connected a Unifi AP - VLAN aware
On port 15 is pfsense (LAN side)

stephenw10

The more I look into this the more it looks like an incorrect QoS setting being applied.

NRgia

@stephenw10

PVID:
https://imgur.com/1hOGcjW

QOS page 1

https://imgur.com/GdPzhEn

QOS page 2

https://imgur.com/jS8Px5Y

QOS page 3

https://imgur.com/ef3h9kF

If this don't work I can hook up a laptop with Manjaro if you tell me what to do

stephenw10

Hmm, well you wouldn't expect it to be doing anything with those settings but try setting QoS to 802.1p mode with no port selected and see if that changes anything in pcaps.

It pretty much has to be the switch doing that since it's just passing the tagged traffic.

NRgia

@stephenw10

I will do as you say, but I remind you that with pfsense 22.01 worked, I did not touch the switches.

So, first change the setting and then to do a dump from where? pfsense or hook up a laptop to vlan2.20 port ?

stephenw10

I'd repeat the previous dump where we could see the double tagged traffic arriving from the the device at .56.

I'm suggesting that this was working in 22.01 at earlier because the driver was incorrectly stripping the tags and now after the fix it is not. FreeBSD now drops the traffic because that's what it's supposed top do with VLAN0.
The last snapshot that worked was built just before that fix was added. On the same day.

NRgia

@stephenw10

19:54:15.069333 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:20.067967 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:25.067447 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:30.256858 dc:f5:05:3d:18:2d (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, LLC, dsap Null (0x00) Individual, ssap Null (0x00) Response, ctrl 0xaf: Unnumbered, xid, Flags [Response], length 46: 01 02
19:54:32.205770 dc:f5:05:3d:18:2d (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 358: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from dc:f5:05:3d:18:2d (oui Unknown), length 308
19:54:36.198452 dc:f5:05:3d:18:2d (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 358: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from dc:f5:05:3d:18:2d (oui Unknown), length 308
19:54:44.184506 dc:f5:05:3d:18:2d (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 358: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from dc:f5:05:3d:18:2d (oui Unknown), length 308
19:54:45.079594 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:46.068199 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:47.067681 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:49.068646 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:49.743799 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype 802.1Q (0x8100), length 86: vlan 20, p 0, ethertype IPv4, 192.168.10.1.mdns > 224.0.0.251.mdns: 0 PTR (QM)? _googlezone._tcp.local. (40)
19:54:49.743972 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype 802.1Q (0x8100), length 123: vlan 20, p 0, ethertype IPv4, 192.168.10.1.mdns > 224.0.0.251.mdns: 0 SRV (QM)? ee41442d-2c14-cc09-fde8-2be16f84be32._googlezone._tcp.local. (77)
19:54:49.744264 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype 802.1Q (0x8100), length 256: vlan 20, p 0, ethertype IPv4, 192.168.10.1.mdns > 224.0.0.251.mdns: 0*- [0q] 4/0/0 PTR ee41442d-2c14-cc09-fde8-2be16f84be32._googlezone._tcp.local., (Cache flush) A 172.18.0.14, (Cache flush) SRV ee41442d-2c14-cc09-fde8-2be16f84be32.local.:10001 1100 0, (Cache flush) TXT "id=3CABD325728E72997BA6735F95651E36" "UDS" (210)
19:54:52.068645 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:55.068665 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:54:59.070888 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:55:00.157354 dc:f5:05:3d:18:2d (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 358: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from dc:f5:05:3d:18:2d (oui Unknown), length 308
19:55:03.070243 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:55:06.797226 dc:f5:05:4d:ec:1a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, LLC, dsap Null (0x00) Individual, ssap Null (0x00) Response, ctrl 0xaf: Unnumbered, xid, Flags [Response], length 46: 01 02
19:55:08.072959 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
19:55:08.751278 dc:f5:05:4d:ec:1a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 358: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from dc:f5:05:4d:ec:1a (oui Unknown), length 308
19:55:10.253923 cc:f4:11:c5:bc:81 (oui Unknown) > 33:33:00:0c:00:0c (oui Unknown), ethertype 802.1Q (0x8100), length 108: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv6, fe80::cef4:11ff:fec5:bc81.10101 > ff02::c:c.10101: UDP, length 38
19:55:10.253930 cc:f4:11:c5:bc:81 (oui Unknown) > 33:33:00:00:0c:0c (oui Unknown), ethertype 802.1Q (0x8100), length 108: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv6, fe80::cef4:11ff:fec5:bc81.10101 > ff05::c0c.10101: UDP, length 38
^C
29 packets captured
424 packets received by filter
0 packets dropped by kernel

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

I'm suggesting that this was working in 22.01 at earlier because the driver was incorrectly stripping the tags and now after the fix it is not. FreeBSD now drops the traffic because that's what it's supposed top do with VLAN0.
The last snapshot that worked was built just before that fix was added. On the same day.

At least I'm not crazy. :)

stephenw10

Ok, no difference.

Ok let's try to verify it is the switch doing this. Can you connect one of the access points to ix2 directly?
Otherwise lets get a laptop on one of the ports and see what's arriving at the other end.

It looks very likely to be the switch. There's probably some combination of QoS settings that will allow it to work. Enabling it on an unused port. Enabling it on the ports we need (pfSense doesn't care about the priority tag).

Steve