pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue

johnpoz

@nrgia fist thing I would do is just change your switch to 802.1p mode... Are you seeing that vlan 0 on your sniff on pfsense.. Doesn't matter if 22.01 or 22.05

NRgia

@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@nrgia on some iot device not going to be possible... Would need to be another downstream switch that allows you to set it or setting it on the device.

With linux you could do with vconfig, you just set a egress map to change the priority. Windows can set it as well.. Just not as easy as with vconfig in linux.

But highly unlikely some smart lightbulb is going to let you set it ;)

:) I'm worried that neither my Hub that controls all the smart things cannot be set to do that, neither the smart tv. I will try what you say but if this priority cannot be set an all devices, how it will solve the issue? Only a laptop will work? Or is just for investigation?

johnpoz

@nrgia solve what remove vlan 0, not sure why you would have to do anything on your device.. I find it hard to believe devices are adding that..

Can you do a simple test and change your switch to 802.1p mode, and sniff - are you seeing that vlan 0?

NRgia

@johnpoz

Sure I can,
Here you go:

pfsense_dump.txt

johnpoz

@nrgia well I don't see any vlan 0 in there.. So your saying still not working?

Your saying when you change to 22.05 you is only time you see that vlan 0?

NRgia

@johnpoz
I'm not just saying, the above dumps shows that.
But that's the idea yes.

Should I upgrade and see what happens ?

I only want for this to work, if it works, tell me how to donate, for wasting your time, no joke intended.

stephenw10

In 22.01, sniffing in pfSense, the VLAN0 tags never show and the traffic works as expected.

We are speculating that is because the ix driver is incorrectly stripping the VLAN 0 tags before tcpdump sees it.

To prove that what we really want to see is the mirror port in the switch showing the VLAN0 tagged traffic but as far as I know we have not been able to catch that.

However if it was the switch tagging the packets incorrectly we would not have seen them with the AP directly on the pfSense port, yet we did.
We also saw them without an AP connected, just using an access port on the switch.
So that leaves.... ???

johnpoz

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

So that leaves.... ???

That is the $64,000 ? it seems... Very odd for sure.. I missed a bit of the thread it seems.. sorry.

What is putting the vlan 0 tag on there?? I just do not see how it could be pfsense, make no sense that would see them in the tcpdump for inbound traffic.

@stephenw10 you have not been able to duplicate with your 22.01 box with ix nics?

Bob.Dig

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

So that leaves.... ???

TAC Support maybe.

NRgia

@johnpoz @stephenw10
Took a tcpdump from a linux host this time, to be sure it can sniff all the traffic.

Connected to the mirror port:

linux_tcp_dump_mirror_port.txt

My opinion is in the title of this post, of who the culprit is, but I don't want point fingers. I'm expecting to find out from one of you guys. :)

johnpoz

@nrgia well clearly something is going on, but if 22.05 broke vlans - I would expect the boards would be a fire with people complaining. Even if was only ix interfaces, have to assume your not the only one using ix and vlans that have moved to 22.05

NRgia

@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@nrgia well clearly something is going on, but if 22.05 broke vlans - I would expect the boards would be a fire with people complaining. Even if was only ix interfaces, have to assume your not the only one using ix and vlans that have moved to 22.05

That's true, that's why I don't know what to say, it's not a no name board. or a realtek driver. But I wonder, what's @stephenw10 conclusion?

johnpoz

@nrgia I'm with you for sure - one of the more strange threads seen in a really long time. @stephenw10 would be for sure my go to guy on anything like this.

stephenw10

It's not generally broken. The 7100 uses ix NICs with VLANs by default and that works fine in 22.05. Also my test here with a 4100 using an individual NIC is not broken.

We know more than just 22.01 worked and 22.05 didn't. The snapshot that first failed was immediately after the patch that went in to fix VLAN0 handling.

The incoming traffic we see in the pcap in 22.05 is tagged VLAN0 and hence is dropped by pfSense/FreeBSD. That is the expected behaviour.

The only thing we don't know is where that tag is coming from.
Experiments seem to show that is isn't coming from the switch (though that still seems most likely) or from the AP.
Unless I've misunderstood the situation the only thing left is the driver or hardware. Neither of which seem at all likely but in the absence of anything else...

stephenw10

What I expect to see if you have any sort of priotiy tagging on a VLAN is something like this:

18:23:28.004183 00:90:7f:b4:38:fa (oui Unknown) > 90:ec:77:1f:8a:5f (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 4, ethertype IPv4, AP300.stevew.lan > 10.101.0.1: ICMP echo request, id 10296, seq 2, length 64
18:23:28.004212 90:ec:77:1f:8a:5f (oui Unknown) > 00:90:7f:b4:38:fa (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 0, ethertype IPv4, 10.101.0.1 > AP300.stevew.lan: ICMP echo reply, id 10296, seq 2, length 64

There I have set egress tagging in OpenWRT to map 0 to 4 so pfSense sees it on the incoming traffic. But there is only one set of tags.

stephenw10

If we can make the switch or AP apply priority tags we could then see which set of tags they appear on in the 22.05 pcap. I expect it to be on the VLAN20 tag. If it's on VLAN0 tag that might prove what's creating it.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

What I expect to see if you have any sort of priotiy tagging on a VLAN is something like this:
18:23:28.004183 00:90:7f:b4:38:fa (oui Unknown) > 90:ec:77:1f:8a:5f (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 4, ethertype IPv4, AP300.stevew.lan > 10.101.0.1: ICMP echo request, id 10296, seq 2, length 64
18:23:28.004212 90:ec:77:1f:8a:5f (oui Unknown) > 00:90:7f:b4:38:fa (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 0, ethertype IPv4, 10.101.0.1 > AP300.stevew.lan: ICMP echo reply, id 10296, seq 2, length 64
There I have set egress tagging in OpenWRT to map 0 to 4 so pfSense sees it on the incoming traffic. But there is only one set of tags.

Just a question, what we except to happen if set some priority to VLANS form pfSense? I know the documentation says to leave that on zero?

I don't know how to provide what you want to see.

NRgia

@stephenw10
From the switch in order to apply priority tags, I must disable the 802.1p mode, and apply per port, for the AP I don't see any option to apply a priority without buying a switch from Ubiquiti. I can provide you some screenshots.

This is how it looks for a VLAN:
https://imgur.com/a/3gBCTIL

johnpoz

@nrgia I have unifi APs, and yeah I just looked I do not see anywhere to set priority on a vlan at all.

Running the lastest firmware on APs 6.2.27 and 7.2.87 controller software. I could try moving to the new UI vs legacy that I use.

NRgia

@johnpoz I'm using the new UI, don't bother. If you don't buy the whole ecosystem you can do so much. But a four eyes check doesn't do any harm. Thank you