pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue

NRgia

@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

The only device that passes traffic to the switch is pfSense

What.. You have no devices connect to the switch? That passes traffic "through" the switch to pfsense to be routed.

My point was

pfsense - switch -- device

You would have to set the 802.1p priority on the device.. We can play semantics all you want - but sorry that traffic flows through the switch..

Ok, then how do I set the priority on the device? Beeing a laptop, smart hub, Smart Tv, etc.?
pfSense - switch --device(laptop)
--device(IOT hub)
--smart TV

johnpoz

@nrgia on some iot device not going to be possible... Would need to be another downstream switch that allows you to set it or setting it on the device.

With linux you could do with vconfig, you just set a egress map to change the priority. Windows can set it as well.. Just not as easy as with vconfig in linux.

But highly unlikely some smart lightbulb is going to let you set it ;)

NRgia

@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

and sniff the mirror port

Why are you sniffing the mirror port? Sorry but there is no way pfsense is adding the vlan 0 p0 onto traffic that is entering it.

Then guide me through a few steps, what should I sniff, from which host, and what port, please.

Here is a test, get a dumb switch.. It would pass tags.. Now connect your AP to the dumb switch, and then from the dumb switch to pfsense port.

I don't have a dumb switch.

If this was my issue I would be looking how to make that vlan 0 go away, or figure out what is doing it.. You see them on 22.01 but it works, you see them on 22.05 and doesn't work - because it shouldn't you have no vlan 0 setup, etc so that traffic should be dropped.

I'm doing that, but when I run something, I just want to reproduce what you are asking of me, I'm not trying to be smart.

Here is the thing - vlan 0 shouldn't be on the traffic in the first place - what is putting it on there? How would it be pfsense when the traffic is inbound into its interface...

johnpoz

@nrgia fist thing I would do is just change your switch to 802.1p mode... Are you seeing that vlan 0 on your sniff on pfsense.. Doesn't matter if 22.01 or 22.05

NRgia

@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@nrgia on some iot device not going to be possible... Would need to be another downstream switch that allows you to set it or setting it on the device.

With linux you could do with vconfig, you just set a egress map to change the priority. Windows can set it as well.. Just not as easy as with vconfig in linux.

But highly unlikely some smart lightbulb is going to let you set it ;)

:) I'm worried that neither my Hub that controls all the smart things cannot be set to do that, neither the smart tv. I will try what you say but if this priority cannot be set an all devices, how it will solve the issue? Only a laptop will work? Or is just for investigation?

johnpoz

@nrgia solve what remove vlan 0, not sure why you would have to do anything on your device.. I find it hard to believe devices are adding that..

Can you do a simple test and change your switch to 802.1p mode, and sniff - are you seeing that vlan 0?

NRgia

@johnpoz

Sure I can,
Here you go:

pfsense_dump.txt

johnpoz

@nrgia well I don't see any vlan 0 in there.. So your saying still not working?

Your saying when you change to 22.05 you is only time you see that vlan 0?

NRgia

@johnpoz
I'm not just saying, the above dumps shows that.
But that's the idea yes.

Should I upgrade and see what happens ?

I only want for this to work, if it works, tell me how to donate, for wasting your time, no joke intended.

stephenw10

In 22.01, sniffing in pfSense, the VLAN0 tags never show and the traffic works as expected.

We are speculating that is because the ix driver is incorrectly stripping the VLAN 0 tags before tcpdump sees it.

To prove that what we really want to see is the mirror port in the switch showing the VLAN0 tagged traffic but as far as I know we have not been able to catch that.

However if it was the switch tagging the packets incorrectly we would not have seen them with the AP directly on the pfSense port, yet we did.
We also saw them without an AP connected, just using an access port on the switch.
So that leaves.... ???

johnpoz

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

So that leaves.... ???

That is the $64,000 ? it seems... Very odd for sure.. I missed a bit of the thread it seems.. sorry.

What is putting the vlan 0 tag on there?? I just do not see how it could be pfsense, make no sense that would see them in the tcpdump for inbound traffic.

@stephenw10 you have not been able to duplicate with your 22.01 box with ix nics?

Bob.Dig

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

So that leaves.... ???

TAC Support maybe.

NRgia

@johnpoz @stephenw10
Took a tcpdump from a linux host this time, to be sure it can sniff all the traffic.

Connected to the mirror port:

linux_tcp_dump_mirror_port.txt

My opinion is in the title of this post, of who the culprit is, but I don't want point fingers. I'm expecting to find out from one of you guys. :)

johnpoz

@nrgia well clearly something is going on, but if 22.05 broke vlans - I would expect the boards would be a fire with people complaining. Even if was only ix interfaces, have to assume your not the only one using ix and vlans that have moved to 22.05

NRgia

@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@nrgia well clearly something is going on, but if 22.05 broke vlans - I would expect the boards would be a fire with people complaining. Even if was only ix interfaces, have to assume your not the only one using ix and vlans that have moved to 22.05

That's true, that's why I don't know what to say, it's not a no name board. or a realtek driver. But I wonder, what's @stephenw10 conclusion?

johnpoz

@nrgia I'm with you for sure - one of the more strange threads seen in a really long time. @stephenw10 would be for sure my go to guy on anything like this.

stephenw10

It's not generally broken. The 7100 uses ix NICs with VLANs by default and that works fine in 22.05. Also my test here with a 4100 using an individual NIC is not broken.

We know more than just 22.01 worked and 22.05 didn't. The snapshot that first failed was immediately after the patch that went in to fix VLAN0 handling.

The incoming traffic we see in the pcap in 22.05 is tagged VLAN0 and hence is dropped by pfSense/FreeBSD. That is the expected behaviour.

The only thing we don't know is where that tag is coming from.
Experiments seem to show that is isn't coming from the switch (though that still seems most likely) or from the AP.
Unless I've misunderstood the situation the only thing left is the driver or hardware. Neither of which seem at all likely but in the absence of anything else...

stephenw10

What I expect to see if you have any sort of priotiy tagging on a VLAN is something like this:

18:23:28.004183 00:90:7f:b4:38:fa (oui Unknown) > 90:ec:77:1f:8a:5f (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 4, ethertype IPv4, AP300.stevew.lan > 10.101.0.1: ICMP echo request, id 10296, seq 2, length 64
18:23:28.004212 90:ec:77:1f:8a:5f (oui Unknown) > 00:90:7f:b4:38:fa (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 0, ethertype IPv4, 10.101.0.1 > AP300.stevew.lan: ICMP echo reply, id 10296, seq 2, length 64

There I have set egress tagging in OpenWRT to map 0 to 4 so pfSense sees it on the incoming traffic. But there is only one set of tags.

stephenw10

If we can make the switch or AP apply priority tags we could then see which set of tags they appear on in the 22.05 pcap. I expect it to be on the VLAN20 tag. If it's on VLAN0 tag that might prove what's creating it.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

What I expect to see if you have any sort of priotiy tagging on a VLAN is something like this:
18:23:28.004183 00:90:7f:b4:38:fa (oui Unknown) > 90:ec:77:1f:8a:5f (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 4, ethertype IPv4, AP300.stevew.lan > 10.101.0.1: ICMP echo request, id 10296, seq 2, length 64
18:23:28.004212 90:ec:77:1f:8a:5f (oui Unknown) > 00:90:7f:b4:38:fa (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 0, ethertype IPv4, 10.101.0.1 > AP300.stevew.lan: ICMP echo reply, id 10296, seq 2, length 64
There I have set egress tagging in OpenWRT to map 0 to 4 so pfSense sees it on the incoming traffic. But there is only one set of tags.

Just a question, what we except to happen if set some priority to VLANS form pfSense? I know the documentation says to leave that on zero?

I don't know how to provide what you want to see.