pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue

stephenw10

In 22.01, sniffing in pfSense, the VLAN0 tags never show and the traffic works as expected.

We are speculating that is because the ix driver is incorrectly stripping the VLAN 0 tags before tcpdump sees it.

To prove that what we really want to see is the mirror port in the switch showing the VLAN0 tagged traffic but as far as I know we have not been able to catch that.

However if it was the switch tagging the packets incorrectly we would not have seen them with the AP directly on the pfSense port, yet we did.
We also saw them without an AP connected, just using an access port on the switch.
So that leaves.... ???

johnpoz

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

So that leaves.... ???

That is the $64,000 ? it seems... Very odd for sure.. I missed a bit of the thread it seems.. sorry.

What is putting the vlan 0 tag on there?? I just do not see how it could be pfsense, make no sense that would see them in the tcpdump for inbound traffic.

@stephenw10 you have not been able to duplicate with your 22.01 box with ix nics?

Bob.Dig

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

So that leaves.... ???

TAC Support maybe.

NRgia

@johnpoz @stephenw10
Took a tcpdump from a linux host this time, to be sure it can sniff all the traffic.

Connected to the mirror port:

linux_tcp_dump_mirror_port.txt

My opinion is in the title of this post, of who the culprit is, but I don't want point fingers. I'm expecting to find out from one of you guys. :)

johnpoz

@nrgia well clearly something is going on, but if 22.05 broke vlans - I would expect the boards would be a fire with people complaining. Even if was only ix interfaces, have to assume your not the only one using ix and vlans that have moved to 22.05

NRgia

@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@nrgia well clearly something is going on, but if 22.05 broke vlans - I would expect the boards would be a fire with people complaining. Even if was only ix interfaces, have to assume your not the only one using ix and vlans that have moved to 22.05

That's true, that's why I don't know what to say, it's not a no name board. or a realtek driver. But I wonder, what's @stephenw10 conclusion?

johnpoz

@nrgia I'm with you for sure - one of the more strange threads seen in a really long time. @stephenw10 would be for sure my go to guy on anything like this.

stephenw10

It's not generally broken. The 7100 uses ix NICs with VLANs by default and that works fine in 22.05. Also my test here with a 4100 using an individual NIC is not broken.

We know more than just 22.01 worked and 22.05 didn't. The snapshot that first failed was immediately after the patch that went in to fix VLAN0 handling.

The incoming traffic we see in the pcap in 22.05 is tagged VLAN0 and hence is dropped by pfSense/FreeBSD. That is the expected behaviour.

The only thing we don't know is where that tag is coming from.
Experiments seem to show that is isn't coming from the switch (though that still seems most likely) or from the AP.
Unless I've misunderstood the situation the only thing left is the driver or hardware. Neither of which seem at all likely but in the absence of anything else...

stephenw10

What I expect to see if you have any sort of priotiy tagging on a VLAN is something like this:

18:23:28.004183 00:90:7f:b4:38:fa (oui Unknown) > 90:ec:77:1f:8a:5f (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 4, ethertype IPv4, AP300.stevew.lan > 10.101.0.1: ICMP echo request, id 10296, seq 2, length 64
18:23:28.004212 90:ec:77:1f:8a:5f (oui Unknown) > 00:90:7f:b4:38:fa (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 0, ethertype IPv4, 10.101.0.1 > AP300.stevew.lan: ICMP echo reply, id 10296, seq 2, length 64

There I have set egress tagging in OpenWRT to map 0 to 4 so pfSense sees it on the incoming traffic. But there is only one set of tags.

stephenw10

If we can make the switch or AP apply priority tags we could then see which set of tags they appear on in the 22.05 pcap. I expect it to be on the VLAN20 tag. If it's on VLAN0 tag that might prove what's creating it.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

What I expect to see if you have any sort of priotiy tagging on a VLAN is something like this:
18:23:28.004183 00:90:7f:b4:38:fa (oui Unknown) > 90:ec:77:1f:8a:5f (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 4, ethertype IPv4, AP300.stevew.lan > 10.101.0.1: ICMP echo request, id 10296, seq 2, length 64
18:23:28.004212 90:ec:77:1f:8a:5f (oui Unknown) > 00:90:7f:b4:38:fa (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 0, ethertype IPv4, 10.101.0.1 > AP300.stevew.lan: ICMP echo reply, id 10296, seq 2, length 64
There I have set egress tagging in OpenWRT to map 0 to 4 so pfSense sees it on the incoming traffic. But there is only one set of tags.

Just a question, what we except to happen if set some priority to VLANS form pfSense? I know the documentation says to leave that on zero?

I don't know how to provide what you want to see.

NRgia

@stephenw10
From the switch in order to apply priority tags, I must disable the 802.1p mode, and apply per port, for the AP I don't see any option to apply a priority without buying a switch from Ubiquiti. I can provide you some screenshots.

This is how it looks for a VLAN:
https://imgur.com/a/3gBCTIL

johnpoz

@nrgia I have unifi APs, and yeah I just looked I do not see anywhere to set priority on a vlan at all.

Running the lastest firmware on APs 6.2.27 and 7.2.87 controller software. I could try moving to the new UI vs legacy that I use.

NRgia

@johnpoz I'm using the new UI, don't bother. If you don't buy the whole ecosystem you can do so much. But a four eyes check doesn't do any harm. Thank you

NRgia

@nrgia How your networks are configured in the AP, I mean in the Networks tab? Are the VLANS set to VLANs only network? Or how? Thank you

stephenw10

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Just a question, what we except to happen if set some priority to VLANS form pfSense?

Applying it on the VLAN should show in the VLAN tag outbound as expected like:

19:16:32.694149 00:90:7f:b4:38:fa (oui Unknown) > 90:ec:77:1f:8a:5f (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 4, ethertype IPv4, AP300.stevew.lan > 10.101.0.1: ICMP echo request, id 10347, seq 1, length 64
19:16:32.694172 90:ec:77:1f:8a:5f (oui Unknown) > 00:90:7f:b4:38:fa (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 3, ethertype IPv4, 10.101.0.1 > AP300.stevew.lan: ICMP echo reply, id 10347, seq 1, length 64

pfSense sets prio 3 and the AP replies prio 4.

Try using port mode on your switch then. If you can set anything that appears in the pcap it might be useful.

Steve

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

ng that appears in the pcap it might be useful.
Steve

First I selected Normal priority on the switch for the pfSense port and the AP port. The switch don't have numbers. Also I select 3 for VLAN 20 on pfSense.

I got this:
tcp_dump_vlan_prioriry_pfsense_and_switch.txt

I lost connection after doing this, I could only connect to devices via their IPs

Then i reset the priority from VLAN 20 from pfSense to zero(0) and I got this:
tcp_dump_vlan_prioriry_on_switch.txt

stephenw10

Those pcaps were on the mirror port again?

In both those pcaps there are no packets from pfSense at all....

Do you still see outbound packets if you tcpdump on ix2 on pfSense in either of those setups?

Steve

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Those pcaps were on the mirror port again?

In both those pcaps there are no packets from pfSense at all....

Do you still see outbound packets if you tcpdump on ix2 on pfSense in either of those setups?

Steve

Ok I redone the tests:
First test:
Steps:

Set priority to 3 in pfSense to VLAN 20
Set Normal Priority in the switch to the port where the AP is connected
Set High priority in the switch to the port where pfSense LAN side is connected.
Done a tcpdump from pfSense.
Results:
tcpdump_priority_set_to pfsense_and_switch.txt

As you can see, only the priority (3) set in pfSense is intercepted.

Second test:

Removed priority 3 set in pfSense to VLAN 20
Left the Normal Priority in the switch to the port where the AP is connected
Left the High priority in the switch to the port where pfSense LAN side is connected.
Done a tcpdump from pfSense.

Results:
tcpdump_priority_set_to_switch.txt

I don't see anu priorities besides (p0) when done from the switch.

I hope the steps are ok, now.

stephenw10

Hmm, OK.

Can I assume traffic was failing during those tests?
We see no ARP replies to pfSense's queries and it looks like one way traffic from the clients.