pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue

johnpoz

@nrgia well clearly something is going on, but if 22.05 broke vlans - I would expect the boards would be a fire with people complaining. Even if was only ix interfaces, have to assume your not the only one using ix and vlans that have moved to 22.05

NRgia

@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@nrgia well clearly something is going on, but if 22.05 broke vlans - I would expect the boards would be a fire with people complaining. Even if was only ix interfaces, have to assume your not the only one using ix and vlans that have moved to 22.05

That's true, that's why I don't know what to say, it's not a no name board. or a realtek driver. But I wonder, what's @stephenw10 conclusion?

johnpoz

@nrgia I'm with you for sure - one of the more strange threads seen in a really long time. @stephenw10 would be for sure my go to guy on anything like this.

stephenw10

It's not generally broken. The 7100 uses ix NICs with VLANs by default and that works fine in 22.05. Also my test here with a 4100 using an individual NIC is not broken.

We know more than just 22.01 worked and 22.05 didn't. The snapshot that first failed was immediately after the patch that went in to fix VLAN0 handling.

The incoming traffic we see in the pcap in 22.05 is tagged VLAN0 and hence is dropped by pfSense/FreeBSD. That is the expected behaviour.

The only thing we don't know is where that tag is coming from.
Experiments seem to show that is isn't coming from the switch (though that still seems most likely) or from the AP.
Unless I've misunderstood the situation the only thing left is the driver or hardware. Neither of which seem at all likely but in the absence of anything else...

stephenw10

What I expect to see if you have any sort of priotiy tagging on a VLAN is something like this:

18:23:28.004183 00:90:7f:b4:38:fa (oui Unknown) > 90:ec:77:1f:8a:5f (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 4, ethertype IPv4, AP300.stevew.lan > 10.101.0.1: ICMP echo request, id 10296, seq 2, length 64
18:23:28.004212 90:ec:77:1f:8a:5f (oui Unknown) > 00:90:7f:b4:38:fa (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 0, ethertype IPv4, 10.101.0.1 > AP300.stevew.lan: ICMP echo reply, id 10296, seq 2, length 64

There I have set egress tagging in OpenWRT to map 0 to 4 so pfSense sees it on the incoming traffic. But there is only one set of tags.

stephenw10

If we can make the switch or AP apply priority tags we could then see which set of tags they appear on in the 22.05 pcap. I expect it to be on the VLAN20 tag. If it's on VLAN0 tag that might prove what's creating it.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

What I expect to see if you have any sort of priotiy tagging on a VLAN is something like this:
18:23:28.004183 00:90:7f:b4:38:fa (oui Unknown) > 90:ec:77:1f:8a:5f (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 4, ethertype IPv4, AP300.stevew.lan > 10.101.0.1: ICMP echo request, id 10296, seq 2, length 64
18:23:28.004212 90:ec:77:1f:8a:5f (oui Unknown) > 00:90:7f:b4:38:fa (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 0, ethertype IPv4, 10.101.0.1 > AP300.stevew.lan: ICMP echo reply, id 10296, seq 2, length 64
There I have set egress tagging in OpenWRT to map 0 to 4 so pfSense sees it on the incoming traffic. But there is only one set of tags.

Just a question, what we except to happen if set some priority to VLANS form pfSense? I know the documentation says to leave that on zero?

I don't know how to provide what you want to see.

NRgia

@stephenw10
From the switch in order to apply priority tags, I must disable the 802.1p mode, and apply per port, for the AP I don't see any option to apply a priority without buying a switch from Ubiquiti. I can provide you some screenshots.

This is how it looks for a VLAN:
https://imgur.com/a/3gBCTIL

johnpoz

@nrgia I have unifi APs, and yeah I just looked I do not see anywhere to set priority on a vlan at all.

Running the lastest firmware on APs 6.2.27 and 7.2.87 controller software. I could try moving to the new UI vs legacy that I use.

NRgia

@johnpoz I'm using the new UI, don't bother. If you don't buy the whole ecosystem you can do so much. But a four eyes check doesn't do any harm. Thank you

NRgia

@nrgia How your networks are configured in the AP, I mean in the Networks tab? Are the VLANS set to VLANs only network? Or how? Thank you

stephenw10

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Just a question, what we except to happen if set some priority to VLANS form pfSense?

Applying it on the VLAN should show in the VLAN tag outbound as expected like:

19:16:32.694149 00:90:7f:b4:38:fa (oui Unknown) > 90:ec:77:1f:8a:5f (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 4, ethertype IPv4, AP300.stevew.lan > 10.101.0.1: ICMP echo request, id 10347, seq 1, length 64
19:16:32.694172 90:ec:77:1f:8a:5f (oui Unknown) > 00:90:7f:b4:38:fa (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 1001, p 3, ethertype IPv4, 10.101.0.1 > AP300.stevew.lan: ICMP echo reply, id 10347, seq 1, length 64

pfSense sets prio 3 and the AP replies prio 4.

Try using port mode on your switch then. If you can set anything that appears in the pcap it might be useful.

Steve

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

ng that appears in the pcap it might be useful.
Steve

First I selected Normal priority on the switch for the pfSense port and the AP port. The switch don't have numbers. Also I select 3 for VLAN 20 on pfSense.

I got this:
tcp_dump_vlan_prioriry_pfsense_and_switch.txt

I lost connection after doing this, I could only connect to devices via their IPs

Then i reset the priority from VLAN 20 from pfSense to zero(0) and I got this:
tcp_dump_vlan_prioriry_on_switch.txt

stephenw10

Those pcaps were on the mirror port again?

In both those pcaps there are no packets from pfSense at all....

Do you still see outbound packets if you tcpdump on ix2 on pfSense in either of those setups?

Steve

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Those pcaps were on the mirror port again?

In both those pcaps there are no packets from pfSense at all....

Do you still see outbound packets if you tcpdump on ix2 on pfSense in either of those setups?

Steve

Ok I redone the tests:
First test:
Steps:

Set priority to 3 in pfSense to VLAN 20
Set Normal Priority in the switch to the port where the AP is connected
Set High priority in the switch to the port where pfSense LAN side is connected.
Done a tcpdump from pfSense.
Results:
tcpdump_priority_set_to pfsense_and_switch.txt

As you can see, only the priority (3) set in pfSense is intercepted.

Second test:

Removed priority 3 set in pfSense to VLAN 20
Left the Normal Priority in the switch to the port where the AP is connected
Left the High priority in the switch to the port where pfSense LAN side is connected.
Done a tcpdump from pfSense.

Results:
tcpdump_priority_set_to_switch.txt

I don't see anu priorities besides (p0) when done from the switch.

I hope the steps are ok, now.

stephenw10

Hmm, OK.

Can I assume traffic was failing during those tests?
We see no ARP replies to pfSense's queries and it looks like one way traffic from the clients.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Hmm, OK.

Can I assume traffic was failing during those tests?
We see no ARP replies to pfSense's queries and it looks like one way traffic from the clients.

Correct, I could communicate only via IP with pfSense. Only that worked.

johnpoz

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Correct, I could communicate only via IP with pfSense. Only that worked.

If you were not getting arp replies then wouldn't work even with IP..

NRgia

@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Correct, I could communicate only via IP with pfSense. Only that worked.

If you were not getting arp replies then wouldn't work even with IP..

If you say so

johnpoz

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

If you say so

Not something making up dude - if you can not arp for the mac of something, it is impossible to talk to it even with an IP.

Mac is what is used to talk to something on a network.. Just not possible if you do not have the mac to talk to.

As Steve mentioned

We see no ARP replies to pfSense's queries and it looks like one way traffic from the clients.

Look at your arp table on your client, arp -a will show you the full table.. If there is no mac for an IP then your not going to talk to it.