pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue

NRgia

So I did like that:

I was connected via ssh to pfSense
Set the priority to 3 from pfSense from GUI
Set the priority from the switch
After that all the hosts lost communication besides pfSense and the host I was running the test
I performed a tcpdump on pfsense LAN interface ix2
I could also connect to pfSense GUI via it's LAN IP, but not with it's domain name.

Other technicalities I don't know how to explain. If the test wasn't good, then this is how I understood to reproduce it, following what Steve has told me.

stephenw10

Right but the LAN interface here is ix2 unatgged, no VLAN right?

The pcap only captured VLAN tagged traffic, the GUEST interface, and it looked like that was failing?

It appeared as though the priority setting in the switch was in fact filtering the traffic in some way rather than applying tags. Which might well be what it's supposed to be doing in that mode.
Are you able to test that on your switch @johnpoz?

johnpoz

@stephenw10 yeah my switch is still up, I could setup some vlans to go through it.

What exactly would you like me to test, doing port based qos? vs the 802.1p

Let me add the vlans to the switch, so I can just connect my AP to it, it has 1 untagged vlan and 3 other tagged vlans. 2 of which are tagged all the way to pfsense, and another that has its own uplink to pfsense that is untagged.

stephenw10

Yeah any test you can do there to see what those QoS settings it has do.

I expected it to apply tags but I have no idea if that's inbound or outbound on the ports.

However it looks more like it's filtering based on the tags from those last pcaps. No ARP replies to queries sent as p 3.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Right but the LAN interface here is ix2 unatgged, no VLAN right?

On the switch it looks like this, port 15, is ix2 and it is a member of all 3:

01 is VLAN1 which is untagged
20 is VLAN 20 tagged
30 is VLAN 30 tagged

The pcap only captured VLAN tagged traffic, the GUEST interface, and it looked like that was failing?

It appeared as though the priority setting in the switch was in fact filtering the traffic in some way rather than applying tags. Which might well be what it's supposed to be doing in that mode.
Are you able to test that on your switch @johnpoz?

I run it like this:

tcpdump -e -i ix2 vlan

Maybe that's why it captured only VLANs traffic. Should I redo the tests ?

stephenw10

No, that's fine we only want to see the VLAN tagged traffic there since that's what is failing.

The traffic on LAN (untagged in pfSense) still worked fine in 22.05.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

No, that's fine we only want to see the VLAN tagged traffic there since that's what is failing.

The traffic on LAN (untagged in pfSense) still worked fine in 22.05.

Yes it worked fine, only VLANs traffic are the problem on 22.05.

johnpoz

@stephenw10 let me find that flexHD ap I have laying around... I can then setup its own ssids so sure connected to it on different vlans.. So I don't have to disrupt my current network at all.

Been meaning to fire it up for updating its firmware and testing it anyway - use to be over at my sons house.. Think its in the garage ;)

stephenw10

I suspect the high priority setting you applied to port 15 in the switch filtered the p 3 tagged traffic coming into it so the clients never saw the ARP requests.

You could repeat it with port 15 set to low and see if ARP replies then come back.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

I suspect the high priority setting you applied to port 15 in the switch filtered the p 3 tagged traffic coming into it so the clients never saw the ARP requests.

You could repeat it with port 15 set to low and see if ARP replies then come back.

Do you want me to set any priorities to other ports like port 5 where the AP gets connected?
Also should I set VLAN 20 priority to 3 in pfSense, or should I work only with the switch now ?

stephenw10

Well you could do either, or both.
But I would do one thing at a time and see what changes.

Of course even after we understand what the switch is doing that doesn't really help us know why it fails in 22.05 since the mirror port didn't show VLAN0 tagged traffic.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Of course even after we understand what the switch is doing that doesn't really help us know why it fails in 22.05 sinc

At least you and @johnpoz are trying to help. I appreciate that.
I will redo the test as you suggested, only that the low setting was the default one.

stephenw10

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

I will redo the test as you suggested, only that the low setting was the default one.

Ah, interesting. Well an alternative might be to leave the switch port 15 set as 'high' and set the prio tag in pfSense to 7. I would expect that to then pass.

NRgia

@stephenw10
First test, only set the low priority only on the switch(the default one), on port 15. I did not touch pfSense.

I got this:
low_priority_on_switch_only.txt

Second test:

Set VLAN20 priority to 3 on pfSense
Set low priority on the switch to port 15

I saw p3 and p0 now.
pfsense_and_switch_set priorities.txt

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

I will redo the test as you suggested, only that the low setting was the default one.

Ah, interesting. Well an alternative might be to leave the switch port 15 set as 'high' and set the prio tag in pfSense to 7. I would expect that to then pass.

This is what I got with VLAN 20 priority set to 7 in pfSense and High priority set to port 15 in the switch.
high_priority.txt

johnpoz

@stephenw10 ok I fired up my new AP, connected to my gs108e on port 8.. Using 802.1p setting for qos and have zero issues connecting to any of the vlans.

I switched it to port mode qos, and still no issues. Now only thing is my gs108 switch is downstream of my cisco sg300, if the switch is adding some odd vlan 0 tag maybe its stripping that?

I have a port on my 4860 I could connect the the netgear switch too, but that would mean creating new vlans on my pfsense as to not disrupt my network, etc. Happy to do so if you feel that could help in anyway.

stephenw10

Maybe you can setup a mirror port and make sure it's seeing all the traffic on the trunk/uplink?

stephenw10

@nrgia Hmm, still no ARP replies shown. I assume with that setting Guest wifi clients fail?

This is slightly odd:

01:31:36.279050 28:6d:97:7f:bb:0c (oui Unknown) > ac:1f:6b:45:fa:8a (oui Unknown), ethertype 802.1Q (0x8100), length 74: vlan 20, p 0, ethertype IPv4, 172.19.15.156.51950 > 192.168.10.1.domain: 54418+ A? gOoGle.cOM. (28)

Why is that client in the 172.19.15.X subnet using he 192.168.10.1 as a server?

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@nrgia Hmm, still no ARP replies shown. I assume with that setting Guest wifi clients fail?

This is slightly odd:
01:31:36.279050 28:6d:97:7f:bb:0c (oui Unknown) > ac:1f:6b:45:fa:8a (oui Unknown), ethertype 802.1Q (0x8100), length 74: vlan 20, p 0, ethertype IPv4, 172.19.15.156.51950 > 192.168.10.1.domain: 54418+ A? gOoGle.cOM. (28)
Why is that client in the 172.19.15.X subnet using he 192.168.10.1 as a server?

That is the smart hub that should've been on 192.168.10.0 subnet, not on 172.18.0.0 subnet

From DHCP leases:

 	172.19.15.156 	28:6d:97:7f:bb:0c (Samjin) 		hubv3-4011027753 		2022/07/06 22:30:47 	2022/07/07 00:30:47 	online 	active 	
	192.168.10.62 	28:6d:97:7f:bb:0c (Samjin) 		192.168.10.62 		2022/07/06 22:08:29 	2022/07/06 22:30:41 	offline 	expired

After I played with what you said, happened the above.
Only after I restart pfSense it gets the proper IP from the proper subnet, after doing the above tests. Otherwise it never happens.

 	172.19.15.156 	28:6d:97:7f:bb:0c (Samjin) 		172.19.15.156 		2022/07/06 22:30:47 	2022/07/06 23:08:28 	offline 	expired 	
	192.168.10.62 	28:6d:97:7f:bb:0c (Samjin) 		hubv3-4011027753 		2022/07/06 23:08:28 	2022/07/07 01:08:28 	online 	active

Please mind the timezone :)

stephenw10

Hmm, I guess the switch removed the tags at some point during the config changes.

That sort of thing is why running tagged and untagged traffic on the same link is better avoided. Though if you were doing that here the LAN would also have failed in 22.05.

Steve