pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue

stephenw10

I suspect the high priority setting you applied to port 15 in the switch filtered the p 3 tagged traffic coming into it so the clients never saw the ARP requests.

You could repeat it with port 15 set to low and see if ARP replies then come back.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

I suspect the high priority setting you applied to port 15 in the switch filtered the p 3 tagged traffic coming into it so the clients never saw the ARP requests.

You could repeat it with port 15 set to low and see if ARP replies then come back.

Do you want me to set any priorities to other ports like port 5 where the AP gets connected?
Also should I set VLAN 20 priority to 3 in pfSense, or should I work only with the switch now ?

stephenw10

Well you could do either, or both.
But I would do one thing at a time and see what changes.

Of course even after we understand what the switch is doing that doesn't really help us know why it fails in 22.05 since the mirror port didn't show VLAN0 tagged traffic.

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Of course even after we understand what the switch is doing that doesn't really help us know why it fails in 22.05 sinc

At least you and @johnpoz are trying to help. I appreciate that.
I will redo the test as you suggested, only that the low setting was the default one.

stephenw10

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

I will redo the test as you suggested, only that the low setting was the default one.

Ah, interesting. Well an alternative might be to leave the switch port 15 set as 'high' and set the prio tag in pfSense to 7. I would expect that to then pass.

NRgia

@stephenw10
First test, only set the low priority only on the switch(the default one), on port 15. I did not touch pfSense.

I got this:
low_priority_on_switch_only.txt

Second test:

Set VLAN20 priority to 3 on pfSense
Set low priority on the switch to port 15

I saw p3 and p0 now.
pfsense_and_switch_set priorities.txt

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

I will redo the test as you suggested, only that the low setting was the default one.

Ah, interesting. Well an alternative might be to leave the switch port 15 set as 'high' and set the prio tag in pfSense to 7. I would expect that to then pass.

This is what I got with VLAN 20 priority set to 7 in pfSense and High priority set to port 15 in the switch.
high_priority.txt

johnpoz

@stephenw10 ok I fired up my new AP, connected to my gs108e on port 8.. Using 802.1p setting for qos and have zero issues connecting to any of the vlans.

I switched it to port mode qos, and still no issues. Now only thing is my gs108 switch is downstream of my cisco sg300, if the switch is adding some odd vlan 0 tag maybe its stripping that?

I have a port on my 4860 I could connect the the netgear switch too, but that would mean creating new vlans on my pfsense as to not disrupt my network, etc. Happy to do so if you feel that could help in anyway.

stephenw10

Maybe you can setup a mirror port and make sure it's seeing all the traffic on the trunk/uplink?

stephenw10

@nrgia Hmm, still no ARP replies shown. I assume with that setting Guest wifi clients fail?

This is slightly odd:

01:31:36.279050 28:6d:97:7f:bb:0c (oui Unknown) > ac:1f:6b:45:fa:8a (oui Unknown), ethertype 802.1Q (0x8100), length 74: vlan 20, p 0, ethertype IPv4, 172.19.15.156.51950 > 192.168.10.1.domain: 54418+ A? gOoGle.cOM. (28)

Why is that client in the 172.19.15.X subnet using he 192.168.10.1 as a server?

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@nrgia Hmm, still no ARP replies shown. I assume with that setting Guest wifi clients fail?

This is slightly odd:
01:31:36.279050 28:6d:97:7f:bb:0c (oui Unknown) > ac:1f:6b:45:fa:8a (oui Unknown), ethertype 802.1Q (0x8100), length 74: vlan 20, p 0, ethertype IPv4, 172.19.15.156.51950 > 192.168.10.1.domain: 54418+ A? gOoGle.cOM. (28)
Why is that client in the 172.19.15.X subnet using he 192.168.10.1 as a server?

That is the smart hub that should've been on 192.168.10.0 subnet, not on 172.18.0.0 subnet

From DHCP leases:

 	172.19.15.156 	28:6d:97:7f:bb:0c (Samjin) 		hubv3-4011027753 		2022/07/06 22:30:47 	2022/07/07 00:30:47 	online 	active 	
	192.168.10.62 	28:6d:97:7f:bb:0c (Samjin) 		192.168.10.62 		2022/07/06 22:08:29 	2022/07/06 22:30:41 	offline 	expired

After I played with what you said, happened the above.
Only after I restart pfSense it gets the proper IP from the proper subnet, after doing the above tests. Otherwise it never happens.

 	172.19.15.156 	28:6d:97:7f:bb:0c (Samjin) 		172.19.15.156 		2022/07/06 22:30:47 	2022/07/06 23:08:28 	offline 	expired 	
	192.168.10.62 	28:6d:97:7f:bb:0c (Samjin) 		hubv3-4011027753 		2022/07/06 23:08:28 	2022/07/07 01:08:28 	online 	active

Please mind the timezone :)

stephenw10

Hmm, I guess the switch removed the tags at some point during the config changes.

That sort of thing is why running tagged and untagged traffic on the same link is better avoided. Though if you were doing that here the LAN would also have failed in 22.05.

Steve

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

Hmm, I guess the switch removed the tags at some point during the config changes.

That sort of thing is why running tagged and untagged traffic on the same link is better avoided. Though if you were doing that here the LAN would also have failed in 22.05.

Steve

So due to the fact that for the same port I have tagged and untagged, maybe the tags for the VLANS get stripped ? And only the traffic to VLAN 0(1) remains?

I my case for a home network and all., I used the default VLAN 1 for management which is untagged. So for port 15 which is where pfSense LAN side connects, it has traffic Untagged from VLAN 1, Tagged from VLAN 20, and Tagged from VLAN 30.

I pfsense I have VLAN 20, VLAN 30, and the management LAN is not part of any VLAN. I am writting this, maybe you find some mistake from my part.

So maybe not beeing part of any VLAN , the traffic is transmitted to VLAN 0 , because it doesn't know to which VLAN to send the traffic to and it sends it to Native VLAN 0 ?

johnpoz

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

So maybe not beeing part of any VLAN , the traffic is transmitted to VLAN 0

That shouldn't ever be able to happen.. I have never seen a switch where there wasn't a pvid on an interface. So if untagged the traffic would be on whatever the pvid is, which normally would be default vlan 1 of the switch.

NRgia

@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

So maybe not beeing part of any VLAN , the traffic is transmitted to VLAN 0

That shouldn't ever be able to happen.. I have never seen a switch where there wasn't a pvid on an interface. So if untagged the traffic would be on whatever the pvid is, which normally would be default vlan 1 of the switch.

So a PVID is set by default to 1 if it's not set to anything else, ok then what is vlan 0? As I read out, some say VLAN 1 is the default, others that VLAN 0 is the default. As you saw my switch can only tag from 1 above...then I'm trying to understand which device in the network sets VLAN 0 tag?

johnpoz

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

then what is vlan 0

That is a special use vlan - it is not commonly used.. Its used normally to set priority when the actual vlan is not known, or whatever can not set the priority on the actual vlan.

It is a special use.. when a switch sees a vlan 0, it should set the priority of the frame to whatever the priority is on on vlan 0 to whatever the default vlan/native vlan for that port is, ie the pvid.. Which normally is 1 on any switch, unless it has been changed by the operator..

Here is what I can tell you how uncommon it is in the normal enterprise - I have never in 30 some years working in the biz, ever had need/want to set that on any sort of switches or routers, and have worked with lots and lots of them over the years. Nor have I ever seen it in the field on any pcaps, or any pcaps sent to me from multiple customer and locations - and I work for a major player, and have gotten pcaps for things they want help on from really all over the globe. Vlan 0 has never been part of any discussion or troubleshooting have ever been involved in. Now is it possible it was there and pcaps sent to me didn't have it - ok sure. But I have to say its not a very common used thing in my personal and professional opinion.

That your seeing them it is odd for sure..

NRgia

@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

then what is vlan 0

That is a special use vlan - it is not commonly used.. Its used normally to set priority when the actual vlan is not known, or whatever can not set the priority on the actual vlan.

It is a special use.. when a switch sees a vlan 0, it should set the priority of the frame to whatever the priority is on on vlan 0 to whatever the default vlan/native vlan for that port is, ie the pvid.. Which normally is 1 on any switch, unless it has been changed by the operator..

Here is what I can tell you how uncommon it is in the normal enterprise - I have never in 30 some years working in the biz, ever had need/want to set that on any sort of switches or routers, and have worked with lots and lots of them over the years. Nor have I ever seen it in the field on any pcaps, or any pcaps sent to me from multiple customer and locations - and I work for a major player, and have gotten pcaps for things they want help on from really all over the globe. Vlan 0 has never been part of any discussion or troubleshooting have ever been involved in. Now is it possible it was there and pcaps sent to me didn't have it - ok sure. But I have to say its not a very common used thing in my personal and professional opinion.

That your seeing them it is odd for sure..

pfff...As deeper we go in, the more weirder it becomes.
The point is, if it was a package, like Suricata, or something, I could live without...but this is something that prevents me, to use pfSense. Sure, for a few months 22.01 is sound, but after those...

I mean you guys are way more experienced than me, and ran out of ideas...
Can I at least ask both of you @johnpoz and @stephenw10 , to just drop me a message, if you stumble upon the same issue on other users posts, and you find a solution.
If you please @stephenw10 and find out, that something changed upstream in FreeBSD, can you let me know to test again?

Thank you guys for your time, I will not dare to keep more on this. If you have any news, ideas please let me know.
Thank you again.

stephenw10

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

So due to the fact that for the same port I have tagged and untagged, maybe the tags for the VLANS get stripped ? And only the traffic to VLAN 0(1) remains?

I was suggesting that what might have happened to get that client a dhcp lease on LAN is that while making changes to the QoS settings at some point the switch removed the VLAN tags from the port long enough for the DHCP sequence to complete.
Where as if you had your LAN assigned as ix2.10, for example, untagged traffic arriving at pfSense would simply be dropped.

That's a completely separate issue to the inexplicable VLAN 0 tags we see in the pcaps though.
And I agree VLAN 0 (priority) tagged traffic is rare. I've only seen it in DHCP traffic arriving from an ISP.

Steve

stephenw10

@johnpoz were you able to get a mirror port working that could see all the tagged traffic on your Netgear switch?

@NRgia Do you have any other interfaces you can use on that box? Or could you add any?

Steve

johnpoz

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

get a mirror port working that could see all the tagged traffic on your Netgear switch?

No I haven't I got tied up with real work stuff.. And have to look into being able to sniff vlan traffic on windows machine before hand anyway.. Or might be easier to just mirror the traffic to a spare interface on pfsense ;) That will be much easier I think.