pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue
-
@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:
Yeah so try running:
tcpdump -e -i ix2 vlanThen try to ping anything in the vlan 20 or 30 subnets from pfSense. You should see at least the ARP traffic and how it's tagged.
Ok did what you asked:
pinged another 192.168.10.58
tcpdump -e -i ix2 vlan tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ix2, link-type EN10MB (Ethernet), capture size 262144 bytes 16:44:07.012079 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, et hertype 802.1Q, vlan 20, p 0, ethertype ARP, Request who-has Sperry.Blueshift tell 169.254.231.56, length 46 16:44:07.089429 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, e thertype ARP, Request who-has 192.168.10.58 tell 192.168.10.1, length 28 16:44:07.531974 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, e thertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 2 8:6d:97:7f:bb:0c (oui Unknown), length 548 16:44:07.751772 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, e thertype ARP, Request who-has 192.168.10.60 tell 192.168.10.1, length 28 16:44:08.090969 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, et hertype 802.1Q, vlan 20, p 0, ethertype ARP, Request who-has Sperry.Blueshift tell 169.254.231.56, length 46 16:44:08.116574 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, e thertype ARP, Request who-has 192.168.10.58 tell 192.168.10.1, length 28 16:44:09.121573 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.58 tell 192.168.10.1, length 28 16:44:09.131032 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype ARP, Request who-has Sperry.Blueshift tell 169.254.231.56, length 46 16:44:09.191236 cc:f4:11:c5:bc:81 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 350: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from cc:f4:11:c5:bc:81 (oui Unknown), length 300 16:44:10.048919 cc:f4:11:c5:bc:81 (oui Unknown) > 33:33:00:00:0c:0c (oui Unknown), ethertype 802.1Q (0x8100), length 108: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv6, fe80::cef4:11ff:fec5:bc81.10101 > ff05::c0c.10101: UDP, length 38 16:44:10.048986 cc:f4:11:c5:bc:81 (oui Unknown) > 33:33:00:0c:00:0c (oui Unknown), ethertype 802.1Q (0x8100), length 108: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv6, fe80::cef4:11ff:fec5:bc81.10101 > ff02::c:c.10101: UDP, length 38 16:44:10.166995 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.58 tell 192.168.10.1, length 28 16:44:10.532377 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548 16:44:11.180544 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.58 tell 192.168.10.1, length 28 16:44:12.204575 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.58 tell 192.168.10.1, length 28 16:44:13.037319 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype ARP, Request who-has Sperry.Blueshift tell 169.254.231.56, length 46 16:44:13.194665 dc:f5:05:3d:18:2d (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 358: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from dc:f5:05:3d:18:2d (oui Unknown), length 308 16:44:13.220498 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.58 tell 192.168.10.1, length 28 16:44:14.091153 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype ARP, Request who-has Sperry.Blueshift tell 169.254.231.56, length 46 16:44:14.222187 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.58 tell 192.168.10.1, length 28 16:44:14.532642 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548 16:44:15.131325 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype ARP, Request who-has Sperry.Blueshift tell 169.254.231.56, length 46 16:44:15.229386 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.58 tell 192.168.10.1, length 28 16:44:16.253933 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.58 tell 192.168.10.1, length 28 16:44:17.074123 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype ARP, Request who-has Sperry.Blueshift tell 169.254.231.56, length 46 16:44:17.285564 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.58 tell 192.168.10.1, length 28 16:44:17.872307 dc:f5:05:70:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 358: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from dc:f5:05:70:fa:8a (oui Unknown), length 308 16:44:18.091162 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype ARP, Request who-has Sperry.Blueshift tell 169.254.231.56, length 46 16:44:18.288921 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.58 tell 192.168.10.1, length 28 16:44:18.532216 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548 16:44:19.131218 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype ARP, Request who-has Sperry.Blueshift tell 169.254.231.56, length 46 16:44:19.290397 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.58 tell 192.168.10.1, length 28 16:44:20.314943 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.58 tell 192.168.10.1, length 28 16:44:20.723599 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype 802.1Q (0x8100), length 140: vlan 20, p 0, ethertype IPv4, 192.168.10.1.mdns > 224.0.0.251.mdns: 0 [2q] PTR (QM)? _%9E5E7C8F47989526C9BCD95D24084F6F0B27C5ED._sub._googlecast._tcp.local. PTR (QM)? _googlecast._tcp.local. (94) 16:44:20.724545 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype 802.1Q (0x8100), length 402: vlan 20, p 0, ethertype IPv4, 192.168.10.1.mdns > 224.0.0.251.mdns: 0*- [0q] 4/0/0 PTR SHIELD-Android-TV-ee41442d2c14cc09fde82be16f84be32._googlecast._tcp.local., (Cache flush) A 172.18.0.14, (Cache flush) SRV ee41442d-2c14-cc09-fde8-2be16f84be32.local.:8009 0 0, (Cache flush) TXT "id=ee41442d2c14cc09fde82be16f84be32" "cd=3CABD325728E72997BA6735F95651E36" "rm=" "ve=05" "md=SHIELD Android TV" "ic=/setup/icon.png" "fn=SHIELD" "ca=463365" "st=0" "bs=FA8F14F198FB" "nf=1" "rs=" (356) 16:44:20.744088 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.60 tell 192.168.10.1, length 28 16:44:21.378435 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.58 tell 192.168.10.1, length 28 ^C 37 packets captured 6046 packets received by filter 0 packets dropped by kernel -
The native (parent) NIC. We need to see the tagged traffic in the pcap and capturing on the VLAN removes that.
Using
tcpdump -e -i ix2 vlanwill show only VLAN tagged traffic which will make things easier to read. -
Hmm, so that clarifies what we saw before. The only incoming traffic is strangely double tagged.
But that does prove we can see incoming tagged traffic. Which implies nothing else is sending VLAN 20 tagged replies.
Do devices at .58 and .60 actually exist? Can you check for the incoming ARP requests there?
There has been some driver changes in ix including that specific fix for VLAN0 traffic but that was before 22.01 not between that and 22.05....
-
@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:
Hmm, so that clarifies what we saw before. The only incoming traffic is strangely double tagged.
But that does prove we can see incoming tagged traffic. Which implies nothing else is sending VLAN 20 tagged replies.
Do devices at .58 and .60 actually exist? Can you check for the incoming ARP requests there?
There has been some driver changes in ix including that specific fix for VLAN0 traffic but that was before 22.01 not between that and 22.05....
This is what I have on DHCP leases screen:
They exist but they are offline
Only IOT devices are on these VLANS, most of them are Wireless, the only one wired is 192.168.10.58 and connected to a specific port.
The last pfSense that worked(besides 22.01), to have timeline, was pfSense-CE-memstick-2.7.0-DEVELOPMENT-amd64-20220314-1916. After that all snapshots behaved like it is now.
-
@stephenw10 Can we try something else...I don't try to be smart or anything, but how do I set those 2 VLANS to have PROMISC tag? I know it works for you. But maybe we can try
-
@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:
VLANS to have PROMISC tag?
just set it with ifconfig
here
-
@johnpoz
Yep figured it out from here https://www.freebsd.org/cgi/man.cgi?ifconfigAs you said, it does not work but I had to try. Don't get my wrong, thank you both for what you did until now. I'm just desperate. :)
I'm open to ideas, if you still have any.
Would it be useful to compare with a dump from 22.01, to see how vlans are handled there in my case? Should I revert back for now? Or should we try something else?
-
I would get a laptop on an access port on one of those VLANs so you can see what's happening at that end.
It's possible something got fixed and that has broken your setup because the double tagged traffic is now dropped as expected.
We don't see double tagged ARP replies from those devices though so it's unclear if they ever see the requests.What is handling the VLANs in between? How is it configured?
Steve
-
@nrgia if I had to "guess" something changed in the driver with that 0 tag.. that is a "guess".. @stephenw10 might have some other stuff to try/look at.
-
We know that something did change: https://github.com/pfsense/FreeBSD-src/commit/9c762cc125c0c2dae9fbf49cc526bb97c14b54a4
Hence my suggestion that it could be the VLAN0 tags were being incorrectly passed in 22.01 and before that 'fix'.
There shouldn't be any VLAN0 tagging happening here though.
-
@stephenw10
A Netgear switch....could it be that for some switches that VLAN 1 to be untagged, and for others VLAN0. Could it be that pfsense 22.01 had VLAN1 for untagged and now pfsense 22.05 have VLAN 0 for untagged ? Or I speak nonsense ? -
Anything is possible! You should never use VLAN1 (or 0 IMO) because some switches do weird things with that.
That double tagged traffic looks wrong and it's hard to see how pfSense could be causing it.Can we see the switch config? Does it have any QinQ or Priority tagging options?
I would have to guess that something in incorrectly applying priority tags to already VLAN tagged traffic.
-
@stephenw10 isn't there a whole other thread going on for quite some time about vlan 0? I have just stayed out of that one - no experience ever using a vlan 0.
But vlan 1 should pretty much never be tagged. That is just the default vlan ID uses use for their default untagged default network.
But yeah that double tag thing doesn't look right to me.
edit: Just me and my ocd I think - but why would you tag 192.168.10 with an ID of 20 and 192.168.20 with 30?, wouldn't 10 be better ;) just so you know hey 192.168.10 - that is vlan 10, I do that with my vlan 4 and 6, they are 192.168.4 and 192.168.6 networks ;)
-
@stephenw10
You can, sure, my native VLAN is untagged with vlan 1 on the switch. It worked before so I did not bother. -
@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:
@stephenw10 isn't there a whole other thread going on for quite some time about vlan 0? I have just stayed out of that one - no experience ever using a vlan 0.
But vlan 1 should pretty much never be tagged. That is just the default vlan ID uses use for their default untagged default network.
But yeah that double tag thing doesn't look right to me.
edit: Just me and my ocd I think - but why would you tag 192.168.10 with an ID of 20, wouldn't 10 be better ;) just so you know hey 192.168.10 - that is vlan 10, I do that with my vlan 4 and 6, they are 192.168.4 and 192.168.6 networks ;)
Watched Tom Lawrence once, and he tagged them that way :) If it matters I can rename them, if you think it matters
VLAN 1 is not tagged in my case. It's only in the switch. All the ports on Group VLAN 1 are untagged.
-
@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:
if you think it matters
no doesn't matter - just odd, it is common practice to use an ID that somehow relates to the IP range is all.. But the vlan ID has zero to do with the IP space used on the vlan..
-
@johnpoz I will rename them, I know it's not logic to follow when debugging.
On your primary switch what do you have for native 1 or 0 ? -
Can we see the other VLAN config tabs? What is that switch? What firmware version?
But I would still get a laptop on to it and take some pcaps there to see what's happening.
Steve
-
@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:
primary switch what do you have for native 1 or 0 ?
My default is 9 ;) common practice to move away from 1 in the enterprise. Have never seen 0 to be honest.. Its more of a special use ID, have never ever seen 0 used on a switch as the default vlan. Every switch that I can remember has always been 1 as the default vlan.
Notice - doesn't allow you to set 0, its 1-4094
-
@stephenw10
The Model is GS116Ev2 firmware version 2.6.0.48VLAN1
https://imgur.com/Js7iYjcVLAN20
https://imgur.com/keYmhMBVLAN30
https://imgur.com/gW0qBhc
