pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue

NRgia

And with tcpdump -e -i ix2.20

]/root: tcpdump -e -i ix2.20
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ix2.20, link-type EN10MB (Ethernet), capture size 262144 bytes
15:45:48.622767 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:46:05.588374 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype IPv4 (0x0800), length 136: 192.168.10.1.mdns > 224.0.0.251.mdns: 0 [2q] PTR (QM)? _%9E5E7C8F47989526C9BCD95D24084F6F0B27C5ED._sub._googlecast._tcp.local. PTR (QM)? _googlecast._tcp.local. (94)
15:46:05.589317 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype IPv4 (0x0800), length 398: 192.168.10.1.mdns > 224.0.0.251.mdns: 0*- [0q] 4/0/0 PTR SHIELD-Android-TV-ee41442d2c14cc09fde82be16f84be32._googlecast._tcp.local., (Cache flush) A 172.18.0.14, (Cache flush) SRV ee41442d-2c14-cc09-fde8-2be16f84be32.local.:8009 0 0, (Cache flush) TXT "id=ee41442d2c14cc09fde82be16f84be32" "cd=3CABD325728E72997BA6735F95651E36" "rm=" "ve=05" "md=SHIELD Android TV" "ic=/setup/icon.png" "fn=SHIELD" "ca=463365" "st=0" "bs=FA8F14F198FB" "nf=1" "rs=" (356)
15:46:05.619590 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:46:06.623982 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:46:08.616921 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:46:18.970438 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype IPv4 (0x0800), length 82: 192.168.10.1.mdns > 224.0.0.251.mdns: 0 PTR (QM)? _googlezone._tcp.local. (40)
15:46:18.970617 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype IPv4 (0x0800), length 119: 192.168.10.1.mdns > 224.0.0.251.mdns: 0 SRV (QM)? ee41442d-2c14-cc09-fde8-2be16f84be32._googlezone._tcp.local. (77)
15:46:18.970973 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype IPv4 (0x0800), length 252: 192.168.10.1.mdns > 224.0.0.251.mdns: 0*- [0q] 4/0/0 PTR ee41442d-2c14-cc09-fde8-2be16f84be32._googlezone._tcp.local., (Cache flush) A 172.18.0.14, (Cache flush) SRV ee41442d-2c14-cc09-fde8-2be16f84be32.local.:10001 1100 0, (Cache flush) TXT "id=3CABD325728E72997BA6735F95651E36" "UDS" (210)
^C
9 packets captured
9 packets received by filter
0 packets dropped by kernel

stephenw10

Sorry I meant: tcpdump -e -i ix2

On the parent interface dircetly

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

tcpdump -e -i ix2

15:49:59.310577 d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, ethertype RRCP (0x8899), length 60: d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, RRCP-0x25 query
15:49:57.147272 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype ARP, Request who-has Sperry.Blueshift tell 169.254.216.169, length 46
15:49:57.248033 00:04:4b:ba:35:05 (oui Unknown) > ac:1f:6b:45:fa:8a (oui Unknown), ethertype IPv4 (0x0800), length 376: Shield.Blueshift.39344 > fra02s19-in-f3.1e100.net.http: Flags [P.], seq 3325661723:3325662033,
ack 288699136, win 685, options [nop,nop,TS val 644858214 ecr 3014227199], length 310: HTTP: HEAD /generate_204 HTTP/1.1
15:49:57.278408 ac:1f:6b:45:fa:8a (oui Unknown) > 00:04:4b:ba:35:05 (oui Unknown), ethertype IPv4 (0x0800), length 149: fra02s19-in-f3.1e100.net.http > Shield.Blueshift.39344: Flags [P.], seq 1:84, ack 310, win 399,
options [nop,nop,TS val 3014287261 ecr 644858214], length 83: HTTP: HTTP/1.1 204 No Content
15:49:57.278930 00:04:4b:ba:35:05 (oui Unknown) > ac:1f:6b:45:fa:8a (oui Unknown), ethertype IPv4 (0x0800), length 66: Shield.Blueshift.39344 > fra02s19-in-f3.1e100.net.http: Flags [.], ack 84, win 685,
options [nop,nop,TS val 644858222 ecr 3014287261], length 0
15:49:57.308952 d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, ethertype RRCP (0x8899), length 60: d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, RRCP-0x25 query
15:49:57.472278 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
15:49:58.187286 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype ARP, Request who-has Sperry.Blueshift tell 169.254.216.169, length 46
15:49:58.297476 08:36:c9:2a:16:e7 (oui Unknown) > Broadcast, ethertype RRCP (0x8899), length 60: 08:36:c9:2a:16:e7 (oui Unknown) > Broadcast, RRCP-0x23 query
15:49:58.309782 d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, ethertype RRCP (0x8899), length 60: d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, RRCP-0x25 query
15:49:58.596361 cc:40:d0:52:32:7d (oui Unknown) > 01:80:c2:00:00:40 (oui Unknown), ethertype Slow Protocols (0x8809), length 60: unknown (136), length 46
        0x0000:  880f 0000 0000 0000 0000 0000 0000 0000
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0020:  0000 0000 0000 0000 0000 0000 0000
15:49:59.310577 d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, ethertype RRCP (0x8899), length 60: d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, RRCP-0x25 query

If it's not enough, tell me what to "grep" for, I dumped it to a file, due to too much lines.

stephenw10

Hmm, ~~so still only outgoing packets~~. At least as far as tcpdump can see.

Are you able to pcap on something upstream to see the tagged traffic that should be arriving there?

NRgia

@stephenw10
Can you give me an example, please.
I don't have Wireshark installed.
I found it, it's in the UI, Packet capture

johnpoz

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20

That seems odd.. why is showing vlan 0 and vlan 20?

What is this guy 28:6d:97:7f:bb:0c, is that pfsense

That isn't outbound from pfsense.. Your other post shows ix2 as ether ac:1f:6b:45:fa:8a

A mac vendor lookup shows it as SAMJIN Co., Ltd.? Never heard of that company.
Seems "The Company provides its products mainly to Samsung Electronics."

stephenw10

@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

That seems odd.. why is showing vlan 0 and vlan 20

Mmm, that is a very good point! Like it's QinQ.

NRgia

@johnpoz If you must know 28:6d:97:7f:bb:0c is a Samsung Smarthings v3 Hub which is on vlan 20 :) It screams for Internet connection, but it doesn't get it :)

johnpoz

@nrgia but showing vlan 0 with a p0? But that is inbound to pfsense.. I don't have a lot of experience with setting priority on vlan 0, etc. But that could be maybe why pfsense not actually seeing the tag 20?

NRgia

@johnpoz
I don't know what to say, but pfSense 22.01 see it just fine.
The native LAN is working just fine, vlan 20 and vlan 30 are dead.

stephenw10

Mmm, can you generate some traffic from pfSense on VLAN 20 and run that again so we can see what outgoing packets look like?

Though I would expect to see some there anyway....

NRgia

@stephenw10
how, if nothing works ?

stephenw10

Try to ping something in the VLAN20 subnet and it will ARP for it.

johnpoz

@nrgia trying pinging some IP from pfsense, it would for sure atleast send arps that would be tagged or should be.

edit: haha jinx :) great minds think a like it seems ;)

stephenw10

Yeah so try running: tcpdump -e -i ix2 vlan

Then try to ping anything in the vlan 20 or 30 subnets from pfSense. You should see at least the ARP traffic and how it's tagged.

stephenw10

Are you running any version of the netgraph vlan0 tagging scripts for your WAN?

NRgia

@stephenw10
So I pinged 192.168.10.56 which is that Smart hub. From pfsense on the native LAN ->VLAN 20

16:22:15.815491 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.56 tell 192.168.10.1, length 28
16:22:15.829472 d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, ethertype RRCP (0x8899), length 60: d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, RRCP-0x25 query
16:22:16.063582 90:e6:ba:31:03:2f (oui Unknown) > 01:00:5e:7f:ff:fa (oui Unknown), ethertype IPv4 (0x0800), length 143: Sperry.Blueshift.63312 > 239.255.255.250.ssdp: UDP, length 101
16:22:16.066237 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype ARP, Request who-has Sperry.Blueshift tell 169.254.129.102, length 46
16:22:16.336837 08:36:c9:2a:16:e7 (oui Unknown) > Broadcast, ethertype RRCP (0x8899), length 60: 08:36:c9:2a:16:e7 (oui Unknown) > Broadcast, RRCP-0x23 reply
16:22:16.507280 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
16:22:16.529972 cc:40:d0:52:32:7d (oui Unknown) > 01:80:c2:00:00:40 (oui Unknown), ethertype Slow Protocols (0x8809), length 60: unknown (136), length 46
        0x0000:  880f 0000 0000 0000 0000 0000 0000 0000
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0020:  0000 0000 0000 0000 0000 0000 0000
16:22:16.823906 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.56 tell 192.168.10.1, length 28
16:22:16.830303 d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, ethertype RRCP (0x8899), length 60: d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, RRCP-0x25 query
16:22:17.105431 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype ARP, Request who-has Sperry.Blueshift tell 169.254.129.102, length 46
16:22:17.213672 80:ee:73:bb:0e:55 (oui Unknown) > ac:1f:6b:45:fa:8a (oui Unknown), ethertype IPv4 (0x0800), length 92: 172.18.0.5.37282 > Entaro.Blueshift.nut: Flags [P.], seq 503429173:503429199, ack 3130284466, win 1027,
options [nop,nop,TS val 1992520480 ecr 754440160], length 26
16:22:17.213730 ac:1f:6b:45:fa:8a (oui Unknown) > 80:ee:73:bb:0e:55 (oui Unknown), ethertype IPv4 (0x0800), length 66: Entaro.Blueshift.nut > 172.18.0.5.37282: Flags [.], ack 26, win 514, options [nop,nop,TS val 754445177
ecr 1992520480], length 0
16:22:17.213804 ac:1f:6b:45:fa:8a (oui Unknown) > 80:ee:73:bb:0e:55 (oui Unknown), ethertype IPv4 (0x0800), length 93: Entaro.Blueshift.nut > 172.18.0.5.37282: Flags [P.], seq 1:28, ack 26, win 514, options [nop,nop,TS val 754445177
ecr 1992520480], length 27
16:22:17.256772 80:ee:73:bb:0e:55 (oui Unknown) > ac:1f:6b:45:fa:8a (oui Unknown), ethertype IPv4 (0x0800), length 66: 172.18.0.5.37282 > Entaro.Blueshift.nut: Flags [.], ack 28, win 1027, options [nop,nop,TS val 1992520523
ecr 754445177], length 0
16:22:17.829565 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.56 tell 192.168.10.1, length 28
16:22:17.831107 d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, ethertype RRCP (0x8899), length 60: d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, RRCP-0x25 query
16:22:18.145452 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype ARP, Request who-has Sperry.Blueshift tell 169.254.129.102, length 46
16:22:18.164338 dc:f5:05:70:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 358: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
Request from dc:f5:05:70:fa:8a (oui Unknown), length 308
16:22:18.336871 08:36:c9:2a:16:e7 (oui Unknown) > Broadcast, ethertype RRCP (0x8899), length 60: 08:36:c9:2a:16:e7 (oui Unknown) > Broadcast, RRCP-0x23 reply
16:22:18.532211 cc:40:d0:52:32:7d (oui Unknown) > 01:80:c2:00:00:40 (oui Unknown), ethertype Slow Protocols (0x8809), length 60: unknown (136), length 46
        0x0000:  880f 0000 0000 0000 0000 0000 0000 0000
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0020:  0000 0000 0000 0000 0000 0000 0000
16:22:18.831961 d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, ethertype RRCP (0x8899), length 60: d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, RRCP-0x25 query
16:22:18.832041 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.56 tell 192.168.10.1, length 28
16:22:18.832237 ac:1f:6b:45:fa:8a (oui Unknown) > 90:e6:ba:31:03:2f (oui Unknown), ethertype IPv4 (0x0800), length 134: Entaro.Blueshift.6675 > Sperry.Blueshift.2463: Flags [P.], seq 177:257, ack 64, win 65535, length 80
16:22:18.873300 90:e6:ba:31:03:2f (oui Unknown) > ac:1f:6b:45:fa:8a (oui Unknown), ethertype IPv4 (0x0800), length 60: Sperry.Blueshift.2463 > Entaro.Blueshift.6675: Flags [.], ack 257, win 63536, length 0
16:22:19.832759 d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, ethertype RRCP (0x8899), length 60: d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, RRCP-0x25 query
16:22:19.844918 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.56 tell 192.168.10.1, length 28
16:22:19.845097 ac:1f:6b:45:fa:8a (oui Unknown) > 90:e6:ba:31:03:2f (oui Unknown), ethertype IPv4 (0x0800), length 134: Entaro.Blueshift.6675 > Sperry.Blueshift.2463: Flags [P.], seq 257:337, ack 64, win 65535, length 80
16:22:19.885283 90:e6:ba:31:03:2f (oui Unknown) > ac:1f:6b:45:fa:8a (oui Unknown), ethertype IPv4 (0x0800), length 60: Sperry.Blueshift.2463 > Entaro.Blueshift.6675: Flags [.], ack 337, win 63456, length 0
16:22:20.106267 90:e6:ba:31:03:2f (oui Unknown) > 01:00:5e:7f:ff:fa (oui Unknown), ethertype IPv4 (0x0800), length 143: Sperry.Blueshift.63312 > 239.255.255.250.ssdp: UDP, length 101
16:22:20.108822 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype ARP, Request who-has Sperry.Blueshift tell 169.254.129.102, length 46
16:22:20.161707 80:ee:73:bb:0e:55 (oui Unknown) > ac:1f:6b:45:fa:8a (oui Unknown), ethertype IPv4 (0x0800), length 82: 172.18.0.5.51157 > Entaro.Blueshift.nut: Flags [P.], seq 3582196994:3582197010, ack 3023361596, win 1027,
options [nop,nop,TS val 3498492293 ecr 4079702838], length 16
16:22:20.161765 ac:1f:6b:45:fa:8a (oui Unknown) > 80:ee:73:bb:0e:55 (oui Unknown), ethertype IPv4 (0x0800), length 66: Entaro.Blueshift.nut > 172.18.0.5.51157: Flags [.], ack 16, win 514, options [nop,nop,TS val 4079712813
ecr 3498492293], length 0

172.18.0.0 is the native LAN

the dump is on ix2

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

netgraph

I saw "netgraph"in that defect. So If I don't know what it means, I think I don't use it.

stephenw10

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

16:22:16.823906 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 20, p 0, ethertype ARP, Request who-has 192.168.10.56 tell 192.168.10.1, length 28

Ok, so we see pfSense sending traffic as expected. But no response. Presumably because that device is behind some sort of odd double tagging somehow.

Can you try pinging something else in that subnet?

NRgia

@stephenw10 how do you want the dump to be, on ix2(Native) or ix2.20(VLAN)?