pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue

johnpoz

@nrgia Steve would be my go to guy on something like this - if he doesn't know about an issue, he will for sure have some stuff to get more info with..

So your seeing the tags hit pfsense? But pfsense just not answering them?

So for example if I do a sniff with -e on from command line with tcpdump - I see traffic with the tag..

When you sniff on the vlan interface via packet capture under diagnostic menu - you see no traffic?

NRgia

@johnpoz unfortunatelly for the moment I reverted back to pfSense 22.01. If you are kind enough to help me investigate, I can upgrade tomorrow and investigate together, or provide the information you asked for.

I can do it via VPN, but I'm afraid that I will lose connection, after that.

Can I ping @stephenw10 directly on private, about this? What is the best practice?
Thanks

johnpoz

@nrgia I am normally around all day, really all the time ;) Unless I get caught up in a real work thing - Fridays normally pretty light ;) Got a couple meetings on the cal, but cross fingers it will be a light day..

When it comes to hardware related stuff, or stuff that is most likely driver or hardware sort of stuff @stephenw10 is the guy! Now if I could duplicate the problem here I would be all in ;) But have not seen this issue and don't have anything that has that interface to play with even.

But always up for helping figure out a problem..

Lot of smart people around here - sure we can figure out what is going on.. Problem is might not be something that people get involved in - especially if they are not seeing the problem or don't have the hardware to try and duplicate. Steve seems to have everything ever made in his lab ;) heheh

The VLAN_HWTAGGING is a thing I would be interesting in looking into, I saw some stuff on the freebsd mailing list about that actually and the promisc setting.. You could put your interface into promisc mode and see if that corrects the problem - but that shouldn't need to be set as you see on my setup.. But it might be a work around for a driver issue?

I should be around trmw though - and happy to help if I can.. But hardware issues not really my "thing" if you will.. Especially if I don't have the hardware to duplicate it with.

stephenw10

I can definitely look into this. I'm not aware of any issue with ix and vlans though.

In fact I have one here and it's working fine:

[22.05-RELEASE][admin@4100.stevew.lan]/root: ifconfig ix3.1001
ix3.1001: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: Guest
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 90:ec:77:1f:8a:5f
	inet6 fe80::92ec:77ff:fe1f:8a5f%ix3.1001 prefixlen 64 scopeid 0x10
	inet 10.101.0.1 netmask 0xffffff00 broadcast 10.101.0.255
	groups: vlan
	vlan: 1001 vlanpcp: 0 parent interface: ix3
	media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[22.05-RELEASE][admin@4100.stevew.lan]/root: tcpdump -i ix3.1001
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ix3.1001, link-type EN10MB (Ethernet), capture size 262144 bytes
01:55:37.636598 IP AP300.stevew.lan.55659 > ntp2.icolo.io.ntp: NTPv4, Client, length 48
01:55:37.773762 IP ntp2.icolo.io.ntp > AP300.stevew.lan.55659: NTPv4, Server, length 48
01:55:41.643175 ARP, Request who-has 10.101.0.1 tell AP300.stevew.lan, length 42
01:55:41.643193 ARP, Reply 10.101.0.1 is-at 90:ec:77:1f:8a:5f (oui Unknown), length 28

Steve

johnpoz

@stephenw10 and no promisc set either ;)

stephenw10

Indeed, and it shouldn't need to be.

Something hardware specific perhaps? That's the SoC NIC in a 4100.

Steve

NRgia

@stephenw10 @johnpoz
I appreciate your time guys.

This is the board if it helps
https://www.supermicro.com/en/products/motherboard/A2SDi-4C-HLN4F

Should I provide a TCP dump on vlan interface or on the native interface ?
I mean in my case ix2 or ix.20 ? Or both ?
Thank you

stephenw10

Look on the VLAN. If there's nothing there look on the parent for the tagged traffic. Or just do both anyway!

If you're doing that in the pfSense GUI be aware that you cannot (currently) apply filters when looking for tagged traffic. https://redmine.pfsense.org/issues/13094

NRgia

@stephenw10 I will use your cli command

tcpdump -i

I see it spits a lot of info, I will try to paste relevant info like in your example.

NRgia

@stephenw10

This all I have:


listening on ix2.20, link-type EN10MB (Ethernet), capture size 262144 bytes
15:22:07.873050 ARP, Request who-has 192.168.10.56 tell 192.168.10.1, length 28
15:22:10.070379 IP 192.168.10.1.mdns > 224.0.0.251.mdns: 0 [2q] PTR (QM)? _%9E5E7C8F47989526C9BCD95D24084F6F0B27C5ED._sub._googlecast._tcp.local. PTR (QM)? _googlecast._tcp.local. (94)
15:22:10.071252 IP 192.168.10.1.mdns > 224.0.0.251.mdns: 0*- [0q] 4/0/0 PTR SHIELD-Android-TV-ee41442d2c14cc09fde82be16f84be32._googlecast._tcp.local., (Cache flush) A 172.18.0.14, (Cache flush) SRV ee41442d-2c14-cc09-fde8-2be16f84be32.local.:8009 0 0, (Cache flush) TXT "id=ee41442d2c14cc09fde82be16f84be32" "cd=3CABD325728E72997BA6735F95651E36" "rm=" "ve=05" "md=SHIELD Android TV" "ic=/setup/icon.png" "fn=SHIELD" "ca=463365" "st=0" "bs=FA8F14F198FB" "nf=1" "rs=" (356)
15:22:10.096988 ARP, Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:22:11.093740 ARP, Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:22:13.091847 ARP, Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:22:30.072687 IP 192.168.10.1.mdns > 224.0.0.251.mdns: 0 [2q] PTR (QM)? _%9E5E7C8F47989526C9BCD95D24084F6F0B27C5ED._sub._googlecast._tcp.local. PTR (QM)? _googlecast._tcp.local. (94)
15:22:30.073588 IP 192.168.10.1.mdns > 224.0.0.251.mdns: 0*- [0q] 4/0/0 PTR SHIELD-Android-TV-ee41442d2c14cc09fde82be16f84be32._googlecast._tcp.local., (Cache flush) A 172.18.0.14, (Cache flush) SRV ee41442d-2c14-cc09-fde8-2be16f84be32.local.:8009 0 0, (Cache flush) TXT "id=ee41442d2c14cc09fde82be16f84be32" "cd=3CABD325728E72997BA6735F95651E36" "rm=" "ve=05" "md=SHIELD Android TV" "ic=/setup/icon.png" "fn=SHIELD" "ca=463365" "st=0" "bs=FA8F14F198FB" "nf=1" "rs=" (356)
15:22:30.093361 ARP, Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:22:31.091257 ARP, Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:22:33.095748 ARP, Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:22:37.097207 ARP, Request who-has 192.168.10.60 tell 192.168.10.1, length 28

johnpoz

@nrgia make sure you add -e on tcpdump or it won't spit out vlan tag info

but isn't that 192.168.10 your vlan 20?

stephenw10

Yeah, so that looks expected for a dump inside the VLAN. Except there's only outbound traffic.

So run tcpdump -e -i ix2.20 and see if the tagged traffic is arriving.

NRgia

@johnpoz @stephenw10

Yes 192.168.10.1 is vlan 20

Here you go:

]/root: tcpdump -i ix2.20 -e
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ix2.20, link-type EN10MB (Ethernet), capture size 262144 bytes
15:40:03.507991 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:40:03.518434 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype IPv4 (0x0800), length 136: 192.168.10.1.mdns > 224.0.0.251.mdns: 0 [2q] PTR (QM)? _%9E5E7C8F47989526C9BCD95D24084F6F0B27C5ED._sub._googlecast._tcp.local. PTR (QM)? _googlecast._tcp.local. (94)
15:40:04.505932 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:40:04.508145 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype IPv4 (0x0800), length 136: 192.168.10.1.mdns > 224.0.0.251.mdns: 0 [2q] PTR (QM)? _%9E5E7C8F47989526C9BCD95D24084F6F0B27C5ED._sub._googlecast._tcp.local. PTR (QM)? _googlecast._tcp.local. (94)
15:40:04.509221 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype IPv4 (0x0800), length 398: 192.168.10.1.mdns > 224.0.0.251.mdns: 0*- [0q] 4/0/0 PTR SHIELD-Android-TV-ee41442d2c14cc09fde82be16f84be32._googlecast._tcp.local., (Cache flush) A 172.18.0.14, (Cache flush) SRV ee41442d-2c14-cc09-fde8-2be16f84be32.local.:8009 0 0, (Cache flush) TXT "id=ee41442d2c14cc09fde82be16f84be32" "cd=3CABD325728E72997BA6735F95651E36" "rm=" "ve=05" "md=SHIELD Android TV" "ic=/setup/icon.png" "fn=SHIELD" "ca=463365" "st=0" "bs=FA8F14F198FB" "nf=1" "rs=" (356)
15:40:05.510287 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype IPv4 (0x0800), length 136: 192.168.10.1.mdns > 224.0.0.251.mdns: 0 [2q] PTR (QM)? _%9E5E7C8F47989526C9BCD95D24084F6F0B27C5ED._sub._googlecast._tcp.local. PTR (QM)? _googlecast._tcp.local. (94)
15:40:05.511240 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype IPv4 (0x0800), length 398: 192.168.10.1.mdns > 224.0.0.251.mdns: 0*- [0q] 4/0/0 PTR SHIELD-Android-TV-ee41442d2c14cc09fde82be16f84be32._googlecast._tcp.local., (Cache flush) A 172.18.0.14, (Cache flush) SRV ee41442d-2c14-cc09-fde8-2be16f84be32.local.:8009 0 0, (Cache flush) TXT "id=ee41442d2c14cc09fde82be16f84be32" "cd=3CABD325728E72997BA6735F95651E36" "rm=" "ve=05" "md=SHIELD Android TV" "ic=/setup/icon.png" "fn=SHIELD" "ca=463365" "st=0" "bs=FA8F14F198FB" "nf=1" "rs=" (356)
15:40:05.526062 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:40:06.506694 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:40:07.530222 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:40:10.516671 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.60 tell 192.168.10.1, length 28
^C
11 packets captured
11 packets received by filter
0 packets dropped by kernel

NRgia

@stephenw10

And with tcpdump -e -i ix2.20

]/root: tcpdump -e -i ix2.20
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ix2.20, link-type EN10MB (Ethernet), capture size 262144 bytes
15:45:48.622767 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:46:05.588374 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype IPv4 (0x0800), length 136: 192.168.10.1.mdns > 224.0.0.251.mdns: 0 [2q] PTR (QM)? _%9E5E7C8F47989526C9BCD95D24084F6F0B27C5ED._sub._googlecast._tcp.local. PTR (QM)? _googlecast._tcp.local. (94)
15:46:05.589317 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype IPv4 (0x0800), length 398: 192.168.10.1.mdns > 224.0.0.251.mdns: 0*- [0q] 4/0/0 PTR SHIELD-Android-TV-ee41442d2c14cc09fde82be16f84be32._googlecast._tcp.local., (Cache flush) A 172.18.0.14, (Cache flush) SRV ee41442d-2c14-cc09-fde8-2be16f84be32.local.:8009 0 0, (Cache flush) TXT "id=ee41442d2c14cc09fde82be16f84be32" "cd=3CABD325728E72997BA6735F95651E36" "rm=" "ve=05" "md=SHIELD Android TV" "ic=/setup/icon.png" "fn=SHIELD" "ca=463365" "st=0" "bs=FA8F14F198FB" "nf=1" "rs=" (356)
15:46:05.619590 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:46:06.623982 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:46:08.616921 ac:1f:6b:45:fa:8a (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.60 tell 192.168.10.1, length 28
15:46:18.970438 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype IPv4 (0x0800), length 82: 192.168.10.1.mdns > 224.0.0.251.mdns: 0 PTR (QM)? _googlezone._tcp.local. (40)
15:46:18.970617 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype IPv4 (0x0800), length 119: 192.168.10.1.mdns > 224.0.0.251.mdns: 0 SRV (QM)? ee41442d-2c14-cc09-fde8-2be16f84be32._googlezone._tcp.local. (77)
15:46:18.970973 ac:1f:6b:45:fa:8a (oui Unknown) > 01:00:5e:00:00:fb (oui Unknown), ethertype IPv4 (0x0800), length 252: 192.168.10.1.mdns > 224.0.0.251.mdns: 0*- [0q] 4/0/0 PTR ee41442d-2c14-cc09-fde8-2be16f84be32._googlezone._tcp.local., (Cache flush) A 172.18.0.14, (Cache flush) SRV ee41442d-2c14-cc09-fde8-2be16f84be32.local.:10001 1100 0, (Cache flush) TXT "id=3CABD325728E72997BA6735F95651E36" "UDS" (210)
^C
9 packets captured
9 packets received by filter
0 packets dropped by kernel

stephenw10

Sorry I meant: tcpdump -e -i ix2

On the parent interface dircetly

NRgia

@stephenw10 said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

tcpdump -e -i ix2

15:49:59.310577 d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, ethertype RRCP (0x8899), length 60: d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, RRCP-0x25 query
15:49:57.147272 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype ARP, Request who-has Sperry.Blueshift tell 169.254.216.169, length 46
15:49:57.248033 00:04:4b:ba:35:05 (oui Unknown) > ac:1f:6b:45:fa:8a (oui Unknown), ethertype IPv4 (0x0800), length 376: Shield.Blueshift.39344 > fra02s19-in-f3.1e100.net.http: Flags [P.], seq 3325661723:3325662033,
ack 288699136, win 685, options [nop,nop,TS val 644858214 ecr 3014227199], length 310: HTTP: HEAD /generate_204 HTTP/1.1
15:49:57.278408 ac:1f:6b:45:fa:8a (oui Unknown) > 00:04:4b:ba:35:05 (oui Unknown), ethertype IPv4 (0x0800), length 149: fra02s19-in-f3.1e100.net.http > Shield.Blueshift.39344: Flags [P.], seq 1:84, ack 310, win 399,
options [nop,nop,TS val 3014287261 ecr 644858214], length 83: HTTP: HTTP/1.1 204 No Content
15:49:57.278930 00:04:4b:ba:35:05 (oui Unknown) > ac:1f:6b:45:fa:8a (oui Unknown), ethertype IPv4 (0x0800), length 66: Shield.Blueshift.39344 > fra02s19-in-f3.1e100.net.http: Flags [.], ack 84, win 685,
options [nop,nop,TS val 644858222 ecr 3014287261], length 0
15:49:57.308952 d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, ethertype RRCP (0x8899), length 60: d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, RRCP-0x25 query
15:49:57.472278 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 598: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
Request from 28:6d:97:7f:bb:0c (oui Unknown), length 548
15:49:58.187286 28:6d:97:7f:bb:0c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20, p 0, ethertype ARP, Request who-has Sperry.Blueshift tell 169.254.216.169, length 46
15:49:58.297476 08:36:c9:2a:16:e7 (oui Unknown) > Broadcast, ethertype RRCP (0x8899), length 60: 08:36:c9:2a:16:e7 (oui Unknown) > Broadcast, RRCP-0x23 query
15:49:58.309782 d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, ethertype RRCP (0x8899), length 60: d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, RRCP-0x25 query
15:49:58.596361 cc:40:d0:52:32:7d (oui Unknown) > 01:80:c2:00:00:40 (oui Unknown), ethertype Slow Protocols (0x8809), length 60: unknown (136), length 46
        0x0000:  880f 0000 0000 0000 0000 0000 0000 0000
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0020:  0000 0000 0000 0000 0000 0000 0000
15:49:59.310577 d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, ethertype RRCP (0x8899), length 60: d8:0d:17:4e:7a:13 (oui Unknown) > Broadcast, RRCP-0x25 query

If it's not enough, tell me what to "grep" for, I dumped it to a file, due to too much lines.

stephenw10

Hmm, ~~so still only outgoing packets~~. At least as far as tcpdump can see.

Are you able to pcap on something upstream to see the tagged traffic that should be arriving there?

NRgia

@stephenw10
Can you give me an example, please.
I don't have Wireshark installed.
I found it, it's in the UI, Packet capture

johnpoz

@nrgia said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

length 68: vlan 0, p 0, ethertype 802.1Q, vlan 20

That seems odd.. why is showing vlan 0 and vlan 20?

What is this guy 28:6d:97:7f:bb:0c, is that pfsense

That isn't outbound from pfsense.. Your other post shows ix2 as ether ac:1f:6b:45:fa:8a

A mac vendor lookup shows it as SAMJIN Co., Ltd.? Never heard of that company.
Seems "The Company provides its products mainly to Samsung Electronics."

stephenw10

@johnpoz said in pfSense 22.05 breaks VLANS, restoring pfSense 22.01 fixes the issue:

That seems odd.. why is showing vlan 0 and vlan 20

Mmm, that is a very good point! Like it's QinQ.