Link-local address flooding logs
-
Have you run the packet capture yet? I can see no reason why you wouldn't be able to do that and it will show the device causing this immediately.
If you run it on both interfaces it will also confirm if the traffic really is arriving from the same device on both. In which case you have something misconfigured with those VLANs.Steve
-
Yeah take all of 10 seconds to run a sniff and get the mac address of what is sending that out.
-
@stephenw10 I wished the firewall logs included the particular subinterface that showed the 169.x broadcast; instead, you see a parent interface in the log. So, for example, vmx3 has 6 subinterfaces and I don't know which particular subinterface the problem is originating from. To troubleshoot, I'd have to connect to each subnet/subinterface and capture packets.
I did manage to capture traffic on one of the sub-interfaces. This particular 169.x entry was originating from an HD HomeRun Tuner. What's very peculiar is that this device has a DHCP allocated IP address (ie: no auto-configuration of 169.x address). Yet it's broadcasting to 169.x domain with the source IP being 169.254.100.100. BTW, the tuner works fine as it has IP connectivity to the outside. The tuner does not allow ssh connectivity into it but does have a web administrative page that I'm able to see. There no mention of 169.x address. All looks well with the Tuner network configuration. For further troubleshooting, I took this device offline and did a packet recapture. As expected, I did not see log entries from the MAC address of the Tuner; however, I continue to see 169.254.100.100 from another MAC address that pointed me to the QNAP. The QNAP is configured with two static IPs (172.24.16.x for general access and 10.56.1.x for iSCSI) and therefore has no auto-configured 169.254.100.100 address; however, ssh'ing into QNAP, I shockingly see:
mgmt0 Link encap:Ethernet HWaddr 00:08:9B:xx:xx:xx inet addr:169.254.100.100 Bcast:169.254.255.255 Mask:255.255.0.0 inet6 addr: fe80::208:9bff:feee:8e46/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2760 errors:0 dropped:0 overruns:0 frame:0 TX packets:34665 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:165682 (161.7 KiB) TX bytes:3009614 (2.8 MiB)I have no idea why the QNAP has the mgmt0 interface.
I'll troubleshoot more as time permits and post any updates here.
-
You will have to look into your HD tuner thing.. But many devices be they have IP or not will look for stuff via link-local... I had a direcTV wireless bridge that did that - pissed me off to be honest as well.. Not a fan of NOISE for no reason.. You have an IP - if you want to search for stuff then use your IP not link-local ;)
So HD tuner thing was looking for plex server via link-local.. Why not look on the network its actually configured for broadcast address? Sometimes I think the people that write the code for these things don't actually think it through..
Why would your Plex server answer a link-local broadcast if actually has an IP.. Is the device also sending out broadcasts to the network its on broadcast address?
-
@johnpoz Agree! Doesn't make sense why any device/application would use link-local address when none of the interfaces on the device itself has a link-local address. Go figure!
-
Is there a packet capture package for pfSense? It would be much easier to capture packets directly on the firewall instead from a client that would need to be connected to the appropriate subnet.
-
It's built in. Diagnostics > Packet Capture.
That would explain your not capturing until now.
It's interesting that you're seeing those blocks in the parent interface of the VLANs. To me that points to something incorrectly stripping the tags off that broadcast traffic.
Steve
-
@stephenw10 A quote is in order
"I've always thought that when they say ignorance is bliss, the converse to that is that knowledge is hell. The more you know, the bleaker things can get."Thanks for pointing out the built-in capture tool.
It's even more interesting that em0/wan (among other interfaces) is showing the 169.x broadcast. All nodes on the internal network function as purposed (ie: no service/connectivity issues). It's just that too much 169.x noise is being generated and the firewall is having to actively block those across different interfaces. It would be easy (& lazy, might I add) for me to add a firewall rule to not log those 169.x packets, but that is not the approach I prefer. I'd be surprised if there are any issues with vlan tag stripping as all switches on the network are cisco switches that are running the latest f/w.
I'll have to dig deeper..
-
Where I've seen similar things it was because the native vlan was always present on all port and could not be removed. Broadcast traffic was sent to evety port regardless of VLANs configured. Crazyness! It functions as an unmamaged switch now and even as that it's failing.
Steve
-
yeah the tplink low end smart switches are like that... the tl-sg108e and 105e did not allow you to remove vlan 1 from ports... So really its just JUNK!!
They finally put out a firmware that was suppose to fix this for their v3 hardware, but v1 and v2 got no love. And not sure if the v4 hardware works nor not.
If your running any of this through a vm sort of switch - like esxi, and don't set the vlan id to say 4095 then those vswitches will strip tags.
BTW you can also just shell into pfsense and run tcpdump directly, -e will show you the vlan stuff.. If you sniff on the parent interface you should be able to see all the traffic with their tags, etc.
example
tcpdump -ni igb2 -eWill show you traffic that is on igb2 along with any vlan traffic on that interface
10:37:55.166622 02:11:32:21:6a:72 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60: vlan 5, p 0, ethertype ARP, Request who-has 192.168.5.23 tell 192.168.5.22, length 42 10:37:55.205238 02:11:32:21:6a:72 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60: vlan 6, p 0, ethertype ARP, Request who-has 192.168.6.17 tell 192.168.6.22, length 42 10:37:55.217230 02:11:32:21:6a:72 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60: vlan 8, p 0, ethertype ARP, Request who-has 192.168.8.15 tell 192.168.8.22, length 42 10:37:55.237575 02:11:32:21:6a:72 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.2.96 tell 192.168.2.22, length 46There you can see vlan traffic on the parent and also non tagged (native) traffic..
-
@johnpoz Thank you!! tcpdump is exactly what I need.
Here's a snip of the capture:
---------em0 capture------------ listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes 11:50:11.908185 00:08:9b:ee:xx:46 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 63: 169.254.100.100.45345 > 169.254.255.255.32414: UDP, length 21 11:50:11.914948 00:08:9b:ee:xx:46 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 63: 169.254.100.100.42249 > 169.254.255.255.32412: UDP, length 21 11:50:12.328153 00:08:9b:ee:xx:46 > 01:00:5e:7f:ff:fa, ethertype IPv4 (0x0800), length 136: 169.254.100.100.47100 > 239.255.255.250.1900: UDP, length 94 3 packets captured 732 packets received by filter 0 packets dropped by kernel ---------vmx3 capture------------ listening on vmx3, link-type EN10MB (Ethernet), capture size 262144 bytes 11:51:02.722666 00:08:9b:ee:xx:46 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 92: 169.254.100.100.137 > 169.254.255.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCASTSource device for captures on both interfaces is QNAP, albeit em0 capture shows destination to be 169.254.255.255 with .32412 & 32414 ports. Ports 32412/4 are Plex discovery related. The vmx3 related capture is NetBIOS.
On QNAP side, I do not know what purpose the mgmt0/169.x serves and therefore do not want to delete that interface. I did delete the route entry for 169.x on QNAP that was pointing to the default gateway,
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 mgmt0This way no 169.x leaks out to the Internet Gateway. This didn't seem to make any difference.
-
did you use -e, not seeing any tagged traffic on em0.. Did you say that was your WAN... How is that traffic getting on your WAN vlan?
As to what mgmt0, is I would get with qnap support. If I had to guess its some sort of management vlan use.. Which maybe you have not setup?
-
@johnpoz I ran the command so it filters only the src ip of 169.x (tcpdump -ni em0 src 169.254.100.100 -e)
As for administrative vlan, I have a non-default (ie: vlan 111) that's used for administrative purposes.
-
Yeah thats fine that you use vlanX for YOUR managment vlan - but did you setup what qnap "thinks" is its management vlan ;)
So how exactly is untagged traffic getting to your WAN layer 2? Must be a problem with switch config would be really only way such a thing could happen.. Or you have something plugged into the wrong port crossing your vlans.
So a quick google for this qnap mgmt0 doesn't find a lot - other than PROBLEMS with it.. I see a thread where plex and kodi having issues where they seem to bind to this mgmt0 that has no IP on it other that than 169.254.100.100
Example
https://forum.qnap.com/viewtopic.php?t=143879I would really suggest you get with qnap support or their forums for info on this interface.. What version of qnap are you running btw?
-
Not sure why the QNAP needs an administrative vlan. Within QNAP Admin UI, there's no option for setting admin. vlan/ip. The only way you know there's a manangement interface is when you ssh into the nas. Also, this mgmt0 has the 169.x address which implies the DHCP setup.
Not all internal traffic is allowed Internet connectivity. The untagged traffic that is allowed Internet access is forwarded to the Internet Gateway interface (em0). If there were ports that were cross-connected to wrong vlans, then there would be connectivity issues from a firewall rules standpoint. All internal connectivity across all subnets seem to be operating as expected.
I hesitate to contact QNAP support as I've not heard great things about them. Though, in the end, I might not be left with any other alternative.
I am tempted to configure a fw rule to drop & not log any 169.x packets..
-
You could do that sure with your not log.. But talk of the 169.254 traffic got me curious if my new direcTV box was sending out any crap ;)
Yup some new noise I found with a simple sniff.
12:02:02.857057 0c:08:b4:48:cc:63 > 01:00:5e:00:90:01, ethertype IPv4 (0x0800), length 96: 169.254.8.104.52613 > 224.0.144.1.52613: UDP, length 54 12:02:02.857556 0c:08:b4:48:cc:63 > 01:00:5e:00:90:01, ethertype IPv4 (0x0800), length 96: 169.254.8.104.52613 > 224.0.144.1.52613: UDP, length 54 12:02:03.358348 0c:08:b4:48:cc:63 > 01:00:5e:00:90:01, ethertype IPv4 (0x0800), length 96: 169.254.8.104.52613 > 224.0.144.1.52613: UDP, length 54 12:02:03.358648 0c:08:b4:48:cc:63 > 01:00:5e:00:90:01, ethertype IPv4 (0x0800), length 96: 169.254.8.104.52613 > 224.0.144.1.52613: UDP, length 54 12:02:03.859561 0c:08:b4:48:cc:63 > 01:00:5e:00:90:01, ethertype IPv4 (0x0800), length 96: 169.254.8.104.52613 > 224.0.144.1.52613: UDP, length 54 12:02:03.859875 0c:08:b4:48:cc:63 > 01:00:5e:00:90:01, ethertype IPv4 (0x0800), length 96: 169.254.8.104.52613 > 224.0.144.1.52613: UDP, length 54I had a ACL already setup in my switch to block this noise from old box to 239.255.255.250 from 169.254 that was SSDP on 1900..
So just edited the ACL on the switch to block this new noise.. So now none of that traffic goes anywhere.. Its stopped right at the switch port the directv box is connected too..
-
@johnpoz Good idea!
Block at the switch level instead of at the firewall. Though, I'm still a bit apprehensive about blocking 169.x all together. The flip side (reason for) is that 169.x only deals with zero configuration situation. So, it should not hinder normal operation. -
I am not blocking the 169.254 - I block the multicast address.
But sure you could block it via 169.254.x.x if you wanted too..I don't want or need any of that multicast noise - and I sure don't want it going out to the wifi, which that vlan is where my roku sticks connect..
169.254 is IPv4 link local and yes is used for APIPA... In a correctly configured network there should never be any need or use for it.. What you posted is clearly what I would call NOISE.. that should be able to be disabled on the device sending it.. But if you can not, then the best place to stop is before it even enters the switch ;)
I had looked all over to how could disable the noise coming out of the directv box.. Could not find anyway.. So block at switch it is then ;) Which reminds me should look to thread I had on plex forums about the noise it was sending out.. Even when dlna and disabled - it was sending out SSDP nonsense.. I had to block that at the switch as well..
edit: F'ing Crickets over there
https://forums.plex.tv/t/stop-pms-from-sending-ssdp-dlna-and-gdm-disabled/321779Thing sends every freaking 10 seconds.. Wonder if any of the beta's after posted that fixed it? On 1.14 something now.. going to remove the acl on the switch and see ;) fingers crossed.
edit: Arrggghhh still chatty kathy..
13:22:57.575617 IP 192.168.9.10.42339 > 239.255.255.250.1900: UDP, length 101 13:22:57.575635 IP 192.168.9.11.45988 > 239.255.255.250.1900: UDP, length 101 -
V vronp referenced this topic on
