Link-local address flooding logs

rsaanon · Dec 14, 2018, 9:38 PM

Firewall logs are getting flooded with the broadcast link-local address (src: 169.254.100.100, dst: 169.254.255.255). I have upwards of 100+ IP devices in the house. How do I locate/isolate this problematic device without physically going to each and every device in the house?

Thank you!
-r

SteveITS · Dec 14, 2018, 9:51 PM

Does Diagnostics/ARP Table show it, and hence a MAC address?

rsaanon · Dec 14, 2018, 9:54 PM

Thanks for responding Steve.

There are no entries for 169.x in the ARP table.

JKnott · Dec 14, 2018, 10:05 PM

@rsaanon said in Link-local address flooding logs:

There are no entries for 169.x in the ARP table.

Fire up Packet Capture or Wireshark to see what the MAC addresses are.

marvosa · Dec 15, 2018, 9:47 AM

First, 169.254.0.0/16 is an APIPA range that devices revert to when they can't find a DHCP server on the network. So, the device is configured with DHCP and is on a segment of your network without a DHCP server, which may narrow down your search.

A quick google search pulls up a few posts that appear to reveal a couple things:

169.254.100.100 is a specific APIPA address that QNAP NAS units revert to when they can't find a DHCP server. Do you have a QNAP NAS? If so, that's where I'd look first.
If you take a closer look at the source and destination of most of those entries, you have 169.254.100.100 sending traffic to 169.254.255.255 on two different ports... that is broadcast traffic. I stumbled across a couple posts that state ports 32414 and 32412 UDP are used by Plex for discovery.

So, it looks like you have a Plex server (possibly virtualized on a QNAP?) that could not find a DHCP server, so it reverted to 169.254.100.100 and is sending broadcasts for discovery.

There is also a post on the Plex forums that says Plex can start spamming UDP if the time is off, so assuming you have a Plex server somewhere, you may want to check the time/date on both your Plex server and the hardware it's running on.

johnpoz · Dec 15, 2018, 9:54 AM

Great post @marvosa

And concur so much linklocal traffic screams lack of configuration, failure of dhcp quite possible...

rsaanon · Dec 18, 2018, 5:43 AM

@marvosa Your analysis is quite impressive. Good forensic work!

Here's what I can confirm:

I do have a QNAP that has 4 ethernet ports. All ports are 802.1ad aggregated with static IP. So, there's no possibility of auto-configuration of 169.x addresses.
I do have Plex, but not virtualized (ie: not running in a container or a vm).

stephenw10 · Dec 17, 2018, 10:20 PM

Mmm, I would run a pcap and filter by that source IP. You should not need to wait long given the frequency. Check it to get the MAC address.

Interesting that they are coming in on two interfaces. If they are from the same source that is of some concern. Does that device also have interfaces in both subnets? Otherwise how are those broadcast packets leaking across?

Steve

rsaanon · Dec 17, 2018, 10:33 PM

@stephenw10 I tried doing a capture, but a little difficult to do for the following reason:

The em0 is a WAN interface where the traffic seems to be originating from
The vmx3 is a parent interface has six subinterfaces VLANs/subnet ports tied to it. The firewall log shows the parent interface and not the subinterface where the problem seems to be coming from.

stephenw10 · Dec 17, 2018, 11:41 PM

As long as em0 is assigned you should be able to capture on it. That will give you the MAC and you can go from there.

If you're seeing that traffic on the parent interface it must be assigned also. Do you have a switch stripping VLAN tags incorrectly maybe?

I have a TP-Link switch that does that with broadcast traffic. I stopped using it for vlans!

Steve

JKnott · Dec 18, 2018, 2:04 AM

BTW, there's no need to obscure those addresses. They're RFC 1918 private addresses, which are not routed over the Internet. All you're doing is making it more difficult for us to help you.

marvosa · Dec 18, 2018, 5:43 AM

@rsaanon said in Link-local address flooding logs:

@marvosa Your analysis is quite impressive. Good forensic work!

Here's what I can confirm:

I do have a QNAP that has 4 ethernet ports. All ports are 802.1ad aggregated with static IP. So, there's no possibility of auto-configuration of 169.x addresses.

I do have Plex, but not virtualized (ie: not running in a container or a vm).

I mentioned Plex possibly being virtualized because some of the traffic is being blocked on your vmx3 interface, which tells me something is virtualized... I assumed Plex, but maybe it's your PFsense?

rsaanon · Dec 18, 2018, 12:20 PM

@jknott I prefer to keep my internal addressing scheme private and thus the reason for masking the rfc1918 addresses I have on my network. Also, I don’t see how masking the address makes it difficult for someone to help. I appreciate you taking time to chime in on the topic.

johnpoz · Dec 18, 2018, 1:40 PM

How exactly are you seeing that on your pfsense WAN and lan side interfaces at the same time? So you have no layer 2 isolation it seems.. Misconfigured switch vlan settings?

Why do you have your nas multihomed? What exactly are you trying to accomplish with such a setup?

rsaanon · Dec 18, 2018, 2:00 PM

@johnpoz Can’t explain why I am seeing the 169.x on two interfaces at the same time.

The WAN/em0 is configured for VLAN 2 as required by the ISP going into a Cisco managed switch. Internet connectivity is good & stable. Internally, the LAN is segmented in to multi-broadcast domains each serving a particular purpose (eg: video, SAN, etc). The NAS is split into two 802.1ad aggregates, one for general access and the other for iscsi connectivity.

stephenw10 · Dec 18, 2018, 2:02 PM

Have you run the packet capture yet? I can see no reason why you wouldn't be able to do that and it will show the device causing this immediately.
If you run it on both interfaces it will also confirm if the traffic really is arriving from the same device on both. In which case you have something misconfigured with those VLANs.

Steve

johnpoz · Dec 18, 2018, 2:08 PM

Yeah take all of 10 seconds to run a sniff and get the mac address of what is sending that out.

rsaanon · Dec 18, 2018, 2:43 PM

@stephenw10 I wished the firewall logs included the particular subinterface that showed the 169.x broadcast; instead, you see a parent interface in the log. So, for example, vmx3 has 6 subinterfaces and I don't know which particular subinterface the problem is originating from. To troubleshoot, I'd have to connect to each subnet/subinterface and capture packets.

I did manage to capture traffic on one of the sub-interfaces. This particular 169.x entry was originating from an HD HomeRun Tuner. What's very peculiar is that this device has a DHCP allocated IP address (ie: no auto-configuration of 169.x address). Yet it's broadcasting to 169.x domain with the source IP being 169.254.100.100. BTW, the tuner works fine as it has IP connectivity to the outside. The tuner does not allow ssh connectivity into it but does have a web administrative page that I'm able to see. There no mention of 169.x address. All looks well with the Tuner network configuration. For further troubleshooting, I took this device offline and did a packet recapture. As expected, I did not see log entries from the MAC address of the Tuner; however, I continue to see 169.254.100.100 from another MAC address that pointed me to the QNAP. The QNAP is configured with two static IPs (172.24.16.x for general access and 10.56.1.x for iSCSI) and therefore has no auto-configured 169.254.100.100 address; however, ssh'ing into QNAP, I shockingly see:

mgmt0     Link encap:Ethernet  HWaddr 00:08:9B:xx:xx:xx
          inet addr:169.254.100.100  Bcast:169.254.255.255  Mask:255.255.0.0
          inet6 addr: fe80::208:9bff:feee:8e46/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2760 errors:0 dropped:0 overruns:0 frame:0
          TX packets:34665 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:165682 (161.7 KiB)  TX bytes:3009614 (2.8 MiB)

I have no idea why the QNAP has the mgmt0 interface.

I'll troubleshoot more as time permits and post any updates here.

johnpoz · Dec 18, 2018, 2:48 PM

You will have to look into your HD tuner thing.. But many devices be they have IP or not will look for stuff via link-local... I had a direcTV wireless bridge that did that - pissed me off to be honest as well.. Not a fan of NOISE for no reason.. You have an IP - if you want to search for stuff then use your IP not link-local ;)

So HD tuner thing was looking for plex server via link-local.. Why not look on the network its actually configured for broadcast address? Sometimes I think the people that write the code for these things don't actually think it through..

Why would your Plex server answer a link-local broadcast if actually has an IP.. Is the device also sending out broadcasts to the network its on broadcast address?

rsaanon · Dec 18, 2018, 3:03 PM

@johnpoz Agree! Doesn't make sense why any device/application would use link-local address when none of the interfaces on the device itself has a link-local address. Go figure!