Link-local address flooding logs
-
First, 169.254.0.0/16 is an APIPA range that devices revert to when they can't find a DHCP server on the network. So, the device is configured with DHCP and is on a segment of your network without a DHCP server, which may narrow down your search.
A quick google search pulls up a few posts that appear to reveal a couple things:
-
169.254.100.100 is a specific APIPA address that QNAP NAS units revert to when they can't find a DHCP server. Do you have a QNAP NAS? If so, that's where I'd look first.
-
If you take a closer look at the source and destination of most of those entries, you have 169.254.100.100 sending traffic to 169.254.255.255 on two different ports... that is broadcast traffic. I stumbled across a couple posts that state ports 32414 and 32412 UDP are used by Plex for discovery.
So, it looks like you have a Plex server (possibly virtualized on a QNAP?) that could not find a DHCP server, so it reverted to 169.254.100.100 and is sending broadcasts for discovery.
There is also a post on the Plex forums that says Plex can start spamming UDP if the time is off, so assuming you have a Plex server somewhere, you may want to check the time/date on both your Plex server and the hardware it's running on.
-
-
Great post @marvosa
And concur so much linklocal traffic screams lack of configuration, failure of dhcp quite possible...
-
@marvosa Your analysis is quite impressive. Good forensic work!
Here's what I can confirm:
-
I do have a QNAP that has 4 ethernet ports. All ports are 802.1ad aggregated with static IP. So, there's no possibility of auto-configuration of 169.x addresses.
-
I do have Plex, but not virtualized (ie: not running in a container or a vm).
-
-
Mmm, I would run a pcap and filter by that source IP. You should not need to wait long given the frequency. Check it to get the MAC address.
Interesting that they are coming in on two interfaces. If they are from the same source that is of some concern. Does that device also have interfaces in both subnets? Otherwise how are those broadcast packets leaking across?
Steve
-
@stephenw10 I tried doing a capture, but a little difficult to do for the following reason:
- The em0 is a WAN interface where the traffic seems to be originating from
- The vmx3 is a parent interface has six subinterfaces VLANs/subnet ports tied to it. The firewall log shows the parent interface and not the subinterface where the problem seems to be coming from.
-
As long as em0 is assigned you should be able to capture on it. That will give you the MAC and you can go from there.
If you're seeing that traffic on the parent interface it must be assigned also. Do you have a switch stripping VLAN tags incorrectly maybe?
I have a TP-Link switch that does that with broadcast traffic. I stopped using it for vlans!
Steve
-
BTW, there's no need to obscure those addresses. They're RFC 1918 private addresses, which are not routed over the Internet. All you're doing is making it more difficult for us to help you.
-
@rsaanon said in Link-local address flooding logs:
@marvosa Your analysis is quite impressive. Good forensic work!
Here's what I can confirm:
-
I do have a QNAP that has 4 ethernet ports. All ports are 802.1ad aggregated with static IP. So, there's no possibility of auto-configuration of 169.x addresses.
-
I do have Plex, but not virtualized (ie: not running in a container or a vm).
I mentioned Plex possibly being virtualized because some of the traffic is being blocked on your vmx3 interface, which tells me something is virtualized... I assumed Plex, but maybe it's your PFsense?
-
-
@jknott I prefer to keep my internal addressing scheme private and thus the reason for masking the rfc1918 addresses I have on my network. Also, I don’t see how masking the address makes it difficult for someone to help. I appreciate you taking time to chime in on the topic.
-
How exactly are you seeing that on your pfsense WAN and lan side interfaces at the same time? So you have no layer 2 isolation it seems.. Misconfigured switch vlan settings?
Why do you have your nas multihomed? What exactly are you trying to accomplish with such a setup?
-
@johnpoz Can’t explain why I am seeing the 169.x on two interfaces at the same time.
The WAN/em0 is configured for VLAN 2 as required by the ISP going into a Cisco managed switch. Internet connectivity is good & stable. Internally, the LAN is segmented in to multi-broadcast domains each serving a particular purpose (eg: video, SAN, etc). The NAS is split into two 802.1ad aggregates, one for general access and the other for iscsi connectivity.
-
Have you run the packet capture yet? I can see no reason why you wouldn't be able to do that and it will show the device causing this immediately.
If you run it on both interfaces it will also confirm if the traffic really is arriving from the same device on both. In which case you have something misconfigured with those VLANs.Steve
-
Yeah take all of 10 seconds to run a sniff and get the mac address of what is sending that out.
-
@stephenw10 I wished the firewall logs included the particular subinterface that showed the 169.x broadcast; instead, you see a parent interface in the log. So, for example, vmx3 has 6 subinterfaces and I don't know which particular subinterface the problem is originating from. To troubleshoot, I'd have to connect to each subnet/subinterface and capture packets.
I did manage to capture traffic on one of the sub-interfaces. This particular 169.x entry was originating from an HD HomeRun Tuner. What's very peculiar is that this device has a DHCP allocated IP address (ie: no auto-configuration of 169.x address). Yet it's broadcasting to 169.x domain with the source IP being 169.254.100.100. BTW, the tuner works fine as it has IP connectivity to the outside. The tuner does not allow ssh connectivity into it but does have a web administrative page that I'm able to see. There no mention of 169.x address. All looks well with the Tuner network configuration. For further troubleshooting, I took this device offline and did a packet recapture. As expected, I did not see log entries from the MAC address of the Tuner; however, I continue to see 169.254.100.100 from another MAC address that pointed me to the QNAP. The QNAP is configured with two static IPs (172.24.16.x for general access and 10.56.1.x for iSCSI) and therefore has no auto-configured 169.254.100.100 address; however, ssh'ing into QNAP, I shockingly see:
mgmt0 Link encap:Ethernet HWaddr 00:08:9B:xx:xx:xx inet addr:169.254.100.100 Bcast:169.254.255.255 Mask:255.255.0.0 inet6 addr: fe80::208:9bff:feee:8e46/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2760 errors:0 dropped:0 overruns:0 frame:0 TX packets:34665 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:165682 (161.7 KiB) TX bytes:3009614 (2.8 MiB)I have no idea why the QNAP has the mgmt0 interface.
I'll troubleshoot more as time permits and post any updates here.
-
You will have to look into your HD tuner thing.. But many devices be they have IP or not will look for stuff via link-local... I had a direcTV wireless bridge that did that - pissed me off to be honest as well.. Not a fan of NOISE for no reason.. You have an IP - if you want to search for stuff then use your IP not link-local ;)
So HD tuner thing was looking for plex server via link-local.. Why not look on the network its actually configured for broadcast address? Sometimes I think the people that write the code for these things don't actually think it through..
Why would your Plex server answer a link-local broadcast if actually has an IP.. Is the device also sending out broadcasts to the network its on broadcast address?
-
@johnpoz Agree! Doesn't make sense why any device/application would use link-local address when none of the interfaces on the device itself has a link-local address. Go figure!
-
Is there a packet capture package for pfSense? It would be much easier to capture packets directly on the firewall instead from a client that would need to be connected to the appropriate subnet.
-
It's built in. Diagnostics > Packet Capture.
That would explain your not capturing until now.
It's interesting that you're seeing those blocks in the parent interface of the VLANs. To me that points to something incorrectly stripping the tags off that broadcast traffic.
Steve
-
@stephenw10 A quote is in order
"I've always thought that when they say ignorance is bliss, the converse to that is that knowledge is hell. The more you know, the bleaker things can get."Thanks for pointing out the built-in capture tool.
It's even more interesting that em0/wan (among other interfaces) is showing the 169.x broadcast. All nodes on the internal network function as purposed (ie: no service/connectivity issues). It's just that too much 169.x noise is being generated and the firewall is having to actively block those across different interfaces. It would be easy (& lazy, might I add) for me to add a firewall rule to not log those 169.x packets, but that is not the approach I prefer. I'd be surprised if there are any issues with vlan tag stripping as all switches on the network are cisco switches that are running the latest f/w.
I'll have to dig deeper..
-
Where I've seen similar things it was because the native vlan was always present on all port and could not be removed. Broadcast traffic was sent to evety port regardless of VLANs configured. Crazyness! It functions as an unmamaged switch now and even as that it's failing.
Steve
