Link-local address flooding logs



  • Firewall logs are getting flooded with the broadcast link-local address (src: 169.254.100.100, dst: 169.254.255.255). I have upwards of 100+ IP devices in the house. How do I locate/isolate this problematic device without physically going to each and every device in the house?

    0_1544823631427_pfsense.linklocal.PNG

    Thank you!
    -r



  • Does Diagnostics/ARP Table show it, and hence a MAC address?



  • Thanks for responding Steve.

    There are no entries for 169.x in the ARP table.



  • @rsaanon said in Link-local address flooding logs:

    There are no entries for 169.x in the ARP table.

    Fire up Packet Capture or Wireshark to see what the MAC addresses are.



  • First, 169.254.0.0/16 is an APIPA range that devices revert to when they can't find a DHCP server on the network. So, the device is configured with DHCP and is on a segment of your network without a DHCP server, which may narrow down your search.

    A quick google search pulls up a few posts that appear to reveal a couple things:

    • 169.254.100.100 is a specific APIPA address that QNAP NAS units revert to when they can't find a DHCP server. Do you have a QNAP NAS? If so, that's where I'd look first.

    • If you take a closer look at the source and destination of most of those entries, you have 169.254.100.100 sending traffic to 169.254.255.255 on two different ports... that is broadcast traffic. I stumbled across a couple posts that state ports 32414 and 32412 UDP are used by Plex for discovery.

    So, it looks like you have a Plex server (possibly virtualized on a QNAP?) that could not find a DHCP server, so it reverted to 169.254.100.100 and is sending broadcasts for discovery.

    There is also a post on the Plex forums that says Plex can start spamming UDP if the time is off, so assuming you have a Plex server somewhere, you may want to check the time/date on both your Plex server and the hardware it's running on.


  • LAYER 8 Global Moderator

    Great post @marvosa

    And concur so much linklocal traffic screams lack of configuration, failure of dhcp quite possible...



  • @marvosa Your analysis is quite impressive. Good forensic work! 👏

    Here's what I can confirm:

    • I do have a QNAP that has 4 ethernet ports. All ports are 802.1ad aggregated with static IP. So, there's no possibility of auto-configuration of 169.x addresses.
      0_1544880820501_qnap_nics.png

    • I do have Plex, but not virtualized (ie: not running in a container or a vm).


  • Netgate Administrator

    Mmm, I would run a pcap and filter by that source IP. You should not need to wait long given the frequency. Check it to get the MAC address.

    Interesting that they are coming in on two interfaces. If they are from the same source that is of some concern. Does that device also have interfaces in both subnets? Otherwise how are those broadcast packets leaking across?

    Steve



  • @stephenw10 I tried doing a capture, but a little difficult to do for the following reason:

    • The em0 is a WAN interface where the traffic seems to be originating from
    • The vmx3 is a parent interface has six subinterfaces VLANs/subnet ports tied to it. The firewall log shows the parent interface and not the subinterface where the problem seems to be coming from.

  • Netgate Administrator

    As long as em0 is assigned you should be able to capture on it. That will give you the MAC and you can go from there.

    If you're seeing that traffic on the parent interface it must be assigned also. Do you have a switch stripping VLAN tags incorrectly maybe?

    I have a TP-Link switch that does that with broadcast traffic. I stopped using it for vlans!

    Steve



  • BTW, there's no need to obscure those addresses. They're RFC 1918 private addresses, which are not routed over the Internet. All you're doing is making it more difficult for us to help you.



  • @rsaanon said in Link-local address flooding logs:

    @marvosa Your analysis is quite impressive. Good forensic work! 👏

    Here's what I can confirm:

    • I do have a QNAP that has 4 ethernet ports. All ports are 802.1ad aggregated with static IP. So, there's no possibility of auto-configuration of 169.x addresses.

    • I do have Plex, but not virtualized (ie: not running in a container or a vm).

    I mentioned Plex possibly being virtualized because some of the traffic is being blocked on your vmx3 interface, which tells me something is virtualized... I assumed Plex, but maybe it's your PFsense?



  • @jknott I prefer to keep my internal addressing scheme private and thus the reason for masking the rfc1918 addresses I have on my network. Also, I don’t see how masking the address makes it difficult for someone to help. I appreciate you taking time to chime in on the topic.


  • LAYER 8 Global Moderator

    How exactly are you seeing that on your pfsense WAN and lan side interfaces at the same time? So you have no layer 2 isolation it seems.. Misconfigured switch vlan settings?

    Why do you have your nas multihomed? What exactly are you trying to accomplish with such a setup?



  • @johnpoz Can’t explain why I am seeing the 169.x on two interfaces at the same time.

    The WAN/em0 is configured for VLAN 2 as required by the ISP going into a Cisco managed switch. Internet connectivity is good & stable. Internally, the LAN is segmented in to multi-broadcast domains each serving a particular purpose (eg: video, SAN, etc). The NAS is split into two 802.1ad aggregates, one for general access and the other for iscsi connectivity.


  • Netgate Administrator

    Have you run the packet capture yet? I can see no reason why you wouldn't be able to do that and it will show the device causing this immediately.
    If you run it on both interfaces it will also confirm if the traffic really is arriving from the same device on both. In which case you have something misconfigured with those VLANs.

    Steve


  • LAYER 8 Global Moderator

    Yeah take all of 10 seconds to run a sniff and get the mac address of what is sending that out.



  • @stephenw10 I wished the firewall logs included the particular subinterface that showed the 169.x broadcast; instead, you see a parent interface in the log. So, for example, vmx3 has 6 subinterfaces and I don't know which particular subinterface the problem is originating from. To troubleshoot, I'd have to connect to each subnet/subinterface and capture packets.

    I did manage to capture traffic on one of the sub-interfaces. This particular 169.x entry was originating from an HD HomeRun Tuner. What's very peculiar is that this device has a DHCP allocated IP address (ie: no auto-configuration of 169.x address). Yet it's broadcasting to 169.x domain with the source IP being 169.254.100.100. BTW, the tuner works fine as it has IP connectivity to the outside. The tuner does not allow ssh connectivity into it but does have a web administrative page that I'm able to see. There no mention of 169.x address. All looks well with the Tuner network configuration. For further troubleshooting, I took this device offline and did a packet recapture. As expected, I did not see log entries from the MAC address of the Tuner; however, I continue to see 169.254.100.100 from another MAC address that pointed me to the QNAP. The QNAP is configured with two static IPs (172.24.16.x for general access and 10.56.1.x for iSCSI) and therefore has no auto-configured 169.254.100.100 address; however, ssh'ing into QNAP, I shockingly see:

    mgmt0     Link encap:Ethernet  HWaddr 00:08:9B:xx:xx:xx
              inet addr:169.254.100.100  Bcast:169.254.255.255  Mask:255.255.0.0
              inet6 addr: fe80::208:9bff:feee:8e46/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:2760 errors:0 dropped:0 overruns:0 frame:0
              TX packets:34665 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:165682 (161.7 KiB)  TX bytes:3009614 (2.8 MiB)
    

    I have no idea why the QNAP has the mgmt0 interface.

    I'll troubleshoot more as time permits and post any updates here.


  • LAYER 8 Global Moderator

    You will have to look into your HD tuner thing.. But many devices be they have IP or not will look for stuff via link-local... I had a direcTV wireless bridge that did that - pissed me off to be honest as well.. Not a fan of NOISE for no reason.. You have an IP - if you want to search for stuff then use your IP not link-local ;)

    So HD tuner thing was looking for plex server via link-local.. Why not look on the network its actually configured for broadcast address? Sometimes I think the people that write the code for these things don't actually think it through..

    Why would your Plex server answer a link-local broadcast if actually has an IP.. Is the device also sending out broadcasts to the network its on broadcast address?



  • @johnpoz Agree! Doesn't make sense why any device/application would use link-local address when none of the interfaces on the device itself has a link-local address. Go figure! 🤷



  • Is there a packet capture package for pfSense? It would be much easier to capture packets directly on the firewall instead from a client that would need to be connected to the appropriate subnet.


  • Netgate Administrator

    It's built in. Diagnostics > Packet Capture.

    That would explain your not capturing until now. 😉

    It's interesting that you're seeing those blocks in the parent interface of the VLANs. To me that points to something incorrectly stripping the tags off that broadcast traffic.

    Steve



  • @stephenw10 A quote is in order 😉

    "I've always thought that when they say ignorance is bliss, the converse to that is that knowledge is hell. 
    The more you know, the bleaker things can get."
    

    Thanks for pointing out the built-in capture tool.

    It's even more interesting that em0/wan (among other interfaces) is showing the 169.x broadcast. All nodes on the internal network function as purposed (ie: no service/connectivity issues). It's just that too much 169.x noise is being generated and the firewall is having to actively block those across different interfaces. It would be easy (& lazy, might I add) for me to add a firewall rule to not log those 169.x packets, but that is not the approach I prefer. I'd be surprised if there are any issues with vlan tag stripping as all switches on the network are cisco switches that are running the latest f/w.

    I'll have to dig deeper..


  • Netgate Administrator

    Where I've seen similar things it was because the native vlan was always present on all port and could not be removed. Broadcast traffic was sent to evety port regardless of VLANs configured. Crazyness! It functions as an unmamaged switch now and even as that it's failing.

    Steve


  • LAYER 8 Global Moderator

    yeah the tplink low end smart switches are like that... the tl-sg108e and 105e did not allow you to remove vlan 1 from ports... So really its just JUNK!!

    They finally put out a firmware that was suppose to fix this for their v3 hardware, but v1 and v2 got no love. And not sure if the v4 hardware works nor not.

    If your running any of this through a vm sort of switch - like esxi, and don't set the vlan id to say 4095 then those vswitches will strip tags.

    BTW you can also just shell into pfsense and run tcpdump directly, -e will show you the vlan stuff.. If you sniff on the parent interface you should be able to see all the traffic with their tags, etc.

    example
    tcpdump -ni igb2 -e

    Will show you traffic that is on igb2 along with any vlan traffic on that interface

    10:37:55.166622 02:11:32:21:6a:72 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60: vlan 5, p 0, ethertype ARP, Request who-has 192.168.5.23 tell 192.168.5.22, length 42
    
    10:37:55.205238 02:11:32:21:6a:72 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60: vlan 6, p 0, ethertype ARP, Request who-has 192.168.6.17 tell 192.168.6.22, length 42
    
    10:37:55.217230 02:11:32:21:6a:72 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60: vlan 8, p 0, ethertype ARP, Request who-has 192.168.8.15 tell 192.168.8.22, length 42
    
    10:37:55.237575 02:11:32:21:6a:72 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.2.96 tell 192.168.2.22, length 46
    

    There you can see vlan traffic on the parent and also non tagged (native) traffic..



  • @johnpoz Thank you!! tcpdump is exactly what I need.

    Here's a snip of the capture:

    ---------em0 capture------------
    listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
    11:50:11.908185 00:08:9b:ee:xx:46 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 63: 169.254.100.100.45345 > 169.254.255.255.32414: UDP, length 21
    11:50:11.914948 00:08:9b:ee:xx:46 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 63: 169.254.100.100.42249 > 169.254.255.255.32412: UDP, length 21
    11:50:12.328153 00:08:9b:ee:xx:46 > 01:00:5e:7f:ff:fa, ethertype IPv4 (0x0800), length 136: 169.254.100.100.47100 > 239.255.255.250.1900: UDP, length 94
    3 packets captured
    732 packets received by filter
    0 packets dropped by kernel
    ---------vmx3 capture------------
    listening on vmx3, link-type EN10MB (Ethernet), capture size 262144 bytes
    11:51:02.722666 00:08:9b:ee:xx:46 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 92: 169.254.100.100.137 > 169.254.255.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
    

    Source device for captures on both interfaces is QNAP, albeit em0 capture shows destination to be 169.254.255.255 with .32412 & 32414 ports. Ports 32412/4 are Plex discovery related. The vmx3 related capture is NetBIOS.

    On QNAP side, I do not know what purpose the mgmt0/169.x serves and therefore do not want to delete that interface. I did delete the route entry for 169.x on QNAP that was pointing to the default gateway,

    169.254.0.0     0.0.0.0    255.255.0.0     U    0 0    0 mgmt0
    

    This way no 169.x leaks out to the Internet Gateway. This didn't seem to make any difference.


  • LAYER 8 Global Moderator

    did you use -e, not seeing any tagged traffic on em0.. Did you say that was your WAN... How is that traffic getting on your WAN vlan?

    As to what mgmt0, is I would get with qnap support. If I had to guess its some sort of management vlan use.. Which maybe you have not setup?



  • @johnpoz I ran the command so it filters only the src ip of 169.x (tcpdump -ni em0 src 169.254.100.100 -e)

    As for administrative vlan, I have a non-default (ie: vlan 111) that's used for administrative purposes.


  • LAYER 8 Global Moderator

    Yeah thats fine that you use vlanX for YOUR managment vlan - but did you setup what qnap "thinks" is its management vlan ;)

    So how exactly is untagged traffic getting to your WAN layer 2? Must be a problem with switch config would be really only way such a thing could happen.. Or you have something plugged into the wrong port crossing your vlans.

    So a quick google for this qnap mgmt0 doesn't find a lot - other than PROBLEMS with it.. I see a thread where plex and kodi having issues where they seem to bind to this mgmt0 that has no IP on it other that than 169.254.100.100

    Example
    https://forum.qnap.com/viewtopic.php?t=143879

    I would really suggest you get with qnap support or their forums for info on this interface.. What version of qnap are you running btw?



  • Not sure why the QNAP needs an administrative vlan. Within QNAP Admin UI, there's no option for setting admin. vlan/ip. The only way you know there's a manangement interface is when you ssh into the nas. Also, this mgmt0 has the 169.x address which implies the DHCP setup.

    Not all internal traffic is allowed Internet connectivity. The untagged traffic that is allowed Internet access is forwarded to the Internet Gateway interface (em0). If there were ports that were cross-connected to wrong vlans, then there would be connectivity issues from a firewall rules standpoint. All internal connectivity across all subnets seem to be operating as expected.

    I hesitate to contact QNAP support as I've not heard great things about them. Though, in the end, I might not be left with any other alternative.

    I am tempted to configure a fw rule to drop & not log any 169.x packets..


  • LAYER 8 Global Moderator

    You could do that sure with your not log.. But talk of the 169.254 traffic got me curious if my new direcTV box was sending out any crap ;)

    Yup some new noise I found with a simple sniff.

    12:02:02.857057 0c:08:b4:48:cc:63 > 01:00:5e:00:90:01, ethertype IPv4 (0x0800), length 96: 169.254.8.104.52613 > 224.0.144.1.52613: UDP, length 54
    12:02:02.857556 0c:08:b4:48:cc:63 > 01:00:5e:00:90:01, ethertype IPv4 (0x0800), length 96: 169.254.8.104.52613 > 224.0.144.1.52613: UDP, length 54
    12:02:03.358348 0c:08:b4:48:cc:63 > 01:00:5e:00:90:01, ethertype IPv4 (0x0800), length 96: 169.254.8.104.52613 > 224.0.144.1.52613: UDP, length 54
    12:02:03.358648 0c:08:b4:48:cc:63 > 01:00:5e:00:90:01, ethertype IPv4 (0x0800), length 96: 169.254.8.104.52613 > 224.0.144.1.52613: UDP, length 54
    12:02:03.859561 0c:08:b4:48:cc:63 > 01:00:5e:00:90:01, ethertype IPv4 (0x0800), length 96: 169.254.8.104.52613 > 224.0.144.1.52613: UDP, length 54
    12:02:03.859875 0c:08:b4:48:cc:63 > 01:00:5e:00:90:01, ethertype IPv4 (0x0800), length 96: 169.254.8.104.52613 > 224.0.144.1.52613: UDP, length 54
    

    I had a ACL already setup in my switch to block this noise from old box to 239.255.255.250 from 169.254 that was SSDP on 1900..

    So just edited the ACL on the switch to block this new noise.. So now none of that traffic goes anywhere.. Its stopped right at the switch port the directv box is connected too..



  • @johnpoz Good idea! 👌 Block at the switch level instead of at the firewall. Though, I'm still a bit apprehensive about blocking 169.x all together. The flip side (reason for) is that 169.x only deals with zero configuration situation. So, it should not hinder normal operation.


  • LAYER 8 Global Moderator

    I am not blocking the 169.254 - I block the multicast address.
    But sure you could block it via 169.254.x.x if you wanted too..

    I don't want or need any of that multicast noise - and I sure don't want it going out to the wifi, which that vlan is where my roku sticks connect..

    169.254 is IPv4 link local and yes is used for APIPA... In a correctly configured network there should never be any need or use for it.. What you posted is clearly what I would call NOISE.. that should be able to be disabled on the device sending it.. But if you can not, then the best place to stop is before it even enters the switch ;)

    I had looked all over to how could disable the noise coming out of the directv box.. Could not find anyway.. So block at switch it is then ;) Which reminds me should look to thread I had on plex forums about the noise it was sending out.. Even when dlna and disabled - it was sending out SSDP nonsense.. I had to block that at the switch as well..

    edit: F'ing Crickets over there
    https://forums.plex.tv/t/stop-pms-from-sending-ssdp-dlna-and-gdm-disabled/321779

    Thing sends every freaking 10 seconds.. Wonder if any of the beta's after posted that fixed it? On 1.14 something now.. going to remove the acl on the switch and see ;) fingers crossed.

    edit: Arrggghhh still chatty kathy..

    13:22:57.575617 IP 192.168.9.10.42339 > 239.255.255.250.1900: UDP, length 101
    13:22:57.575635 IP 192.168.9.11.45988 > 239.255.255.250.1900: UDP, length 101
    

Log in to reply