Link-local address flooding logs
-
@stephenw10 I tried doing a capture, but a little difficult to do for the following reason:
- The em0 is a WAN interface where the traffic seems to be originating from
- The vmx3 is a parent interface has six subinterfaces VLANs/subnet ports tied to it. The firewall log shows the parent interface and not the subinterface where the problem seems to be coming from.
-
As long as em0 is assigned you should be able to capture on it. That will give you the MAC and you can go from there.
If you're seeing that traffic on the parent interface it must be assigned also. Do you have a switch stripping VLAN tags incorrectly maybe?
I have a TP-Link switch that does that with broadcast traffic. I stopped using it for vlans!
Steve
-
BTW, there's no need to obscure those addresses. They're RFC 1918 private addresses, which are not routed over the Internet. All you're doing is making it more difficult for us to help you.
-
@rsaanon said in Link-local address flooding logs:
@marvosa Your analysis is quite impressive. Good forensic work!
Here's what I can confirm:
-
I do have a QNAP that has 4 ethernet ports. All ports are 802.1ad aggregated with static IP. So, there's no possibility of auto-configuration of 169.x addresses.
-
I do have Plex, but not virtualized (ie: not running in a container or a vm).
I mentioned Plex possibly being virtualized because some of the traffic is being blocked on your vmx3 interface, which tells me something is virtualized... I assumed Plex, but maybe it's your PFsense?
-
-
@jknott I prefer to keep my internal addressing scheme private and thus the reason for masking the rfc1918 addresses I have on my network. Also, I don’t see how masking the address makes it difficult for someone to help. I appreciate you taking time to chime in on the topic.
-
How exactly are you seeing that on your pfsense WAN and lan side interfaces at the same time? So you have no layer 2 isolation it seems.. Misconfigured switch vlan settings?
Why do you have your nas multihomed? What exactly are you trying to accomplish with such a setup?
-
@johnpoz Can’t explain why I am seeing the 169.x on two interfaces at the same time.
The WAN/em0 is configured for VLAN 2 as required by the ISP going into a Cisco managed switch. Internet connectivity is good & stable. Internally, the LAN is segmented in to multi-broadcast domains each serving a particular purpose (eg: video, SAN, etc). The NAS is split into two 802.1ad aggregates, one for general access and the other for iscsi connectivity.
-
Have you run the packet capture yet? I can see no reason why you wouldn't be able to do that and it will show the device causing this immediately.
If you run it on both interfaces it will also confirm if the traffic really is arriving from the same device on both. In which case you have something misconfigured with those VLANs.Steve
-
Yeah take all of 10 seconds to run a sniff and get the mac address of what is sending that out.
-
@stephenw10 I wished the firewall logs included the particular subinterface that showed the 169.x broadcast; instead, you see a parent interface in the log. So, for example, vmx3 has 6 subinterfaces and I don't know which particular subinterface the problem is originating from. To troubleshoot, I'd have to connect to each subnet/subinterface and capture packets.
I did manage to capture traffic on one of the sub-interfaces. This particular 169.x entry was originating from an HD HomeRun Tuner. What's very peculiar is that this device has a DHCP allocated IP address (ie: no auto-configuration of 169.x address). Yet it's broadcasting to 169.x domain with the source IP being 169.254.100.100. BTW, the tuner works fine as it has IP connectivity to the outside. The tuner does not allow ssh connectivity into it but does have a web administrative page that I'm able to see. There no mention of 169.x address. All looks well with the Tuner network configuration. For further troubleshooting, I took this device offline and did a packet recapture. As expected, I did not see log entries from the MAC address of the Tuner; however, I continue to see 169.254.100.100 from another MAC address that pointed me to the QNAP. The QNAP is configured with two static IPs (172.24.16.x for general access and 10.56.1.x for iSCSI) and therefore has no auto-configured 169.254.100.100 address; however, ssh'ing into QNAP, I shockingly see:
mgmt0 Link encap:Ethernet HWaddr 00:08:9B:xx:xx:xx inet addr:169.254.100.100 Bcast:169.254.255.255 Mask:255.255.0.0 inet6 addr: fe80::208:9bff:feee:8e46/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2760 errors:0 dropped:0 overruns:0 frame:0 TX packets:34665 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:165682 (161.7 KiB) TX bytes:3009614 (2.8 MiB)I have no idea why the QNAP has the mgmt0 interface.
I'll troubleshoot more as time permits and post any updates here.
-
You will have to look into your HD tuner thing.. But many devices be they have IP or not will look for stuff via link-local... I had a direcTV wireless bridge that did that - pissed me off to be honest as well.. Not a fan of NOISE for no reason.. You have an IP - if you want to search for stuff then use your IP not link-local ;)
So HD tuner thing was looking for plex server via link-local.. Why not look on the network its actually configured for broadcast address? Sometimes I think the people that write the code for these things don't actually think it through..
Why would your Plex server answer a link-local broadcast if actually has an IP.. Is the device also sending out broadcasts to the network its on broadcast address?
-
@johnpoz Agree! Doesn't make sense why any device/application would use link-local address when none of the interfaces on the device itself has a link-local address. Go figure!
-
Is there a packet capture package for pfSense? It would be much easier to capture packets directly on the firewall instead from a client that would need to be connected to the appropriate subnet.
-
It's built in. Diagnostics > Packet Capture.
That would explain your not capturing until now.
It's interesting that you're seeing those blocks in the parent interface of the VLANs. To me that points to something incorrectly stripping the tags off that broadcast traffic.
Steve
-
@stephenw10 A quote is in order
"I've always thought that when they say ignorance is bliss, the converse to that is that knowledge is hell. The more you know, the bleaker things can get."Thanks for pointing out the built-in capture tool.
It's even more interesting that em0/wan (among other interfaces) is showing the 169.x broadcast. All nodes on the internal network function as purposed (ie: no service/connectivity issues). It's just that too much 169.x noise is being generated and the firewall is having to actively block those across different interfaces. It would be easy (& lazy, might I add) for me to add a firewall rule to not log those 169.x packets, but that is not the approach I prefer. I'd be surprised if there are any issues with vlan tag stripping as all switches on the network are cisco switches that are running the latest f/w.
I'll have to dig deeper..
-
Where I've seen similar things it was because the native vlan was always present on all port and could not be removed. Broadcast traffic was sent to evety port regardless of VLANs configured. Crazyness! It functions as an unmamaged switch now and even as that it's failing.
Steve
-
yeah the tplink low end smart switches are like that... the tl-sg108e and 105e did not allow you to remove vlan 1 from ports... So really its just JUNK!!
They finally put out a firmware that was suppose to fix this for their v3 hardware, but v1 and v2 got no love. And not sure if the v4 hardware works nor not.
If your running any of this through a vm sort of switch - like esxi, and don't set the vlan id to say 4095 then those vswitches will strip tags.
BTW you can also just shell into pfsense and run tcpdump directly, -e will show you the vlan stuff.. If you sniff on the parent interface you should be able to see all the traffic with their tags, etc.
example
tcpdump -ni igb2 -eWill show you traffic that is on igb2 along with any vlan traffic on that interface
10:37:55.166622 02:11:32:21:6a:72 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60: vlan 5, p 0, ethertype ARP, Request who-has 192.168.5.23 tell 192.168.5.22, length 42 10:37:55.205238 02:11:32:21:6a:72 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60: vlan 6, p 0, ethertype ARP, Request who-has 192.168.6.17 tell 192.168.6.22, length 42 10:37:55.217230 02:11:32:21:6a:72 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60: vlan 8, p 0, ethertype ARP, Request who-has 192.168.8.15 tell 192.168.8.22, length 42 10:37:55.237575 02:11:32:21:6a:72 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.2.96 tell 192.168.2.22, length 46There you can see vlan traffic on the parent and also non tagged (native) traffic..
-
@johnpoz Thank you!! tcpdump is exactly what I need.
Here's a snip of the capture:
---------em0 capture------------ listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes 11:50:11.908185 00:08:9b:ee:xx:46 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 63: 169.254.100.100.45345 > 169.254.255.255.32414: UDP, length 21 11:50:11.914948 00:08:9b:ee:xx:46 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 63: 169.254.100.100.42249 > 169.254.255.255.32412: UDP, length 21 11:50:12.328153 00:08:9b:ee:xx:46 > 01:00:5e:7f:ff:fa, ethertype IPv4 (0x0800), length 136: 169.254.100.100.47100 > 239.255.255.250.1900: UDP, length 94 3 packets captured 732 packets received by filter 0 packets dropped by kernel ---------vmx3 capture------------ listening on vmx3, link-type EN10MB (Ethernet), capture size 262144 bytes 11:51:02.722666 00:08:9b:ee:xx:46 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 92: 169.254.100.100.137 > 169.254.255.255.137: NBT UDP PACKET(137): QUERY; REQUEST; BROADCASTSource device for captures on both interfaces is QNAP, albeit em0 capture shows destination to be 169.254.255.255 with .32412 & 32414 ports. Ports 32412/4 are Plex discovery related. The vmx3 related capture is NetBIOS.
On QNAP side, I do not know what purpose the mgmt0/169.x serves and therefore do not want to delete that interface. I did delete the route entry for 169.x on QNAP that was pointing to the default gateway,
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 mgmt0This way no 169.x leaks out to the Internet Gateway. This didn't seem to make any difference.
-
did you use -e, not seeing any tagged traffic on em0.. Did you say that was your WAN... How is that traffic getting on your WAN vlan?
As to what mgmt0, is I would get with qnap support. If I had to guess its some sort of management vlan use.. Which maybe you have not setup?
-
@johnpoz I ran the command so it filters only the src ip of 169.x (tcpdump -ni em0 src 169.254.100.100 -e)
As for administrative vlan, I have a non-default (ie: vlan 111) that's used for administrative purposes.
