Questions about log messages
-
Add a note to that bug report.
It's more of an issue because sshguard spams the system log at rotation IMO.
-
@stephenw10 said in Questions about log messages:
Add a note to that bug report.
It's more of an issue because sshguard spams the system log at rotation IMO.
I updated that bug report and created another one for the mangled link-local addresses.
https://redmine.pfsense.org/issues/14692
-
I have an update on the mangled link-local messages. I updated the bug report Mangled link-local addresses are being logged, but I'm following up here to get more visibility.
When I originally posted about this problem, I was only seeing messages being received from the WAN interface toward the probe on the LAN interface.
Since then, I confirmed using Wireshark that the actual messages being received on the WAN interface were not mangled, so they must be getting mangled in FreeBSD / pfSense.
I also noticed similar messages being sent to other addresses on the LAN. I unfortunately don't have any examples of these messages.
I also noticed messages being sent from a Pixel mobile phone toward Google and WhatsApp. I posted examples of these messages in the bug report.
The inbound messages are all mangled the same way: fe80:5:: as opposed to fe80::.
The outbound messages are all mangled the same way: fe80:6:: as opposed to fe80::.
The messages are not random, with respect to time or address, so something must be doing this.
Does anyone have a suggestion of anything I can do to troubleshoot this?
-
@bimmerdriver said in Questions about log messages:
The inbound messages are all mangled the same way: fe80:5:: as opposed to fe80::.
The outbound messages are all mangled the same way: fe80:6:: as opposed to fe80::.
Both
fe80:5::/128(i.e.,fe80:0005:0000:0000:0000:0000:0000:0000) andfe80:6::/128(i.e.,fe80:0006:0000:0000:0000:0000:0000:0000) are valid IPv6 host addresses and are within thefe80::/10link-local reserved address block (i.e.,fe80:0000:0000:0000:0000:0000:0000:0000-febf:ffff:ffff:ffff:ffff:ffff:ffff:ffff). -
@bimmerdriver to continue with what @tinfoilmatt was pointing out - both fe80:5 and fe80:6 are valid link-local.. But pfsense is not blocking it because its not on the same fe80:X address - but the fact that you can not route a link-local address to a GUA..
link-local address space fe80::/10
If you have some device that wants to talk to GUA address, it would need to be coming from a GUA address
If you assigned a ULA (fc00::/7), you could do some natting and route it and change it GUA.. but link-local by designed to only be used locally on the same network.
If you know what is generating the traffic - it most likely does not have a gua to use, and is stupidly trying to talk via its link-local which is never going to work.
edit: oh I see your on a pixel, so android - this is bad coding on their part.. It is not possible to talk to a gua from a link local, unless maybe the gua was actually on the same local network, but it sure isn't going to route..
-
I agree with @johnpoz's diagnosis of
dropdue to link-local source destined for globally routable unicast.This appears to be the root issue going back almost a couple years now (the improperly configured system logging aside)? And the subject of a Redmine?
@bimmerdriver, if I understand, you believe pfSense is 'mangling' your packets because of the "
fe80:5[…]" and "fe80:6[…]" part of the IPv6 addresses you're observing as source addresses in these log messages. But these are valid link-local addresses.Remember that the second-part of a link-local IPv6 address can sometimes help identify which LAN host is sending these packets. Meaning the "[…]
02:61b6" part offe80:5::1cce:5fff:fe02:616b, and the "[…]xx:3a14" part offe80:6::e479:xxxx:xxxx:3a14of these two (valid) link-local addresses.In many cases, by-default a device will use the MAC address of an attached NIC as the final 48 bits of the 128 bit IPv6 link-local address it creates for it. So those LAN hosts may be identifiable if you observe any with the MAC addresses "
xx:xx:xx:02:61:B6" or "xx:xx:xx:xx:3A:14". -
@tinfoilmatt yeah I don't think many devices do that any more - but if you should be able to see the mac address of these link local addresses in the NDP table.. There was just recently a thread where someone was trying to track down a device sending dhcpv6 on his network.. NDP table should give you the mac, or if pfsense is seeing the traffic you can do a packet capture and get the mac address.
https://forum.netgate.com/topic/197269/unknown-dhcp-ping
They were trying track down what was sending it - which turns out was the hardware his pfsense was running on ipmp port plugged into the same switch so he was seeing the blocks in his firewall lan log.
-
Ok so you have public IPv6 addresses on your internal devices and most of this traffic is from some WAN side devices sending traffic to those internal devices.
That traffic is unroutable it should never be sent by those devices.
Do you actually see that traffic in a pcap on WAN? With the correct localhost source address?
And now you are also seeing this outbound in a pcap on LAN?
-
Hi All,
Thank you for the replies.
I understand link-local messages are not routable. There is no need to keep restating this.
When I originally noticed these messages in the log, they were all fe80:5::X addressed toward the Atlas probe that's connected to my LAN. I used Wireshark on the WAN and found that they were all originally fe80::X, yet they were appearing in the log as fe80:5::X, hence why I referred to them as "mangled" either by FreeBSD or pfSense. If they are fe80:5::X on the WAN, yet they are being reported in the pfSense log as fe80:5::x, then the bits in the address must have been changed in FreeBSD or pfSense.
I later noticed that there were occasionally some of these inbound messages to other addresses (not the probe). I unfortunately don't have any of them captured.
I also later noticed that there are occasionally some of the logged messages are originating from a Pixel phone (4a) on my LAN, with the destinations being Google or WhatsApp, based on whois. I don't have any of them captured. I searched on the internet and found some instances of Android phones doing this, so I will disregard these messages.
However, I think the incoming messages must have a different explanation. I've had the probe since 2019 and the messages only started to appear a couple of years ago when I first reported it.
-
@bimmerdriver please show a packet capture of this traffic on your WAN.. just because your seeing traffic on your wan and lan that seems sim doesn't mean its the same traffic..
There is no pfsense would send in unsolicited inbound traffic on wan from a source IP of link-local to some internal IP and change the source IP and send it out your lan.
Could be return traffic from dhcpv6 server on your wan.. Could be all kinds of anything traffic.. But pfsense sure isn't going to forward traffic to your lan and change the source IP to its link-local address. Please show a packet capture on your wan an lan at same time showing this traffic. Or the state table - if pfsense is forwarding traffic from anything there would be a state.
But its possible to see all kinds of NOISE on the wan side.. you have no idea what either the isp or other users on your segment are spewing out.
-
Ok so the main concern here is that the logged source IP doesn't match what is seen in a pcap?
Accepting that some rogue devices are sending traffic incorrectly that pfSense can do nothing about. Except logging the failure.
-
@stephenw10 Sorry for the slow reply. I ran wireshark on the WAN and filtered for fe80:5. There were no instances of it. However, there were instances of the messages in the log. I'm trying to understand how this could happen, because it doesn't seem it should be possible. It's not like there should be a lot of "rogue devices" on the WAN. My internet service is GPON, so AFAIK, the device on the other end of the fiber is an ISP router.
-
@bimmerdriver said in Questions about log messages:
other end of the fiber is an ISP router
Does this ISP router have other devices connected to it ? By cable or Wifi ?
-
No he means the remote end of the GPON link at the ISP.
Which definitely does have other devices connected to it. I wouldn't normally expect to see layer2 traffic from other customers. However I'd bet that's what it is. I don't (yet) have a GPON connection to test that.
-
@stephenw10 said in Questions about log messages:
I wouldn't normally expect to see layer2 traffic from other customers.
yeah would hope your isp would filter this.. But the one I notice is just loads and loads of arp traffic - from all bunch of other L3 networks as well - isp is clearly running multiple L3 on the same L2 and they are not filtering arp that is for damn sure.
If they are not filtering that - I bet you there is other stuff as well.. But the traffic I have really noticed is arp, since its just a flood of it.
-
The fiber is connected to an ONT which has a single gigabit ethernet interface. There is a direct ethernet connection to the ISP router. The hyper-v server running pfSense is connected to a bridged port on the ISP router using a dedicated NIC.
Within the hyper-v server, the NIC being used for the WAN has a virtual switch. 2X pfSense plus 1X OPNsense are connected to it. The second pfSense and the OPNsense are both completely virtual. I can be sure the traffic in question is not coming from the virtual pfSense or OPNsense VMs, because it happens even if both are shut down. Wireshark was connected to the same virtual switch that is being used for the WAN.
I spoke with a very knowledgeable tech from the ISP and he thinks it should be fine to put a switch between the ONT and the router and then connect my hyper-v server to it. This would bypass the ISP router completely. I don't think this will make any difference, but it would at least eliminate the ISP router as a cause.
-
@bimmerdriver so your isp is going to provide multiple IPs? an ont isn't a nat router. Its not a gateway device that isp give you where it a modem/router combo. What is the model number of what your calling your ont?
-
Ah, that's not the scenario I was imagining. But, sure, if you ca remove the local ISP router entirely that's a better setup IMO.
-
@johnpoz said in Questions about log messages:
@bimmerdriver so your isp is going to provide multiple IPs? an ont isn't a nat router. Its not a gateway device that isp give you where it a modem/router combo. What is the model number of what your calling your ont?
The ONT is a Nokia G-240G-A. It's a media converter, not a router, and it also provides POTS. The ISP provides multiple IP addresses / prefixes. There is probably a limit, but it's more than I need / use. The ISP router, which currently has port 1 bridged, uses one. The other 2X pfSense and OPNsense use three more. Normally, the ISP doesn't "support" a switch being placed between the ONT and their router, but the technician I spoke with confirmed there is no reason why a switch couldn't be put between them.
-
@stephenw10 said in Questions about log messages:
Ah, that's not the scenario I was imagining. But, sure, if you ca remove the local ISP router entirely that's a better setup IMO.
The reason I haven't bypassed the ISP router using a switch is because according to the ISP it's "unsupported". If there is an issue with the IPTV service, they not only might refuse to touch it, but they might claim it's cause for a service call being billable.
The other reason I haven't bypassed the ISP router is because I have no reason to believe it's causing any problems.
