how to stop logging blocked LAN IGMP?
-
PMs with igmp.pcap sent with my thanks.
This afternoon I removed the two disabled IGMP pass rules, which I had set up to try to stop this logging, from pfSense and the mysterious logging still has not returned.
This seems to indicate the disabled (and even when enabled) pass rules were having no discernible effect on the logging of IGMP packets from a Roku box on the Ent network attributed to, at varying times, a non-existent or previously deleted rule, or, this morning after a reboot, the policy routing rule (!) of all things, which also was configured to not log.
This may be one of those things that is very difficult to catch and fix given it will probably happen on a production system that cannot be experimented with easily and given I don't know what triggered this to begin with. However, the three reboots seem to have cleared it for now, much like the OP discovered cleared his issue as well.
-
@Mission-Ghost I took a look at your pcap. Your Netgear (192.168.10.22) switch or WAP is operating as an IGMP querier, soliciting IGMP messages from hosts in the network. Nothing particularly unusual about that assuming you have the Netgear configured for IGMP snooping. I was curious to line this up with some of your error messages and went back to look at detail information in your prior logs.
Then, the sky fell. The Netgear has an address of 192.168.10.22. Looking at your prior posts, packets from address 192.168.10.22 are arriving on at least 4 different pfSense interfaces:
- 30_UTIL
- 40_CAM
- 50_ENT
- 60_GUEST_GM
Obviously, that should not happen, and is indicative of a significant issue. Either you have multicast packet flooding, or the VLANs are not configured/functioning as expected.
I recommend setting aside IGMP logging issues for a moment, and going back to validate the basic configuration of switch/WAP VLANs and pfSense interfaces. It's a good idea to start a new forum topic for that, with a link to this topic in your initial post.
-
Yikes, nice catch.
-
@dennypage Wow! Thank you for making the time and effort to examine the capture file. This is an interesting revelation.
I noticed the Netgear records in the capture file but did not recognize the meaning and importance of them. It does makes sense that the packets would arrive on all the interfaces: the Netgears are on trunk ports and host all six VLANs.
The Netgears are WAX610 WAPs. I'll go through them (there are four) and shut off or verify multicast functions are shut off.
Thanks again. Networking is so much more complex now than when I did it at work (Thick- and Thin-net!) 30 years ago. It's almost not the same thing at all anymore.
-
Yeah great catch @dennypage I knew something was off when was seeing that much multicast traffic from a roku - I see zero.. But thought maybe when he mentioned the hdhomerun device - was like maybe it does the multicast, I have played with them a tiny bit.. I had set one up for my son but he was on a flat network and it just worked.
Not really a fan of multicast - sure it has some great uses.. Hey if you want to send a video/audio stream to multiple devices at the same time - then yeah it rocks! But most of it is just noise, mdns, ssdp, llnmr - etc.. pure and utter crap noise that I want nowhere near my network. I have zero use for those protocols, and do everything I can to keep the noise at a min.. One thing I hate is opening up a pcap and seeing just noise, and more noise ;)
But great digging into details there - who was the guy that recently said we have to much free time <hahaha> clearly you have some - and fantastic that you help others and dig into a problem like that!! Impressive!
-
@Mission-Ghost said in how to stop logging blocked LAN IGMP?:
It does makes sense that the packets would arrive on all the interfaces: the Netgears are on trunk ports and host all six VLANs.
It does not make sense that packets arrive on multiple interfaces from a single IP address. The packets should come from a VLAN specific IP address in each VLAN.
-
@dennypage Would it make more sense given each of six SSIDs has an Enable/Disable Multicast/Broadcast setting and associated other settings on these APs, and each SSID goes through their own VLAN each of which has an interface on pfSense?
-
The "blocked" logging from 50.2 returned this morning without any pfSense changes:
Rule *3276 is my Entertainment VLAN pass rule for policy routing:
I have a IGMP pass rule at the top with Allow IP Options enabled and logging disabled:
-
@Mission-Ghost you may be encountering issues simply because the source IP address is not valid for the subnet defined on the interface. Fix the basic VLAN issues first.
-
@dennypage said in how to stop logging blocked LAN IGMP?:
@Mission-Ghost you may be encountering issues simply because the source IP address is not valid for the subnet defined on the interface. Fix the basic VLAN issues first.
Perhaps. Something I forgot to mention is that .10.1 is the pfSense router/firewall (which is also .50.1 on the .50 Entertainment VLAN), so while it showed up in the pcap it is the router, so while maybe it should have shown as .50.1, it's the same node as .10.1.
I spent time today shutting off all of the multicast functionality in the four access points. It turns out the HDHomeRun device needs multicast to function even on and within the same VLAN (.50 in this case) so I had to turn it all back on with the exception of IGMP Snooping, which was not required. Doing so enabled HDHomeRun to work properly within the .50 VLAN, which is sufficient. An IGMP pass rule is required in the .50 VLAN as well. I got occasional blocked log from non-existent or policy routing rules but they've stopped again for no apparent reason.
Thank you again for your help along with @johnpoz and @stephenw10. I appreciate the insights.
Setting all of this stuff above aside, what hasn't been acknowledged is that pfSense is, in this case, posting log messages, except when they're attributed to the policy routing rule, that can't be attributed to a rule the administrator can locate or do anything about and can't be stopped using documented procedures.
I think this is reasonable to call a bug and an investigation into why it could happen and making a fix would be worthwhile at improving the stability and trustworthiness of the product.
-
Spoke too soon. The mystery log messages have started up again while I was doing another pcap against igc0.50:
[.50.3 is the other Roku box (hard wired....50.2 is the WiFi Roku box)].
Rule 1761573276 is the policy routing rule for the .50 VLAN, which is set to not log and is below the IGMP pass rule. I modified the IGMP pass rule (with the Allow IP Options enabled) to match on the log source and destination addresses specifically, the log still comes out from the policy routing rule.
This just seems buggy. I'll have to reboot again when I get an maintenance hour and see if that stops it again.
-
@Mission-Ghost said in how to stop logging blocked LAN IGMP?:
Setting all of this stuff above aside, what hasn't been acknowledged is that pfSense is, in this case, posting log messages, except when they're attributed to the policy routing rule, that can't be attributed to a rule the administrator can locate or do anything about and can't be stopped using documented procedures.
I think this is reasonable to call a bug and an investigation into why it could happen and making a fix would be worthwhile at improving the stability and trustworthiness of the product.
It's not my call, but in my opinion it's a bit premature to waive the "there's a bug" flag. You have packets arriving on the wrong interface, with source addresses are outside the local address range of that subnet. By default, this alone will trigger logging.
Have a look at the checkbox options on the options Status / System Logs / Firewall (wrench icon). One of those options may affect what you are seeing. Also read here. @stephenw10 may have additional suggestions.
This really should be broken off into a new thread. Be sure to include information about your interface configurations and rules.
-
@dennypage Thanks again for the helpful discussion.
All of the check boxes for logging bogons etc. under Status/System Logs/Firewall > Manage Firewall Log are unchecked, and unchanged from prior to this issue (I have not set the system to do Firewall logging which is why this issue is noticeable. The firewall log should be empty on my system all the time.)
I re-read the section you pointed me to (I have read it before--but I did not have anything logging prior to this mystery so it was unneeded before). So to be scientific about it, I put a block-all-no-logging at the end of the .50 VLAN rule set, to no effect. First I put a block no log rule to 192.165.50.255. When that didn't work, I changed it to the one shown below to block all no-logging to the IGMP broadcast address. The mystery logs continue to be posted from the prior policy routing rule that is being attributed to by the log records.
So I moved this block-no-log rule above the policy routing rule, to no effect.
Still getting log records attributed to the policy routing rule.
To be thorough and consistent with the documentation example you referenced, I put a block-no-log rule on the Starlink WAN rule set to no effect (I set it to log, first, then to no log, neither one made a difference as inbound Starlink traffic has not been logging anyway):
Because overkill is underrated, I also then copied the rule to the TMHI backup-internet interface, to no effect.
Of course, the problem isn't incoming from the Internet, but I want to demonstrate that I have no firewall logging enabled anywhere I can think of.
This issue is frustrating because I've done nothing to deliberately start this logging and have apparently no direct control over stopping it.
A couple of days ago I rebooted three times and the log entries stopped on the third reboot, until they restarted today, both for no apparent reason.
(Aside, I disagree this should be a separate thread; it's consistent with the O.P.'s problem and unresolved.)
If this isn't a bug, it's not clear why it's a feature!
-
@Mission-Ghost so I am not sure what you have going on.. But I found something (my iphone) that is sending out igmp on one of my wifi vlans its currently connected to.. It is sending it out every few minutes - I really hadn't noticed it before. So not sure if normally does it or not. But if you notice when I filter on igmp - there are some 378 log entries (all from my iphone 192.168.2.198) in the last 24 hours.
I set it to normal block - because there is zero anything pfsense is going to do with traffic sent to 224.0.0.2 that is for sure.. I can play with setting it to allow later. But for now blocking it, clearly stops it from showing up in the logs.
So you can see my default allow on my wlan was logging it, you can see previous logs going back to at least 21:23:15, and the last log entry at 21:27:59, you can see my new rules loaded at 21:30:26.. There are way more than that - but no reason to show huge list of the same thing.
I no longer see any hits in the firewall for igmp.. And you can see my rule has triggered at least a couple of times via the states going up from 0/0. And you can see firewall is still seeing igmp traffic via packet capture I fired up - notice the time 21:48 in the packet capture.
Its getting a bit late here, but tmrw if the phone is still sending it out, be happy to change that to an allow rule. But this is really how simple it should be to prevent logging of that noise, another option would be just to allow IP options on your allow rule that is currently blocking the traffic. I will fire up a test of that as well.
You got something weird going on - you should never see traffic from one network(vlan) on a different interface other than that networks interface. So either you don't actually have isolation at layer 2, or you have something repeating the traffic into a different network/vlan.
edit: btw this is the only thing that seems to send igmp in my whole network, at least in the last 24 hours that is the only IP see in my logs. Or as far back as 5000 entries go back. And I really don't do much logging, I only log syn hits to my wan, and some specific ports on udp, and some specific rules. As you can see I have many networks, and lots and lots of devices.. Just looked an there are 41 different wifi devices connect to multiple different vlans(ssids).. And quite a few wired devices as well, and also vms and dockers running connected via wire.. It seems insane to me you should see that much igmp traffic - unless you were specifically using it for something - my rokus sure don't send any of it out. Maybe its your hdhomerun thing? But again you should only see that on the network that device is actually one.. That you seeing that much seems to me you have some sort of loop or rebroadcaster that is putting that traffic out on multiple interfaces.
edit2: before someone chimes in on my open rules on this wlan interface ;) This is my trusted wifi network, you have to have auth via wpa3 enterprise via eap-tls (so you have to have cert).. Only my phone and my tablet ever log into this network.
-
@johnpoz thanks for the detailed post. It gave me a couple of ideas but it's getting late here too so I didn't do too much. It's a fascinating, if frustrating, problem.
I created a blocking rule at the top of the 50 VLAN blocking IGMP with allow ip options to no effect. There are a few packets and kb showing up in the block rule but the logs keep coming from my pass rule. The pass rule is set to not log. I made sure all the pass rules in the 50 VLAN have allow ip options enabled, to no effect. The logs just keep coming.
I tried a similar fast floating block rule on all VLAN interfaces to no effect.
I unplugged the HDHomeRun box associated with all this starting (but not apparently causing it) to no effect.
So I finally tried going nuclear and put a block any protocol, any source, any destination with allow ip options at the top of the 50 VLAN rule set and the logs keep coming. From the PASS rule at the bottom of the list! No effect.
Theoretically, that block should block all 50 VLAN traffic. Full stop. Right? It shows zero states and a couple of kb.
Something funky going on...
I'll try reboot(s) again next chance I get, but that won't really diagnose what's going on even if it works.
-
@Mission-Ghost said in how to stop logging blocked LAN IGMP?:
50 VLAN blocking IGMP with allow ip options to no effect
There shouldn't be any reason to set that if your blocking igmp before it gets to your allow rule. my rule is just block igmp, I set no advanced options in the rule.
If you put in a nuclear block rule like any any at the top, keep in mind states would still allow traffic. But a block that would match for igmp wouldn't create a state. So yeah that should have stopped igmp traffic from getting to any allow rules that would of allowed it and said hey wait that is ip options set and shown the block log.
If you put in a nuclear rule at the top - and your still seeing log, kind of points to the rules are are not actually loading.
-
@johnpoz I haven't been clearing the state table, so that's something I should try. I always forget about states of the living dead. While it shouldn't wreck everything like a reboot I'll need to do it during a more quiet time.
I have been watching the rule reload to be sure it's done before checking the logs. I haven't read the whole rule reload output but maybe there's something in there.
This isn't an awful problem, but it adds wear to the SSD and fills up the firewall log with nothing useful and it should be something I should have easy control over. Coulda shoulda woulda.
-
@Mission-Ghost said in how to stop logging blocked LAN IGMP?:
fills up the firewall log with nothing useful
exactly - a full log, makes it hard to spot problems that could/should be addressed, etc.
And with seeing traffic on multiple interfaces that shouldn't be there - its clear your network is not running optimally that is for sure. In a proper setup you should never see such traffic.. There should be no way traffic from X network shows up as source into the Y network interface..
With that much multicast - I can't believe your wifi is running the best it could.. Multicast/Broadcast is not wifi friend.. This sort traffic runs at much lower speed (basic rate). Which slows down the whole wifi network..
If you looking for the best wifi - you would look to minimize/remove any multicast or broadcast that is not actually required. This is why many wifi AP have the ability to actually block multicast, or convert multicast to unicast..
While you might not notice it on a wired network if you have a ton of multicast - you will notice it on your wifi that is for sure.
-
Interesting end to this chapter: unbidden logging stopped early this morning when Starlink forced a reboot of our dish. At the same second pfsense killed the states during failover to the THMI backup.
-
@Mission-Ghost I would still look into why your seeing traffic on different interfaces, even if not logged. Just not logging noise doesn't mean its still not happening. You should never see traffic from vlan X as source into vlan Y interface.. Unless vlan Y interface was a transit/connector network and you had a router downstream that these networks were attached too.
And that wouldn't be multicast that is for sure.
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 25.07.1 | Lab VMs 2.8.1, 25.07.1
