disappearing pings
-
Hello,
I have icmp echo requests arriving at my external interface which i believe should be making it through my firewall but are not. It seems that some remote ip addresses simultaneously get responses while others do not. My first firewall rule on this interface is allow icmp from anywhere but I can see the ignored packets with a promiscuous capture.
I upgraded to 2.5.0 on Tuesday and first experience this issue wednesday and then again today. The specific affected addresses on wednesday eventually resolved and now get responses. I am not seeing any errors on the interface. I tried reloading the filter table and the issue persists after that. I would be grateful for any advice or assistance.
-
@dmd6tx If you can see the blocks in the firewall log, does it say what rule is blocking them if you click on the red X?
-
Guess would be floating rule - say a geoip/list rule via something like pfblocker.
Or could be bogon rule on the wan? Or are you running IPS?
-
I'm Not running IPS or pfblocker.
No floating rules are in place.
I do have the bogon rule in place but at one point I disabled it, reloaded rules and was still having unanswered pings
Are the firewall rules evaluated in order, and do they not short circuit further evaluation (outside of the match action on floating rules) ?
In this case my first rule on the WAN interface is accept any ICMP from any source to WAN address, is it possible for subsequent rules to have an effect in this case?
Thanks!
-
@dmd6tx said in disappearing pings:
is it possible for subsequent rules to have an effect in this case?
No... So your saying pings are not being answered - but the blocks are not being logged either?
Please post up the sniff of this traffic your saying is not being answered, but not logged.
Could you post up this rule please... For example here is mine..
Notice I icmp allowed there on the top.. have never seen anything that couldn't ping my wan..
And I have a couple different external monitoring sites pinging me - that ping from different source IPs all the time. .from all over status cake and uptime robot. That I get alerted on if don't answer
-
This has been intermittent for me, I have 4 specific remote IP addresses over a couple of days where i was able to verify that they could not ping and that the pings were getting to me but in each case the issue resolved after a couple of hours. I dd not think to check the firewall logs at the time. I'm going to enable logging for pass as well just to clarify what is going on if this crops up again. I also use uptime robot and it did not go down through this time last week, it was only a few discrete addresses that seemed to be impacted.
I'll come back with those screen shots in a bit.
Thanks!
-
Was the ips from some known tool like status cake or uptime robot?
You sure it was a echo req, maybe it was some sort of other icmp packet? But if the packet didn't meet your rule parameters for some reason. Then it should of been logged by the default block rule.
-
this was from manual invocation of ping utility on windows or debian terminal.
Sadly I'm not retaining logs long enough to look back to last week, but I'm going to go up the log size / log rotation settings.
Here is the top of my current WAN rules:
-
Well can you duplicate with your remote host pinging, while you do a sniff? And then look at the logs right away?
-
I can't always duplicate this issue on demand but it did crop up this morning again so I was able to dig in a little bit.
Yes the packets are getting logged as blocked, here is a line from the filter log:
Mar 16 13:19:15 <name> filterlog[13681]: 22,,,1000000400,em0,match,block,in,4,0x48,,53,53753,0,none,1,icmp,84,<remote ip> ,<my ip> ,request,27751,26164
Is 1000000400 the default deny rule?
Here is a packet capture from the same approximate time:
-
@dmd6tx Default Deny for me shows:
The rule that triggered this action is: @5(1000000103) block drop in log inet all label "Default deny rule IPv4" -
Oh I see how to get that description from the gui by hovering
here is the explanation for my drop:
A quick google search seems to indicate that this is associated with rate limits and I do have rate limits enforced on a different port for all of the effected IP addresses...
-
@dmd6tx said in disappearing pings:
Oh I see how to get that description from the gui by hovering
here is the explanation for my drop:
A quick google search seems to indicate that this is associated with rate limits and I do have rate limits enforced on a different port for all of the effected IP addresses...
I think that'll give you the direction you're looking for. Rate limits or maybe AV set up in Squidguard. I think that's the only place that has virusprotection included since your logs show "virusprot". Unless that's what you call your rate limit. I'm not sure since I've never used rate limiters.
-
I'm not using squidguard right now but I am using max-src-conn-rate and max-src-conn-rates in a differernt rule for each of the addresses I've had this problem with. It did not occur to me that the rate limiting would apply to all communication from the source. even if this is the case though it seems like maybe the block is lasting longer than I have configured in max-src-conn-rates. I will have to look closer in that. And maybe I can use this to trigger the issue now anyway. Thanks!
-
Ok so I set up a test, verified that I can ping my WAN ip from outside then triggered a max-src-conn block from the same ip on another port. After that I cannot ping.
The disturbing part is that the block is not expiring on schedule. max-src-conn-rates is 30 seconds but Traffic to the original port and icmp is still blocked over half an hour later. I went away and had lunch in the meanwhile in case prodding it extends the block. Does anyone have ideas on where to go troubleshooting this?
Thanks!
-
This post is deleted! -
OK after reading around some more, I see that I didn't realize the full implications of the options I am using here.
I see that I can manually delete this affected IP from diagnostics => tables => virusprot and that the rules there are supposed to persist for an hour.
I see there is an open feature request from 2011 for more control around this functionality but I take it that that is not going to happen.
Thanks for helping me work through this.
Is there any supported method to limit the rate of new connections without hour long global blocks for users who exceed the threshold?
-
off the top of my head no - I have never had need to play with any of the rate limiting stuff.
If was going to block some IP - it would be a perm block ;)
The only thing I could see it might being something I would look into would be the ntp I provide to the pool.. But have never seen anything really do anything that would require me to rate limit someone, etc. The amount of ntp traffic is low..
compared to my plex
But I would take it as a win that you figured out what was going on...
-
@johnpoz said in disappearing pings:
I would take it as a win that you figured out what was going on...
Yes, thanks to you and Stewart to pointing me in the right direction. I certainly learned some useful things along the way too.
I do have to say that this virusprot arrangement feels like a violation of the law of least surprise. I see now that it is documented but I guess I thought i understood rate limiting well enough to skip the fine print. Oh well.
