Snort clearing block hosts ahead of schedule

vbentley

My Snort installation is configured to block hosts for 28 days and collect ruleset update daily. This has been working fine for all of the 2.0.x releases. However, since doing an upgrade to 2.1 from the console, Snort appears to clear blocked hosts from the list daily. Perhaps it is doing this when the updated ruleset is loaded.

fragged

Snort's block list gets cleared on filter reload.

bmeeks

@fragged:

Snort's block list gets cleared on filter reload.

fragged is correct. Snort uses the pf block table mechanism within pfSense to actually perform the host blocking. Snort inserts the offending IP address into the table "snort2c" and then forgets about it. There are internal pfSense processes that take the IPs in that table and then do the actual traffic blocking. Anything that causes pfSense to clear out the pf tables (including "snort2c") will cause the Snort blocked hosts to be purged.

There are some internal processes that can happen in pfSense (filter_reload is one) that result in the snort2c block table getting cleared. Snort has no control over this. There is a cron job created by the Snort package upon installation that runs every 5 minutes to test entries in the snort2c table against the configured "expire time". If an entry's block time has expired, the cron job will remove that particular entry from the snort2c table and that blocked host will be cleared. This is a different process than filter_reload, though.

Bill

vbentley

I was aware that whenever I rebooted pfSense that Snort would clear the blocklist. My uptimes were typically measured in months on 2.0.x and I would regularly see 200+ hosts in Snort's blocklist with some hosts having multiple log entry dates for successive intrusions a week or so apart. I was able to identify persistent threats from the log.

I regularly checked the rules update log and it was working daily as expected. Some days there wouldn't be an update to collect but on the whole, I could expect to see more than a 100 hosts in my blocklist whenever I viewed it. Since, upgrading to 2.1 there is usually less than five hosts being blocked and sometimes none. I just checked the blocklist and I have four today.

I have temporarily changed my auto-update interval from 1 day to 7 days to see if there is any difference.

If this is now the normal behaviour of Snort of pfSense perhaps I need to develop a package that will look for persistent threats and maintain a separate blocklist that can be reloaded as a firewall rule or pfBlocker rule.

Hmm… I wonder if it's pfBlocker clearing the blocklist? I recently re-enabled pfBlocker after having it switched off for some time.

vbentley

Well, I can't work out what's forcing a filter reload. I have disabled rule updates and set remove blocked hosts to never in an attempt to eliminate these processes from my investigation. Something is forcing a filter reload. The system log doesn't state why the filter was reloaded but at least i can see when it happened.

bmeeks

@vbentley:

Well, I can't work out what's forcing a filter reload. I have disabled rule updates and set remove blocked hosts to never in an attempt to eliminate these processes from my investigation. Something is forcing a filter reload. The system log doesn't state why the filter was reloaded but at least i can see when it happened.

Nothing in Snort will cause a filter reload. So things you have done inside Snort itself are of no benefit. Things that cause a filter reload are changes to firewall rules, Aliases, change in WAN IP, rule schedule changes, etc. Again, Snort is totally out of the picture here.

Its blocking table just happens to be a victim of the filter reload process.

Bill

vbentley

My firewall rule, aliases and schedules are static now and I rarely need to make changes so it's probably OK to eliminate them. It could be WAN IP changing but the log states that dynamic dns hasn't noticed a change. I have a timestamp in the log for the last filter reload so I guess that's the starting point.

I have two gateways configured. I checked the log and it looks like 'apinger: ALARM:WAN(nnn.nnn.nnn.nnn) *** loss ***' is the culprit. It looks like I am regularly getting momentary packet losses to the monitor IP on each gateway at least twice a day and at different time to each other.

Thanks for your help.

I'm seeing this behavior as well. Snort IS clearing blocked hosts ahead of schedule. I fully understand that the clearing has nothing at all to do with snort and is relying on other mechanisms, and in no way and/or shape and/or form am I implying that snort is broken, but on 2.0.3 it behaved exactly as it was intentioned. A filter reload did NOT cause the blocked hosts to be cleared. A blocked host remained on the blocked tab even through multiple rule changes, manual filter reload, CARP failover events, etc. etc. It was only removed when the firewall rebooted or when it reached the snort ban time limit.
As far as I can tell, something else broke during the upgrade to 2.1, since I'm also seing Code 2: Invalid return payload: enable debugging to examine incoming payload on the system logs, along with traffic on the sync interface being blocked although it shouldn't.
Will continue fighting with it and see if I can find any more info.

vbentley

I extended the gateway monitoring intervals and this has helped reduce the number of filter reloads, but it is still annoying that the block list is being cleared more frequently than I want.

I have noticed that Snort does show some returning intruders in the current block list with the dates of previous intrusion attempts going back to before the 2.1 update. So, past intrusions are not completely forgotten, just not actively blocked until they try again.

bmeeks

@vbentley:

I extended the gateway monitoring intervals and this has helped reduce the number of filter reloads, but it is still annoying that the block list is being cleared more frequently than I want.

I have noticed that Snort does show some returning intruders in the current block list with the dates of previous intrusion attempts going back to before the 2.1 update. So, past intrusions are not completely forgotten, just not actively blocked until they try again.

Those "past intruders" are coming from the Snort logs. These are maintained separately from the "blocked table". The block table is named "snort2c". An internal, periodic pfSense process is clearing the snort2c table and thus removing Snort's blocked IP addresses. One of the pfSense team members has committed to take a look at fixing the problem. It was introduced in teh 2.1 code base.

Bill

sebna

@vbentley:

I extended the gateway monitoring intervals and this has helped reduce the number of filter reloads

Could you tell me where is the option and to what value have you changed for optimal resoults?

What I have noticed that it was working pretty well for me (300+ blocked) until I did two things. Enabled Pfblocker with good few lists and also switched snort to "Use IPS Policy". I will try to test in coming days which one of those two has caused the big change.

DB

I'm having the exact same issue with Pfsense 2.1 and Snort 2.6.0.

Do you have any update?

Thank you!

@sebna:

Could you tell me where is the option and to what value have you changed for optimal resoults?

What I have noticed that it was working pretty well for me (300+ blocked) until I did two things. Enabled Pfblocker with good few lists and also switched snort to "Use IPS Policy". I will try to test in coming days which one of those two has caused the big change.

I'm using pfblocker (with a few tens of thousands IPs in the table), all snort rules enabled (except FPs, as mentioned in setup threads) and a few (tens) of custom rules (most IPs blocked are by these). I'm regularly seeing 4500+ blocked IPs, so pfblocker and IPS Policy is (IMHO) unrelated, but today I got a huge packet loss (10%) on one network and the hosts were cleared. Just saying, the problem might have something to do with the gateway monitoring. Or the gateway monitoring just causes a reload and is completely unrelated.

It's an annoying bug but hopefully will get fixed.

edirob

I see there has been no update on this thread for 2 months, but I have had the same problem since the pfSense update to:

2.1-RELEASE (i386)
built on Wed Sep 11 18:16:50 EDT 2013
FreeBSD 8.3-RELEASE-p11

I will occasionally see a few blocks in a row, but they disappear almost instantly. I also use apinger to monitor 2 gateways that I load balance/fail over on. I have tried stopping apinger, to no avail. It is hard to judge the value I'm getting out of SNORT if I cannot see whether the alerts I receive are being blocked. This also makes it almost impossible to suppress or whitelist issues, since I can't really see if I need to. Fortunately, no unusual blocking activity is affecting my users - or is this: unfortunately, my users are not being blocked??

Assuming apinger and the last pfSense update is the problem, what can the pfSense folks do to help us?

Thanks for the erstwhile awesome product! Rob…

edirob

Hey, SOLUTION time!

After reading a bunch of Bill's posts and what others of you have experienced I checked to see what ELSE might have been causing my filters to reload. It turns out apinger and gateway monitoring had nothing to do with it.

I had a broken installation of Squid 3 that looked like it was OK and always appeared to be running, but was causing very regular filter reloads (check your Status: System Logs: General tab for "Squid_Alarm[######]: Reconfiguring filter…")

Since my post less than 90 minutes ago I now have over 130 blocks in my Blocked tab! Wait. Is that a good thing?? :)

Rob...

bmeeks

@edirob:

Hey, SOLUTION time!

After reading a bunch of Bill's posts and what others of you have experienced I checked to see what ELSE might have been causing my filters to reload. It turns out apinger and gateway monitoring had nothing to do with it.

I had a broken installation of Squid 3 that looked like it was OK and always appeared to be running, but was causing very regular filter reloads (check your Status: System Logs: General tab for "Squid_Alarm[######]: Reconfiguring filter…")

Since my post less than 90 minutes ago I now have over 130 blocks in my Blocked tab! Wait. Is that a good thing?? :)

Rob...

There have also been reports that the constant filter_reloads and subsequent clearing of Snort's block table are fixed in the 2.1.1-PRERELEASE snapshots.

Bill

Ramosel

@jflsakfja:

I'm using pfblocker (with a few tens of thousands IPs in the table), all snort rules enabled (except FPs, as mentioned in setup threads) and a few (tens) of custom rules (most IPs blocked are by these). I'm regularly seeing 4500+ blocked IPs, so pfblocker and IPS Policy is (IMHO) unrelated, but today I got a huge packet loss (10%) on one network and the hosts were cleared. Just saying, the problem might have something to do with the gateway monitoring. Or the gateway monitoring just causes a reload and is completely unrelated.

I'm a believer in your pfBlocker/Snort setup schema. Care to elaborate on your custom rules?

My system is alerting and blocking better than ever but I'm in single digit percentages of 4500+.

Rick

@Ramosel:

I'm a believer in your pfBlocker/Snort setup schema. Care to elaborate on your custom rules?

My system is alerting and blocking better than ever but I'm in single digit percentages of 4500+.

Rick

Contrary to industry accepted practices and hundreds,nope thousands, nope trillions of RFC documents (and whole galaxy systems with their sys admins supporting their use) detailing why you should allow privileged ports to connect to privileged ports, I personally want to be alerted when something like that happens. Most reflection attacks work on a form of this, but then again the security industry leaders say "who are you to say that?". I have been known though to campaign for the industry wide banning of HTTP requests for numeric IP numbers (requesting a webpage off a server without a hostname), but then again nobody listens to me. Statistically speaking, based on previous decades, the industry will accept this at some point around the year 2049.

I'm sorry but with regards to elaborating on custom rules (flicks through Company policy) "I am not at liberty to discuss that" ;-)

Ramosel

@jflsakfja:

[I'm sorry but with regards to elaborating on custom rules (flicks through Company policy) "I am not at liberty to discuss that" ;-)
[/quote]

Thanks for trying…
Firewall rules that are proprietary??? Hmmm... I'm so glad I'm retired and don't have to put up with that level of BS any more. Don't get me wrong, I fully understand the need to keep specifics of internal ports and addresses close to the vest... but when the bean counters and lawyers view firewall rules as an asset to be coveted, hoarded and locked down... Sorry, I'm just from a time gone by when as an admin if you found something that worked you made it known within the community. IT was so much more efficient and FUN when it was run by folks that actually KNEW IT. That was then, this is now... and now is ME, ME, ME.

Rick

When you open up something to the community, with good intentions, but the vultures that be (all other "companies"* in your country) circle above your head, copying your every move and declaring they are using tomorrow's technology today (technology you happened to be using first, even before they even knew how to plug a computer into a power outlet,let alone switch it on), and constantly probing you for vulnerabilities (both metaphorically and literally speaking), one tends to get a bit "proprietary" and "locked down".

BTW, bean counters and lawyers have nothing to do with it. It's my call to declare it proprietary and a trade secret, with all the accompanying safety mechanisms that entails. I'm in the security industry after all, so by definition they are trade secrets ;-)

*example of one such said company: Transfered a client's domain to me, just about a month ago. His hosting plan run out just >before< the "Industry Leader with tens of years of domain transfer experience, and billions of domains transfered" insisted that to transfer a .com you don't need an EPP code. A month later the hosting plan still operates as if it was "forgotten" and left there for all eternity. Conclusion: If you don't bother removing plans you no longer host, then you don't even bother updating your servers anymore. Do the industry a favor and consider a career change before someone gets hurt by your negligence (you is directed to said "Industry Leader").