Snort unblocks IPs when it shouldn't

ConfusedUser

Hmm…I'm not sure how exactly, but perhaps the fact 443 is generally encrypted SSL traffic comes into play somehow. You said Snort "sees the event" and logs an alert for it, but it just does not insert a corresponding block. Do I have that correct?

Yes, that is correct.
In my first post you can see the alert as visible in the GUI (only replaced my fixed public WAN IP). So the alert is working.
But the IP address (some Chinese 218.77.79.34) didn't show up in the block list.

Initially I thought that's related to the Snort rules but in my example the same rule (1:2402000) triggered an alert for another IP using another destination port (TCP 33462). This one got blocked. This port was closed on my firewall anyway but that shouldn't make a difference. As you wrote snort catches all traffic on the interface doesn't matter if pf blocks or forwards it.

Thanks,
Mike

ConfusedUser

Hi Bill,

I am sorry but it looks like I was wrong.
I have not seen 443 ever being blocked but I was clearing everything this morning and I was comparing the alerts with the block list after three hours. There are IPs for other destination ports that aren't blocked either (but should be). Please see attached screenshot. For whatever reason three of the source IPs have not made it to the block list (red frame in the screenshot).
A very good example is the 4th line from the top and the 2nd from the bottom (green marker in the screenshot). Same rule, same source port, same destination port, same destination IP but one is blocked and one isn't.

Btw, I have set the timeout to remove blocked IPs to 6 hours so everything in the list on the source side should be blocked. On the destination side this is one of my IP addresses.
Is there any way how I can trace what is happening there? I was checking snort.conf but I didn't find any documentation or source code for the "alert_pf" output (and even if I had found it… not sure if it would help me because I'm lousy programmer :) ). Anyway... I'm out of ideas really.

Thanks,
Mike

Clipboard01.jpg_thumb

bmeeks

ConfusedUser:

I believe I can explain what you are seeing with IP address 116.10.191.195. The same process may be at play with the others as well.

Remember the promiscuous mode stuff I mentioned earlier. Snort will continue to see packets even from blocked hosts. However, the packet filter firewall table that Snort inserts IP addresses into for blocking can only hold a single unique instance of a given IP. There is no reason to have the same IP in there multiple times as the block is on all ports from the given IP. So if a given IP address attacks you multiple times, but with a different port each time, you will see Alerts on the ALERTS tab for each probe, but there will only be one block representing the very first probe (and whatever port that probe might have used).

As for displaying things on the BLOCKED tab, there is a little bit of magic happening under the covers. See, when Snort inserts an IP into the block table in the packet filter, it can only insert the IP. It can't insert any reason. Thus just by looking at the block table you can't tell why the IP was put there. To try and show some context, when you open the BLOCKED tab Snort first grabs all the currently being blocked IP addresses from the <snort2c>table. You can view this table under DIAGNOSTICS…TABLES to see what IP addresses are in it. After loading an array with the currently blocked IPs, Snort then opens the alerts log file and reads the most recent lines from it. It tries to match up IP addresses in there with those in the block list and then does a sort of grouping sort. So what you sometimes see is a single IP address with multiple events listed for it in the ALERT DESCRIPTION column on the BLOCKED tab.

Bill</snort2c>

ConfusedUser

Hi Bill,

Thank you for the explanation. This is exactly how I expected to Snort to work with pfSense. But if I didn't misunderstand your reply it still doesn't explain the behaviour.
I'm not worrying too much about the alert view. Of course I was comparing against the snort2c table but this matches the alert view so everything seems fine in the GUI.

But… Not sure if you missed that part:
In my example that I marked in green in the screenshot there are two different source IPs (116.10.191.195 and 116.10.191.164).
So both should be blocked. But only 116.10.191.195 was showing up in snort2c where 116.10.191.164 wasn't. The rest was absolutely identical, same source port, same destination port, same destination IP, same rule.

Thanks,
Mike

bmeeks

@ConfusedUser:

Hi Bill,

Thank you for the explanation. This is exactly how I expected to Snort to work with pfSense. But if I didn't misunderstand your reply it still doesn't explain the behaviour.
I'm not worrying too much about the alert view. Of course I was comparing against the snort2c table but this matches the alert view so everything seems fine in the GUI.

But… Not sure if you missed that part:
In my example that I marked in green in the screenshot there are two different source IPs (116.10.191.195 and 116.10.191.164).
So both should be blocked. But only 116.10.191.195 was showing up in snort2c where 116.10.191.164 wasn't. The rest was absolutely identical, same source port, same destination port, same destination IP, same rule.

Thanks,
Mike

I'm sorry. I did misinterpret those two IP addresses. I don't have a good answer without more data to review. Of these two IPs (116.10.191.164 and .195), which one showed up in the block table?

Bill

ConfusedUser

@bmeeks:

I'm sorry. I did misinterpret those two IP addresses. I don't have a good answer without more data to review. Of these two IPs (116.10.191.164 and .195), which one showed up in the block table?

Only 116.10.191.195 was showing up in snort2c.

Do you know where I can find some documentation or source code for the "alert_pf" output that is used in "snort.conf"? Maybe that would help me to find the issue. I did a lot search but I couldn't find any sources or documentation.

Thanks,
Mike

bmeeks

@ConfusedUser:

@bmeeks:

I'm sorry. I did misinterpret those two IP addresses. I don't have a good answer without more data to review. Of these two IPs (116.10.191.164 and .195), which one showed up in the block table?

Only 116.10.191.195 was showing up in snort2c.

Do you know where I can find some documentation or source code for the "alert_pf" output that is used in "snort.conf"? Maybe that would help me to find the issue. I did a lot search but I couldn't find any sources or documentation.

Thanks,
Mike

Here is a link to the patch file for the Snort binary source code. This is from my old fork of the pfsense-tools repository. This particular file has not changed, though, since the official updated repository changed locations.

https://github.com/bmeeks8/pfsense-tools/tree/master/pfPorts/snort/files

The main code for the alert-pf output plugin is down at the bottom of the patch-spoink-integration-diff file.

Bill

ConfusedUser

Hi Bill,

I didn't find anything special in the output that I would immediately consider as bug but like I wrote earlier, I'm a lousy programmer (if I do programming then only vb.NET).

But… I found something in the package that is 99.9% a bug and I can re-produce it any time: The aging adds entries in /etc/crontab but never removes old entries. Maybe this is related to the issue I experience - not sure yet.
I was changing the aging in the snort/pfSense web GUI entries from "never" to "28 days", to "7 days", etc. See my /etc/crontab below:

SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin
HOME=/var/log
#minute hour    mday    month   wday    who      command
#
#
# pfSense specific crontab entries
# Created: April 29, 2014, 1:55 pm
#

1,31	0-5	*	*	*	root	/usr/bin/nice -n20 adjkerntz -a
1	3	*	*	0	root	/usr/bin/nice -n20 /etc/rc.update_bogons.sh
*/60	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 sshlockout
1	1	*	*	*	root	/usr/bin/nice -n20 /etc/rc.dyndns.update
*/60	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot
30	12	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_urltables
*/5	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_cron_misc.inc
*/15	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -t 10800 snort2c
30	11,23	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/www/snort/snort_check_for_rule_updates.php
*/30	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -t 21600 snort2c
2	*/1	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -t 43200 snort2c
2	0	*/2	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -t 2419200 snort2c
2	*/14	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -t 604800 snort2c
#
# If possible do not add items to this file manually.
# If you do so, this file must be terminated with a blank line (e.g. new line)
#

There are now five entries for expiretable snort2c. That can't be right?!

Thanks,
Mike

bmeeks

@ConfusedUser:

Hi Bill,

I didn't find anything special in the output that I would immediately consider as bug but like I wrote earlier, I'm a lousy programmer (if I do programming then only vb.NET).

But… I found something in the package that is 99.9% a bug and I can re-produce it any time: The aging adds entries in /etc/crontab but never removes old entries. Maybe this is related to the issue I experience - not sure yet.
I was changing the aging in the snort/pfSense web GUI entries from "never" to "28 days", to "7 days", etc. See my /etc/crontab below:
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin
HOME=/var/log
#minute hour    mday    month   wday    who      command
#
#
# pfSense specific crontab entries
# Created: April 29, 2014, 1:55 pm
#

1,31	0-5	*	*	*	root	/usr/bin/nice -n20 adjkerntz -a
1	3	*	*	0	root	/usr/bin/nice -n20 /etc/rc.update_bogons.sh
*/60	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 sshlockout
1	1	*	*	*	root	/usr/bin/nice -n20 /etc/rc.dyndns.update
*/60	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot
30	12	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_urltables
*/5	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_cron_misc.inc
*/15	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -t 10800 snort2c
30	11,23	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/www/snort/snort_check_for_rule_updates.php
*/30	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -t 21600 snort2c
2	*/1	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -t 43200 snort2c
2	0	*/2	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -t 2419200 snort2c
2	*/14	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -t 604800 snort2c
#
# If possible do not add items to this file manually.
# If you do so, this file must be terminated with a blank line (e.g. new line)
#
There are now five entries for expiretable snort2c. That can't be right?!

Thanks,
Mike

Yep…you are correct. This is a bug. I will fix it. Thanks for reporting it.

Bill

ConfusedUser

Hi Bill,

I currently got three virtual test installs of pfSense with snort on different IPs and all three keep blocking everything since I removed the cron entries manually. Probably that was the bug that I experienced with all test installs… I will keep monitoring for a few days.

Btw, make sure you read your PMs! :)

bmeeks

@ConfusedUser:

Hi Bill,

I currently got three virtual test installs of pfSense with snort on different IPs and all three keep blocking everything since I removed the cron entries manually. Probably that was the bug that I experienced with all test installs… I will keep monitoring for a few days.

Btw, make sure you read your PMs! :)

I did see you PM and replied. Thank you… :)

I will push a fix soon for the cron bug.

Bill

bmeeks

@ConfusedUser:

Hi Bill,

I didn't find anything special in the output that I would immediately consider as bug but like I wrote earlier, I'm a lousy programmer (if I do programming then only vb.NET).

But… I found something in the package that is 99.9% a bug and I can re-produce it any time: The aging adds entries in /etc/crontab but never removes old entries. Maybe this is related to the issue I experience - not sure yet.
I was changing the aging in the snort/pfSense web GUI entries from "never" to "28 days", to "7 days", etc. See my /etc/crontab below:
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin
HOME=/var/log
#minute hour    mday    month   wday    who      command
#
#
# pfSense specific crontab entries
# Created: April 29, 2014, 1:55 pm
#

1,31	0-5	*	*	*	root	/usr/bin/nice -n20 adjkerntz -a
1	3	*	*	0	root	/usr/bin/nice -n20 /etc/rc.update_bogons.sh
*/60	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 sshlockout
1	1	*	*	*	root	/usr/bin/nice -n20 /etc/rc.dyndns.update
*/60	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot
30	12	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_urltables
*/5	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_cron_misc.inc
*/15	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -t 10800 snort2c
30	11,23	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/www/snort/snort_check_for_rule_updates.php
*/30	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -t 21600 snort2c
2	*/1	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -t 43200 snort2c
2	0	*/2	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -t 2419200 snort2c
2	*/14	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -t 604800 snort2c
#
# If possible do not add items to this file manually.
# If you do so, this file must be terminated with a blank line (e.g. new line)
#
There are now five entries for expiretable snort2c. That can't be right?!

Thanks,
Mike

The fix for this bug has been posted in a Pull Request here: https://github.com/pfsense/pfsense-packages/pull/653/files

As soon as the Core Team Developers approve and merge the request, an updated Snort package will appear under System…Packages for installation.

Bill

ConfusedUser

Hi Bill,

Many thanks!

For now I manually modified the crontab and config.xml and I removed all expiretable entries for snort.
Now I can see that everything that should be blocked is blocked really. But there might be another issue with expiretable (and I might be wrong): I was running a command to remove entries older than 6 hrs manually and the result was that also an entry disappeared which was alerted only two hours ago.

I will look closer into it and properly document it and post it here if I find out that's another issue (and not my fault by typing something wrong).

Mike

simby

Can you please update snort to 2.9.6.1 ? :)

bmeeks

@simby:

Can you please update snort to 2.9.6.1 ? :)

Yes, I will put that on my TODO list. My goal is to stay as close to current as possible with Snort. Sometimes for various reasons the package may lag behind for one minor version for a while.

Bill

ConfusedUser

Hi Bill,
Do you know if the cron bug fix for Snort is already live?

In the meantime I found the second issue with the blocking feature.
Even after manually fixing the Cron entries there were still IPs unblocked that shouldn't be. I found out that this is caused by the "expiretable" command. This command cleans up IPs by the time they were entered in the snort2c table. That means if I set the blocking time to 6 hrs an IP address that was creating an alert only 10 minutes ago will be unblocked if it also created an alert 6 hours ago.

The blocking time should always depend on the last generated alert - not on the first one.

This might not sound like a real issue but I was monitoring 3 test systems in 3 different WANs for one week and in total it happened 5 times that packets were allowed to pass to the PBX even they should be blocked by Snort. I was first surprised how this could happen but this is caused by the behavior that I described above.
I have several senders from a big country in Asia ;) that try to get access to SIP for example. These frequent talkers get blocked for example because they are on a bad reputation IP list. Now they still continue trying but Snort doesn't create an alert every time a packet is sent (which makes sense of course).
And this combination (frequent sender + suppressed alerts) causes IPs to be removed from the block lists and then being able to reach the VoIP PBX because expiretable unblocks them even they are still sending - just because they have an older timestamp on the snort2c table.

Is there any way to get this fixed or any idea for a workaround?

Thanks,
Mike

bmeeks

@ConfusedUser:

Hi Bill,
Do you know if the cron bug fix for Snort is already live?

In the meantime I found the second issue with the blocking feature.
Even after manually fixing the Cron entries there were still IPs unblocked that shouldn't be. I found out that this is caused by the "expiretable" command. This command cleans up IPs by the time they were entered in the snort2c table. That means if I set the blocking time to 6 hrs an IP address that was creating an alert only 10 minutes ago will be unblocked if it also created an alert 6 hours ago.

The blocking time should always depend on the last generated alert - not on the first one.

This might not sound like a real issue but I was monitoring 3 test systems in 3 different WANs for one week and in total it happened 5 times that packets were allowed to pass to the PBX even they should be blocked by Snort. I was first surprised how this could happen but this is caused by the behavior that I described above.
I have several senders from a big country in Asia ;) that try to get access to SIP for example. These frequent talkers get blocked for example because they are on a bad reputation IP list. Now they still continue trying but Snort doesn't create an alert every time a packet is sent (which makes sense of course).
And this combination (frequent sender + suppressed alerts) causes IPs to be removed from the block lists and then being able to reach the VoIP PBX because expiretable unblocks them even they are still sending - just because they have an older timestamp on the snort2c table.

Is there any way to get this fixed or any idea for a workaround?

Thanks,
Mike

Yes, the cron fix should be "live" in the current package, v3.0.8.

As for the expiretable command, it may have a bug as you say. That binary is actually quite old and its functionality can now be duplicated directly with pfctl itself. Suricata uses this new technique, but within Snort I did not change it from how it was when I inherited it. Perhaps it's now time to make the change. I will do that in the next update.

In the meantime, since you have an easily reproducible case for testing, try this for me.

Edit the cron job command so that instead of using this string:

/usr/local/sbin/expiretable -t 21600 snort2c

it uses this one:

/sbin/pfctl -t snort2c -T expire 21600

Leave the part with "/usr/bin/nice -n20" alone.

Supposedly this utility clears all IP addresses from the specified table if the statistics for the entry have not been cleared in the time interval specified (in seconds). So 21,600 seconds equals 6 hours. Here is what the FreeBSD man page says about the -T expire sub-command:

Delete addresses which had their statistics cleared more than number seconds ago. For entries which have never had their statistics cleared, number refers to the time they were added to the table.

Bill

ConfusedUser

Tried that already a while ago - didn't work. Well, it works exactly the same way as expiretable. So changing it will not make a difference.

The command looks at the time when the statistics for an IP were last cleared. When an IP is already in the snort2c table and an alert is triggered again for the IP the date/time doesn't change.

One good example (currently I got block time set to 24 hours):
IP address 210.22.81.89 triggered an alert at the following times:
05/14/14 20:23:38
05/14/14 06:04:58

So this IP was still blocked. In "pfctl -t snort2c -T show -v" I can see the following details:
210.22.81.89
Cleared: Wed May 14 06:04:59 2014

After running "/sbin/pfctl -t snort2c -T expire 21600" at 21:30 the IP disappeared from snort2c even last alert was triggered at 20:23 (only a bit more than one hour ago).

So… unfortunately no. Changing to pfctl expire won't fix that issue.

ConfusedUser

Btw, I have tried several things to get the statistics cleared for one single IP address in the snort2c table - I didn't get it done. The command "-T zero" works only for the whole table.

The only way I have managed to do to get it cleared is to
pfctl -t snort2c -T delete 1.2.3.4
pfctl -t snort2c -T add 1.2.3.4

So whenever an alert is triggered and both commands are run it would allow a correct expiry of entries. But… I don't consider this to be a good idea to remove an entry first.
I have not found any other way to clear the statistics of an IP (but it does't mean it is not possible, I just didn't find anything).

A possible workaround would be to use two tables for snort in an alternating order.
Example:

IP 1.2.3.4 triggers an alert.
pfctl -t snort2c1 -T add 1.2.3.4
pfctl -t snort2c2 -T delete 1.2.3.4

Next IP 5.6.7.8 triggers an alert.
pfctl -t snort2c2 -T add 5.6.7.8
pfctl -t snort2c1 -T delete 5.6.7.8

Next IP 5.6.7.8 triggers another alert.
pfctl -t snort2c1 -T add 5.6.7.8
pfctl -t snort2c2 -T delete 5.6.7.8

So whatever comes in - the tables would need to be used in an alternating order. And of course a second set of block rules would need to be added and the expire commands would need to run for both tables.
Not sure if that is any practical (especially with the "blocked" tab in the GUI which would require some extra coding) but it would allow the expiry function to work properly.

bmeeks

Hmm…I was sort of afraid you might find that both worked the same way. I originally thought (due to not carefully reading the MAN page) that anytime a packet arrived for an IP in the table that would "update" the stats and thus reset the timer. Turns out that's not true when I read the MAN page more thoroughly.

I don't know if there is any easy way out of this problem with the current setup. Unless there is some other command available via the system call to the packet filter used by the Snort plugin, this may not be fixable. At least not without a patch to the packet filter code itself, and I doubt there is much appetite for that with the Core Team developers.

Bill