Snort unblocks IPs when it shouldn't
-
Hi all,
I am running pfSense test installations in different networks with Snort 2.9.6.0 pkg v3.0.6 on pfSense 2.1.2 release (Amd64).My setup is pretty straight forward. On all devices I have two port forwards from WAN to a server (443 and 25 TCP) and everything else inbound from WAN is blocked with a block all firewall rule.
Blocking in Snort is activated and it is set up for both IPs (src and dst), remove block interval is 6 hrs.Now the issue: Snort seems to block all traffic when an alert is created which is mainly ports that are blocked by the firewall anyway but I have never seen it blocking any traffic that generates an alert for port 443. Am I missing a setting?
Example alert:
04/26/14 15:48:43 / TCP / Misc Attack / Source 218.77.79.34, 56115 / Destination [My WAN IP], 443 / 1:2402000 / ET DROP Dshield Block Listed Source group 1The same alert 1:2402000 for a different port (for example 33462) is creating a block.
What I can see so far only port 443 TCP is affected where Snort fails to create a block (but I may be wrong). Any idea what that could be? -
Hi all,
I am running pfSense test installations in different networks with Snort 2.9.6.0 pkg v3.0.6 on pfSense 2.1.2 release (Amd64).My setup is pretty straight forward. On all devices I have two port forwards from WAN to a server (443 and 25 TCP) and everything else inbound from WAN is blocked with a block all firewall rule.
Blocking in Snort is activated and it is set up for both IPs (src and dst), remove block interval is 6 hrs.Now the issue: Snort seems to block all traffic when an alert is created which is mainly ports that are blocked by the firewall anyway but I have never seen it blocking any traffic that generates an alert for port 443. Am I missing a setting?
Example alert:
04/26/14 15:48:43 / TCP / Misc Attack / Source 218.77.79.34, 56115 / Destination [My WAN IP], 443 / 1:2402000 / ET DROP Dshield Block Listed Source group 1The same alert 1:2402000 for a different port (for example 33462) is creating a block.
What I can see so far only port 443 TCP is affected where Snort fails to create a block (but I may be wrong). Any idea what that could be?Here is what is probably happening. First, let me explain why Snort can still alert on blocked traffic. Snort puts any interface it runs on in promiscuous mode, so all packets are examined by Snort. This means on the WAN interface Snort will even continue to see packets that the firewall is blocking. Snort sees the packets at the same time they hit the firewall input, so even though later down the chain the firewall will drop the packet, Snort has already gotten an "early copy" and thus will still alert on it.
Now your issue with the port forward may be as follows. The automatic Pass List for Snort includes all locally-attached networks. This will pick up the LAN and any other firewall interfaces. I'm assuming your server on the receiving end of the port forward is in a locally-attached network (likely your LAN). Since the LAN is automatically put on a Pass List, and because the 443 port forward rule will change the destination IP to the target server's address, Snort won't block the host. You will see the alert, though. Do you have the block IP set for SRC, DST or BOTH on the INTERFACE SETTNIGS tab?
Bill
-
Now your issue with the port forward may be as follows. The automatic Pass List for Snort includes all locally-attached networks. This will pick up the LAN and any other firewall interfaces. I'm assuming your server on the receiving end of the port forward is in a locally-attached network (likely your LAN). Since the LAN is automatically put on a Pass List, and because the 443 port forward rule will change the destination IP to the target server's address, Snort won't block the host. You will see the alert, though. Do you have the block IP set for SRC, DST or BOTH on the INTERFACE SETTNIGS tab?
Hi Bill,
Your assumption is correct, the server is sitting in a LAN which can reach the internet via NAT using a single public WAN IPv4 address. Pretty "standard" setup.
The problem I experienced is when a bad IP from the internet trying to access my server via the WAN address of pfSense and being forwarded to the server within the LAN.
I set up the block for BOTH so I expected that the source IP will be blocked. This works at least fine for port 25 TCP (exact same setup in the pf ruleset and same forward destination IP of the server in the LAN). It only seems to fail for 443 TCP from what I experienced so far. -
Hi Bill,
Your assumption is correct, the server is sitting in a LAN which can reach the internet via NAT using a single public WAN IPv4 address. Pretty "standard" setup.
The problem I experienced is when a bad IP from the internet trying to access my server via the WAN address of pfSense and being forwarded to the server within the LAN.
I set up the block for BOTH so I expected that the source IP will be blocked. This works at least fine for port 25 TCP (exact same setup in the pf ruleset and same forward destination IP of the server in the LAN). It only seems to fail for 443 TCP from what I experienced so far.Hmm…I'm not sure how exactly, but perhaps the fact 443 is generally encrypted SSL traffic comes into play somehow. You said Snort "sees the event" and logs an alert for it, but it just does not insert a corresponding block. Do I have that correct?
Bill
-
Hmm…I'm not sure how exactly, but perhaps the fact 443 is generally encrypted SSL traffic comes into play somehow. You said Snort "sees the event" and logs an alert for it, but it just does not insert a corresponding block. Do I have that correct?
Yes, that is correct.
In my first post you can see the alert as visible in the GUI (only replaced my fixed public WAN IP). So the alert is working.
But the IP address (some Chinese 218.77.79.34) didn't show up in the block list.Initially I thought that's related to the Snort rules but in my example the same rule (1:2402000) triggered an alert for another IP using another destination port (TCP 33462). This one got blocked. This port was closed on my firewall anyway but that shouldn't make a difference. As you wrote snort catches all traffic on the interface doesn't matter if pf blocks or forwards it.
Thanks,
Mike -
Hi Bill,
I am sorry but it looks like I was wrong.
I have not seen 443 ever being blocked but I was clearing everything this morning and I was comparing the alerts with the block list after three hours. There are IPs for other destination ports that aren't blocked either (but should be). Please see attached screenshot. For whatever reason three of the source IPs have not made it to the block list (red frame in the screenshot).
A very good example is the 4th line from the top and the 2nd from the bottom (green marker in the screenshot). Same rule, same source port, same destination port, same destination IP but one is blocked and one isn't.Btw, I have set the timeout to remove blocked IPs to 6 hours so everything in the list on the source side should be blocked. On the destination side this is one of my IP addresses.
Is there any way how I can trace what is happening there? I was checking snort.conf but I didn't find any documentation or source code for the "alert_pf" output (and even if I had found it… not sure if it would help me because I'm lousy programmer :) ). Anyway... I'm out of ideas really.Thanks,
Mike
-
ConfusedUser:
I believe I can explain what you are seeing with IP address 116.10.191.195. The same process may be at play with the others as well.
Remember the promiscuous mode stuff I mentioned earlier. Snort will continue to see packets even from blocked hosts. However, the packet filter firewall table that Snort inserts IP addresses into for blocking can only hold a single unique instance of a given IP. There is no reason to have the same IP in there multiple times as the block is on all ports from the given IP. So if a given IP address attacks you multiple times, but with a different port each time, you will see Alerts on the ALERTS tab for each probe, but there will only be one block representing the very first probe (and whatever port that probe might have used).
As for displaying things on the BLOCKED tab, there is a little bit of magic happening under the covers. See, when Snort inserts an IP into the block table in the packet filter, it can only insert the IP. It can't insert any reason. Thus just by looking at the block table you can't tell why the IP was put there. To try and show some context, when you open the BLOCKED tab Snort first grabs all the currently being blocked IP addresses from the <snort2c>table. You can view this table under DIAGNOSTICS…TABLES to see what IP addresses are in it. After loading an array with the currently blocked IPs, Snort then opens the alerts log file and reads the most recent lines from it. It tries to match up IP addresses in there with those in the block list and then does a sort of grouping sort. So what you sometimes see is a single IP address with multiple events listed for it in the ALERT DESCRIPTION column on the BLOCKED tab.
Bill</snort2c>
-
Hi Bill,
Thank you for the explanation. This is exactly how I expected to Snort to work with pfSense. But if I didn't misunderstand your reply it still doesn't explain the behaviour.
I'm not worrying too much about the alert view. Of course I was comparing against the snort2c table but this matches the alert view so everything seems fine in the GUI.But… Not sure if you missed that part:
In my example that I marked in green in the screenshot there are two different source IPs (116.10.191.195 and 116.10.191.164).
So both should be blocked. But only 116.10.191.195 was showing up in snort2c where 116.10.191.164 wasn't. The rest was absolutely identical, same source port, same destination port, same destination IP, same rule.Thanks,
Mike -
Hi Bill,
Thank you for the explanation. This is exactly how I expected to Snort to work with pfSense. But if I didn't misunderstand your reply it still doesn't explain the behaviour.
I'm not worrying too much about the alert view. Of course I was comparing against the snort2c table but this matches the alert view so everything seems fine in the GUI.But… Not sure if you missed that part:
In my example that I marked in green in the screenshot there are two different source IPs (116.10.191.195 and 116.10.191.164).
So both should be blocked. But only 116.10.191.195 was showing up in snort2c where 116.10.191.164 wasn't. The rest was absolutely identical, same source port, same destination port, same destination IP, same rule.Thanks,
MikeI'm sorry. I did misinterpret those two IP addresses. I don't have a good answer without more data to review. Of these two IPs (116.10.191.164 and .195), which one showed up in the block table?
Bill
-
I'm sorry. I did misinterpret those two IP addresses. I don't have a good answer without more data to review. Of these two IPs (116.10.191.164 and .195), which one showed up in the block table?
Only 116.10.191.195 was showing up in snort2c.
Do you know where I can find some documentation or source code for the "alert_pf" output that is used in "snort.conf"? Maybe that would help me to find the issue. I did a lot search but I couldn't find any sources or documentation.
Thanks,
Mike -
I'm sorry. I did misinterpret those two IP addresses. I don't have a good answer without more data to review. Of these two IPs (116.10.191.164 and .195), which one showed up in the block table?
Only 116.10.191.195 was showing up in snort2c.
Do you know where I can find some documentation or source code for the "alert_pf" output that is used in "snort.conf"? Maybe that would help me to find the issue. I did a lot search but I couldn't find any sources or documentation.
Thanks,
MikeHere is a link to the patch file for the Snort binary source code. This is from my old fork of the pfsense-tools repository. This particular file has not changed, though, since the official updated repository changed locations.
https://github.com/bmeeks8/pfsense-tools/tree/master/pfPorts/snort/files
The main code for the alert-pf output plugin is down at the bottom of the patch-spoink-integration-diff file.
Bill
-
Hi Bill,
I didn't find anything special in the output that I would immediately consider as bug but like I wrote earlier, I'm a lousy programmer (if I do programming then only vb.NET).
But… I found something in the package that is 99.9% a bug and I can re-produce it any time: The aging adds entries in /etc/crontab but never removes old entries. Maybe this is related to the issue I experience - not sure yet.
I was changing the aging in the snort/pfSense web GUI entries from "never" to "28 days", to "7 days", etc. See my /etc/crontab below:SHELL=/bin/sh PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin HOME=/var/log #minute hour mday month wday who command # # # pfSense specific crontab entries # Created: April 29, 2014, 1:55 pm # 1,31 0-5 * * * root /usr/bin/nice -n20 adjkerntz -a 1 3 * * 0 root /usr/bin/nice -n20 /etc/rc.update_bogons.sh */60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 sshlockout 1 1 * * * root /usr/bin/nice -n20 /etc/rc.dyndns.update */60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot 30 12 * * * root /usr/bin/nice -n20 /etc/rc.update_urltables */5 * * * * root /usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_cron_misc.inc */15 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -t 10800 snort2c 30 11,23 * * * root /usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/www/snort/snort_check_for_rule_updates.php */30 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -t 21600 snort2c 2 */1 * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -t 43200 snort2c 2 0 */2 * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -t 2419200 snort2c 2 */14 * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -t 604800 snort2c # # If possible do not add items to this file manually. # If you do so, this file must be terminated with a blank line (e.g. new line) #There are now five entries for expiretable snort2c. That can't be right?!
Thanks,
Mike -
Hi Bill,
I didn't find anything special in the output that I would immediately consider as bug but like I wrote earlier, I'm a lousy programmer (if I do programming then only vb.NET).
But… I found something in the package that is 99.9% a bug and I can re-produce it any time: The aging adds entries in /etc/crontab but never removes old entries. Maybe this is related to the issue I experience - not sure yet.
I was changing the aging in the snort/pfSense web GUI entries from "never" to "28 days", to "7 days", etc. See my /etc/crontab below:SHELL=/bin/sh PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin HOME=/var/log #minute hour mday month wday who command # # # pfSense specific crontab entries # Created: April 29, 2014, 1:55 pm # 1,31 0-5 * * * root /usr/bin/nice -n20 adjkerntz -a 1 3 * * 0 root /usr/bin/nice -n20 /etc/rc.update_bogons.sh */60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 sshlockout 1 1 * * * root /usr/bin/nice -n20 /etc/rc.dyndns.update */60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot 30 12 * * * root /usr/bin/nice -n20 /etc/rc.update_urltables */5 * * * * root /usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_cron_misc.inc */15 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -t 10800 snort2c 30 11,23 * * * root /usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/www/snort/snort_check_for_rule_updates.php */30 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -t 21600 snort2c 2 */1 * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -t 43200 snort2c 2 0 */2 * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -t 2419200 snort2c 2 */14 * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -t 604800 snort2c # # If possible do not add items to this file manually. # If you do so, this file must be terminated with a blank line (e.g. new line) #There are now five entries for expiretable snort2c. That can't be right?!
Thanks,
MikeYep…you are correct. This is a bug. I will fix it. Thanks for reporting it.
Bill
-
Hi Bill,
I currently got three virtual test installs of pfSense with snort on different IPs and all three keep blocking everything since I removed the cron entries manually. Probably that was the bug that I experienced with all test installs… I will keep monitoring for a few days.
Btw, make sure you read your PMs! :)
-
Hi Bill,
I currently got three virtual test installs of pfSense with snort on different IPs and all three keep blocking everything since I removed the cron entries manually. Probably that was the bug that I experienced with all test installs… I will keep monitoring for a few days.
Btw, make sure you read your PMs! :)
I did see you PM and replied. Thank you… :)
I will push a fix soon for the cron bug.
Bill
-
Hi Bill,
I didn't find anything special in the output that I would immediately consider as bug but like I wrote earlier, I'm a lousy programmer (if I do programming then only vb.NET).
But… I found something in the package that is 99.9% a bug and I can re-produce it any time: The aging adds entries in /etc/crontab but never removes old entries. Maybe this is related to the issue I experience - not sure yet.
I was changing the aging in the snort/pfSense web GUI entries from "never" to "28 days", to "7 days", etc. See my /etc/crontab below:SHELL=/bin/sh PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin HOME=/var/log #minute hour mday month wday who command # # # pfSense specific crontab entries # Created: April 29, 2014, 1:55 pm # 1,31 0-5 * * * root /usr/bin/nice -n20 adjkerntz -a 1 3 * * 0 root /usr/bin/nice -n20 /etc/rc.update_bogons.sh */60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 sshlockout 1 1 * * * root /usr/bin/nice -n20 /etc/rc.dyndns.update */60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot 30 12 * * * root /usr/bin/nice -n20 /etc/rc.update_urltables */5 * * * * root /usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_cron_misc.inc */15 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -t 10800 snort2c 30 11,23 * * * root /usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/www/snort/snort_check_for_rule_updates.php */30 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -t 21600 snort2c 2 */1 * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -t 43200 snort2c 2 0 */2 * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -t 2419200 snort2c 2 */14 * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -t 604800 snort2c # # If possible do not add items to this file manually. # If you do so, this file must be terminated with a blank line (e.g. new line) #There are now five entries for expiretable snort2c. That can't be right?!
Thanks,
MikeThe fix for this bug has been posted in a Pull Request here: https://github.com/pfsense/pfsense-packages/pull/653/files
As soon as the Core Team Developers approve and merge the request, an updated Snort package will appear under System…Packages for installation.
Bill
-
Hi Bill,
Many thanks!
For now I manually modified the crontab and config.xml and I removed all expiretable entries for snort.
Now I can see that everything that should be blocked is blocked really. But there might be another issue with expiretable (and I might be wrong): I was running a command to remove entries older than 6 hrs manually and the result was that also an entry disappeared which was alerted only two hours ago.I will look closer into it and properly document it and post it here if I find out that's another issue (and not my fault by typing something wrong).
Mike
-
Can you please update snort to 2.9.6.1 ? :)
-
Can you please update snort to 2.9.6.1 ? :)
Yes, I will put that on my TODO list. My goal is to stay as close to current as possible with Snort. Sometimes for various reasons the package may lag behind for one minor version for a while.
Bill
-
Hi Bill,
Do you know if the cron bug fix for Snort is already live?In the meantime I found the second issue with the blocking feature.
Even after manually fixing the Cron entries there were still IPs unblocked that shouldn't be. I found out that this is caused by the "expiretable" command. This command cleans up IPs by the time they were entered in the snort2c table. That means if I set the blocking time to 6 hrs an IP address that was creating an alert only 10 minutes ago will be unblocked if it also created an alert 6 hours ago.The blocking time should always depend on the last generated alert - not on the first one.
This might not sound like a real issue but I was monitoring 3 test systems in 3 different WANs for one week and in total it happened 5 times that packets were allowed to pass to the PBX even they should be blocked by Snort. I was first surprised how this could happen but this is caused by the behavior that I described above.
I have several senders from a big country in Asia ;) that try to get access to SIP for example. These frequent talkers get blocked for example because they are on a bad reputation IP list. Now they still continue trying but Snort doesn't create an alert every time a packet is sent (which makes sense of course).
And this combination (frequent sender + suppressed alerts) causes IPs to be removed from the block lists and then being able to reach the VoIP PBX because expiretable unblocks them even they are still sending - just because they have an older timestamp on the snort2c table.Is there any way to get this fixed or any idea for a workaround?
Thanks,
Mike
