Snort rules order
-
Been using snort for awhile, but finally realized that I see multiple alerts for the same blocked ip's upon accessing the supposed blocked IP's. I was stunned to find I still have access. So in frustration I added these ips manually to my block list. But I see that snort "STILL" creates alerts for them.
It dawned on me that snort is probably first in line in the ruleset and still processes ip's regardless if they will be eventually blocked in the alias rules.
Not to mention the wasted overhead, but it also makes it difficult to read the alerts page as it just fills up with alerts from the same ip.
-
Been using snort for awhile, but finally realized that I see multiple alerts for the same blocked ip's upon accessing the supposed blocked IP's. I was stunned to find I still have access.
kevin067:
I don't fully understand the statement I quoted from your post. Can you explain what you mean by "…stunned to find I still have access…"?
Also, when displaying BLOCKED IPs, the code will read the number of entries specified from the alert log file (the default is 500). It then sorts these by IP address into a list. Next, it gets the currently blocked IP addresses by reading the snort2c pf table using pfctl. It then finds all the block-table matching entries from the list read from the alerts log and prints them. This is why you see multiple hits on the same IP. Each should show a different timestamp, though. In some cases probably markedly different timestamps depending on the amount of bad traffic from the IP.
It has to read the stuff from the alerts log so it can get the GID, SID and Description because that data cannot be store in the snort2c block table. All it can store is just the IP address. It just sort of shows you the current blocks and a short history of blocks on that IP address.
Bill
-
Hi Bill, Initially snort looks all well and good. I can trigger it to cause an alert and I see an associated block. Testing access to the ip shows it's indeed blocked. The block list grows happily. And you go away thinking this thing is working great. Then if the block list is cleared an hour later or a day. Either by me, or clears by the timer etc.
It all looks good still and offending ip tries again and it fires off an alert, and you can see a block in the block list. So you go away thinking it worked.
But then you go and test the blocked ip by trying to access the offending ip and you find it is happily still working! And snort is still generating Alerts for the same IP, because it's not really blocked.
Nothing short of re-installing snort will make it block that ip again.So first time an ip is blocked it works. clear the list. and any future blocks the IP can sail thru even though it shows it's in the snort2c table.
To make things worse, the block list is mysteriously getting cleared every couple of hours and I have set the "Never" clear option.
Kevin
-
Hi Bill, Initially snort looks all well and good. I can trigger it to cause an alert and I see an associated block. Testing access to the ip shows it's indeed blocked. The block list grows happily. And you go away thinking this thing is working great. Then if the block list is cleared an hour later or a day. Either by me, or clears by the timer etc.
It all looks good still and offending ip tries again and it fires off an alert, and you can see a block in the block list. So you go away thinking it worked.
But then you go and test the blocked ip by trying to access the offending ip and you find it is happily still working! And snort is still generating Alerts for the same IP, because it's not really blocked.
Nothing short of re-installing snort will make it block that ip again.So first time an ip is blocked it works. clear the list. and any future blocks the IP can sail thru even though it shows it's in the snort2c table.
To make things worse, the block list is mysteriously getting cleared every couple of hours and I have set the "Never" clear option.
Kevin
Thank you for the detailed reply. I will set up a test scenario in VMware to see if I can reproduce the same behavior. What you described is certainly not how it is supposed to work. A block should be a true block. Unfortunately I may be at some disadvantage in troubleshooting this because the Snort part appears to be working in that the offending IP is added back to to the block table (the snort2c table). Could be the problem is deeper in the pfSense code somewhere. All this premature clearing of the block table started in the 2.1 code. That code is based on FreeBSD 8.3 instead of the 8.1 that 2.0.x versions were based on.
Bill
-
Hi,
Have been using pfsense for a couple of years after trying freesco, ipcop etc over the years, It has the most configuration options and fexibility, but at the same time can be brought up to a usable state in less than half an hour, without even reading the manual to start with. All evidence of a properly sorted system. I build enbedded systems, hardware and software and appreciate the amount of work that must have gone into this. I neither have the time or knowledge to do it myself at present.
Anyway, first post here, but perhaps related. Found that 2.03 wouldn't upgrade to 2.1 without locking up after the update, so no problem, bare metal install, reinstall snort and a couple of other packages and fire it up. Snort on wan interface only, with path from wan to webserver in the rules table. Seems to work fine, with loads of alerts and entries in the block list, which is never auto cleared, nor are there any auto updates enabled. Then cleared the the block list manually a few days ago. I still get alerts, but no entries added to the block list I don't remember this problem in 2.03, which I used for many months, perhaps even a year, without any problem.
Have had a poke around the system, but sure what files are relevant, but could download any of the xml / config files / log files if that's any help…
Regards,
Chris
-
Hi,
…
Seems to work fine, with loads of alerts and entries in the block list, which is never auto cleared, nor are there any auto updates enabled. Then cleared the the block list manually a few days ago. I still get alerts, but no entries added to the block list I don't remember this problem in 2.03, which I used for many months, perhaps even a year, without any problem.Have had a poke around the system, but sure what files are relevant, but could download any of the xml / config files / log files if that's any help...
Regards,
Chris
Hi Chris:
That block list behavior is definitely not as expected. Have you verified the same alerts are present that formerly triggered the blocks? At this time I don't see how clearing the block list could prevent new blocks from being added, but I will see if I can test this in my VMware lab.
Bill
-
Thanks, i'm going to keep an eye on this for the next few days. In the past, I saw this sort of thing as a minor inconsistency, but never really dug into it. For example, not all alerts seem to get blocked. Also, there seemed to be a delay between an entry appearing in the alert list before being seen in the block list ?.
Will dig a bit more and report back on specifics…
Regards,
Chris
-
Thanks, i'm going to keep an eye on this for the next few days. In the past, I saw this sort of thing as a minor inconsistency, but never really dug into it. For example, not all alerts seem to get blocked. Also, there seemed to be a delay between an entry appearing in the alert list before being seen in the block list ?.
Will dig a bit more and report back on specifics…
Regards,
Chris
Let me know what you find out. I did some very limited testing last night in my VMware lab and could not reproduce this problem. I could scan a VM from a "hostile VM", get blocked, clear the block, scan again and get blocked again. I did, however, notice a few weird things I want to investigate closer.
UPDATE: I had some time and was able to do some fairly extensive testing against the Snort package on a pfSense VM from the kali-Linux kit of exploits. I threw a substantial set of exploits against the WAN side of the firewall VM, and Snort caught them and blocked my exploit host's IP address. I periodically cleared the blocks during the tests, and each time the block came back as the exploit kit cycled through its various modules. So bottom line is I was unable to reproduce a scenario where Snort did not alert and block on an attack, and did not alert on and block subsequent "attacks" from the same attacker IP address.
Bill
-
Bill,
Thanks for the reply.
I was having a look at the relationship between the alert list and the block list and not all the alerts sem to make it to the block list, but have had another problem today. I have two isp's driving two pfsense boxes, 1.6Ghz and throttled clock thinkcentre P4 machines each with 3 network ports. In both cases, the m/b port for the outer wan and 2 more cards on each for the inner side. I originally tried an upgrade from 2.03 -> 2.1, on one of the machines some time ago. However, this locked up after a couple of days in that the lan / management port became deaf to all incoming traffic, but still allowed ping (from the pfsense console) to other machines on that network. No incoming, but outgoing ok, so this meant that it was impossible to log into the web management console. Rebooting the machine didn't cure the problem. No idea how to fix it and google had no answers, so reinstalled 2.03 for a while, as this had been rock solid for many months.
Finally got round to reinstalling 2.1 on both machines and all seemed well. Once I had both running right with all the rules and snort etc, did a complete .tgz script backup to a new directory under /root, with the tar files ftp'd to the lab server just to make sure. Both machines get rebooted via cron every Sunday at 4.00am. I was thinking that the original problem must have been related to the upgrade procedure, then today at around 16:00, both machines locked up on the inner lan port, almost like in sync. No outgoing internet access at all. One of the ports is used for a webserver and incoming traffic still working as normal. with access from wan -> web ports working fine, as does the non lan internal port on the other machine. Console on both machines looks ok, as does ifconfig -a.
So we have two pfsense boxen, both working fine on the aux none lan ports, but completely deaf on the lan / management port. Wireshark shows no traffic at all from the the port. I was going to point zenmap at it to see if anything at all was alive, but really had work to do today so had to fix the problem fast. Ran the .tgz restore script on both machines, rebooted and everything back to normal in around 10 minutes.
Thoughts:
-
Both machines rebooted at the same time (last Sunday, 4am) and locked up at more or less the same time today.
-
Normally logged in to the dashboard / home on both machines to keep an eye on things. Could this be a problem ?.
-
Restoring from good backup fixed the problem right away, so something must have got corrupted, maybe some file size limit
exceeded or overwritten ?. -
No problem with memory or disk space. Nil swap usage
Apologies if this is steered off topic, but just wanted to get it down while still fresh in the mind. Will try and help debug this if you can tell me what areas to look at / monitor…
Regards,
Chris
-
-
A bit more info for you, just to confirm it wasn't my imagination :-)…
I checked the logs for the webserver pfsense box at ~12am last night and
there was a whole page (250 entries) of alerts, but no entries in the block
list. Mostly from the same src address, 85.xx.17.5, Description: (Emerging
threats), ET WEB_SERVER, various PHP related attempts to gain access, post etc.Checking again this morning, there was the same list, with more of the same,
but this time just two of the addresses in the alert list show up in the block
list.Checking the web server access log, one of the alerted but not blocked
addresses is shown as attempting a POST 4 times. However, there are
dozens of alerts for that address, under different snort categories, that
are not appearing in the webserver access log, so at least some are being
blocked.Can't really see what's happening here, but one thing that did occur to me
is that there seems to be a delay between an entry appearing in the alert
and appearing in the block list. Almost as though the alerts are being cached
in a tmp file and only being appended to the real block list very slowly.The good news is that while many of the alerts are not appearing in the block
list, they don't seem to be reaching the webserver, according to the access
log, so it looks like block works, even if bloxk list reporting is broken.Again, let me know if you want any of the log files or any other info.
Have found the snort alert list, which seems to be a plain text file, but
could you please tell me where the block list lives ?. Also, are there any
docs on pfsense internals and how all the components relate and connect to
each other ?...Chris
-
A bit more info for you, just to confirm it wasn't my imagination :-)…
I checked the logs for the webserver pfsense box at ~12am last night and
there was a whole page (250 entries) of alerts, but no entries in the block
list. Mostly from the same src address, 85.xx.17.5, Description: (Emerging
threats), ET WEB_SERVER, various PHP related attempts to gain access, post etc.Checking again this morning, there was the same list, with more of the same,
but this time just two of the addresses in the alert list show up in the block
list.Checking the web server access log, one of the alerted but not blocked
addresses is shown as attempting a POST 4 times. However, there are
dozens of alerts for that address, under different snort categories, that
are not appearing in the webserver access log, so at least some are being
blocked.Can't really see what's happening here, but one thing that did occur to me
is that there seems to be a delay between an entry appearing in the alert
and appearing in the block list. Almost as though the alerts are being cached
in a tmp file and only being appended to the real block list very slowly.The good news is that while many of the alerts are not appearing in the block
list, they don't seem to be reaching the webserver, according to the access
log, so it looks like block works, even if bloxk list reporting is broken.Again, let me know if you want any of the log files or any other info.
Have found the snort alert list, which seems to be a plain text file, but
could you please tell me where the block list lives ?. Also, are there any
docs on pfsense internals and how all the components relate and connect to
each other ?...Chris
There is no literal block list in Snort. To understand how Snort blocks on pfSense, you have to also understand the pf (packet filter) engine in the FreeBSD kernel and how the API for it works. Snort on pfSense uses an output plugin compiled into the binary that interfaces with the packet filter API to directly insert IP addresses into a table in the filter. That table name is snort2c. You can view its current contents under Diagnostics…Tables in the pfSense menu.
You are correct that Snort maintains a plain-text alert log for each interface. As stated, though, there is no corresponding blocked log. Instead, offending IP addresses are added using an API call to the snort2c table in the packet filter engine of the kernel. It is this snort2c table that is getting cleared prematurely by the filter_reload() code within pfSense.
Something else to understand is that the list of alerts is cumulative. This means it contains ALL the past alerts since the last time you cleared it or it exceeded the set maximum log size. Each entry will have an associated time stamp. You need to matchup those time stamps in the Alert log with any entries in your web server logs to make any sense out of the sequence of events.
Finally, when the Snort package GUI populates the BLOCKED tab page, it gets all the IP addresses currently in the snort2c table (discussed earlier), then gets the most recent 250 (or 500, depending on the option setting) alerts from that plain text file of alerts. It then sorts the alerts and groups by IP with the blocked IPs being the key. So what you see on the BLOCKED tab display is all the previous alerts from that currently blocked IP. Pay attention to the time stamps and you will see they differ.
I tested pretty thoroughly last weekend using the Kali Linux pen-test suite against a 2.1 pfSense firewall with the latest Snort package. It blocked everything it was supposed to when it was supposed to. I noticed no problems. It is true that seemingly randomly the filter_reload() process will come along and clear the snort2c table in the packet filter. That will indeed clear that block. However, on the next offending packet from that IP address the Snort output plugin will call the API function and insert the IP address again into the snort2c table. This can happen over and over and over if necessary. I saw this during my testing last weekend.
Bill
-
Bill,
Thanks v much for the detailed reply, which gives much more info. Can
visualise the little boxes and data flow. With snort being an optional package,
I guess the only way it can work is to use some sort of hooks in the packet
filter for snort inputs and outputs. Thinking streams, but maybe wrong.
My confusion was the snort block list report, which sort of suggested that
block info was snort generated only.Pfsense is a really great piece of work and it would be good to contribute to
the effort at some stage. In particular, getting pfsense running on old Sun
hardware. There's a lot of it about, it's cheap, good and it's not X86, which
gives added security. Have never worked with or programmed FreeBSD and almost
nil experience of interpreted languages, but that could be fixed. Have plenty
of old Sun hardware to contribute if anyone is interested in starting a project
and can provide support on hardware related issues..I guess the first thing to do is to try a FreeBSD install. Debian runs fine
and installed just about out of the box on a V240 class machine, but no idea
how good or stable is the support for FreeBSD on Sparc…Chris
