Snort uses up all memory (12GB) [SOLVED]
-
Software:
- pfSense 2.1.4-RELEASE (amd64)
- Snort 2.9.6.0 pkg v3.0.13
- Lenovo Thinkpad T400 with 4GB RAM + 8GB Swap
Description of my problem:
(1) On a fresh pfSense I install the snort package, then I choose the rules (and update them):
Install Snort VRT rules (free)
Install Snort Community rules
Install Emerging Threats rules (ETOpen)(2) Set the field 'Search Method' to
AC or
AC-STD(3) Start Snort
Snort starts filling up the memory until the 4GB are full, then it starts filling up the swap space either up to around 80% with AC-STD (then it is successfully started) or with AC it fills it up completely, retries and then gives up (service not started).
If I select AC-SPLIT Snort works fine, but pfSense reports 13% of 3970 MB used, swap not used at all. I put 4GB of memory into the machine for Snort, and now it's either using less than 512MB or the 4GB are not enough… the T400 replaced an old Thinkpad with 1GB of memory and snort with AC worked fine, using up around 800-1000MB of memory.
Why is this happening? I suspect that it's because of too many rules, on the other hand it worked fine on a much slower Thinkpad with less memory. I have never seen Snort on pfSense needing more than 12GB of memory.
What I want in the end is snort making use of the 4GB of memory in the machine - simply because I assume (without having too much knowledge about it) that snort performs better with AC or AC-STD than with AC-SPLIT and the memory is not used at all otherwise (simple home network with a 100Mbit/s internet connection). Perhaps I am wrong and there is no need to use AC over AC-SPLIT?
Thank you for suggestions, information or corrections - I do appreciate it.
Semetrothi -
Try the AC-SPARSEBANDS and report back
-
Thanks for the quick reply. AC-SPARSEBANDS fills up the memory, then it's successfully started. With this mode it basically behaves the same way it did with the older Thinkpad. Furthermore I looked up the regular memory usage for AC and apparently one should have 16-32GB of memory for that mode. Perhaps I used less rule sets on the older Thinkpad and that's why it worked better there.
I still welcome suggestions and more info, but I think the main issue (snort not using my memory or using too much) is solved. I will mark the thread solved in a few hours, perhaps someone else will look at it until then.
Semetrothi
-
Thanks for the quick reply. AC-SPARSEBANDS fills up the memory, then it's successfully started. With this mode it basically behaves the same way it did with the older Thinkpad. Furthermore I looked up the regular memory usage for AC and apparently one should have 16-32GB of memory for that mode. Perhaps I used less rule sets on the older Thinkpad and that's why it worked better there.
I still welcome suggestions and more info, but I think the main issue (snort not using my memory or using too much) is solved. I will mark the thread solved in a few hours, perhaps someone else will look at it until then.
Semetrothi
The recommended setting is AC-BNFA-NQ (but AC-BNFA is also fine). Several of the other settings can lead to memory exhaustion with lots of enabled rules. Using the setup you described leads to tons of duplicate rules between the Snort VRT rules and the ET-Open rules (not to mention the GPLv2 Community rules, depending on how many of those you enable). You have your box doing a fair amount of redundant work (that is, inspecting a packet against a number of pretty much identical rules).
Bill
-
I see - I guess I should look into deciding which one of the three rule sets I want to use instead of enabling all three. Well, thanks again, I'll mark the thread as solved but will keep an eye on it in case there are more replies. Quick support on a sunday, on the first glance this forum seems to be great, perhaps I'll stick around.
Semetrothi
-
Can we configure Snort or get a GUI update so that the duplicate rules can be disabled therefore leaving Snort as most efficient?
-
Can we configure Snort or get a GUI update so that the duplicate rules can be disabled therefore leaving Snort as most efficient?
No, there is really no way for code to easily determine which rule duplicates another. For starters, they have different SIDs, then the exact sequence of text after that can also differ between the VRT rules and the ET rule, for example.
Unfortunately it is up to the admin to pick the rules. Taking the easy way out and just enabling all of them can work if you have tons of CPU horsepower and memory (and don't need 10G performance).
Bill
-
Thanks mate! Currently pushing just below 870Mb/s average bandwith, so no issues yet.
-
I see - I guess I should look into deciding which one of the three rule sets I want to use instead of enabling all three. Well, thanks again, I'll mark the thread as solved but will keep an eye on it in case there are more replies. Quick support on a sunday, on the first glance this forum seems to be great, perhaps I'll stick around.
Semetrothi
Here is what I do for Snort. I have the paid subscription for the VRT rules and have both those and the ET-Open rule downloads enabled. On my LAN interface I run the Snort VRT Balanced Security Policy supplemented by just the Malware and Trojan categories from the Emerging Threats rule set. I don't enable every single rule since my protected networks do not contain many of the assets a lot of rules protect. For example, I don't have public web servers or FTP servers or DB servers, so I gain nothing by enabling all the rules designed to protect those kinds of assets. Keep in mind I'm talking about my home network. Things can obviously be different in a large corporate or other commercial environment. But even there, you rarely if ever would want to enable every single rule in all the rule sets.
Bill
-
We had some fist fights over this issue: :P
https://forum.pfsense.org/index.php?topic=64674.120
I haven't tried the "AC-SPARSEBANDS". would be nice to see if there is any good literature on these "Search Methods"
I do notice that on boxes with less than 4GB that "AC-BNFA-NQ" seems to perform the best. But on systems with higher memory availability not sure if the others provide any benefit?
-
We had some fist fights over this issue: :P
https://forum.pfsense.org/index.php?topic=64674.120
I haven't tried the "AC-SPARSEBANDS". would be nice to see if there is any good literature on these "Search Methods"
I do notice that on boxes with less than 4GB that "AC-BNFA-NQ" seems to perform the best. But on systems with higher memory availability not sure if the others provide any benefit?
I think it depends on the enabled rules. The Snort devs say stick with AC-BNFA or AC-BNFA-NQ and you are golden.
Bill
-
It seems to be best to use AC-BNFA-NQ then. Quite honestly I don't really know what the difference between 1GB of memory usage and 4GB or 16GB+ usage is for a regular 100Mbit/s home connection - the speed difference will probably be noticable when it comes to a Gbit/s connections. And what does 'speed difference' mean with snort? Again, I don't know and just assume something: It could be the difference between 'snort is able to work without crashing' and 'snort crashes.'
-
I have 10Gbit internet connection and doesnt see improvement using AC-BNFA-NQ over AC-SPARSEBANDS…but it uses less memory.
-
It seems to be best to use AC-BNFA-NQ then. Quite honestly I don't really know what the difference between 1GB of memory usage and 4GB or 16GB+ usage is for a regular 100Mbit/s home connection - the speed difference will probably be noticable when it comes to a Gbit/s connections. And what does 'speed difference' mean with snort? Again, I don't know and just assume something: It could be the difference between 'snort is able to work without crashing' and 'snort crashes.'
There is absolutely NO difference in speed. No difference whatsoever. The way snort currently works on pfsense behaves the same way if you are pushing 1Mbps, 10Mbps, 100Mbps, or 1Gbps. If your box can route and spare the cycles for snort to do its processing, there is absolutely NO difference between a p3 and a quad socket 12cores/cpu monster, running on a raid0 of 8x1TB ssds.
The key to pushing monstrous bandwidths through pfsense+snort is to get the gateway to use as little CPU as possible doing the routing. If it normally maxes out at 2% while you max out your download, that means that 98% if the CPU is left for snort. If you are running a decent CPU (let's say a core 2 era one) that means snort can analyze a couple of Gbps before things start getting too slow for it to respond to normal routing, which is what will cause your download to drop off.
A simple diagram to (hopefully) stop future discussions on the so called "speed difference".
WAN > WAN Interface > The actual packet is being decided on > The actual packet moves happily along the routing stack > A copy is made in real time (causing an extremely small load) > The copy is passed on to snort for analysis
As you can see from the diagram, as long as you are NOT starving the system of processing capability to do its normal routing duties, snort will handle everything you throw at it.
The difference between the matcher options is this: NONE except the amount of RAM they use.
I can already hear the "ARE YOU F***ING STUPID??? THE DOCUMENTATION SAYS AC-NQ IS BEST PERFORMANCE SETTING?!?!?!?!"
Calm down, take a pill to relax, then re-read the rest of the documentation. I'll save you the trouble and put the small print in sight: NO difference (other than RAM) while running normal snort. HUGE difference while running inline. Since snort-inline is dead (as I've mentioned multiple times) there is no need to use anything else than AC-BNFA-NQ. Use the setting that uses the least RAM, so you can enable multiple interfaces for snort, or use a single interface with minimal load.
For the thousandth time (not directed at the quote's author, directed at the entire forum): THERE IS NO SPEED DIFFERENCE WHILE RUNNING SNORT IF YOU ARE NOT STARVING THE SYSTEM OF RESOURCES TO DO ITS NORMAL ROUTING.
I just wish everybody stopped "experimenting" and trying to re-invent the wheel. There are those of us that were there on the forefront of research while IDS/IPS systems were created, there are those of us that put them through extensive testing, and there are those of us that truely understand the way they work.
Take those people's advice: AC-BNFA-NQ.
-
This is mostly valid if Snort is not configured to block.
If you want to block, quicker Snort can alert - better.
This all depends on hardware as well - I/O, memory. People tend to run pfSense on very old garbage, where the delay can be noticeable, however that hardware tend to have limited RAM, so lowmem might be better option there.
And on newer hardware, CPU and bus are fast enough that AC-BNFA will react as fast as any high-memory configuration. -
This is mostly valid if Snort is not configured to block.
If you want to block, quicker Snort can alert - better.
This all depends on hardware as well - I/O, memory. People tend to run pfSense on very old garbage, where the delay can be noticeable, however that hardware tend to have limited RAM, so lowmem might be better option there.
And on newer hardware, CPU and bus are fast enough that AC-BNFA will react as fast as any high-memory configuration.Please re-read my post. block=snort inline, which I've mentioned.
-
No. Snort setting:
"Block Offenders" - Checking this option will automatically block hosts that generate a Snort alert.
Looks like you do not know about that option…
I would also suggest using Snort's IP lists block instead of blocking via aliases - much better to manage. -
No. Snort setting:
"Block Offenders" - Checking this option will automatically block hosts that generate a Snort alert.
Looks like you do not know about that option…
I would also suggest using Snort's IP lists block instead of blocking via aliases - much better to manage.facepalm sure, the forum's senior expert on how to configure snort (me) doesn't know about the block option ;)
Snort's blocking ability (block offenders) doesn't depend so much on the processing you do to the packet, but on the priority of the rule. The higher the priority, the faster the packet is pushed through processing (actually the lower the number in the queue that gets assigned to it, but don't let technicalities get in the way). In plain words, the higher the priority (technically lower number), the less packets get through before the block is active (see end for further explanation as to why).
The difference between processing a packet, adding the IP to the snort2c table (for the block offenders setting to actually work), pfsense getting notified about the change and acknowledging it, between the matchers is so negligible, it's universally agreed by all races in the known universe to ignore it. Since I don't know about the option ;)As I mentioned in my previous post, by the time you are done processing the >COPY< of the packet, added the host to the blocked table and all that, the >ACTUAL< packet has already passed through pfsense and it's extremely (read:almost guaranteed) that it has gone to it's original destination (if pfsense's rules allow it).
The confusion about the snort package lies in the misconception that snort is acting on the >ACTUAL< packet. It's NOT acting on the >ACTUAL< packet! It's acting on a >COPY< of the packet. The >ACTUAL< packet (ignoring routing bottlenecks, rules, etc) is NOT affected in ANY way by snort. Not even seen by snort. Snort doesn't even know it exists.
-
@jflsakfja:
Snort doesn't even know it exists.
This made my day, thank you :)
-
Hope others understand it as well. Lost count of how many times I've explained how snort works on pfsense ;D
As for the IP lists, please have a look at the suricata topic on why IP lists via snort is bad for kittens. I think there's a bounty for the head of the developer that added that functionality upstream, but don't quote me on that. (why let a perfectly normal conversation settle down, let's stir it up :p)