Snort uses up all memory (12GB) [SOLVED]



  • Software:

    • pfSense 2.1.4-RELEASE (amd64)
    • Snort 2.9.6.0 pkg v3.0.13
    • Lenovo Thinkpad T400 with 4GB RAM + 8GB Swap

    Description of my problem:

    (1) On a fresh pfSense I install the snort package, then I choose the rules (and update them):
    Install Snort VRT rules (free)
    Install Snort Community rules
    Install Emerging Threats rules (ETOpen)

    (2) Set the field 'Search Method' to
    AC or
    AC-STD

    (3) Start Snort

    Snort starts filling up the memory until the 4GB are full, then it starts filling up the swap space either up to around 80% with AC-STD (then it is successfully started) or with AC it fills it up completely, retries and then gives up (service not started).

    If I select AC-SPLIT Snort works fine, but pfSense reports 13% of 3970 MB used, swap not used at all. I put 4GB of memory into the machine for Snort, and now it's either using less than 512MB or the 4GB are not enough… the T400 replaced an old Thinkpad with 1GB of memory and snort with AC worked fine, using up around 800-1000MB of memory.

    Why is this happening? I suspect that it's because of too many rules, on the other hand it worked fine on a much slower Thinkpad with less memory. I have never seen Snort on pfSense needing more than 12GB of memory.

    What I want in the end is snort making use of the 4GB of memory in the machine - simply because I assume (without having too much knowledge about it) that snort performs better with AC or AC-STD than with AC-SPLIT and the memory is not used at all otherwise (simple home network with a 100Mbit/s internet connection). Perhaps I am wrong and there is no need to use AC over AC-SPLIT?

    Thank you for suggestions, information or corrections - I do appreciate it.
    Semetrothi


  • Banned

    Try the AC-SPARSEBANDS and report back



  • Thanks for the quick reply. AC-SPARSEBANDS fills up the memory, then it's successfully started. With this mode it basically behaves the same way it did with the older Thinkpad. Furthermore I looked up the regular memory usage for AC and apparently one should have 16-32GB of memory for that mode. Perhaps I used less rule sets on the older Thinkpad and that's why it worked better there.

    I still welcome suggestions and more info, but I think the main issue (snort not using my memory or using too much) is solved. I will mark the thread solved in a few hours, perhaps someone else will look at it until then.

    Semetrothi



  • @semetrothi:

    Thanks for the quick reply. AC-SPARSEBANDS fills up the memory, then it's successfully started. With this mode it basically behaves the same way it did with the older Thinkpad. Furthermore I looked up the regular memory usage for AC and apparently one should have 16-32GB of memory for that mode. Perhaps I used less rule sets on the older Thinkpad and that's why it worked better there.

    I still welcome suggestions and more info, but I think the main issue (snort not using my memory or using too much) is solved. I will mark the thread solved in a few hours, perhaps someone else will look at it until then.

    Semetrothi

    The recommended setting is AC-BNFA-NQ (but AC-BNFA is also fine).  Several of the other settings can lead to memory exhaustion with lots of enabled rules.  Using the setup you described leads to tons of duplicate rules between the Snort VRT rules and the ET-Open rules (not to mention the GPLv2 Community rules, depending on how many of those you enable).  You have your box doing a fair amount of redundant work (that is, inspecting a packet against a number of pretty much identical rules).

    Bill



  • I see - I guess I should look into deciding which one of the three rule sets I want to use instead of enabling all three. Well, thanks again, I'll mark the thread as solved but will keep an eye on it in case there are more replies. Quick support on a sunday, on the first glance this forum seems to be great, perhaps I'll stick around.

    Semetrothi


  • Banned

    Can we configure Snort or get a GUI update so that the duplicate rules can be disabled therefore leaving Snort as most efficient?



  • @Supermule:

    Can we configure Snort or get a GUI update so that the duplicate rules can be disabled therefore leaving Snort as most efficient?

    No, there is really no way for code to easily determine which rule duplicates another.  For starters, they have different SIDs, then the exact sequence of text after that can also differ between the VRT rules and the ET rule, for example.

    Unfortunately it is up to the admin to pick the rules.  Taking the easy way out and just enabling all of them can work if you have tons of CPU horsepower and memory (and don't need 10G performance).

    Bill


  • Banned

    Thanks mate! Currently pushing just below 870Mb/s average bandwith, so no issues yet.



  • @semetrothi:

    I see - I guess I should look into deciding which one of the three rule sets I want to use instead of enabling all three. Well, thanks again, I'll mark the thread as solved but will keep an eye on it in case there are more replies. Quick support on a sunday, on the first glance this forum seems to be great, perhaps I'll stick around.

    Semetrothi

    Here is what I do for Snort.  I have the paid subscription for the VRT rules and have both those and the ET-Open rule downloads enabled.  On my LAN interface I run the Snort VRT Balanced Security Policy supplemented by just the Malware and Trojan categories from the Emerging Threats rule set.  I don't enable every single rule since my protected networks do not contain many of the assets a lot of rules protect.  For example, I don't have public web servers or FTP servers or DB servers, so I gain nothing by enabling all the rules designed to protect those kinds of assets.  Keep in mind I'm talking about my home network.  Things can obviously be different in a large corporate or other commercial environment.  But even there, you rarely if ever would want to enable every single rule in all the rule sets.

    Bill


  • Moderator

    We had some fist fights over this issue:  :P

    https://forum.pfsense.org/index.php?topic=64674.120

    I haven't tried the "AC-SPARSEBANDS". would be nice to see if there is any good literature on these "Search Methods"

    I do notice that on boxes with less than 4GB that "AC-BNFA-NQ" seems to perform the best. But on systems with higher memory availability not sure if the others provide any benefit?



  • @BBcan177:

    We had some fist fights over this issue:  :P

    https://forum.pfsense.org/index.php?topic=64674.120

    I haven't tried the "AC-SPARSEBANDS". would be nice to see if there is any good literature on these "Search Methods"

    I do notice that on boxes with less than 4GB that "AC-BNFA-NQ" seems to perform the best. But on systems with higher memory availability not sure if the others provide any benefit?

    I think it depends on the enabled rules.  The Snort devs say stick with AC-BNFA or AC-BNFA-NQ and you are golden.

    Bill



  • It seems to be best to use AC-BNFA-NQ then. Quite honestly I don't really know what the difference between 1GB of memory usage and 4GB or 16GB+ usage is for a regular 100Mbit/s home connection - the speed difference will probably be noticable when it comes to a Gbit/s connections. And what does 'speed difference' mean with snort? Again, I don't know and just assume something: It could be the difference between 'snort is able to work without crashing' and 'snort crashes.'


  • Banned

    I have 10Gbit internet connection and doesnt see improvement using AC-BNFA-NQ over AC-SPARSEBANDS…but it uses less memory.



  • @semetrothi:

    It seems to be best to use AC-BNFA-NQ then. Quite honestly I don't really know what the difference between 1GB of memory usage and 4GB or 16GB+ usage is for a regular 100Mbit/s home connection - the speed difference will probably be noticable when it comes to a Gbit/s connections. And what does 'speed difference' mean with snort? Again, I don't know and just assume something: It could be the difference between 'snort is able to work without crashing' and 'snort crashes.'

    There is absolutely NO difference in speed. No difference whatsoever. The way snort currently works on pfsense behaves the same way if you are pushing 1Mbps, 10Mbps, 100Mbps, or 1Gbps. If your box can route and spare the cycles for snort to do its processing, there is absolutely NO difference between a p3 and a quad socket 12cores/cpu monster, running on a raid0 of 8x1TB ssds.

    The key to pushing monstrous bandwidths through pfsense+snort is to get the gateway to use as little CPU as possible doing the routing. If it normally maxes out at 2% while you max out your download, that means that 98% if the CPU is left for snort. If you are running a decent CPU (let's say a core 2 era one) that means snort can analyze a couple of Gbps before things start getting too slow for it to respond to normal routing, which is what will cause your download to drop off.

    A simple diagram to (hopefully) stop future discussions on the so called "speed difference".

    
    WAN > WAN Interface > The actual packet is being decided on > The actual packet moves happily along the routing stack
                        > A copy is made in real time (causing an extremely small load) > The copy is passed on to snort for analysis
    
    

    As you can see from the diagram, as long as you are NOT starving the system of processing capability to do its normal routing duties, snort will handle everything you throw at it.

    The difference between the matcher options is this: NONE except the amount of RAM they use.

    I can already hear the "ARE YOU F***ING STUPID??? THE DOCUMENTATION SAYS AC-NQ IS BEST PERFORMANCE SETTING?!?!?!?!"

    Calm down, take a pill to relax, then re-read the rest of the documentation. I'll save you the trouble and put the small print in sight: NO difference (other than RAM) while running normal snort. HUGE difference while running inline. Since snort-inline is dead (as I've mentioned multiple times) there is no need to use anything else than AC-BNFA-NQ. Use the setting that uses the least RAM, so you can enable multiple interfaces for snort, or use a single interface with minimal load.

    For the thousandth time (not directed at the quote's author, directed at the entire forum): THERE IS NO SPEED DIFFERENCE WHILE RUNNING SNORT IF YOU ARE NOT STARVING THE SYSTEM OF RESOURCES TO DO ITS NORMAL ROUTING.

    I just wish everybody stopped "experimenting" and trying to re-invent the wheel. There are those of us that were there on the forefront of research while IDS/IPS systems were created, there are those of us that put them through extensive testing, and there are those of us that truely understand the way they work.

    Take those people's advice: AC-BNFA-NQ.



  • This is mostly valid if Snort is not configured to block.
    If you want to block, quicker Snort can alert - better.
    This all depends on hardware as well - I/O, memory. People tend to run pfSense on very old garbage, where the delay can be noticeable, however that hardware tend to have limited RAM, so lowmem might be better option there.
    And on newer hardware, CPU and bus are fast enough that AC-BNFA will react as fast as any high-memory configuration.



  • @dgcom:

    This is mostly valid if Snort is not configured to block.
    If you want to block, quicker Snort can alert - better.
    This all depends on hardware as well - I/O, memory. People tend to run pfSense on very old garbage, where the delay can be noticeable, however that hardware tend to have limited RAM, so lowmem might be better option there.
    And on newer hardware, CPU and bus are fast enough that AC-BNFA will react as fast as any high-memory configuration.

    Please re-read my post. block=snort inline, which I've mentioned.



  • No. Snort setting:

    "Block Offenders" - Checking this option will automatically block hosts that generate a Snort alert.

    Looks like you do not know about that option…
    I would also suggest using Snort's IP lists block instead of blocking via aliases - much better to manage.



  • @dgcom:

    No. Snort setting:

    "Block Offenders" - Checking this option will automatically block hosts that generate a Snort alert.

    Looks like you do not know about that option…
    I would also suggest using Snort's IP lists block instead of blocking via aliases - much better to manage.

    facepalm sure, the forum's senior expert on how to configure snort (me) doesn't know about the block option ;)

    Snort's blocking ability (block offenders) doesn't depend so much on the processing you do to the packet, but on the priority of the rule. The higher the priority, the faster the packet is pushed through processing (actually the lower the number in the queue that gets assigned to it, but don't let technicalities get in the way). In plain words, the higher the priority (technically lower number), the less packets get through before the block is active (see end for further explanation as to why).
    The difference between processing a packet, adding the IP to the snort2c table (for the block offenders setting to actually work), pfsense getting notified about the change and acknowledging it, between the matchers is so negligible, it's universally agreed by all races in the known universe to ignore it. Since I don't know about the option ;)

    As I mentioned in my previous post, by the time you are done processing the >COPY< of the packet, added the host to the blocked table and all that, the >ACTUAL< packet has already passed through pfsense and it's extremely (read:almost guaranteed) that it has gone to it's original destination (if pfsense's rules allow it).

    The confusion about the snort package lies in the misconception that snort is acting on the >ACTUAL< packet. It's NOT acting on the >ACTUAL< packet! It's acting on a >COPY< of the packet. The >ACTUAL< packet (ignoring routing bottlenecks, rules, etc) is NOT affected in ANY way by snort. Not even seen by snort. Snort doesn't even know it exists.



  • @jflsakfja:

    Snort doesn't even know it exists.

    This made my day, thank you  :)



  • Hope others understand it as well. Lost count of how many times I've explained how snort works on pfsense  ;D

    As for the IP lists, please have a look at the suricata topic on why IP lists via snort is bad for kittens. I think there's a bounty for the head of the developer that added that functionality upstream, but don't quote me on that. (why let a perfectly normal conversation settle down, let's stir it up :p)



  • @jflsakfja:

    facepalm sure, the forum's senior expert on how to configure snort (me) doesn't know about the block option ;)

    ;D ;D ;D

    (I always have to laugh about how cynical you can write  :P ).

    Snort's blocking ability (block offenders) doesn't depend so much on the processing you do to the packet, but on the priority of the rule. The higher the priority, the faster the packet is pushed through processing (actually the lower the number in the queue that gets assigned to it, but don't let technicalities get in the way). In plain words, the higher the priority (technically lower number), the less packets get through before the block is active (see end for further explanation as to why).
    The difference between processing a packet, adding the IP to the snort2c table (for the block offenders setting to actually work), pfsense getting notified about the change and acknowledging it
    , between the matchers is so negligible, it's universally agreed by all races in the known universe to ignore it. Since I don't know about the option ;)

    As I mentioned in my previous post, by the time you are done processing the >COPY< of the packet, added the host to the blocked table and all that, the >ACTUAL< packet has already passed through pfsense and it's extremely (read:almost guaranteed) that it has gone to it's original destination (if pfsense's rules allow it).

    The confusion about the snort package lies in the misconception that snort is acting on the >ACTUAL< packet. It's NOT acting on the >ACTUAL< packet! It's acting on a >COPY< of the packet. The >ACTUAL< packet (ignoring routing bottlenecks, rules, etc) is NOT affected in ANY way by snort. Not even seen by snort. Snort doesn't even know it exists.

    About the parts in bold, because I didn't realize this (I have not seen it documented elsewhere so clearly, but then again, I suggested to you before you go write books  ;) ): so actually what you are saying Snort in the current way of working will not block all offensive packets, only parts of them?



  • Snort doesn't have anything to do with the blocking, to put it simply. Think of snort like your scout. He can spot enemies, but can't engage them directly. I've already described the way the packets get copied and passed to snort, I'll explain what snort does when an alert is generated.

    When an alert is raised by snort, the offending IP is added to pfsense's snort2c table. It's just a normal table, think of it like a table that gets created when you add an alias. It's job is to hold many IPs, nothing more. When that IP is added to the table, pfsense needs to become aware of the change, therefore the previously cached table is refreshed. There are hidden rules (like dark forces wanting to take over the world?) that say any IP in the snort2c table, either source or destination, gets blocked. As soon as pfsense becomes aware of the change (DON'T tick the reset states under snort/suricata) it effectively stop transmitting packets for those IPs. If you selected to reset the states, then the states are reset, killing the connection in a more RFC compliant way (the proper way). You do NOT want to EVER do this. The attacker shouldn't know what system is your firewall (the one that does the reset).

    Snort isn't involved in any way with the actual packet, from the point the initial actual packet is first received, to the moment it results in a ban. You do understand that going through the previous paragraph takes a while. It's rarely above 2 seconds though, so don't worry. During that time pfsense (the way I like to call it) leaks packets along the network. Using a theoretical 1 packet exploit attack scenario (very very unlikely), no host is directly protected, since at least 1 packet WILL get through. Snort spots what's wrong with the packets, the blocking is all pfsense's job.



  • @jflsakfja:

    Snort doesn't have anything to do with the blocking, to put it simply.

    If you would write books I would be one of the first to buy them  ;D

    So, what you explained now: and is this why everybody is 'eagerly' awaiting the inline filtering? Then it would be a situation of no packet getting through whatsoever?

    With the reset states you mean 'kill states' in the GUI?



  • Yes and yes  ;D

    Inline will bring the current functionality + not having packets leak through the network gateway.

    Technically kill is performed by an RST (reset) flag being set (a reset packet).  ;D


  • Banned

    For what reason shouldnt you kill the states?

    Just asking since I would imagine that killing the connection is better than keeping the state and this could get congested by a massive attack…?

    @jflsakfja:

    Snort doesn't have anything to do with the blocking, to put it simply. Think of snort like your scout. He can spot enemies, but can't engage them directly. I've already described the way the packets get copied and passed to snort, I'll explain what snort does when an alert is generated.

    When an alert is raised by snort, the offending IP is added to pfsense's snort2c table. It's just a normal table, think of it like a table that gets created when you add an alias. It's job is to hold many IPs, nothing more. When that IP is added to the table, pfsense needs to become aware of the change, therefore the previously cached table is refreshed. There are hidden rules (like dark forces wanting to take over the world?) that say any IP in the snort2c table, either source or destination, gets blocked. As soon as pfsense becomes aware of the change (DON'T tick the reset states under snort/suricata) it effectively stop transmitting packets for those IPs. If you selected to reset the states, then the states are reset, killing the connection in a more RFC compliant way (the proper way). You do NOT want to EVER do this. The attacker shouldn't know what system is your firewall (the one that does the reset).

    Snort isn't involved in any way with the actual packet, from the point the initial actual packet is first received, to the moment it results in a ban. You do understand that going through the previous paragraph takes a while. It's rarely above 2 seconds though, so don't worry. During that time pfsense (the way I like to call it) leaks packets along the network. Using a theoretical 1 packet exploit attack scenario (very very unlikely), no host is directly protected, since at least 1 packet WILL get through. Snort spots what's wrong with the packets, the blocking is all pfsense's job.


  • Moderator

    Snort/Suricata inline will be much better as it will be able to block faster than how it works now. However, once the first packet from a malicious IP is blocked, any packets that follow are subsequently blocked immediately by the packet filter as the IP is now in the snort2c file.


  • Moderator

    Blocking the IP and killing the states for "Both" WAN and LAN is recommended. Any IPs on the lan side won't be added to the snort2c file as they are on the "pass list".

    You don't want to leave the states alive that were involved in an "alert" as you leave an attack vector open into your network.



  • pfsense handles the killing, no need to tell the attacker "hey, I just blocked your connection, kthxbye"


  • Banned

    2 opposite ideas….

    What to believe?



  • Only trust the little voice inside your head.

    EDIT: I'm assuming we are talking about a home type system. For a large network then yes, killing the states could save you from a massive attack.



  • @jflsakfja:

    If you selected to reset the states, then the states are reset, killing the connection in a more RFC compliant way (the proper way). You do NOT want to EVER do this. The attacker shouldn't know what system is your firewall (the one that does the reset).

    This is incorrect. Killing states does not involve sending RST packets, it just clears firewall's internal state for the connection in question.
    As BBcan177 said, checking this box is recommended.



  • In that case I understand that I have broken the cluster's quorum and agree to re-sync with the rest of the cluster. See why clusters with 2 members never work? :p

    In plain english: I stand corrected, enable killing the states.


  • Moderator

    Take a look at this thread: I have seen others but hard to find on you mobile browser ;)

    On the other hand for firewall rules "block" on WAN and "reject" on LAN.

    https://forum.pfsense.org/index.php?topic=75199.msg410525#msg410525


  • Banned

    Thanks :)