Snort uses up all memory (12GB) [SOLVED]
-
Thanks mate! Currently pushing just below 870Mb/s average bandwith, so no issues yet.
-
I see - I guess I should look into deciding which one of the three rule sets I want to use instead of enabling all three. Well, thanks again, I'll mark the thread as solved but will keep an eye on it in case there are more replies. Quick support on a sunday, on the first glance this forum seems to be great, perhaps I'll stick around.
Semetrothi
Here is what I do for Snort. I have the paid subscription for the VRT rules and have both those and the ET-Open rule downloads enabled. On my LAN interface I run the Snort VRT Balanced Security Policy supplemented by just the Malware and Trojan categories from the Emerging Threats rule set. I don't enable every single rule since my protected networks do not contain many of the assets a lot of rules protect. For example, I don't have public web servers or FTP servers or DB servers, so I gain nothing by enabling all the rules designed to protect those kinds of assets. Keep in mind I'm talking about my home network. Things can obviously be different in a large corporate or other commercial environment. But even there, you rarely if ever would want to enable every single rule in all the rule sets.
Bill
-
We had some fist fights over this issue: :P
https://forum.pfsense.org/index.php?topic=64674.120
I haven't tried the "AC-SPARSEBANDS". would be nice to see if there is any good literature on these "Search Methods"
I do notice that on boxes with less than 4GB that "AC-BNFA-NQ" seems to perform the best. But on systems with higher memory availability not sure if the others provide any benefit?
-
We had some fist fights over this issue: :P
https://forum.pfsense.org/index.php?topic=64674.120
I haven't tried the "AC-SPARSEBANDS". would be nice to see if there is any good literature on these "Search Methods"
I do notice that on boxes with less than 4GB that "AC-BNFA-NQ" seems to perform the best. But on systems with higher memory availability not sure if the others provide any benefit?
I think it depends on the enabled rules. The Snort devs say stick with AC-BNFA or AC-BNFA-NQ and you are golden.
Bill
-
It seems to be best to use AC-BNFA-NQ then. Quite honestly I don't really know what the difference between 1GB of memory usage and 4GB or 16GB+ usage is for a regular 100Mbit/s home connection - the speed difference will probably be noticable when it comes to a Gbit/s connections. And what does 'speed difference' mean with snort? Again, I don't know and just assume something: It could be the difference between 'snort is able to work without crashing' and 'snort crashes.'
-
I have 10Gbit internet connection and doesnt see improvement using AC-BNFA-NQ over AC-SPARSEBANDS…but it uses less memory.
-
It seems to be best to use AC-BNFA-NQ then. Quite honestly I don't really know what the difference between 1GB of memory usage and 4GB or 16GB+ usage is for a regular 100Mbit/s home connection - the speed difference will probably be noticable when it comes to a Gbit/s connections. And what does 'speed difference' mean with snort? Again, I don't know and just assume something: It could be the difference between 'snort is able to work without crashing' and 'snort crashes.'
There is absolutely NO difference in speed. No difference whatsoever. The way snort currently works on pfsense behaves the same way if you are pushing 1Mbps, 10Mbps, 100Mbps, or 1Gbps. If your box can route and spare the cycles for snort to do its processing, there is absolutely NO difference between a p3 and a quad socket 12cores/cpu monster, running on a raid0 of 8x1TB ssds.
The key to pushing monstrous bandwidths through pfsense+snort is to get the gateway to use as little CPU as possible doing the routing. If it normally maxes out at 2% while you max out your download, that means that 98% if the CPU is left for snort. If you are running a decent CPU (let's say a core 2 era one) that means snort can analyze a couple of Gbps before things start getting too slow for it to respond to normal routing, which is what will cause your download to drop off.
A simple diagram to (hopefully) stop future discussions on the so called "speed difference".
WAN > WAN Interface > The actual packet is being decided on > The actual packet moves happily along the routing stack > A copy is made in real time (causing an extremely small load) > The copy is passed on to snort for analysisAs you can see from the diagram, as long as you are NOT starving the system of processing capability to do its normal routing duties, snort will handle everything you throw at it.
The difference between the matcher options is this: NONE except the amount of RAM they use.
I can already hear the "ARE YOU F***ING STUPID??? THE DOCUMENTATION SAYS AC-NQ IS BEST PERFORMANCE SETTING?!?!?!?!"
Calm down, take a pill to relax, then re-read the rest of the documentation. I'll save you the trouble and put the small print in sight: NO difference (other than RAM) while running normal snort. HUGE difference while running inline. Since snort-inline is dead (as I've mentioned multiple times) there is no need to use anything else than AC-BNFA-NQ. Use the setting that uses the least RAM, so you can enable multiple interfaces for snort, or use a single interface with minimal load.
For the thousandth time (not directed at the quote's author, directed at the entire forum): THERE IS NO SPEED DIFFERENCE WHILE RUNNING SNORT IF YOU ARE NOT STARVING THE SYSTEM OF RESOURCES TO DO ITS NORMAL ROUTING.
I just wish everybody stopped "experimenting" and trying to re-invent the wheel. There are those of us that were there on the forefront of research while IDS/IPS systems were created, there are those of us that put them through extensive testing, and there are those of us that truely understand the way they work.
Take those people's advice: AC-BNFA-NQ.
-
This is mostly valid if Snort is not configured to block.
If you want to block, quicker Snort can alert - better.
This all depends on hardware as well - I/O, memory. People tend to run pfSense on very old garbage, where the delay can be noticeable, however that hardware tend to have limited RAM, so lowmem might be better option there.
And on newer hardware, CPU and bus are fast enough that AC-BNFA will react as fast as any high-memory configuration. -
This is mostly valid if Snort is not configured to block.
If you want to block, quicker Snort can alert - better.
This all depends on hardware as well - I/O, memory. People tend to run pfSense on very old garbage, where the delay can be noticeable, however that hardware tend to have limited RAM, so lowmem might be better option there.
And on newer hardware, CPU and bus are fast enough that AC-BNFA will react as fast as any high-memory configuration.Please re-read my post. block=snort inline, which I've mentioned.
-
No. Snort setting:
"Block Offenders" - Checking this option will automatically block hosts that generate a Snort alert.
Looks like you do not know about that option…
I would also suggest using Snort's IP lists block instead of blocking via aliases - much better to manage. -
No. Snort setting:
"Block Offenders" - Checking this option will automatically block hosts that generate a Snort alert.
Looks like you do not know about that option…
I would also suggest using Snort's IP lists block instead of blocking via aliases - much better to manage.facepalm sure, the forum's senior expert on how to configure snort (me) doesn't know about the block option ;)
Snort's blocking ability (block offenders) doesn't depend so much on the processing you do to the packet, but on the priority of the rule. The higher the priority, the faster the packet is pushed through processing (actually the lower the number in the queue that gets assigned to it, but don't let technicalities get in the way). In plain words, the higher the priority (technically lower number), the less packets get through before the block is active (see end for further explanation as to why).
The difference between processing a packet, adding the IP to the snort2c table (for the block offenders setting to actually work), pfsense getting notified about the change and acknowledging it, between the matchers is so negligible, it's universally agreed by all races in the known universe to ignore it. Since I don't know about the option ;)As I mentioned in my previous post, by the time you are done processing the >COPY< of the packet, added the host to the blocked table and all that, the >ACTUAL< packet has already passed through pfsense and it's extremely (read:almost guaranteed) that it has gone to it's original destination (if pfsense's rules allow it).
The confusion about the snort package lies in the misconception that snort is acting on the >ACTUAL< packet. It's NOT acting on the >ACTUAL< packet! It's acting on a >COPY< of the packet. The >ACTUAL< packet (ignoring routing bottlenecks, rules, etc) is NOT affected in ANY way by snort. Not even seen by snort. Snort doesn't even know it exists.
-
@jflsakfja:
Snort doesn't even know it exists.
This made my day, thank you :)
-
Hope others understand it as well. Lost count of how many times I've explained how snort works on pfsense ;D
As for the IP lists, please have a look at the suricata topic on why IP lists via snort is bad for kittens. I think there's a bounty for the head of the developer that added that functionality upstream, but don't quote me on that. (why let a perfectly normal conversation settle down, let's stir it up :p)
-
@jflsakfja:
facepalm sure, the forum's senior expert on how to configure snort (me) doesn't know about the block option ;)
;D ;D ;D
(I always have to laugh about how cynical you can write :P ).
Snort's blocking ability (block offenders) doesn't depend so much on the processing you do to the packet, but on the priority of the rule. The higher the priority, the faster the packet is pushed through processing (actually the lower the number in the queue that gets assigned to it, but don't let technicalities get in the way). In plain words, the higher the priority (technically lower number), the less packets get through before the block is active (see end for further explanation as to why).
The difference between processing a packet, adding the IP to the snort2c table (for the block offenders setting to actually work), pfsense getting notified about the change and acknowledging it, between the matchers is so negligible, it's universally agreed by all races in the known universe to ignore it. Since I don't know about the option ;)As I mentioned in my previous post, by the time you are done processing the >COPY< of the packet, added the host to the blocked table and all that, the >ACTUAL< packet has already passed through pfsense and it's extremely (read:almost guaranteed) that it has gone to it's original destination (if pfsense's rules allow it).
The confusion about the snort package lies in the misconception that snort is acting on the >ACTUAL< packet. It's NOT acting on the >ACTUAL< packet! It's acting on a >COPY< of the packet. The >ACTUAL< packet (ignoring routing bottlenecks, rules, etc) is NOT affected in ANY way by snort. Not even seen by snort. Snort doesn't even know it exists.
About the parts in bold, because I didn't realize this (I have not seen it documented elsewhere so clearly, but then again, I suggested to you before you go write books ;) ): so actually what you are saying Snort in the current way of working will not block all offensive packets, only parts of them?
-
Snort doesn't have anything to do with the blocking, to put it simply. Think of snort like your scout. He can spot enemies, but can't engage them directly. I've already described the way the packets get copied and passed to snort, I'll explain what snort does when an alert is generated.
When an alert is raised by snort, the offending IP is added to pfsense's snort2c table. It's just a normal table, think of it like a table that gets created when you add an alias. It's job is to hold many IPs, nothing more. When that IP is added to the table, pfsense needs to become aware of the change, therefore the previously cached table is refreshed. There are hidden rules (like dark forces wanting to take over the world?) that say any IP in the snort2c table, either source or destination, gets blocked. As soon as pfsense becomes aware of the change (DON'T tick the reset states under snort/suricata) it effectively stop transmitting packets for those IPs. If you selected to reset the states, then the states are reset, killing the connection in a more RFC compliant way (the proper way). You do NOT want to EVER do this. The attacker shouldn't know what system is your firewall (the one that does the reset).
Snort isn't involved in any way with the actual packet, from the point the initial actual packet is first received, to the moment it results in a ban. You do understand that going through the previous paragraph takes a while. It's rarely above 2 seconds though, so don't worry. During that time pfsense (the way I like to call it) leaks packets along the network. Using a theoretical 1 packet exploit attack scenario (very very unlikely), no host is directly protected, since at least 1 packet WILL get through. Snort spots what's wrong with the packets, the blocking is all pfsense's job.
-
@jflsakfja:
Snort doesn't have anything to do with the blocking, to put it simply.
If you would write books I would be one of the first to buy them ;D
So, what you explained now: and is this why everybody is 'eagerly' awaiting the inline filtering? Then it would be a situation of no packet getting through whatsoever?
With the reset states you mean 'kill states' in the GUI?
-
Yes and yes ;D
Inline will bring the current functionality + not having packets leak through the network gateway.
Technically kill is performed by an RST (reset) flag being set (a reset packet). ;D
-
For what reason shouldnt you kill the states?
Just asking since I would imagine that killing the connection is better than keeping the state and this could get congested by a massive attack…?
@jflsakfja:
Snort doesn't have anything to do with the blocking, to put it simply. Think of snort like your scout. He can spot enemies, but can't engage them directly. I've already described the way the packets get copied and passed to snort, I'll explain what snort does when an alert is generated.
When an alert is raised by snort, the offending IP is added to pfsense's snort2c table. It's just a normal table, think of it like a table that gets created when you add an alias. It's job is to hold many IPs, nothing more. When that IP is added to the table, pfsense needs to become aware of the change, therefore the previously cached table is refreshed. There are hidden rules (like dark forces wanting to take over the world?) that say any IP in the snort2c table, either source or destination, gets blocked. As soon as pfsense becomes aware of the change (DON'T tick the reset states under snort/suricata) it effectively stop transmitting packets for those IPs. If you selected to reset the states, then the states are reset, killing the connection in a more RFC compliant way (the proper way). You do NOT want to EVER do this. The attacker shouldn't know what system is your firewall (the one that does the reset).
Snort isn't involved in any way with the actual packet, from the point the initial actual packet is first received, to the moment it results in a ban. You do understand that going through the previous paragraph takes a while. It's rarely above 2 seconds though, so don't worry. During that time pfsense (the way I like to call it) leaks packets along the network. Using a theoretical 1 packet exploit attack scenario (very very unlikely), no host is directly protected, since at least 1 packet WILL get through. Snort spots what's wrong with the packets, the blocking is all pfsense's job.
-
Snort/Suricata inline will be much better as it will be able to block faster than how it works now. However, once the first packet from a malicious IP is blocked, any packets that follow are subsequently blocked immediately by the packet filter as the IP is now in the snort2c file.
-
Blocking the IP and killing the states for "Both" WAN and LAN is recommended. Any IPs on the lan side won't be added to the snort2c file as they are on the "pass list".
You don't want to leave the states alive that were involved in an "alert" as you leave an attack vector open into your network.
