Snort Consuming All Memory
-
Feb 21 20:50:40 kernel swap_pager_getswapspace(16): failed
Feb 21 20:50:40 kernel swap_pager_getswapspace(8): failed
Feb 21 20:50:40 kernel swap_pager_getswapspace(16): failed
Feb 21 20:50:40 kernel swap_pager_getswapspace(3): failed
Feb 21 21:02:32 kernel pid 77694 (snort), uid 0, was killed: out of swap space -
Feb 21 20:50:40 kernel swap_pager_getswapspace(16): failed
Feb 21 20:50:40 kernel swap_pager_getswapspace(8): failed
Feb 21 20:50:40 kernel swap_pager_getswapspace(16): failed
Feb 21 20:50:40 kernel swap_pager_getswapspace(3): failed
Feb 21 21:02:32 kernel pid 77694 (snort), uid 0, was killed: out of swap spaceThe only theory I can offer is your setup is triggering some weird "edge case" bug in the Snort binary. I've seen nothing like that on my system, and if it was a widespread issue you can bet the pfSense Snort users would be in here posting up their out-of-memory problems. Just for grins, what happens if you run Snort with no rules enabled other than the out-of-the-box default preprocessor and decoder rules? Go to the CATEGORIES tab and de-select everything, save it, and then restart Snort. See if it runs then. If it does, then you can start adding categories back one at the time to see which one breaks it.
Bill
-
It is already disabled. I am keeping just the Snort Subscriber Rules enabled and nothing is checked in the categories. Ive just reinstalled the Snort package keeping the same settings … lets see how it will behave now. Thanks for your help Bill!
-
Same issue … reinstalling the package did not help.
-
It is already disabled. I am keeping just the Snort Subscriber Rules enabled and nothing is checked in the categories.
This part of your reply is not 100% clear to me. When you say "Snort Subscriber Rules" do you mean you have the downloading/updating enabled on the GLOBAL SETTINGS tab, but you have nothing checked on the CATEGORIES. What we want is for no rules to be loaded at all for Snort to inspect traffic with other than the default preprocessor and decoder rules. There are just a very, very few of those.
You can also try disabling the automatic rule updates for troubleshooting by setting the update interval to either "none" or "never" (can't remember what the option is called).
With no rules enabled and even downloads/updates disabled, Snort really should use very little memory and it should not creep up. I have never seen Snort do this nor have there been any other reports of the same (except a long time ago when folks were experimenting with some of the other MPM settings like AC and
such).Last troubleshooting step to 100% isolate it to just Snort would be to uninstall the package and then see if the issue recurs. Don't recall if you have done that yet or not.
Bill
-
Global Settings -> Snort Subscriber Rules -> Enable Snort VRT. This is the only set of rules being downloaded, the other rules like GPLv2 Community, emerging threats and etc are all disabled. Under RTVLA Categories -> rulesets, there is nothing checked.
I am suspecting now that HTTP inspection can be related to this issue, as I have around 800 websites (online stores) running in my infrastructure. I am disabling the HTTP inspection to see how it goes. Will keep you posted.
Thanks!
Andre
-
Global Settings -> Snort Subscriber Rules -> Enable Snort VRT. This is the only set of rules being downloaded, the other rules like GPLv2 Community, emerging threats and etc are all disabled. Under RTVLA Categories -> rulesets, there is nothing checked.
I am suspecting now that HTTP inspection can be related to this issue, as I have around 800 websites (online stores) running in my infrastructure. I am disabling the HTTP inspection to see how it goes. Will keep you posted.
Thanks!
Andre
With the configuration you have, the only rules running are the preprocessor and decoder rules. If you are indeed getting memory exhaustion from those rules, I'm sure the Snort team upstream would like to hear about it. If you can prove that's what it is, you can post to the Snort mailing list about the issue.
Bill
-
Good news … the issue is in fact related to HTTP inspection. The server runs fine (4% memory usage avg) with this feature disabled.
HTTP inspect has several settings. I will enable one at a time to determine exactly which setting is causing the issue.
Thanks!
Andre -
I have finally determined the cause of this issue.
The server runs fine disabling the feature "Snort Interfaces -> Edit -> RTVLA Preprocs -> HTTP Inspect -> Edit -> Inspect gzip".
Any suggestions so I can re-enable this option?
Thanks!
Andre -
I have finally determined the cause of this issue.
The server runs fine disabling the feature "Snort Interfaces -> Edit -> RTVLA Preprocs -> HTTP Inspect -> Edit -> Inspect gzip".
Any suggestions so I can re-enable this option?
Thanks!
AndreThere is a parameter associated with this setting that controls the maximum amount of memory it should consume when inspecting gzip streams. The parameter defaults to 838860 bytes and is found under the HTTP_INSPECT preprocess section of the PREPROCESSORS tab. What value is yours set to? Is yours changed from the default, and if so, is the value set to something considerably ess than the maximum memory in the firewall?
Bill
-
Its default - 838860
-
Its default - 838860
Well then, I'm officially stumped as to the cause. That parameter should limit the total amount of memory consumed by the HTTP_INSPECT preprocessor when it unzips and attempts to analyze a gzip file. If it runs up against that limit, it should log a warning message and just forget further unzipping of that file stream. It definitely should not keep gobbling up memory.
Bill
-
Should I message the developers mail list? For me its clear we have a bug on this package.
-
Should I message the developers mail list? For me its clear we have a bug on this package.
You can, but if you mention "_package on pfSens_e" or "pfSense" anywhere in your report they will just send you right back here as they will assume somehow pfSense is to blame. The Snort package on pfSense uses the same Snort binary as is used on any other Linux/FreeBSD machine. So just tell them you think you have a memory leak issue with Snort using the HTTP_INSPECT preprocessor with gzip encoding enabled and don't mention the platform you are running on unless they ask. If they ask, just say "FreeBSD 11" and don't mention pfSense.
I'm not trying to be coy and hide information, but just pointing out that the default response will likely be to send you right back to the pfSense forums if you mention pfSense in your bug report. Your issue is not with the pfSense package per se, it is potentially with the Snort binary, and that binary is the same as any other FreeBSD user would be using without pfSense in the mix.
Bill