Snort Consuming All Memory

afagund

Hi,

I have installed a new machine with PFSense 2.4.2 and Snort package. The server runs fine for hours and then suddenly Snort process consumes all the server memory and swap (memory usage report attached). Usually when that happens Snort crashes and stop.

The server has 6 GB of RAM and I have enabled just Snort VRT. Under categories, I haven't selected any rules yet. All the other settings are default.

Please can someone help me figure why Snort is consuming all the memory?

Regards,
Andre

pfsense_mem.JPG_thumb

mudmanc4

I'm sure there are default rules, elder yet might assist in getting started with snort.
https://forum.pfsense.org/index.php?topic=61018.0

dales

It looks like the memory shortage occurred at 12:15.

Does the problem always occur at :15 after the hour? Do you have snort configured to update rules every 6/12/24 hours at :15?

afagund

The issue does not occur at specific intervals, it's random.

Thanks for the help so far!

Andre

bmeeks

@afagund:

Hi,

I have installed a new machine with PFSense 2.4.2 and Snort package. The server runs fine for hours and then suddenly Snort process consumes all the server memory and swap (memory usage report attached). Usually when that happens Snort crashes and stop.

The server has 6 GB of RAM and I have enabled just Snort VRT. Under categories, I haven't selected any rules yet. All the other settings are default.

Please can someone help me figure why Snort is consuming all the memory?

Regards,
Andre

Did you change the default for the Multi-Pattern Matcher for the interface? You should use only AC-BNFA or AC-BNFA-NQ. If you use any other MPM, then memory problems will occur. I run Snort (and the latest version) with no issues and memory use is relatively constant.

Bill

afagund

Hi Bill,

I am using the defaul AC-BNFA patter matcher.

Andre

bmeeks

With no rules selected, Snort really should not use much memory at all. Are you absolutely sure it is Snort that is consuming the memory? I've not had any reports like that about Snort in several years. The last time that happened was back a few years ago when folks were trying different pattern matchers. AC was notorious for eating up available RAM.

Bill

afagund

I am sure it is Snort, because I can see the memory usage by process, and when I stop the snort interface the memory is released. I don't know what else to do, the problem is still happening and I don't know where I can get information from to troubleshoot this problem.

bmeeks

Do you have Snort configured on more than one interface? If so, check and make sure that only a single instance of Snort is running on each interface by running this command from the CLI (command line interface) of the firewall –

ps -ax | grep snort

You should see one snort process per enabled interface. If you see more than that, then you have duplicate instances running and we will need to troubleshoot that.

I run Snort and have three enabled interfaces. I run the IPS Balanced policy on my LAN along with a smattering of ET rule categories. I run a handful of ET Open rules on the DMZ interface and I run the ET Open CIARMY and DSHIELD rule categories on my WAN just to see Snort hits so I know its working. I've never had any memory issues. My checks for rule updates twice per day.

Bill

afagund

I double checked and I see just one snort process consuming the whole memory. I have been trying the AC-BNFA-NQ pattern match for almost a whole day now and no issues so far. Hope this fix the issue.

afagund

Same issue using AC-BNFA-NQ.

afagund

Feb 21 20:50:40 kernel swap_pager_getswapspace(16): failed
Feb 21 20:50:40 kernel swap_pager_getswapspace(8): failed
Feb 21 20:50:40 kernel swap_pager_getswapspace(16): failed
Feb 21 20:50:40 kernel swap_pager_getswapspace(3): failed
Feb 21 21:02:32 kernel pid 77694 (snort), uid 0, was killed: out of swap space

bmeeks

@afagund:

Feb 21 20:50:40 kernel swap_pager_getswapspace(16): failed
Feb 21 20:50:40 kernel swap_pager_getswapspace(8): failed
Feb 21 20:50:40 kernel swap_pager_getswapspace(16): failed
Feb 21 20:50:40 kernel swap_pager_getswapspace(3): failed
Feb 21 21:02:32 kernel pid 77694 (snort), uid 0, was killed: out of swap space

The only theory I can offer is your setup is triggering some weird "edge case" bug in the Snort binary. I've seen nothing like that on my system, and if it was a widespread issue you can bet the pfSense Snort users would be in here posting up their out-of-memory problems. Just for grins, what happens if you run Snort with no rules enabled other than the out-of-the-box default preprocessor and decoder rules? Go to the CATEGORIES tab and de-select everything, save it, and then restart Snort. See if it runs then. If it does, then you can start adding categories back one at the time to see which one breaks it.

Bill

afagund

It is already disabled. I am keeping just the Snort Subscriber Rules enabled and nothing is checked in the categories. Ive just reinstalled the Snort package keeping the same settings … lets see how it will behave now. Thanks for your help Bill!

afagund

Same issue … reinstalling the package did not help.

bmeeks

@afagund:

It is already disabled. I am keeping just the Snort Subscriber Rules enabled and nothing is checked in the categories.

This part of your reply is not 100% clear to me. When you say "Snort Subscriber Rules" do you mean you have the downloading/updating enabled on the GLOBAL SETTINGS tab, but you have nothing checked on the CATEGORIES. What we want is for no rules to be loaded at all for Snort to inspect traffic with other than the default preprocessor and decoder rules. There are just a very, very few of those.

You can also try disabling the automatic rule updates for troubleshooting by setting the update interval to either "none" or "never" (can't remember what the option is called).

With no rules enabled and even downloads/updates disabled, Snort really should use very little memory and it should not creep up. I have never seen Snort do this nor have there been any other reports of the same (except a long time ago when folks were experimenting with some of the other MPM settings like AC and
such).

Last troubleshooting step to 100% isolate it to just Snort would be to uninstall the package and then see if the issue recurs. Don't recall if you have done that yet or not.

Bill

afagund

Global Settings -> Snort Subscriber Rules -> Enable Snort VRT. This is the only set of rules being downloaded, the other rules like GPLv2 Community, emerging threats and etc are all disabled. Under RTVLA Categories -> rulesets, there is nothing checked.

I am suspecting now that HTTP inspection can be related to this issue, as I have around 800 websites (online stores) running in my infrastructure. I am disabling the HTTP inspection to see how it goes. Will keep you posted.

Thanks!

Andre

bmeeks

@afagund:

Global Settings -> Snort Subscriber Rules -> Enable Snort VRT. This is the only set of rules being downloaded, the other rules like GPLv2 Community, emerging threats and etc are all disabled. Under RTVLA Categories -> rulesets, there is nothing checked.

I am suspecting now that HTTP inspection can be related to this issue, as I have around 800 websites (online stores) running in my infrastructure. I am disabling the HTTP inspection to see how it goes. Will keep you posted.

Thanks!

Andre

With the configuration you have, the only rules running are the preprocessor and decoder rules. If you are indeed getting memory exhaustion from those rules, I'm sure the Snort team upstream would like to hear about it. If you can prove that's what it is, you can post to the Snort mailing list about the issue.

Bill

afagund

Good news … the issue is in fact related to HTTP inspection. The server runs fine (4% memory usage avg) with this feature disabled.

HTTP inspect has several settings. I will enable one at a time to determine exactly which setting is causing the issue.

Thanks!
Andre

afagund

I have finally determined the cause of this issue.

The server runs fine disabling the feature "Snort Interfaces -> Edit -> RTVLA Preprocs -> HTTP Inspect -> Edit -> Inspect gzip".

Any suggestions so I can re-enable this option?

Thanks!
Andre