Suricata silently crashes with an "Out of swap space" error

[sloppyjoe]

Hey all,

I've been running PfSense for 8 months and its been rock solid and amazingly useful. I love it. Thanks for all of your hard work!

Recently however, Suricata has started crashing seemingly randomly every few weeks. The logs show a lot of "swap_pager_getswapspace(##): failed" messages and then "pid 21229 (suricata), jid 0, uid 0, was killed: out of swap space". Honestly I'm not sure if this is right place to post this because I'm not totally sure this is a Suricata problem but it seems to be the first thing to crash when it (thinks?) it is out of memory.

Here is an excerpt from the PfSense log before the long list of swap_pager kernel messages (note the logs are in reverse chronological order):

Here is an except from the PfSense log after the long string of swap_pager messages:

The suricata logs don't show anything useful at all, to me at least.

I'm running a pretty basic but I'd assumed over-the-top setup on bare metal lenovo SFF with lots of memory, SSD space and CPU power, two Intel NICs, 1 GB symmetrical fibre WAN. Here's the details of the box:

As you can see I'm no where close to maxing out memory, SWAP or SSD space at the time when I noticed the crash.

The graph of memory over time doesn't look like it is running out of memory either during the time of the crash:

When it crashes, I have to manually remove the suricata pid file and restart the service to get it up and running again.

I've searched here and Reddit and can't find anything recent with similar symptoms.

So I'm wondering if anyone has any ideas why this might be happening and how to stop it crashing? Is it as simple as re-installing PfSense and allocating more Swap space?

Thanks in advanced for any insights you might have! Happy to provide any additional information I might have missed.

[bmeeks]

It's obvious your machine is simply running out of memory. Why is the question.

One of the first things to check in Suricata is to make sure you have NOT altered the MPM (multi-pattern matching) algorithm from its default value. Changing the MPM value will definitely lead to a rapid consumption of memory in most instances. If you changed that parameter from the default, put it back to the default setting, restart Suricata, and then monitor.

It's also quite possible that Suricata is not the reason RAM is getting used up. Could easily be another application and Suricata is just a victim. I see pfBlockerNG messages in the log snippet. What other packages are installed on this box?

A box with 8 GB of RAM should never need to use swap.

[sloppyjoe]

Thanks for your reply.

Is it obvious that its running out of memory? The memory graph I included never dips below 60% free memory. I can't find a way to see the state of the swap over time but as you say, it shouldn't be used anyway assuming I have enough RAM.

The Suricata Pattern Matching Algorithm is set to Auto. Don't think I've ever touched those settings.

I agree that it could be another service but can't find a why to determine which service it might be. Here is my list of services. I've tried to keep it as tight as possible given my usecase:

Thanks again for your insights!

[bmeeks]

I see three packages that are not "standard": Suricata, arpwatch, and pfBlockerNG. One of those would be my first suspect.

You can start a process of elimination by disabling all but one of the services and monitoring for a time. Or perhaps disable all of them first to see if it is something else entirely.

If you satisfy yourself that the base system is stable, then start adding the packages back one at the time to see when the problem recurs. This will take some time to accomplish as you say the out-of-memory condition takes perhaps a few weeks to develop.

It might be some large IP list getting processed by pfBlockerNG during an update that triggers the out-of-memory state. Those lists (and Suricata's rules files as well) change in content over time as new versions are automatically downloaded and used. Usually, though, with either pfBlocker or Suricata, because their rule updates are fundamentally handled via PHP coding, you would see a PHP out-of-memory error first and those are non-fatal. You are seeing an out-of-swap-space error which means something used enough system RAM to cause the OS to swap out portions of active RAM to disk. And it is running out of disk space to swap to (because swap is only 1 GB). But as we said, you should normally never need to use swap in the first place. The fact swap is being used means something is leaking RAM.

[sloppyjoe]

That makes a lot of sense, I'll definitely try that starting with pfBlocker.

I'm still confused as to why I'm not seeing PfSense run out of memory in on the Status / Monitoring graph though...

Thanks again!

[bmeeks]

@sloppyjoe said in Suricata silently crashes with an "Out of swap space" error:

I'm still confused as to why I'm not seeing PfSense run out of memory in on the Status / Monitoring graph though...

I don't have an answer to that question. I would expect to see the graph mirror the log messages.

[sloppyjoe]

Ok, thanks again for your help with this!

[sloppyjoe]

Hi, wanted to update to this post because @bmeeks really helped me out here and maybe this will help someone else.

I played with disabling various services and it turns out that my problem was caused by vnstatd. I removed that package and my pfsense and suricata has been 100% stable ever since. Not sure what the issue was with that package but I wasn't really using it anyway.

Thanks again for the help with this! Really nice to have everything staying stable.