Suricata - swap_pager: out of swap space | Please help troubleshoot

br8bruno

Hello community,

I am running pfsense on a proxmox VM.

Hardware:
CPU: i7-8700k (6+6 cores)
RAM: 32GB DDR4
MOBO: Asus Maximus Hero X (onboard lan disabled)
NIC: X550-T2
PFSENSE: latest version: 2.7.2-RELEASE (amd64)

Basic config:
2x Interfaces vtnet (X550-T2)
4x VLANs
Everything has been working fine and I managed to separate devices inside my network which was my goal.

Suricata:
I set it up using youtube videos like everything else. I initially setup suricata for each interface and vlan separatelly, just to test since my pc has only this VM running for testing (resources should not be a problem).
I selected all rules and let it only alert without blocking anything for a week.
After that period I started switching each interface to INLINE IPS and choosing rules to drop. Mostly every BAD IP REPUTATION rule. And they were dropping a lot of stuff.
Today I had 5 interfaces enabled with INLINE IPS (LAN, WAN, 3x Vlans). And I was very happy with the results.

Problem:
I was then trying to forward a port for a NAS in my network and the router stopped working. No internet, no connection to other users. Only proxmox was still reachable.

I went to console and saw this through proxmox:

Failed to replenish seemed like a memory leak. I restarted the VM and watched it. I was thinking of restoring to a backup, but the error came very fast again. This time I saw a "swap_pager: out of swap space" before the "Failed to replenish" error took over the screen again:
67a8f57a-d8ac-4f05-92c2-a505311ceaad-pfsense-crash2.jpg

I read online on my cell phone and someone was talking about suricata using too much ram and filling swap. So I increased RAM to 30GB and gave 10 cores to pfsense. But the error still came every time after only a few seconds.

I managed to "Halt system" from the console after restart and it stopped suricata. I logged in and disabled all interfaces inside suricata.
If I start more than 1 again, the error comes really fast.

I tried monitoring "System Activity" when reproducing the error and swap was always at 1024M Total, 1024M Free. RAM was at 1,5GB Active and that is still very far from full.
This is what it looks like now without Suricata:

I am suprised with the number of threads sleeping and waiting. Other than that I don't understand where the problem came from. And why it keeps happening.

I would appreciate any help at all, as well as any answers to the questions I have upfront.

How do I stop Suriticata from loading together with pfsense? When I restart pfsense I don't want it to run automatically.
What are commands that I can use to monitor the system while reproducing the error?
What kind of information can I provide more to help find the cause of the problem?
Why are there so many threads with the same PID and should I kill them? How?

Thanks in advance!
Bruno

br8bruno

I removed most of the categories (rule sets) in the configuration, leaving only the ET active.
I also removed all VLAN interfaces from Suriticata, leaving only LAN still there.

And now I get the same error every time I turn Suricata on.
It does not make much sense to me. If I have 10 cores at 4,5GHz and 30GB of RAM, I should be able to run all ET rule categories on one interface, right?

bmeeks

How much RAM is allocated to the VM? I assume the RAM and CPU specs you stated are those of the hypervisor host and not an individual VM.

To effectively run Suricata you really want at least 4 GB of allocated RAM in the VM.

There have also been some issues reported with Proxmox and certain NIC drivers on FreeBSD. You may be hitting that problem.

Your VM should never be atempting to allocate any swap space. That indicates the VM is running out of available RAM and is trying to swap memory in and out to disk. That is very undesirable from a performance perspective.

br8bruno

@bmeeks

Hi, thanks for helping.
Those numbers were the actually available to the VM.
I was originally running 2 cores (4 Threads) and 8GB Ram on the VM. After the error I upped it to 5 cores (10 Threads) and 30GB Ram as you see in the picture of "System Activity" from Suriticata. And the errors keep on coming.
The proxmox host is a test environment. I have only this one VM running in it until I can get pfsense to run stable.
My thinking was to test what it needs first and then actually replicate the VM on another host.

The NIC X550-T2 was chosen for being one of the most reliable on FreeBSD according to my research prior to building a pfsense VM.
Every post I find online about this talks about not enough ressources available. But with 30GB Ram that should not be the case.

I kind of wish that people would help me with ideas on how to troubleshoot and find the problem.
I am missing references to know how the system should behave with Suricata.

bmeeks

@br8bruno said in Suricata - swap_pager: out of swap space | Please help troubleshoot:

The NIC X550-T2 was chosen for being one of the most reliable on FreeBSD according to my research prior to building a pfsense VM.

You are not actually using that NIC in Proxmox virtual machines unless you enable hardware passthrough. You are using the virtual adapter as logged here:

vtnet_netmap_rxsync ....

vtnet is the virtual adaptor. If you want to use the Intel NIC natively you will need to configure hardware passthrough of that NIC to the specific Suricata VM. That will, of course, remove the NIC from being shared by any other virtual machine or the host itself.

You are not the first user to post with problems using netmap devices with FreeBSD on Proxmox. It seeems there may be a current bug or incompatibility in Proxmox.

I run Suricata in VMware test virtual machines all the time without incident. Those VMs have usually 4 GB of RAM configured and a handful of CPU cores. Anecdotally, the two most often mentioned hypervisors when users post with Suricata problems are Hyper-V and Proxmox. I almost never see a posted issue where the hypervisor is VMware.

Since I do not use Proxmox, I can't really give you much help in troubleshooting your particular setup. All I can tell you for certain is that Suricata on pfSense works quite well on bare metal and also on VMware virtual machines.

I will offer one more suggestion. Suricata and VLANs are not good friends, especially with netmap in the mix. I do not recommend using VLANs with Suricata. And most certainly you will encounter problems if using Inline IPS Mode where the netmap device is pulled into the mix. The problem is that the netmap device and VLANs are fundamentally incompatible with the way Suricata inline IPS mode works.

SteveITS

@br8bruno FYI, @bmeeks is the package maintainer.

IIRC, if you have stopped Suricata on an interface, it won't load at boot. But I am not sure as I've never tried to not run it like that. You could of course uncheck the Enable box for each Suricata interface.

We routinely run Suricata in 4 GB of RAM, on bare metal. Our data center routers are using about 1 GB currently for pfSense, total. It should not need anywhere near 30 GB.

Check System/Activity or run top via SSH to view process information.

bmeeks

When you uncheck the Enable checkbox on the Suricata INTERFACES tab, that Suricata instance will never auto-start (even on a pfSense reboot). But if the Enable box is checked, the interface will auto-start upon a reboot of pfSense even if the interface had been manually stopped before the reboot.

One perverse thing with Suricata and the way it handles TCP sessions and flows is that the more CPU cores you throw at it, the more RAM it demands for the TCP Flow/Memcap parameter. Start simple with just 4 cores assigned to Suricata and 4 GB of RAM in the virtual machine.

And start with Legacy Blocking Mode instead of Inline IPS Mode. Legacy Blocking does not bring in the netmap kernel device and thus will likely bypass any issues existing there in Proxmox. And as I said earlier, you really can't successfully use VLANs with Inline IPS Mode (at least not without a lot of weirdness up to and including random crashes).