Suricata v7.0.7_5 abruptly stops

[bmeeks]

@anishkgt:
Your problem is a dynamic one. That means the issue occurs when certain things align in terms of high memory utilization. It appears from circumstantial evidence that it happens when your pfBlockerNG automated cron task runs coincident with Suricata operating.

Once the problem is triggered and the Out-of-Memory Reaper process in FreeBSD's kernel kills Suricata and/or unbound to free up RAM, the problem is gone until the next time the stars align in the software. So, you can run top or any other diagnostic tool all you want during the interim, but you will not see the problem unless you are actively running the tool and watching the screen output at the instant the OOM Reaper process activates and begins killing user processes to reclaim RAM. But even if you catch it I doubt you will learn much from it. The problem is simply an exhaustion of available RAM, and that triggers FreeBSD's kernel code to go into a self-protection mode where it will kill user processes in an attempt to free up RAM for use by the more critical kernel processes.

Here is a thread from the FreeBSD Forums explaining memory management in FreeBSD: https://forums.freebsd.org/threads/understanding-memory-management.84695/. The thread in turn has some links to addtional information. Read through this thread and follow some of the links in it to gain a better understanding of how FreeBSD handles memory allocation, recovery, and protection. This should help you understand that your issue is a dynamic one and is triggered only when a certain set of conditions come together. The root cause is processing too much data, but to find out which thing exactly is responsible will require a lengthy process of elimination by disabling all DNSBL lists and then slowly adding them back one by one to see which breaks the firewall. Processes that use Python or Perl regex pattern matching can be real memory hogs -- especially if the search algorithm is poorly implemented.

What I see a lot of pfBlockerNG and DNSBL users misunderstand here on the forum is that something that works just fine for a few hundred items does not necessarily scale to 10s of thousands of items (or even more!). These users fail to grasp that a list with 10,000 or 50,000 or even 100,000 items is not going to process as efficiently as a list with only 100 or 200 items. And with regex pattern matching, the growth in memory is not necessarily linear (meaning twice the number of list lines does not necessarily equal twice the required RAM -- it could be 4, 8 or 16 times more RAM).

This quote from an earlier reply hints that you are using some lists that need a lot memory resources to handle:

I remember changing the 'Firewall Maximum States' from 338,000 to 500,000 and 'Firewall Maximum Table Entries' from default to 800000.

[JonathanLee]

I have a 2100 and I use to have issues with memory on snort updates. I installed a swap partition on a dedicated external HDD drive that was designed for heavy use, and it fixed all my update resource issues. Long story short you have to free up memory or have a plan for when it is used the most. do not rely on swap all the time, but I admit I rely on it for ClamAV updates and snort updates should the happen at the same time, Murphys law when can go wrong will go wrong, some times my blacklist, snort and clamav all attempt to update at the same time it is very rare sometimes on reboots or package reinstalls but you got to plan for it. The 2100 should have an 8GBs ram option to function perfectly, again nothing is perfect so we got to roll with it. Do a flash drive and set it up as a swap.

“the SSD manufacture had this to say about me using it like this...
"Hi Jonathan,

This will damage the drive, it is not safe. Moreover, the response speed and read and write speed are far inferior to RAM. We recommend you not to use it this way, it will probably cause the SSD to become defective."

I really want to use something long term as I am limited on what I can do with this box it has hard set ram without any way to add or remove them. The NVMe drive is the only solution outside of a USB based HDD however that like you said is very slow.

The ZFS yes is a concern with the drive however it shows with gpart as FreeBSD

I triggered a panic and it works with crash dumps also. I had Netgate forum help me with this and FreeBSD forum. I am thinking I should use a actual USB based HDD in the long run to abuse it with swap use however with a firewall that would really slow it down.

Check out
ada0s3
Shell Output - gpart list -a”

Warning Do not use your internal SSD for swap.

Ref:

https://forums.freebsd.org/threads/resolved-usb-based-swap.93362/#post-654423

This FreeBSD research I did got me going also Netgate forum if you want to make a swap.