Suricata - increase in CPU use after upgrade to v6

bmeeks · Oct 15, 2021, 5:56 PM

@darcey said in Suricata - increase in CPU use after upgrade to v6:

I noticed that bug has been copied to 4421 which, unlike 4379, has been updated recently.

Yes, I've seen a few other posts about this issue. Hopefully it's something the Suricata team can address in the near future. It appears from reading the notes that a one-size-fits-all solution is not available. It's more likely this becomes some type of configurable parameter users can customize to fit their hardware and environment.

digdug3 · Nov 8, 2021, 1:19 PM

Same issue here. CPU usage of pfSense/Suricata 6 in Proxmox KVM tripled.
Is it possible to downgrade to Suricata 5?

digdug3 · Nov 8, 2021, 12:23 PM

vjulien is currently looking into this. Please supply him with the info needed

bmeeks · Nov 8, 2021, 1:19 PM

@digdug3 said in Suricata - increase in CPU use after upgrade to v6:

Same issue here. CPU usage of pfSense/Suricata 6 in Proxmox KVM tripled.
Is it possible to downgrade to Suricata 5?

No, unfortunately it is not possible to downgrade to Suricata 5 on pfSense. There are several under-the-hood differences between Suricata 5 and Suricata 6 that would require rewriting portions of the GUI package if the underlying binary were changed.

digdug3 · Nov 24, 2021, 6:10 PM

@bmeeks Looks like suricata 6.0.4 has been released with a "fix":
Suricata master

bmeeks · Nov 24, 2021, 9:51 PM

@digdug3 said in Suricata - increase in CPU use after upgrade to v6:

@bmeeks Looks like suricata 6.0.4 has been released with a "fix":
Suricata master

No, you are misreading the way Victor marks things. That change was actually released in Suricata 6.0.3 back in the earlier part of this year. That particular change was merged into what was then the 6.0.x master branch back in February, so it came out with 6.0.3. You can verify that by looking in the source code for Suricata 6.0.3. Here is the line from the flow-manager.c file in the 6.0.3 source code:

#ifdef FM_PROFILE
        struct timeval sleep_startts;
        memset(&sleep_startts, 0, sizeof(sleep_startts));
        gettimeofday(&sleep_startts, NULL);
#endif
        usleep(250);

Notice the change to the usleep() timer value is the same 250 microseconds. So that fix is already in the pfSense Suricata package. The change had minimal impact according to later postings on the Suricata forum. The overall ticket is still open for some kind of dynamic solution to the issue.

Here is the full list of actual changes in Suricata 6.0.4: https://redmine.openinfosecfoundation.org/versions/169. There is no mention in there of the usleep() CPU utilization bug.

digdug3 · Nov 25, 2021, 7:14 AM

@bmeeks You are right (of course!), I misread... Hope they find a solution soon.

bmeeks · Nov 25, 2021, 2:02 PM

@digdug3 said in Suricata - increase in CPU use after upgrade to v6:

@bmeeks You are right (of course!), I misread... Hope they find a solution soon.

Hopefully they will.

NollipfSense · Nov 26, 2021, 1:31 AM

@darcey said in Suricata - increase in CPU use after upgrade to v6:

pfSense is virtualised, in Proxmox, with 2 cores i7-3770S and 4GB RAM.

I would think the 4GB RAM for host and guest could be your problem unless that's allotted for guest only and even then, how many rules you have enabled could play significantly.

darcey · Nov 26, 2021, 9:09 AM

@nollipfsense Each suricata process uses around 700MB. The PVE host has an i7-3770S and 32GB RAM and serves me well. Of that, the pfsense guest is allocated 4GB and 2 cpus and even that level of resource allocation is somewhat under utilised. But cpu demand changes considerably with suricata 6. I'm sticking with suricata 5 for the time being.

digdug3 · Nov 26, 2021, 3:18 PM

@darcey @NollipfSense It's a KVM issue with usleep in suricata 6.x, but also some low level bare metal machines have it.

darcey · Nov 28, 2021, 6:50 PM

@digdug3 said in Suricata - increase in CPU use after upgrade to v6:

It's a KVM issue with usleep in suricata 6.x, but also some low level bare metal machines have it.

Yes, I'm regularly revisiting that thread. I wonder what, if any, options might mitigate this issue in the meantime. kvm module options were hinted at but they seem limited and would affect all VMs. Then there are qemu options, but I wouldn't know where to start. I'm currently running pfsense in a standard proxmox i440fx vm with host cpu, all network interfaces based on linux bridges, and no passthrough devices. Nothing fancy.

digdug3 · Nov 29, 2021, 6:48 AM

@darcey Could you try to disable "Enable HTTP log" and restart Suricata? It looks like the load is cut down by +50%

darcey · Nov 29, 2021, 9:39 AM

@digdug3 said:

Could you try to disable "Enable HTTP log" and restart Suricata? It looks like the load is cut down by +50%

Thanks @digdug3. I only have EVE logging enabled and, of the two interfaces monitored by suricata, only one is logging traffic. Should I have seen a noticable difference in the CPU utilisation between the two suricata processes? I don't recall that being the case. I'm back on version 5 for now so cannot test it right now. I will look more closely at the impact of logging when I next attempt the upgrade to v6. Do you have more info on that recommendation?
Something that seem to crop up in the discussion of this was the difference in cpu use reported in the guest vs the host. I don't recall seeing a significant difference in my case.

digdug3 · Nov 29, 2021, 2:34 PM

@darcey "Enable HTTP log" is enabled by default. I too have EVE logging on four interfaces and after disabling the "HTTP log" I saw a change in load from 5+ to a load between 2.26 and 2.70.
Even with EVE logging still enabled. I use pfSense on Proxmox, so your and my system are almost equal.

Also CPU usage was sometimes 100% and now between 20% and 50% (in pfSense itself, not in Proxmox, there the change from Suricata 5.x at 15% to Suricata 6.x at 50% is still very visible)

Just wanted to make sure it's not something else I changed.

darcey · Nov 29, 2021, 2:36 PM

@digdug3 I do have the standalone http log option disabled. I have basic logging (for http and several other ptotcols) enabled for eve output on one interface.
If I disable/reduce logging on an interface, I'd expect to see a load reduction in proportion to the volume of traffic on the interface, be it suricata 5 or 6. However the interface concerned is low traffic and the proportion of http is fairly low. I'm going to play around with it next time though. Thanks.