Suricata - increase in CPU use after upgrade to v6

digdug3 · Nov 25, 2021, 7:14 AM

@bmeeks You are right (of course!), I misread... Hope they find a solution soon.

bmeeks · Nov 25, 2021, 2:02 PM

@digdug3 said in Suricata - increase in CPU use after upgrade to v6:

@bmeeks You are right (of course!), I misread... Hope they find a solution soon.

Hopefully they will.

NollipfSense · Nov 26, 2021, 1:31 AM

@darcey said in Suricata - increase in CPU use after upgrade to v6:

pfSense is virtualised, in Proxmox, with 2 cores i7-3770S and 4GB RAM.

I would think the 4GB RAM for host and guest could be your problem unless that's allotted for guest only and even then, how many rules you have enabled could play significantly.

darcey · Nov 26, 2021, 9:09 AM

@nollipfsense Each suricata process uses around 700MB. The PVE host has an i7-3770S and 32GB RAM and serves me well. Of that, the pfsense guest is allocated 4GB and 2 cpus and even that level of resource allocation is somewhat under utilised. But cpu demand changes considerably with suricata 6. I'm sticking with suricata 5 for the time being.

digdug3 · Nov 26, 2021, 3:18 PM

@darcey @NollipfSense It's a KVM issue with usleep in suricata 6.x, but also some low level bare metal machines have it.

darcey · Nov 28, 2021, 6:50 PM

@digdug3 said in Suricata - increase in CPU use after upgrade to v6:

It's a KVM issue with usleep in suricata 6.x, but also some low level bare metal machines have it.

Yes, I'm regularly revisiting that thread. I wonder what, if any, options might mitigate this issue in the meantime. kvm module options were hinted at but they seem limited and would affect all VMs. Then there are qemu options, but I wouldn't know where to start. I'm currently running pfsense in a standard proxmox i440fx vm with host cpu, all network interfaces based on linux bridges, and no passthrough devices. Nothing fancy.

digdug3 · Nov 29, 2021, 6:48 AM

@darcey Could you try to disable "Enable HTTP log" and restart Suricata? It looks like the load is cut down by +50%

darcey · Nov 29, 2021, 9:39 AM

@digdug3 said:

Could you try to disable "Enable HTTP log" and restart Suricata? It looks like the load is cut down by +50%

Thanks @digdug3. I only have EVE logging enabled and, of the two interfaces monitored by suricata, only one is logging traffic. Should I have seen a noticable difference in the CPU utilisation between the two suricata processes? I don't recall that being the case. I'm back on version 5 for now so cannot test it right now. I will look more closely at the impact of logging when I next attempt the upgrade to v6. Do you have more info on that recommendation?
Something that seem to crop up in the discussion of this was the difference in cpu use reported in the guest vs the host. I don't recall seeing a significant difference in my case.

digdug3 · Nov 29, 2021, 2:34 PM

@darcey "Enable HTTP log" is enabled by default. I too have EVE logging on four interfaces and after disabling the "HTTP log" I saw a change in load from 5+ to a load between 2.26 and 2.70.
Even with EVE logging still enabled. I use pfSense on Proxmox, so your and my system are almost equal.

Also CPU usage was sometimes 100% and now between 20% and 50% (in pfSense itself, not in Proxmox, there the change from Suricata 5.x at 15% to Suricata 6.x at 50% is still very visible)

Just wanted to make sure it's not something else I changed.

darcey · Nov 29, 2021, 2:36 PM

@digdug3 I do have the standalone http log option disabled. I have basic logging (for http and several other ptotcols) enabled for eve output on one interface.
If I disable/reduce logging on an interface, I'd expect to see a load reduction in proportion to the volume of traffic on the interface, be it suricata 5 or 6. However the interface concerned is low traffic and the proportion of http is fairly low. I'm going to play around with it next time though. Thanks.