Suricata - increase in CPU use after upgrade to v6

darcey

I just upgraded the pfsense suricata package via the web UI to 6.0.3_2. This seems to introduce suricata-6.

Under suricata-5, total CPU was around 3% at idle. Now, it is 40% with each suricata process consuming virtually identical useage at around 17.5%. I have suricata running on two internal interfaces. Something, other than traffic analysis, would seem to be casuing the CPU utilisation. I looked through the config to see if there were any new settings introduced that might account for this but nothing struck me. Should I expect to see this increase in CPU utilisation with the move to surucata-6?
pfSense is virtualised, in Proxmox, with 2 cores i7-3770S and 4GB RAM.
Any hints much appreciated.

darcey

The increase in CPU use I've seen with Suricata 6 seems to be a consequence of changes to suricata's flow manager in conjunction with running pfsense as a kvm guest. Compounded by the fact I'm running on low end hardware, by 2021 standards.

https://forum.suricata.io/t/cpu-usage-of-version-6-0-0/706
https://redmine.openinfosecfoundation.org/issues/4096

bmeeks

@darcey said in Suricata - increase in CPU use after upgrade to v6:

https://forum.suricata.io/t/cpu-usage-of-version-6-0-0/706

Thank you for providing your research on this issue in your post. It is an upstream problem for them to solve. I found this tracking issue here for it on the Suricata Redmine site: https://redmine.openinfosecfoundation.org/issues/4379.

Looks like no real progress on resolving it to date, though. But in their defense, this is a difficult problem to solve because there are some diametrically opposite concerns to resolve. Read the notes in the linked issue for details.

darcey

@bmeeks There seems to be increased overhead with suricata6 running on any virtualisation platform. But it seems most acute with KVM. I wonder if there are qemu options that may help.

bmeeks

@darcey said in Suricata - increase in CPU use after upgrade to v6:

@bmeeks There seems to be increased overhead with suricata6 running on any virtualisation platform. But it seems most acute with KVM. I wonder if there are qemu options that may help.

Sorry, but I"m not a KVM expert. I run all my stuff on VMware, but still I do not consider myself a virtualization expert even there.

If you have followed all the linked posts both on the Suricata forum and the Suricata Redmine site, you can see that the flow manager threading code was changed quite significantly between version 5.x and 6.x of Suricata. Seems that helped things with some systems, but definitely hurt other systems. Virtualized systems seem to be particularly impacted as you note.

Obviously this problem will have to be resolved by the upstream Suricata team, so maybe it would help if you "ring their bell" a bit by posting in the Redmine Issue #4379.

darcey

I noticed that bug has been copied to 4421 which, unlike 4379, has been updated recently.

bmeeks

@darcey said in Suricata - increase in CPU use after upgrade to v6:

I noticed that bug has been copied to 4421 which, unlike 4379, has been updated recently.

Yes, I've seen a few other posts about this issue. Hopefully it's something the Suricata team can address in the near future. It appears from reading the notes that a one-size-fits-all solution is not available. It's more likely this becomes some type of configurable parameter users can customize to fit their hardware and environment.

digdug3

Same issue here. CPU usage of pfSense/Suricata 6 in Proxmox KVM tripled.
Is it possible to downgrade to Suricata 5?

digdug3

vjulien is currently looking into this. Please supply him with the info needed

bmeeks

@digdug3 said in Suricata - increase in CPU use after upgrade to v6:

Same issue here. CPU usage of pfSense/Suricata 6 in Proxmox KVM tripled.
Is it possible to downgrade to Suricata 5?

No, unfortunately it is not possible to downgrade to Suricata 5 on pfSense. There are several under-the-hood differences between Suricata 5 and Suricata 6 that would require rewriting portions of the GUI package if the underlying binary were changed.

digdug3

@bmeeks Looks like suricata 6.0.4 has been released with a "fix":
Suricata master

bmeeks

@digdug3 said in Suricata - increase in CPU use after upgrade to v6:

@bmeeks Looks like suricata 6.0.4 has been released with a "fix":
Suricata master

No, you are misreading the way Victor marks things. That change was actually released in Suricata 6.0.3 back in the earlier part of this year. That particular change was merged into what was then the 6.0.x master branch back in February, so it came out with 6.0.3. You can verify that by looking in the source code for Suricata 6.0.3. Here is the line from the flow-manager.c file in the 6.0.3 source code:

#ifdef FM_PROFILE
        struct timeval sleep_startts;
        memset(&sleep_startts, 0, sizeof(sleep_startts));
        gettimeofday(&sleep_startts, NULL);
#endif
        usleep(250);

Notice the change to the usleep() timer value is the same 250 microseconds. So that fix is already in the pfSense Suricata package. The change had minimal impact according to later postings on the Suricata forum. The overall ticket is still open for some kind of dynamic solution to the issue.

Here is the full list of actual changes in Suricata 6.0.4: https://redmine.openinfosecfoundation.org/versions/169. There is no mention in there of the usleep() CPU utilization bug.

digdug3

@bmeeks You are right (of course!), I misread... Hope they find a solution soon.

bmeeks

@digdug3 said in Suricata - increase in CPU use after upgrade to v6:

@bmeeks You are right (of course!), I misread... Hope they find a solution soon.

Hopefully they will.

NollipfSense

@darcey said in Suricata - increase in CPU use after upgrade to v6:

pfSense is virtualised, in Proxmox, with 2 cores i7-3770S and 4GB RAM.

I would think the 4GB RAM for host and guest could be your problem unless that's allotted for guest only and even then, how many rules you have enabled could play significantly.

darcey

@nollipfsense Each suricata process uses around 700MB. The PVE host has an i7-3770S and 32GB RAM and serves me well. Of that, the pfsense guest is allocated 4GB and 2 cpus and even that level of resource allocation is somewhat under utilised. But cpu demand changes considerably with suricata 6. I'm sticking with suricata 5 for the time being.

digdug3

@darcey @NollipfSense It's a KVM issue with usleep in suricata 6.x, but also some low level bare metal machines have it.

darcey

@digdug3 said in Suricata - increase in CPU use after upgrade to v6:

It's a KVM issue with usleep in suricata 6.x, but also some low level bare metal machines have it.

Yes, I'm regularly revisiting that thread. I wonder what, if any, options might mitigate this issue in the meantime. kvm module options were hinted at but they seem limited and would affect all VMs. Then there are qemu options, but I wouldn't know where to start. I'm currently running pfsense in a standard proxmox i440fx vm with host cpu, all network interfaces based on linux bridges, and no passthrough devices. Nothing fancy.

digdug3

@darcey Could you try to disable "Enable HTTP log" and restart Suricata? It looks like the load is cut down by +50%

darcey

@digdug3 said:

Could you try to disable "Enable HTTP log" and restart Suricata? It looks like the load is cut down by +50%

Thanks @digdug3. I only have EVE logging enabled and, of the two interfaces monitored by suricata, only one is logging traffic. Should I have seen a noticable difference in the CPU utilisation between the two suricata processes? I don't recall that being the case. I'm back on version 5 for now so cannot test it right now. I will look more closely at the impact of logging when I next attempt the upgrade to v6. Do you have more info on that recommendation?
Something that seem to crop up in the discussion of this was the difference in cpu use reported in the guest vs the host. I don't recall seeing a significant difference in my case.