Suricata process dying due to hyperscan problem
-
@bmeeks
yup, still running, i didn't check the memory usage before the patch, i'm around44% of 16321 MiB
maybe i have 4% 5% more ram usage in total but i have other services like telegraf / haproxy / 5 openvpn / 1 wireguard and at this time there are alot of traffic
-
@kiokoman said in Suricata process dying due to hyperscan problem:
@bmeeks
yup, still running, i didn't check the memory usage before the patch, i'm around44% of 16321 MiB
maybe i have 4% 5% more ram usage in total but i have other services like telegraf / haproxy / 5 openvpn / 1 wireguard and at this time there are alot of traffic
Great news!
As you see immediately above, I've posted the pull request containing the bug fix for the Netgate developer team to review and merge. Also sent them an email request. Hopefully they get to it soon.
The new package version will be 7.0.2_3 (for the GUI) and will be a normal non-debug production build.
-
@bmeeks
Great news indeed. Bill, do you have a paetron or some method that we, the community, can contribute to you? Your hard work, your efforts, are not unnoticed, and truly appreciate the efforts you place into making pfSense better with the integration of Suricata. -
@michmoor said in Suricata process dying due to hyperscan problem:
@bmeeks
Great news indeed. Bill, do you have a paetron or some method that we, the community, can contribute to you? Your hard work, your efforts, are not unnoticed, and truly appreciate the efforts you place into making pfSense better with the integration of Suricata.No, I do not have a Paetron account. I do this in the spirit of open-source free software, so I don't expect to make money from it. But thank you for offering.
-
Just installed 7.0.2_3 a few minutes ago, thanks for getting it fixed....
-
Just installed 7.0.2_3 on two 6100 in HA. Suricata on three interfaces with MPM: Auto and ET Pro ruleset. No Signal 11 segfault on suricata start now but the three processes of suricata takes 100% cpu and interface goes down with failover to secondary node. After failover, primary node gui hangs and ssh reveals that the three suricata processes takes 100% cpu. Around 30 seconds after failover on secondary node gui hangs there aswell and suricata shows 100% and never let go. After gui hang traffic stops on the interfaces. Suricata 7.0.2_2 was removed and 7.0.2_3 was installed and fw was rebooted after install.
Same behaviour was observed with AC-BS on 7.0.2_2.
No issues with 21.05.1 and that suricata version.
-
@bmeeks said in Suricata process dying due to hyperscan problem:
1, How much RAM is installed in the firewall?
4GB
- Are you running with a ZFS or UFS installation?
ZFS
After uninstalling the debug Suricata package and installing the new Suricata plugin, memory usage is back down to around 30%.
Thank you so much @bmeeks for getting to the bottom of this Hyperscan issue!
-
@btspce said in Suricata process dying due to hyperscan problem:
Just installed 7.0.2_3 on two 6100 in HA. Suricata on three interfaces with MPM: Auto and ET Pro ruleset. No Signal 11 segfault on suricata start now but the three processes of suricata takes 100% cpu and interface goes down with failover to secondary node. After failover, primary node gui hangs and ssh reveals that the three suricata processes takes 100% cpu. Around 30 seconds after failover on secondary node gui hangs there aswell and suricata shows 100% and never let go. After gui hang traffic stops on the interfaces. Suricata 7.0.2_2 was removed and 7.0.2_3 was installed and fw was rebooted after install.
Same behaviour was observed with AC-BS on 7.0.2_2.
No issues with 21.05.1 and that suricata version.
This is more like what I'm seeing as well on my 2100, only one WAN, though, so no failover, and AC-BS works for me. Suricata uses 100% CPU for a while after it launches, then the kernel eventually kills it, if I'm lucky. If not, the system gets VERY unstable for a while, eventually ending with Suricata and unbound both dead, and I have to log in from the LAN side to the IP address directly to fix things, since there's no DNS response.
As I've noted several times above, this is not just Hyperscan for me - it's all algorithms except for AC-BS. The latest update did not fix it. :(
-
I installed package version 7.0.2_3 as soon as it was available, I set the Multi-Pattern Matcher Algorithm to Auto again and so far everything is working perfectly. I haven't had any errors caused by hyperscan.
Great job @bmeeks! Thank you for your efforts to identify the problem! -
@Maltz said in Suricata process dying due to hyperscan problem:
This is more like what I'm seeing as well on my 2100, only one WAN, though, so no failover, and AC-BS works for me. Suricata uses 100% CPU for a while after it launches, then the kernel eventually kills it, if I'm lucky. If not, the system gets VERY unstable for a while, eventually ending with Suricata and unbound both dead, and I have to log in from the LAN side to the IP address directly to fix things, since there's no DNS response.
As I've noted several times above, this is not just Hyperscan for me - it's all algorithms except for AC-BS. The latest update did not fix it. :(
The SG-2100 has an ARM Cortex CPU (not Intel architecture), so Hyperscan does not and cannot work on that platform nor any other ARM platform. The Hyperscan library is completely excluded from the Suricata binary build on ARM platforms. Hyperscan is a technology written by Intel exclusively for use on their CPUs.
When you choose Auto for the Multi-Pattern Matcher algorithm, Suricata will use Hyperscan if it is available, then default to use AC otherwise. Because you have an ARM CPU in the SG-2100, then Suricata will never choose Hyperscan when set to Auto. It will instead automatically use AC.
Fiddling with the Pattern Matcher settings can lead to huge increases in RAM usage, and your SG-2100 has a very limited amount of RAM to begin with. Leave it on Auto. If you are also using ZFS, that will compound the limited RAM problem because of competition from the ARC cache. And during the rules update process, the amount of RAM needed by Suricata will sharply increase (especially if you have "Live Rule Swap" enabled).
-
@bmeeks
7.0.2_3 system all green
-
@bmeeks I am set to Auto when the error occurs. And the error doesn't just happen when I choose Hyperscan - I've never set it to that except when testing these new builds. None of the other AC choices even work, it's strictly AC-BS that works. When I choose it, my RAM usage is around 30%. When I choose anything else, Auto, Hyperscan, or any of the other AC flavors, I see the symptoms I described above.
These issues first appeared when I upgraded to 23.09 and its accompanying Suricata version, and none of the patches to pfSense or Suricata since has helped so far. Maybe this is a third issue? If so, there's another more specific thread about this, so maybe we should pick up there...
https://forum.netgate.com/topic/184119/suricata-7-0-0-being-killed-by-kernel-in-23-09/11
-
@Maltz said in Suricata process dying due to hyperscan problem:
@bmeeks I am set to Auto when the error occurs. And the error doesn't just happen when I choose Hyperscan - I've never set it to that except when testing these new builds. None of the other AC choices even work, it's strictly AC-BS that works. When I choose it, my RAM usage is around 30%. When I choose anything else, Auto, Hyperscan, or any of the other AC flavors, I see the symptoms I described above.
These issues first appeared when I upgraded to 23.09 and its accompanying Suricata version, and none of the patches to pfSense or Suricata since has helped so far. Maybe this is a third issue? If so, there's another more specific thread about this, so maybe we should pick up there...
https://forum.netgate.com/topic/184119/suricata-7-0-0-being-killed-by-kernel-in-23-09/11
Is this with the SG-2100? If so, then you are on the very ragged edge of not having enough free RAM. Suricata 7 requires more RAM than the previous 6.x version (mostly due to the increased requirements for TCP Stream Memory and reassembly space).
I don't recall which one does what, but the various available Pattern Matcher choices differ in the amount of RAM they consume to operate. Some are slower with pattern matches but stingy with RAM, while others can match patterns much faster but at the expense of allocating and using large amounts of RAM to accomplish this. The fact only one of the available pattern matchers works for you indicates you really do not have enough RAM in the firewall to run Suricata 7.x with the rules you have enabled. You can try drastically trimming back the number of enabled rules to see if that helps. If you are using ZFS (which the is the default since pfSense Plus 22.01, then that puts further pressure on the available RAM as ZFS needs memory for the ARC cache.
-
@bmeeks With 7.0.2_3 Suricata now starts and runs on our two 6100 in HA but as soon as there is traffic on the interface cpu goes to 100% and firewall gui becomes unresponsive after about 30 seconds. No problems when we used 23.05.1 which we had before update. Memory around 20-30% which is normal and as it was under 23.05.1.
AC-BS or Hyperscan does not matter. Suricata runs on secondary without cpu spikes until we initiate a failover and it begins to recieve traffic.
Primary hangs gui within 30 seconds if suricata is started.
No config changes since update from 23.05.1Any tips on what we could test or check? Is there anything in the new version of suricata that could load the cpu like this ?
Traffic on the 3 interfaces was between 10-100Mbit when we were testing. -
AC-BS was working fine, but that also died on me today. I'm switching back to auto and see what happens with Hyper scan.
-
@btspce said in Suricata process dying due to hyperscan problem:
Any tips on what we could test or check?
No, I'm afraid not.
If you have ZFS and Boot Environments, then rollback to 23.05.1 and run that Suricata version.
If you want to stay on 23.09.1, then you will need to uninstall Suricata it seems.
-
@btspce Maybe try disabling a category at a time? Or a binary test/search of disabling categories? We have a 6100 pair on 7.0.2_2 an AC-BS without issue and a 4860 on 7.0.2_3/Auto.
-
@SteveITS Thanks for this info. So that seems to point to one of the new features being the culprit and a config difference between our 6100 pair and yours. Its very hard to diagnose as these firewalls are in production and the gui freeze so fast.
It could simply be that with the new version of suricata it does more scanning and the cpu hits the roof but we were only seeing around 100Mbit of traffic over 3 interfaces in total.
Are your 6100 pair running inline or legacy blocking mode? -
@btspce Legacy mode.
-
Here is what I tested that did not work on our 6100 ha pair:
Disabled Quic parser
Switching to AC-BS
Disabled blocking
Deleted all log filesSuricata loads on interface ix1 and igc0 without problems which is also carp/ha interfaces.
Suricata loads on ix0 but around 5 seconds after loading is complete and cpu load drops, traffic stops to flow and both firewalls is stuck as master for that particular interface until reboot of both firewalls.
So only interface ix0 seems to have problem.Tried to raise the Carp Advertising Frequency Base from 1 to 5 seconds but that did not help.
Either Suricata is blocking carp heartbeats on this interface (could not see anything in the suricata logs) or I need to up the Carp Advertising Frequency Base more so it doesn't failover during loading.
