Suricata process dying due to hyperscan problem
-
@bmeeks
still running man .. still running ....
for me it's a record, 10 min without any hyperscan error -
@kiokoman said in Suricata process dying due to hyperscan problem:
@bmeeks
still running man .. still running ....
for me it's a record, 10 min without any hyperscan errorThis is good! Maybe I did find it this time.
The ASAN-enabled debug package is going to run quite a bit slower. If your results continue to be positive into the next day, then I will prepare a pull request with the fixes and get it submitted to the Netgate developer team so we can get a new package posted for production use.
There were two places in the custom blocking module code where an IPv4 and an IPv6 address were being copied using
memcpy()and I had left off the address-of operator on one of the parameters (the destination memory address). That resulted in heap buffer overflows that would have produced random results depending on the runtime memory configuration of a particular installation. One manifestation could be corrupted pointers that were later passed to Hyperscan.This would explain the connection between disabling Legacy Blocking Mode and successful execution with Hyperscan enabled.
-
My Suricata instances have mostly been running fine since I installed your latest debug package. However, I've noticed a major change in the memory usage and during a rule update tonight both instances were killed due to lack of memory. I assume the memory usage will come back down once a non-debug enabled package is re-deployed, but I just thought I should point this out in case it's helpful.
This graph shows memory usage for the past two weeks.
This graph shows memory usage before and after installing the debug package.
Finally, this graph shows the Suricata instances being killed following a rule update. I had to delete the PID files in order to get Suricata to start.
-
@masons:
1, How much RAM is installed in the firewall?
2. Are you running with a ZFS or UFS installation?The ASAN-enabled build will run quite a bit slower than a regular build, but I would not expect it to consume all that much more memory.
Changing the multi-pattern matcher algorithm can have a very large impact on memory usage, though.
-
@kiokoman: do you still have good news for me about the patched Suricata code
?If you do, I will create the pull requests for the production update and get them submitted.
-
I have submitted the pull request that hopefully contains the final fix for this Hyperscan crash. It should also address the instances of Signal 11 segfault crashes some other users have experienced in Suricata.
Here is the request: https://github.com/pfsense/FreeBSD-ports/pull/1337.
There is also a GUI package update to match: https://github.com/pfsense/FreeBSD-ports/pull/1338.
Look for a new 7.0.2_3 version of the Suricata package to show up soon.
-
@bmeeks
yup, still running, i didn't check the memory usage before the patch, i'm around44% of 16321 MiB
maybe i have 4% 5% more ram usage in total but i have other services like telegraf / haproxy / 5 openvpn / 1 wireguard and at this time there are alot of traffic
-
@kiokoman said in Suricata process dying due to hyperscan problem:
@bmeeks
yup, still running, i didn't check the memory usage before the patch, i'm around44% of 16321 MiB
maybe i have 4% 5% more ram usage in total but i have other services like telegraf / haproxy / 5 openvpn / 1 wireguard and at this time there are alot of traffic
Great news!
As you see immediately above, I've posted the pull request containing the bug fix for the Netgate developer team to review and merge. Also sent them an email request. Hopefully they get to it soon.
The new package version will be 7.0.2_3 (for the GUI) and will be a normal non-debug production build.
-
@bmeeks
Great news indeed. Bill, do you have a paetron or some method that we, the community, can contribute to you? Your hard work, your efforts, are not unnoticed, and truly appreciate the efforts you place into making pfSense better with the integration of Suricata. -
@michmoor said in Suricata process dying due to hyperscan problem:
@bmeeks
Great news indeed. Bill, do you have a paetron or some method that we, the community, can contribute to you? Your hard work, your efforts, are not unnoticed, and truly appreciate the efforts you place into making pfSense better with the integration of Suricata.No, I do not have a Paetron account. I do this in the spirit of open-source free software, so I don't expect to make money from it. But thank you for offering.
-
Just installed 7.0.2_3 a few minutes ago, thanks for getting it fixed....
-
Just installed 7.0.2_3 on two 6100 in HA. Suricata on three interfaces with MPM: Auto and ET Pro ruleset. No Signal 11 segfault on suricata start now but the three processes of suricata takes 100% cpu and interface goes down with failover to secondary node. After failover, primary node gui hangs and ssh reveals that the three suricata processes takes 100% cpu. Around 30 seconds after failover on secondary node gui hangs there aswell and suricata shows 100% and never let go. After gui hang traffic stops on the interfaces. Suricata 7.0.2_2 was removed and 7.0.2_3 was installed and fw was rebooted after install.
Same behaviour was observed with AC-BS on 7.0.2_2.
No issues with 21.05.1 and that suricata version.
-
@bmeeks said in Suricata process dying due to hyperscan problem:
1, How much RAM is installed in the firewall?
4GB
- Are you running with a ZFS or UFS installation?
ZFS
After uninstalling the debug Suricata package and installing the new Suricata plugin, memory usage is back down to around 30%.
Thank you so much @bmeeks for getting to the bottom of this Hyperscan issue!
-
@btspce said in Suricata process dying due to hyperscan problem:
Just installed 7.0.2_3 on two 6100 in HA. Suricata on three interfaces with MPM: Auto and ET Pro ruleset. No Signal 11 segfault on suricata start now but the three processes of suricata takes 100% cpu and interface goes down with failover to secondary node. After failover, primary node gui hangs and ssh reveals that the three suricata processes takes 100% cpu. Around 30 seconds after failover on secondary node gui hangs there aswell and suricata shows 100% and never let go. After gui hang traffic stops on the interfaces. Suricata 7.0.2_2 was removed and 7.0.2_3 was installed and fw was rebooted after install.
Same behaviour was observed with AC-BS on 7.0.2_2.
No issues with 21.05.1 and that suricata version.
This is more like what I'm seeing as well on my 2100, only one WAN, though, so no failover, and AC-BS works for me. Suricata uses 100% CPU for a while after it launches, then the kernel eventually kills it, if I'm lucky. If not, the system gets VERY unstable for a while, eventually ending with Suricata and unbound both dead, and I have to log in from the LAN side to the IP address directly to fix things, since there's no DNS response.
As I've noted several times above, this is not just Hyperscan for me - it's all algorithms except for AC-BS. The latest update did not fix it. :(
-
I installed package version 7.0.2_3 as soon as it was available, I set the Multi-Pattern Matcher Algorithm to Auto again and so far everything is working perfectly. I haven't had any errors caused by hyperscan.
Great job @bmeeks! Thank you for your efforts to identify the problem! -
@Maltz said in Suricata process dying due to hyperscan problem:
This is more like what I'm seeing as well on my 2100, only one WAN, though, so no failover, and AC-BS works for me. Suricata uses 100% CPU for a while after it launches, then the kernel eventually kills it, if I'm lucky. If not, the system gets VERY unstable for a while, eventually ending with Suricata and unbound both dead, and I have to log in from the LAN side to the IP address directly to fix things, since there's no DNS response.
As I've noted several times above, this is not just Hyperscan for me - it's all algorithms except for AC-BS. The latest update did not fix it. :(
The SG-2100 has an ARM Cortex CPU (not Intel architecture), so Hyperscan does not and cannot work on that platform nor any other ARM platform. The Hyperscan library is completely excluded from the Suricata binary build on ARM platforms. Hyperscan is a technology written by Intel exclusively for use on their CPUs.
When you choose Auto for the Multi-Pattern Matcher algorithm, Suricata will use Hyperscan if it is available, then default to use AC otherwise. Because you have an ARM CPU in the SG-2100, then Suricata will never choose Hyperscan when set to Auto. It will instead automatically use AC.
Fiddling with the Pattern Matcher settings can lead to huge increases in RAM usage, and your SG-2100 has a very limited amount of RAM to begin with. Leave it on Auto. If you are also using ZFS, that will compound the limited RAM problem because of competition from the ARC cache. And during the rules update process, the amount of RAM needed by Suricata will sharply increase (especially if you have "Live Rule Swap" enabled).
-
@bmeeks
7.0.2_3 system all green
-
@bmeeks I am set to Auto when the error occurs. And the error doesn't just happen when I choose Hyperscan - I've never set it to that except when testing these new builds. None of the other AC choices even work, it's strictly AC-BS that works. When I choose it, my RAM usage is around 30%. When I choose anything else, Auto, Hyperscan, or any of the other AC flavors, I see the symptoms I described above.
These issues first appeared when I upgraded to 23.09 and its accompanying Suricata version, and none of the patches to pfSense or Suricata since has helped so far. Maybe this is a third issue? If so, there's another more specific thread about this, so maybe we should pick up there...
https://forum.netgate.com/topic/184119/suricata-7-0-0-being-killed-by-kernel-in-23-09/11
-
@Maltz said in Suricata process dying due to hyperscan problem:
@bmeeks I am set to Auto when the error occurs. And the error doesn't just happen when I choose Hyperscan - I've never set it to that except when testing these new builds. None of the other AC choices even work, it's strictly AC-BS that works. When I choose it, my RAM usage is around 30%. When I choose anything else, Auto, Hyperscan, or any of the other AC flavors, I see the symptoms I described above.
These issues first appeared when I upgraded to 23.09 and its accompanying Suricata version, and none of the patches to pfSense or Suricata since has helped so far. Maybe this is a third issue? If so, there's another more specific thread about this, so maybe we should pick up there...
https://forum.netgate.com/topic/184119/suricata-7-0-0-being-killed-by-kernel-in-23-09/11
Is this with the SG-2100? If so, then you are on the very ragged edge of not having enough free RAM. Suricata 7 requires more RAM than the previous 6.x version (mostly due to the increased requirements for TCP Stream Memory and reassembly space).
I don't recall which one does what, but the various available Pattern Matcher choices differ in the amount of RAM they consume to operate. Some are slower with pattern matches but stingy with RAM, while others can match patterns much faster but at the expense of allocating and using large amounts of RAM to accomplish this. The fact only one of the available pattern matchers works for you indicates you really do not have enough RAM in the firewall to run Suricata 7.x with the rules you have enabled. You can try drastically trimming back the number of enabled rules to see if that helps. If you are using ZFS (which the is the default since pfSense Plus 22.01, then that puts further pressure on the available RAM as ZFS needs memory for the ARC cache.
-
@bmeeks With 7.0.2_3 Suricata now starts and runs on our two 6100 in HA but as soon as there is traffic on the interface cpu goes to 100% and firewall gui becomes unresponsive after about 30 seconds. No problems when we used 23.05.1 which we had before update. Memory around 20-30% which is normal and as it was under 23.05.1.
AC-BS or Hyperscan does not matter. Suricata runs on secondary without cpu spikes until we initiate a failover and it begins to recieve traffic.
Primary hangs gui within 30 seconds if suricata is started.
No config changes since update from 23.05.1Any tips on what we could test or check? Is there anything in the new version of suricata that could load the cpu like this ?
Traffic on the 3 interfaces was between 10-100Mbit when we were testing.
