Suricata process dying due to hyperscan problem

michmoor

@bmeeks
Great news indeed. Bill, do you have a paetron or some method that we, the community, can contribute to you? Your hard work, your efforts, are not unnoticed, and truly appreciate the efforts you place into making pfSense better with the integration of Suricata.

bmeeks

@michmoor said in Suricata process dying due to hyperscan problem:

@bmeeks
Great news indeed. Bill, do you have a paetron or some method that we, the community, can contribute to you? Your hard work, your efforts, are not unnoticed, and truly appreciate the efforts you place into making pfSense better with the integration of Suricata.

No, I do not have a Paetron account. I do this in the spirit of open-source free software, so I don't expect to make money from it. But thank you for offering.

Negan

Just installed 7.0.2_3 a few minutes ago, thanks for getting it fixed....

btspce

Just installed 7.0.2_3 on two 6100 in HA. Suricata on three interfaces with MPM: Auto and ET Pro ruleset. No Signal 11 segfault on suricata start now but the three processes of suricata takes 100% cpu and interface goes down with failover to secondary node. After failover, primary node gui hangs and ssh reveals that the three suricata processes takes 100% cpu. Around 30 seconds after failover on secondary node gui hangs there aswell and suricata shows 100% and never let go. After gui hang traffic stops on the interfaces. Suricata 7.0.2_2 was removed and 7.0.2_3 was installed and fw was rebooted after install.

Same behaviour was observed with AC-BS on 7.0.2_2.

No issues with 21.05.1 and that suricata version.

masons

@bmeeks said in Suricata process dying due to hyperscan problem:

1, How much RAM is installed in the firewall?

4GB

Are you running with a ZFS or UFS installation?

ZFS

After uninstalling the debug Suricata package and installing the new Suricata plugin, memory usage is back down to around 30%.

Thank you so much @bmeeks for getting to the bottom of this Hyperscan issue!

Maltz

@btspce said in Suricata process dying due to hyperscan problem:

Just installed 7.0.2_3 on two 6100 in HA. Suricata on three interfaces with MPM: Auto and ET Pro ruleset. No Signal 11 segfault on suricata start now but the three processes of suricata takes 100% cpu and interface goes down with failover to secondary node. After failover, primary node gui hangs and ssh reveals that the three suricata processes takes 100% cpu. Around 30 seconds after failover on secondary node gui hangs there aswell and suricata shows 100% and never let go. After gui hang traffic stops on the interfaces. Suricata 7.0.2_2 was removed and 7.0.2_3 was installed and fw was rebooted after install.

Same behaviour was observed with AC-BS on 7.0.2_2.

No issues with 21.05.1 and that suricata version.

This is more like what I'm seeing as well on my 2100, only one WAN, though, so no failover, and AC-BS works for me. Suricata uses 100% CPU for a while after it launches, then the kernel eventually kills it, if I'm lucky. If not, the system gets VERY unstable for a while, eventually ending with Suricata and unbound both dead, and I have to log in from the LAN side to the IP address directly to fix things, since there's no DNS response.

As I've noted several times above, this is not just Hyperscan for me - it's all algorithms except for AC-BS. The latest update did not fix it. :(

paulp

I installed package version 7.0.2_3 as soon as it was available, I set the Multi-Pattern Matcher Algorithm to Auto again and so far everything is working perfectly. I haven't had any errors caused by hyperscan.
Great job @bmeeks! Thank you for your efforts to identify the problem!

bmeeks

@Maltz said in Suricata process dying due to hyperscan problem:

This is more like what I'm seeing as well on my 2100, only one WAN, though, so no failover, and AC-BS works for me. Suricata uses 100% CPU for a while after it launches, then the kernel eventually kills it, if I'm lucky. If not, the system gets VERY unstable for a while, eventually ending with Suricata and unbound both dead, and I have to log in from the LAN side to the IP address directly to fix things, since there's no DNS response.

As I've noted several times above, this is not just Hyperscan for me - it's all algorithms except for AC-BS. The latest update did not fix it. :(

The SG-2100 has an ARM Cortex CPU (not Intel architecture), so Hyperscan does not and cannot work on that platform nor any other ARM platform. The Hyperscan library is completely excluded from the Suricata binary build on ARM platforms. Hyperscan is a technology written by Intel exclusively for use on their CPUs.

When you choose Auto for the Multi-Pattern Matcher algorithm, Suricata will use Hyperscan if it is available, then default to use AC otherwise. Because you have an ARM CPU in the SG-2100, then Suricata will never choose Hyperscan when set to Auto. It will instead automatically use AC.

Fiddling with the Pattern Matcher settings can lead to huge increases in RAM usage, and your SG-2100 has a very limited amount of RAM to begin with. Leave it on Auto. If you are also using ZFS, that will compound the limited RAM problem because of competition from the ARC cache. And during the rules update process, the amount of RAM needed by Suricata will sharply increase (especially if you have "Live Rule Swap" enabled).

kiokoman

@bmeeks
7.0.2_3 system all green

Maltz

@bmeeks I am set to Auto when the error occurs. And the error doesn't just happen when I choose Hyperscan - I've never set it to that except when testing these new builds. None of the other AC choices even work, it's strictly AC-BS that works. When I choose it, my RAM usage is around 30%. When I choose anything else, Auto, Hyperscan, or any of the other AC flavors, I see the symptoms I described above.

These issues first appeared when I upgraded to 23.09 and its accompanying Suricata version, and none of the patches to pfSense or Suricata since has helped so far. Maybe this is a third issue? If so, there's another more specific thread about this, so maybe we should pick up there...

https://forum.netgate.com/topic/184119/suricata-7-0-0-being-killed-by-kernel-in-23-09/11

bmeeks

@Maltz said in Suricata process dying due to hyperscan problem:

@bmeeks I am set to Auto when the error occurs. And the error doesn't just happen when I choose Hyperscan - I've never set it to that except when testing these new builds. None of the other AC choices even work, it's strictly AC-BS that works. When I choose it, my RAM usage is around 30%. When I choose anything else, Auto, Hyperscan, or any of the other AC flavors, I see the symptoms I described above.

These issues first appeared when I upgraded to 23.09 and its accompanying Suricata version, and none of the patches to pfSense or Suricata since has helped so far. Maybe this is a third issue? If so, there's another more specific thread about this, so maybe we should pick up there...

https://forum.netgate.com/topic/184119/suricata-7-0-0-being-killed-by-kernel-in-23-09/11

Is this with the SG-2100? If so, then you are on the very ragged edge of not having enough free RAM. Suricata 7 requires more RAM than the previous 6.x version (mostly due to the increased requirements for TCP Stream Memory and reassembly space).

I don't recall which one does what, but the various available Pattern Matcher choices differ in the amount of RAM they consume to operate. Some are slower with pattern matches but stingy with RAM, while others can match patterns much faster but at the expense of allocating and using large amounts of RAM to accomplish this. The fact only one of the available pattern matchers works for you indicates you really do not have enough RAM in the firewall to run Suricata 7.x with the rules you have enabled. You can try drastically trimming back the number of enabled rules to see if that helps. If you are using ZFS (which the is the default since pfSense Plus 22.01, then that puts further pressure on the available RAM as ZFS needs memory for the ARC cache.

btspce

@bmeeks With 7.0.2_3 Suricata now starts and runs on our two 6100 in HA but as soon as there is traffic on the interface cpu goes to 100% and firewall gui becomes unresponsive after about 30 seconds. No problems when we used 23.05.1 which we had before update. Memory around 20-30% which is normal and as it was under 23.05.1.

AC-BS or Hyperscan does not matter. Suricata runs on secondary without cpu spikes until we initiate a failover and it begins to recieve traffic.
Primary hangs gui within 30 seconds if suricata is started.
No config changes since update from 23.05.1

Any tips on what we could test or check? Is there anything in the new version of suricata that could load the cpu like this ?
Traffic on the 3 interfaces was between 10-100Mbit when we were testing.

ajohnson353

AC-BS was working fine, but that also died on me today. I'm switching back to auto and see what happens with Hyper scan.

bmeeks

@btspce said in Suricata process dying due to hyperscan problem:

Any tips on what we could test or check?

No, I'm afraid not.

If you have ZFS and Boot Environments, then rollback to 23.05.1 and run that Suricata version.

If you want to stay on 23.09.1, then you will need to uninstall Suricata it seems.

SteveITS

@btspce Maybe try disabling a category at a time? Or a binary test/search of disabling categories? We have a 6100 pair on 7.0.2_2 an AC-BS without issue and a 4860 on 7.0.2_3/Auto.

btspce

@SteveITS Thanks for this info. So that seems to point to one of the new features being the culprit and a config difference between our 6100 pair and yours. Its very hard to diagnose as these firewalls are in production and the gui freeze so fast.

It could simply be that with the new version of suricata it does more scanning and the cpu hits the roof but we were only seeing around 100Mbit of traffic over 3 interfaces in total.
Are your 6100 pair running inline or legacy blocking mode?

SteveITS

@btspce Legacy mode.

btspce

Here is what I tested that did not work on our 6100 ha pair:
Disabled Quic parser
Switching to AC-BS
Disabled blocking
Deleted all log files

Suricata loads on interface ix1 and igc0 without problems which is also carp/ha interfaces.
Suricata loads on ix0 but around 5 seconds after loading is complete and cpu load drops, traffic stops to flow and both firewalls is stuck as master for that particular interface until reboot of both firewalls.
So only interface ix0 seems to have problem.

Tried to raise the Carp Advertising Frequency Base from 1 to 5 seconds but that did not help.
Either Suricata is blocking carp heartbeats on this interface (could not see anything in the suricata logs) or I need to up the Carp Advertising Frequency Base more so it doesn't failover during loading.

Maltz

@bmeeks said in Suricata process dying due to hyperscan problem:

then you are on the very ragged edge of not having enough free RAM.

In my testing, that does appear to be the case, but only with some algorithms. Is it really normal that I should expect all the algorithms other than AC-BS to use >2GB of additional RAM above what AC-BS uses? I have assumed that was a bug, but if it really uses THAT much more RAM, perhaps the auto setting should take the device's available RAM into account. Suricata continues to be only barely squeaking by on this device with default settings and all other packages disabled, where it was fine before alongside pfBlocker.

So to sum up again: My NG-2100 w/ 4GB RAM with pfBlocker/DNSBL and Suricata using AC-BS uses about 30-40% of RAM. With any other algorithm, I have to disable pfBlocker entirely, and even then it sits at around about 80-90% of RAM. So I guess I just want to verify if that is expected behavior and just something I have to live with?

bmeeks

@Maltz said in Suricata process dying due to hyperscan problem:

@bmeeks said in Suricata process dying due to hyperscan problem:

then you are on the very ragged edge of not having enough free RAM.

In my testing, that does appear to be the case, but only with some algorithms. Is it really normal that I should expect all the algorithms other than AC-BS to use >2GB of additional RAM above what AC-BS uses? I have assumed that was a bug, but if it really uses THAT much more RAM, perhaps the auto setting should take the device's available RAM into account. Suricata continues to be only barely squeaking by on this device with default settings and all other packages disabled, where it was fine before alongside pfBlocker.

So to sum up again: My NG-2100 w/ 4GB RAM with pfBlocker/DNSBL and Suricata using AC-BS uses about 30-40% of RAM. With any other algorithm, I have to disable pfBlocker entirely, and even then it sits at around about 80-90% of RAM. So I guess I just want to verify if that is expected behavior and just something I have to live with?

Here are the comments from the suricata.yaml template file distributed with the binary. This explains each option available, and advises that Auto is best for most situations.

# Select the multi pattern algorithm you want to run for scan/search the
# in the engine.
#
# The supported algorithms are:
# "ac"      - Aho-Corasick, default implementation
# "ac-bs"   - Aho-Corasick, reduced memory implementation
# "ac-ks"   - Aho-Corasick, "Ken Steele" variant
# "hs"      - Hyperscan, available when built with Hyperscan support
#
# The default mpm-algo value of "auto" will use "hs" if Hyperscan is
# available, "ac" otherwise.
#
# The mpm you choose also decides the distribution of mpm contexts for
# signature groups, specified by the conf - "detect.sgh-mpm-context".
# Selecting "ac" as the mpm would require "detect.sgh-mpm-context"
# to be set to "single", because of ac's memory requirements, unless the
# ruleset is small enough to fit in memory, in which case one can
# use "full" with "ac".  The rest of the mpms can be run in "full" mode.

You are really asking a lot from a little SG-2100 with pfBlockerNG and DNSBL along with Suricata. On top of that, you have the RAM requirements of ZFS and its ARC cache. I assume that you are running ZFS as that is the default since 22.01.

I have an SG-5100, also with 4GB of RAM, and I don't run any IDS/IPS on it. If I did, it would be with an extremely limited ruleset. You can see from your own experiment that disabling pfBlockerNG frees up just enough RAM for Suricata to run. But there really is not much cushion. To run the packages you have there reliably, you should have 8 GB of RAM at the minimum- and 16 GB would not be a bad idea.