Suricata process dying due to hyperscan problem

btspce

Here is what I tested that did not work on our 6100 ha pair:
Disabled Quic parser
Switching to AC-BS
Disabled blocking
Deleted all log files

Suricata loads on interface ix1 and igc0 without problems which is also carp/ha interfaces.
Suricata loads on ix0 but around 5 seconds after loading is complete and cpu load drops, traffic stops to flow and both firewalls is stuck as master for that particular interface until reboot of both firewalls.
So only interface ix0 seems to have problem.

Tried to raise the Carp Advertising Frequency Base from 1 to 5 seconds but that did not help.
Either Suricata is blocking carp heartbeats on this interface (could not see anything in the suricata logs) or I need to up the Carp Advertising Frequency Base more so it doesn't failover during loading.

Maltz

@bmeeks said in Suricata process dying due to hyperscan problem:

then you are on the very ragged edge of not having enough free RAM.

In my testing, that does appear to be the case, but only with some algorithms. Is it really normal that I should expect all the algorithms other than AC-BS to use >2GB of additional RAM above what AC-BS uses? I have assumed that was a bug, but if it really uses THAT much more RAM, perhaps the auto setting should take the device's available RAM into account. Suricata continues to be only barely squeaking by on this device with default settings and all other packages disabled, where it was fine before alongside pfBlocker.

So to sum up again: My NG-2100 w/ 4GB RAM with pfBlocker/DNSBL and Suricata using AC-BS uses about 30-40% of RAM. With any other algorithm, I have to disable pfBlocker entirely, and even then it sits at around about 80-90% of RAM. So I guess I just want to verify if that is expected behavior and just something I have to live with?

bmeeks

@Maltz said in Suricata process dying due to hyperscan problem:

@bmeeks said in Suricata process dying due to hyperscan problem:

then you are on the very ragged edge of not having enough free RAM.

In my testing, that does appear to be the case, but only with some algorithms. Is it really normal that I should expect all the algorithms other than AC-BS to use >2GB of additional RAM above what AC-BS uses? I have assumed that was a bug, but if it really uses THAT much more RAM, perhaps the auto setting should take the device's available RAM into account. Suricata continues to be only barely squeaking by on this device with default settings and all other packages disabled, where it was fine before alongside pfBlocker.

So to sum up again: My NG-2100 w/ 4GB RAM with pfBlocker/DNSBL and Suricata using AC-BS uses about 30-40% of RAM. With any other algorithm, I have to disable pfBlocker entirely, and even then it sits at around about 80-90% of RAM. So I guess I just want to verify if that is expected behavior and just something I have to live with?

Here are the comments from the suricata.yaml template file distributed with the binary. This explains each option available, and advises that Auto is best for most situations.

# Select the multi pattern algorithm you want to run for scan/search the
# in the engine.
#
# The supported algorithms are:
# "ac"      - Aho-Corasick, default implementation
# "ac-bs"   - Aho-Corasick, reduced memory implementation
# "ac-ks"   - Aho-Corasick, "Ken Steele" variant
# "hs"      - Hyperscan, available when built with Hyperscan support
#
# The default mpm-algo value of "auto" will use "hs" if Hyperscan is
# available, "ac" otherwise.
#
# The mpm you choose also decides the distribution of mpm contexts for
# signature groups, specified by the conf - "detect.sgh-mpm-context".
# Selecting "ac" as the mpm would require "detect.sgh-mpm-context"
# to be set to "single", because of ac's memory requirements, unless the
# ruleset is small enough to fit in memory, in which case one can
# use "full" with "ac".  The rest of the mpms can be run in "full" mode.

You are really asking a lot from a little SG-2100 with pfBlockerNG and DNSBL along with Suricata. On top of that, you have the RAM requirements of ZFS and its ARC cache. I assume that you are running ZFS as that is the default since 22.01.

I have an SG-5100, also with 4GB of RAM, and I don't run any IDS/IPS on it. If I did, it would be with an extremely limited ruleset. You can see from your own experiment that disabling pfBlockerNG frees up just enough RAM for Suricata to run. But there really is not much cushion. To run the packages you have there reliably, you should have 8 GB of RAM at the minimum- and 16 GB would not be a bad idea.

btspce

@bmeeks
The issue is IP's on the passlist is still being blocked with this suricata version. Our CARP WAN VIP is being blocked within seconds after suricata starts on that interface.
Our Internal DNS was also blocked today which is the reason we detected this issue.

Suricata logged that it blocked DST our WAN VIP which makes both firewall become master and all traffic stops. Please look into this asap.

We are running 7.0.2_3 now and the issue still exists

tim_co

I hate to be the last person to the show, but I'm still having the problem and I still haven't seen anything indicating there's a new package for Suricata. My system is still showing Suricata 7.0.2 as the latest version. I've tried reinstalling, uninstalling and reinstalling, rebooting and several combinations therein. Would someone please kindly point me in the right direction to getting this resolved?

Thanks,
Tim Welty
Broomfield, CO

bmeeks

@tim_co said in Suricata process dying due to hyperscan problem:

I hate to be the last person to the show, but I'm still having the problem and I still haven't seen anything indicating there's a new package for Suricata. My system is still showing Suricata 7.0.2 as the latest version. I've tried reinstalling, uninstalling and reinstalling, rebooting and several combinations therein. Would someone please kindly point me in the right direction to getting this resolved?

Thanks,
Tim Welty
Broomfield, CO

Are you on the very latest version of pfSense?

The fix for the Hyperscan error is in Suricata package version 7.0.2_3 (the underscore "3" is the minor revision number and is key). The accompanying binary version is 7.0.2_6.

The fix was published around December 19, 2023

tim_co

@bmeeks

Thanks for the reply. It says I'm on the latest version of pfSense?

Tim

SteveITS

@tim_co you are not!
https://docs.netgate.com/pfsense/en/latest/releases/2-7-1.html#troubleshooting

michmoor

@SteveITS
Devils advocate…
Unless they are paying attention to Netgate blog posts it’s reasonable to see why they thought they were on the latest version.
Known issue I believe

bmeeks

@tim_co:
As some others have stated -- you are not on the newest pfSense version. That would be 2.7.2 as of right now. Update to that and you should see the new Suricata package with the Hyperscan fix.

Unfortunately there is an issue with pfSense versions older than 2.7.1 seeing the "latest" version. So, your 2.7.0 thinks it is the current version, but it is not. Follow the links others provided to work around the quirk and install the latest version.

SteveITS

@michmoor said in Suricata process dying due to hyperscan problem:

@SteveITS
Devils advocate…
Unless they are paying attention to Netgate blog posts it’s reasonable to see why they thought they were on the latest version.
Known issue I believe

Oh agree. And I linked the note about it. ;)

Probably going to be people 3 years from now with the same problem. [edit: it’s been years since a new pfSense release! How awful!]

tim_co

@SteveITS said in Suricata process dying due to hyperscan problem:

https://docs.netgate.com/pfsense/en/latest/releases/2-7-1.html#troubleshooting

Thank you for the link. This resolved my problem.

Regards,
Tim