Suricata process dying due to hyperscan problem
-
@bmeeks With 7.0.2_3 Suricata now starts and runs on our two 6100 in HA but as soon as there is traffic on the interface cpu goes to 100% and firewall gui becomes unresponsive after about 30 seconds. No problems when we used 23.05.1 which we had before update. Memory around 20-30% which is normal and as it was under 23.05.1.
AC-BS or Hyperscan does not matter. Suricata runs on secondary without cpu spikes until we initiate a failover and it begins to recieve traffic.
Primary hangs gui within 30 seconds if suricata is started.
No config changes since update from 23.05.1Any tips on what we could test or check? Is there anything in the new version of suricata that could load the cpu like this ?
Traffic on the 3 interfaces was between 10-100Mbit when we were testing. -
AC-BS was working fine, but that also died on me today. I'm switching back to auto and see what happens with Hyper scan.
-
@btspce said in Suricata process dying due to hyperscan problem:
Any tips on what we could test or check?
No, I'm afraid not.
If you have ZFS and Boot Environments, then rollback to 23.05.1 and run that Suricata version.
If you want to stay on 23.09.1, then you will need to uninstall Suricata it seems.
-
@btspce Maybe try disabling a category at a time? Or a binary test/search of disabling categories? We have a 6100 pair on 7.0.2_2 an AC-BS without issue and a 4860 on 7.0.2_3/Auto.
-
@SteveITS Thanks for this info. So that seems to point to one of the new features being the culprit and a config difference between our 6100 pair and yours. Its very hard to diagnose as these firewalls are in production and the gui freeze so fast.
It could simply be that with the new version of suricata it does more scanning and the cpu hits the roof but we were only seeing around 100Mbit of traffic over 3 interfaces in total.
Are your 6100 pair running inline or legacy blocking mode? -
@btspce Legacy mode.
-
Here is what I tested that did not work on our 6100 ha pair:
Disabled Quic parser
Switching to AC-BS
Disabled blocking
Deleted all log filesSuricata loads on interface ix1 and igc0 without problems which is also carp/ha interfaces.
Suricata loads on ix0 but around 5 seconds after loading is complete and cpu load drops, traffic stops to flow and both firewalls is stuck as master for that particular interface until reboot of both firewalls.
So only interface ix0 seems to have problem.Tried to raise the Carp Advertising Frequency Base from 1 to 5 seconds but that did not help.
Either Suricata is blocking carp heartbeats on this interface (could not see anything in the suricata logs) or I need to up the Carp Advertising Frequency Base more so it doesn't failover during loading. -
@bmeeks said in Suricata process dying due to hyperscan problem:
then you are on the very ragged edge of not having enough free RAM.
In my testing, that does appear to be the case, but only with some algorithms. Is it really normal that I should expect all the algorithms other than AC-BS to use >2GB of additional RAM above what AC-BS uses? I have assumed that was a bug, but if it really uses THAT much more RAM, perhaps the auto setting should take the device's available RAM into account. Suricata continues to be only barely squeaking by on this device with default settings and all other packages disabled, where it was fine before alongside pfBlocker.
So to sum up again: My NG-2100 w/ 4GB RAM with pfBlocker/DNSBL and Suricata using AC-BS uses about 30-40% of RAM. With any other algorithm, I have to disable pfBlocker entirely, and even then it sits at around about 80-90% of RAM. So I guess I just want to verify if that is expected behavior and just something I have to live with?
-
@Maltz said in Suricata process dying due to hyperscan problem:
@bmeeks said in Suricata process dying due to hyperscan problem:
then you are on the very ragged edge of not having enough free RAM.
In my testing, that does appear to be the case, but only with some algorithms. Is it really normal that I should expect all the algorithms other than AC-BS to use >2GB of additional RAM above what AC-BS uses? I have assumed that was a bug, but if it really uses THAT much more RAM, perhaps the auto setting should take the device's available RAM into account. Suricata continues to be only barely squeaking by on this device with default settings and all other packages disabled, where it was fine before alongside pfBlocker.
So to sum up again: My NG-2100 w/ 4GB RAM with pfBlocker/DNSBL and Suricata using AC-BS uses about 30-40% of RAM. With any other algorithm, I have to disable pfBlocker entirely, and even then it sits at around about 80-90% of RAM. So I guess I just want to verify if that is expected behavior and just something I have to live with?
Here are the comments from the
suricata.yamltemplate file distributed with the binary. This explains each option available, and advises that Auto is best for most situations.# Select the multi pattern algorithm you want to run for scan/search the # in the engine. # # The supported algorithms are: # "ac" - Aho-Corasick, default implementation # "ac-bs" - Aho-Corasick, reduced memory implementation # "ac-ks" - Aho-Corasick, "Ken Steele" variant # "hs" - Hyperscan, available when built with Hyperscan support # # The default mpm-algo value of "auto" will use "hs" if Hyperscan is # available, "ac" otherwise. # # The mpm you choose also decides the distribution of mpm contexts for # signature groups, specified by the conf - "detect.sgh-mpm-context". # Selecting "ac" as the mpm would require "detect.sgh-mpm-context" # to be set to "single", because of ac's memory requirements, unless the # ruleset is small enough to fit in memory, in which case one can # use "full" with "ac". The rest of the mpms can be run in "full" mode.You are really asking a lot from a little SG-2100 with pfBlockerNG and DNSBL along with Suricata. On top of that, you have the RAM requirements of ZFS and its ARC cache. I assume that you are running ZFS as that is the default since 22.01.
I have an SG-5100, also with 4GB of RAM, and I don't run any IDS/IPS on it. If I did, it would be with an extremely limited ruleset. You can see from your own experiment that disabling pfBlockerNG frees up just enough RAM for Suricata to run. But there really is not much cushion. To run the packages you have there reliably, you should have 8 GB of RAM at the minimum- and 16 GB would not be a bad idea.
-
@bmeeks
The issue is IP's on the passlist is still being blocked with this suricata version. Our CARP WAN VIP is being blocked within seconds after suricata starts on that interface.
Our Internal DNS was also blocked today which is the reason we detected this issue.Suricata logged that it blocked DST our WAN VIP which makes both firewall become master and all traffic stops. Please look into this asap.
We are running 7.0.2_3 now and the issue still exists
-
I hate to be the last person to the show, but I'm still having the problem and I still haven't seen anything indicating there's a new package for Suricata. My system is still showing Suricata 7.0.2 as the latest version. I've tried reinstalling, uninstalling and reinstalling, rebooting and several combinations therein. Would someone please kindly point me in the right direction to getting this resolved?
Thanks,
Tim Welty
Broomfield, CO -
@tim_co said in Suricata process dying due to hyperscan problem:
I hate to be the last person to the show, but I'm still having the problem and I still haven't seen anything indicating there's a new package for Suricata. My system is still showing Suricata 7.0.2 as the latest version. I've tried reinstalling, uninstalling and reinstalling, rebooting and several combinations therein. Would someone please kindly point me in the right direction to getting this resolved?
Thanks,
Tim Welty
Broomfield, COAre you on the very latest version of pfSense?
The fix for the Hyperscan error is in Suricata package version 7.0.2_3 (the underscore "3" is the minor revision number and is key). The accompanying binary version is 7.0.2_6.
The fix was published around December 19, 2023
-
-
@tim_co you are not!
https://docs.netgate.com/pfsense/en/latest/releases/2-7-1.html#troubleshooting -
@SteveITS
Devils advocate…
Unless they are paying attention to Netgate blog posts it’s reasonable to see why they thought they were on the latest version.
Known issue I believe -
@tim_co:
As some others have stated -- you are not on the newest pfSense version. That would be 2.7.2 as of right now. Update to that and you should see the new Suricata package with the Hyperscan fix.Unfortunately there is an issue with pfSense versions older than 2.7.1 seeing the "latest" version. So, your 2.7.0 thinks it is the current version, but it is not. Follow the links others provided to work around the quirk and install the latest version.
-
@michmoor said in Suricata process dying due to hyperscan problem:
@SteveITS
Devils advocate…
Unless they are paying attention to Netgate blog posts it’s reasonable to see why they thought they were on the latest version.
Known issue I believeOh agree. And I linked the note about it. ;)
Probably going to be people 3 years from now with the same problem. [edit: it’s been years since a new pfSense release! How awful!]
-
@SteveITS said in Suricata process dying due to hyperscan problem:
https://docs.netgate.com/pfsense/en/latest/releases/2-7-1.html#troubleshooting
Thank you for the link. This resolved my problem.
Regards,
Tim
