Suricata not working after update 4.0.13_9
-
What other packages do you have running on your machine? How much RAM is in the machine? Essentially you ran the PHP process out of memory. There are some hard limits set by the pfSense configuration files, but the limit is usually enough for any situation.
Your "Log File Path: Not Available" error message is strange. The logs are always stored in /var/log/suricata. If that path was not available during Suricata startup you have something really weird going on with your firewall setup.
-
@bmeeks said in Suricata not working after update 4.0.13_9:
/var/log/suricata
I only have pfBlocker and Suricata installed. I use 4GB DDR3 RAM. In the log folder are some files, but suricata never shows me. I only can see this by WinSCP.
-
The log file directory should never be missing. About the only way that could conceivably happen is if you are using RAM Disks. If you are, DO NOT do that with an IDS/IPS such as Suricata or Snort. Either of these applications will log a lot of data and can need a bunch of disk space to do that.
If you have big IP lists for pfBlocker and are using a large rule set for Suricata, then you can easily exhaust the available allocated memory for PHP processes on the firewall.
-
I never had this issues before the update of Suricata. My RAM usage usually is at around 1,5 GB. And no, I don't use RAM disks.
-
I am experiencing this problem, too. When I attempt to access suricata.log, I get the message "Log File Path: Not Available". Like mrsunfire, the only packages I have installed are pfBlockerNG (Ver. 2.2.5_19). I do not use RAM disks and upgraded Suricata to Ver. 4.0.13_9 this evening.
-
I am unable to replicate this problem in my virtual machine testing. Have you changed something in your configuration such as replacing the NIC hardware? Suricata creates sub-directories for logging based on the physical interface name and stores those in the configuration when you create a Suricata interface. However, I would expect changing the physical NIC to also prevent Suricata from even starting up.
This may also be a file permissions issue. Are you logging in to view the file as a user with limited permissions? By default, the root account is used for the Suricata process. The suricata.log file for an interface should be in a sub-directory under /var/log/suricata, and the sub-directory will be named with the physical interface name and a random GUID as part of the path name.
-
Ok. So I went to /var/log/suricata and took a peek in one of the sub-directories for one of the interfaces. Sure enough, the corresponding suricata.log file was there. I took a look at the file using clog system.log | less. There seemed to be quite a lot of ERRCODE entries related to rule signatures.
I am using Suricata with a Snort VRT subscription (Personal). The whole reason I wanted to look at the Suricata log files to begin with was to ensure the Snort VRT rules were being adopted properly. So what I was seeing via the command line was not exactly reassuring. Moreover, nothing I seemed to do allowed me to look at the log file via the GUI.
So I uninstalled Suricata and re-installed a fresh copy (i.e., without saving the settings from the previous install). I setup up the new installation on the WAN interface to keep things relatively simple. Now I can see the suricata.log file via the GUI with no problem. Mind you, at this point I have not turned Suricata on.
-
I reviewed the suricata.log file after re-booting pfSense. Suricata will not activate. I have set up the WAN interface to run in Legacy mode. I have also set up the Snort IPS Policy as Security and further set the IPS Policy Mode to Policy. I am running off of snortrules-snapshot-3000.tar.gz. Virtually all settings in Suricata are at default.
I am hoping that you can provide some help in diagnosing the error entries in my suricata.log file. Right off the bat I see:
<Notice> -- using flow hash instead of active packets
I am not sure what to make of this entry. Other threads about this notice don't seem applicable to me. However, I am not sure why Suricata would use hash instead of active packets. Virtually all of my settings are default. I also see a very large number of ERRCODE entries related to rule signatures. For example, I am referring to:
<Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)]
These ERRCODE entries terminate with:
<Info> -- 2 rule files processed. 257 rules successfully loaded, 14099 rules failed
This many failed rules renders my Snort subscription pretty much worthless. Something is not right here. The suricata.log file closes with:
<Error> -- [ERRCODE: SC_ERR_POOL_INIT(66)] - alloc error
<Error> -- [ERRCODE: SC_ERR_POOL_INIT(66)] - pool grow failed
<Error> -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] - failed to setup/expand stream session pool. Expand stream.memcap?
<Info> -- RunModeIdsPcapAutoFp initialised
<Error> -- [ERRCODE: SC_ERR_THREAD_INIT(49)] - thread "W#08" failed to initialize: flags 0145
<Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - Engine initialization failed, aborting...Obviously, Suricata refuses to start. I am rather flummoxed what to do here. I am not setting up Suricata in any complex manner. Virtually everything is set on default. Incidently, if I attempt to re-start Suricata after a failure, I get the following message:
<Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata_ix014870.pid' exists but appears stale. Make sure Suricata is not running and then remove /var/run/suricata_ix014870.pid. Aborting!
While it is straight-forward for me to delete this file via the command line, it does become laborious during troubleshooting situations.
-
@bclothier said in Suricata not working after update 4.0.13_9:
I reviewed the suricata.log file after re-booting pfSense. Suricata will not activate. I have set up the WAN interface to run in Legacy mode. I have also set up the Snort IPS Policy as Security and further set the IPS Policy Mode to Policy. I am running off of snortrules-snapshot-3000.tar.gz. Virtually all settings in Suricata are at default.
I am hoping that you can provide some help in diagnosing the error entries in my suricata.log file. Right off the bat I see:
<Notice> -- using flow hash instead of active packets
I am not sure what to make of this entry. Other threads about this notice don't seem applicable to me. However, I am not sure why Suricata would use hash instead of active packets. Virtually all of my settings are default. I also see a very large number of ERRCODE entries related to rule signatures. For example, I am referring to:
<Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)]
These ERRCODE entries terminate with:
<Info> -- 2 rule files processed. 257 rules successfully loaded, 14099 rules failed
This many failed rules renders my Snort subscription pretty much worthless. Something is not right here. The suricata.log file closes with:
<Error> -- [ERRCODE: SC_ERR_POOL_INIT(66)] - alloc error
<Error> -- [ERRCODE: SC_ERR_POOL_INIT(66)] - pool grow failed
<Error> -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] - failed to setup/expand stream session pool. Expand stream.memcap?
<Info> -- RunModeIdsPcapAutoFp initialised
<Error> -- [ERRCODE: SC_ERR_THREAD_INIT(49)] - thread "W#08" failed to initialize: flags 0145
<Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - Engine initialization failed, aborting...Obviously, Suricata refuses to start. I am rather flummoxed what to do here. I am not setting up Suricata in any complex manner. Virtually everything is set on default. Incidently, if I attempt to re-start Suricata after a failure, I get the following message:
<Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata_ix014870.pid' exists but appears stale. Make sure Suricata is not running and then remove /var/run/suricata_ix014870.pid. Aborting!
While it is straight-forward for me to delete this file via the command line, it does become laborious during troubleshooting situations.
You have two problems with your setup.
First, you do not have enough TCP Stream Memory allocated for the number of CPU cores in your box. That's what those three errors that say ERRCODE:SC_ERR_POOL_INIT and ERRCODE:SC_ERR_MEM_ALLOC mean. You will need to at least double, and perhaps triple or quadruple, the amount of memory allocated for Stream Reassembly on the FLOW/STREAM tab. The Suricata upstream folks have made some changes in that part of the code in the recent past and the amount of required memory drastically increases for some multi-core CPUs.
Your second problem is attempting to run Snort3 rules. Those are not compatible with Suricata. You should run the snortrules-snapshot-29120 file for Suricata. Even running the 2.9.x version of Snort rules, you will still have quite few that fail to load. This is expected. Remember, Suricata is NOT Snort. While the Suricata team mimics the actions of most of the Snort binary when it comes to decoding rule keywords, not everything is parsed. The Emerging Threats crew produces a rule set specifically optimized for Suricata.
-
@mrsunfire said in Suricata not working after update 4.0.13_9:
I did an uninstall of suricata and installed it new. Now it's working again. The suricata log only shows "Log File Path: Not Available"
EDIT: oh now I got a crash report:
Crash report begins. Anonymous machine information: amd64 11.2-RELEASE-p3 FreeBSD 11.2-RELEASE-p3 #17 e6b497fa0a3(RELENG_2_4_4): Thu Sep 20 09:04:45 EDT 2018 root@buildbot3:/crossbuild/ce-244/obj/amd64/WvDslnYb/crossbuild/ce-244/pfSense/tmp/FreeBSD-src/sys/pfSense Crash report details: PHP Errors: [06-Nov-2018 14:57:59 Europe/Berlin] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 160684000 bytes) in /usr/local/www/csrf/csrf-magic.php on line 149 No FreeBSD crash data found.Whats wrong?
I figured out what's wrong. The suricata.log file was growing unabated with each time Suricata was started/restarted. After enough restarts, it could become quite large. A fix was just posted in the latest version of the package, version 4.0.13_10, which should appear as a package update for your box anytime now.
-
Thanks for your detailed explanation! I assumed that my closing errors had something to do with the TCP stream memory being too low. But I wanted to check here first, in case there was a different root cause. I won't have a chance to address this matter until later in the day. But I will follow-up with a post. Hopefully, everything will work. If I still get large numbers of failed rules, I will have to consider either moving to the Snort package or staying with Suricata, but switching to the ET ruleset.
-
A fix was just posted in the latest version of the package
That is good news. It looks like we are knocking out some bugs. Thanks for posting a patch so quickly. I really do appreciate your effort!
-
Suricata is up and running. Per your recommendations, I did the following:
[1] I updated the package to 4.0.13_10 Of course, at present, my suricata.log is relatively short. But I presume the overflow PHP error is resolved.
[2] I went to Interfaces --> WAN --> WAN Flow/Stream. In Stream Engine Settings, I quadrupled the Stream Memory Cap to 268435456 (i.e., 256MB). I read elsewhere that four times the default value is necessary to completely eliminate the allocation error (i.e., https://chrislazari.com/pfsense-suricata-service-fails-resolved/). I did not bother testing other values.
[3] I changed the Snort VRT subscription rules from snortrules-snapshot-3000.tar.gz to snortrules-snapshot-29120.tar.gz. Now I get:
<Info> -- 2 rule files processed. 14273 rules successfully loaded, 83 rules failed
which is quite the opposite of my previous post. So this issue is resolved. I can absolutely live with 83 failed rules.
I also made one other change, namely, I switched Suricata to run in inline mode. In doing so, I also needed to change Max Pending Packets in the Detection Engine Settings to a value equal to or greater than 2048. I choose 4096. Otherwise suricata.log would gerenate error messages. My system is based on an Intel C3758 8-Core Denverton Atom SoC (i.e., a Supermicro A2SDi-8C+-HLN4F motherboard). This SoC incorporates Intel's X553 NIC silicon which supports netmap functionality. So far,at 4096, everything is running smoothly. I will tweak Suricata a little more on the WAN interface before setting up the LAN interface.
Anyway, thank you so much for your help. You quick responses are very much appreciated!
-
I also made the update but again after the update I can't see any new alerts in the altert tab. Why is this happening? It seems that suricata stopped working, also restarting the service didn't help. I now uninstalled and reinstalled it and see if it works.
-
Did you get an error in your suricata.log file something like this:
<Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata_ix014870.pid' exists but appears stale. Make sure Suricata is not running and then remove /var/run/suricata_ix014870.pid. Aborting!
If so, you need to manually delete this file or Suricata will just turn off. I found this article helpful in addition to the feedback from bmeeks:
https://chrislazari.com/pfsense-suricata-service-fails-resolved/
-
No I didn't. Everything was fine, but it's not working. After fresh install it works again.
-
@bclothier said in Suricata not working after update 4.0.13_9:
Suricata is up and running. Per your recommendations, I did the following:
[1] I updated the package to 4.0.13_10 Of course, at present, my suricata.log is relatively short. But I presume the overflow PHP error is resolved.
[2] I went to Interfaces --> WAN --> WAN Flow/Stream. In Stream Engine Settings, I quadrupled the Stream Memory Cap to 268435456 (i.e., 256MB). I read elsewhere that four times the default value is necessary to completely eliminate the allocation error (i.e., https://chrislazari.com/pfsense-suricata-service-fails-resolved/). I did not bother testing other values.
[3] I changed the Snort VRT subscription rules from snortrules-snapshot-3000.tar.gz to snortrules-snapshot-29120.tar.gz. Now I get:
<Info> -- 2 rule files processed. 14273 rules successfully loaded, 83 rules failed
which is quite the opposite of my previous post. So this issue is resolved. I can absolutely live with 83 failed rules.
I also made one other change, namely, I switched Suricata to run in inline mode. In doing so, I also needed to change Max Pending Packets in the Detection Engine Settings to a value equal to or greater than 2048. I choose 4096. Otherwise suricata.log would gerenate error messages. My system is based on an Intel C3758 8-Core Denverton Atom SoC (i.e., a Supermicro A2SDi-8C+-HLN4F motherboard). This SoC incorporates Intel's X553 NIC silicon which supports netmap functionality. So far,at 4096, everything is running smoothly. I will tweak Suricata a little more on the WAN interface before setting up the LAN interface.
Anyway, thank you so much for your help. You quick responses are very much appreciated!
Glad you got things sorted out. The PHP overflow error is resolved. Now, with each restart of Suricata, that suricata.log file will get truncated to zero length. This means it will contain startup info for just the currently running (or just "failed to start") Suricata instance on the interface. Each interface configured to run Suricata has its own instance of this log file.
