Suricata 5.0.2 Will Not Start on pfSense 2.4.5
-
Yesterday, I installed a fresh copy of pfSense 2.4.5-RELEASE (amd64) and configured it with no issues. I then installed the Suricata 5.0.2 package. Interestingly, Suricata will not start. I've tried starting it from the Services > Suricata > Interfaces page for the interface I've configured and I've tried starting the Suricata IDS/IPS Daemon from the Dashboard > Services Status. The Services > Suricata > Logs View shows the below:
2/4/2020 -- 21:32:35 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
2/4/2020 -- 21:32:35 - <Info> -- CPUs/cores online: 8
2/4/2020 -- 21:32:35 - <Info> -- HTTP memcap: 67108864
2/4/2020 -- 21:32:35 - <Notice> -- using flow hash instead of active packets
2/4/2020 -- 21:32:35 - <Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata_igb732313.pid' exists but appears stale. Make sure Suricata is not running and then remove /var/run/suricata_igb732313.pid. Aborting!So I open a shell and remove the suricata_igb732313.pid file. I then go back to the Dashboard and start the Suricata IDS/IPS Daemon and it starts. I then check the Services > Suricata > Interfaces page and the interface is started. However, this is short lived. The start only lasts for less than a minute.
I then go back to a shell and delete this file once again and then start Suricata again. The same repetitive cycle occurs - Suricata stops and I get the same file with the same log file information except for different dates and times. Anyone know what may be occurring and how to fix it so I can start Suricata? Any suggestions would be most helpful. Thank you.
-
Suricata is crashing (the process is aborting) and not cleaning up after itself. That's why the PID file gets left behind.
You need to examine the pfSense system log to see what it being logged there. My guess is a Signal 4 abort (SIGILL) which stands for ILLEGAL INSTRUCTION. This is a known issue on Netgate non-Intel hardware such as the SG-1100 with the aarch64 CPU.
Are you running that type of hardware? If so, don't look for a quick fix for this. It is a complicated problem caused by the non-Intel hardware. I suggest you consider switching over to Snort for the foreseeable future.
The above all assumes you have an aarch64 CPU platform and that you see the Signal 4 error in the system log. If that is not the case, then post back here with what you do find in the system log.
-
Hi Bill...
I have pfSense installed on my own hardware. I'm running an Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz, 4 core hyperthreaded to 8. The NIC is a 4 port Supermicro based on Intel's chipset; I believe I purchased it from Netgate as I recall.
There are only 50 entries in the Status > System Logs > System > General and there appears to be nothing out of the ordinary.
For whatever reason, there is one setting that I changed in Suricata and after this change I deleted the PID file and restarted Suricata. Suricata then seemed to remain working.
Services > Suricata > Edit Interface Settings > Performance and Detection Engine Settings > Run Mode was changed to Workers.I now see there is a Suricata update from 5.0.2 to 5.0.2_1. Anything significant that changed? I uninstalled 5.0.2 and installed 5.0.2_1 with no issues. When I checked the interface, Suricata was running automagically.
-
No, nothing at all significant changed in the last update. That just altered the rules update schedule to occur on a random minute past the hour to alleviate loading on the rules update sites caused by all the pfSense users hitting them at the same minute past the hour. This was because users rarely changed the default value.
If you have 8 CPU cores, then TCP stream_memcap memory is not going to be large enough in most cases. The defaults are for average home machines. Servers with high core count (or logical core count) CPUs will need to make changes due to the way Suricata automatically allocates RAM. Changing the Run Mode will alter this behavior some, and in your case likely let you scrimp by and start.
The
suricata.log
file is wiped on every restart attempt, so to see why it's not starting, you need to attempt a start (with no existing PID file in/var/run
) and then immediately go examine thesuricata.log
for the interface before you attempt any other start. I'm betting you will find a memory allocation error and a suggestion to increase stream_memcap. -
So I stopped Suricata, deleted the PID file, restarted Suricata and looked at the suricata.log and this is what was shown:
3/4/2020 -- 16:06:50 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
3/4/2020 -- 16:06:50 - <Info> -- CPUs/cores online: 8
3/4/2020 -- 16:06:50 - <Info> -- HTTP memcap: 67108864
3/4/2020 -- 16:06:50 - <Notice> -- using flow hash instead of active packets
3/4/2020 -- 16:06:50 - <Info> -- Netmap: Setting IPS mode
3/4/2020 -- 16:06:50 - <Info> -- fast output device (regular) initialized: alerts.log
3/4/2020 -- 16:06:50 - <Info> -- http-log output device (regular) initialized: http.log
3/4/2020 -- 16:06:50 - <Info> -- stats output device (regular) initialized: stats.logNothing about increase stream_memcap that I found in the entire log.
-
@newUser2pfSense said in Suricata 5.0.2 Will Not Start on pfSense 2.4.5:
So I stopped Suricata, deleted the PID file, restarted Suricata and looked at the suricata.log and this is what was shown:
3/4/2020 -- 16:06:50 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
3/4/2020 -- 16:06:50 - <Info> -- CPUs/cores online: 8
3/4/2020 -- 16:06:50 - <Info> -- HTTP memcap: 67108864
3/4/2020 -- 16:06:50 - <Notice> -- using flow hash instead of active packets
3/4/2020 -- 16:06:50 - <Info> -- Netmap: Setting IPS mode
3/4/2020 -- 16:06:50 - <Info> -- fast output device (regular) initialized: alerts.log
3/4/2020 -- 16:06:50 - <Info> -- http-log output device (regular) initialized: http.log
3/4/2020 -- 16:06:50 - <Info> -- stats output device (regular) initialized: stats.logNothing about increase stream_memcap that I found in the entire log.
So is that the entire log and Suricata has crashed, or is Suricata running? I'm confused. Changing the Run Mode is a significant change you made, and that will possibly mask the stream_memcap error.
And if Suricata is currently running, then by definition there is no startup error.
-
Doh! Ok, I changed the Run Mode back to AutoFP and went through the process again. Suricata will not start.
3/4/2020 -- 16:21:44 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
3/4/2020 -- 16:21:44 - <Info> -- CPUs/cores online: 8
3/4/2020 -- 16:21:44 - <Info> -- HTTP memcap: 67108864
3/4/2020 -- 16:21:44 - <Notice> -- using flow hash instead of active packets
3/4/2020 -- 16:21:44 - <Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata_igb732313.pid' exists but appears stale. Make sure Suricata is not running and then remove /var/run/suricata_igb732313.pid. Aborting!That's the whole file.
I've got 64 gig of memory in this computer.
-
So I changed the Stream Memory Cap and the Reassembly Memory Cap to 88080384 (84MB). I kept the Run Mode as AutoFP. Restarted the process again and Suricata started.
-
Yes, I suspected that was an issue.
I think you failed to catch the error because you misunderstand what I'm saying about the log getting wiped with each start. So when it tries to start and fails, it will potentially leave a dangling PID file, but there should also be in the
suricata.log
at that instant the memory error. But you must look at the file BEFORE you attempt any kind of restart.So in your case the sequence would be this --
- Clean out all Suricata-related PID files in
/var/run
. - Attempt to start Suricata.
- If it fails, now go to the LOGS VIEW tab, select the Interface where it is failing to start, and then look in the
suricata.log
file to see what is shown. That's when I would expect you to find the memory allocation error.
Now if Suricata starts successfully (such as when you changed the Run Mode), then there will be no memory error in the file. The error would only be there if Suricata fails to start AND if you don't try any other start BEFORE you look at the file.
No matter, if you have it running now you're good.
- Clean out all Suricata-related PID files in
-
Thanks for the help Bill. I appreciate it.
-
-