Suricata Not Starting, Blank Log File (Resolved)

[bmeeks]

If you have a blank suricata.log file immediately after attempting an interface start, then Suricata is never even getting launched. That would point to a missing shared library or the wrong version of a required shared library.

So when you did the /usr/local/bin/suricata -V command did anything else print out in terms of an error about missing libraries?

There is a separate suricata.log file for each configured Suricata interface, so if you have more than one interface configured, are you checking the correct suricata.log file? Just asking the simple questions first to get those out of the way.

Did you by chance recently update Suricata before you updated pfSense? You should always update pfSense before updating or installing any packages (when a pfSense update is available).

You can try deleting and then reinstalling the package. Perhaps your installation failed to fully complete.

[5cub4f1y]

Thank you for the reply.

Nothing printed out other than the version of Suricata when I typed suricata -V.

I am checking the log files for the WAN interface, which is blank.

This was a fresh install of Suricata. I have reinstalled the package twice and still nothing. I am really hoping this can be resolved as from what I have read about Suricata, I would rather use it than Snort, if for nothing else than the support for multi-core CPUs.

I checked the System Logs/ System/ General and this is what I found:

It says "START", but it immediately goes to the red "X"...doesn't even show the green check mark at all, not even a glimpse of it.

[bmeeks]

@5cub4f1y said in Suricata Not Starting, Blank Log File:

Thank you for the reply.

Nothing printed out other than the version of Suricata when I typed suricata -V.

I am checking the log files for the WAN interface, which is blank.

This was a fresh install of Suricata. I have reinstalled the package twice and still nothing. I am really hoping this can be resolved as from what I have read about Suricata, I would rather use it than Snort, if for nothing else than the support for multi-core CPUs.

I checked the System Logs/ System/ General and this is what I found:

It says "START", but it immediately goes to the red "X"...doesn't even show the green check mark at all, not even a glimpse of it.

Did you actually attempt to start it twice in quick succession? The log shows a start at 08:59:49, and then another start at 09:00:30. So roughly 40 seconds or so apart.

Without attempting another start, go to the LOGS VIEW tab and make sure the WAN interface is selected in the drop-down selector there. Then choose the suricata.log file in the Which File To View selector. There should be something in that file giving a hint of what's wrong if the binary even tried to start. Post back whatever you find in that file. That file almost can't be blank if the binary even begins to start. One of the very first things that happens is the version string is written to that log. The fact you say this command works with no errors --

/usr/local/bin/suricata -V

indicates to me that the binary at least is starting and can find its basic dependent shared libraries.

Are there any other Suricata related messages in your system log? If so, post those as well. In particular are there any errors related to Signal 11 or Signal 10 with Suricata?

[5cub4f1y]

This is what happens with suricata -V

Here is what the log screen looks like, without starting another session.

Then I hit start, this is what is in the system log:

Then after a few minutes, I hit start again, I also quit Snort (I have snort running because suricata is not working..):

And again it went to the red X. I hti start one more time...

And still red X, still no logs in suricata.

[bmeeks]

You have something trying to start two identical instances of Suricata on that interface. See that line that says "...Ignoring additional START command since Suricata is already starting..."? That is an unexpected condition. There should not be two starts happening. That message is coming from the shell script in /usr/local/etc/rc.d that is used to start Suricata when the firewall reboots or when you click the icon in the SERVICES widget.

Do you have the Service Watchdog package installed, and if so, is it configured to watch Suricata?

You have something weird going on in your system if the shell script is trying to start Suricata while you are using the GUI icons to do the same. That's why you are having problems.

[5cub4f1y]

Not that I know of. These are the services I have running:

Should I try starting only from the command line?

[bmeeks]

Are you maybe trying to run Snort and Suricata on the same interface?

I'm really struggling to envision a situation where the Suricata binary will start and print out version information from the command line, but won't at least start populating the suricata.log file when you start it from the GUI.

You are starting it from the INTERFACES tab in Suricata, correct? You aren't trying to start it from the SERVICES widget icon are you? You can't start Suricata from the SERVICES widget until you have at least configured everything and successfully saved settings on the INTERFACE SETTINGS tab.

[5cub4f1y]

I tried starting from both. The last couple of times only from the interfaces tab. I have Snort disabled.

[bmeeks]

At this point, I'm sorry to say I am out of ideas. I've never seen this problem before with the package. The problems with non-starting have always been either missing or incorrect versions of shared libraries or problems with TCP stream memcap limits being exceeded on multi-core CPUs. The former will show up as error messages when trying to run the suricata -V command from a shell prompt, and the latter will show up as errors in the suricata.log for the interface.

One last thing you can try, just in case your browser has some plugin causing an issue with opening and viewing the text log, is to go to DIAGNOSTICS > EDIT and then browse to and open this file --

/var/log/suricata/suricata_igb0_xxxxx/suricata.log

where the xxxxx is a random UUID number. See if you see the log file there and if it has anything in it. Another wild guess is perhaps the permissions in that directory are wrong (but that is really highly unlikely as Suricata runs as root).

One last thing that just hit my mind, is /var a RAM disk? And if it is, do you have enough space allocated?

[5cub4f1y]

I went to the file through the Diagnostics area, and it was in fact blank also. How do I check for RAM disk? This pfSense machine is it's own PC, doesn't share anything and is not virtualized if that's what you mean. When I SSH in to the machine, and go to the /var/log/suricata file, it tells me permission denied. A can get to the log directory, but not suricata directory, and it won't let me sudo as it says "sudo: command not found"... the permissions for the suricata directory appear to be 700, owner: root, group: wheel. I logged in as admin to the GUI, and I can now get to the file through SSH, but it is still empty.

[bmeeks]

@5cub4f1y said in Suricata Not Starting, Blank Log File:

I went to the file through the Diagnostics area, and it was in fact blank also. How do I check for RAM disk? This pfSense machine is it's own PC, doesn't share anything and is not virtualized if that's what you mean. When I SSH in to the machine, and go to the /var/log/suricata file, it tells me permission denied. A can get to the log directory, but not suricata directory, and it won't let me sudo as it says "sudo: command not found"... the permissions for the suricata directory appear to be 700, owner: root, group: wheel. I logged in as admin to the GUI, and I can now get to the file through SSH, but it is still empty.

Since I do not know your skill level with Unix type systems, I need you to get very specific with exactly which file you tried to view. Saying /var/log/suricata is not descriptive enough. That is the root-level logging directory for all Suricata instances that you may configure. It is not a file. Each interface has its own private sub-directory underneath /var/log/suricata where it writes interface-specific logs. So please give me the entire pathname and then final filename of the file you tried to view the contents of.

As I said in an earlier post, I really can't envision a failure where Suricata will start and print out version information from the command line, but then fail to start from the GUI and also fail to log a single thing to its suricata.log file.

I am the programmer who created the package, so I am extremely familiar with how it works ... .

[5cub4f1y]

The path to the log file was /var/log/suricata/suricata_igb043430/suricata.log

And thanks for taking the time to help with this. Is there a chance I may have misconfigured something? The “Log Facility” when in the Edit Interface Settings/WAN is set to LOCAL1, this is one thing I am not sure about.

[bmeeks]

@5cub4f1y said in Suricata Not Starting, Blank Log File:

The path to the log file was /var/log/suricata/suricata_igb043430/suricata.log

And thanks for taking the time to help with this. Is there a chance I may have misconfigured something? The “Log Facility” when in the Edit Interface Settings/WAN is set to LOCAL1, this is one thing I am not sure about.

No, that Log Facility setting should have no bearing on writing the suricata.log file. That setting just determines how Suricata logs alerts to syslog (when enabled). All of the startup and other diagnostic information is always written to the suricata.log regardless of any other settings.

It is apparent that Suricata is not starting since the log file is empty and the GUI shows the red X. Why it is not starting is the big mystery. I assume you have done at least a minimal configuration by selecting an interface, enabling the download of some rules and then configured some rules on the CATEGORIES tab (or else used the SID MGMT tab features to enabled rules). And that you clicked Save each time after doing these things.

So examining your pfSense system log after attempting a Suricata start, do you see any messages saying Suricate exited on Signal 11, or Signal 10 or anything else like that?

You can try this command sequence to launch the binary and test the suricata.yaml configuration from a shell prompt on the firewall --

cd /usr/local/etc/suricata/suricata_43430_igb0
/usr/local/bin/suricata -T -c ./suricata.yaml

So the first line simply changes the directory to the local interface sub-directory for your WAN. The second line executes the Suricata binary and tells it to test the suricata.yaml file for errors.

Run that sequence and see what you get. Report back here.

[5cub4f1y]

Now we may be getting somewhere.

[bmeeks]

@5cub4f1y said in Suricata Not Starting, Blank Log File:

Now we may be getting somewhere.

Yep, there is an error in that file on line 454. The file format is very picky about indentations and other issues. However, if you have an error in that file it would actually indicate something entered incorrectly in the GUI boxes. That's because what you enter into the GUI boxes is used to create the suricata.yaml file on-the-fly when you click Start. So if you look at that line number in the file, you can get a clue about what may be wrong on the GUI side.

[5cub4f1y]

Here is the line number in when I 'cat' the file. Indentation issue?

[bmeeks]

Yes, I think it is an indentation issue. YAML files are very picky about that, and so is the parser inside the Suricata binary.

Try this edit and see what effect it has. Post back with the results.

1. Use either a shell text editor or the DIAGNOSTICS > EDIT command in the pfSense menu to open this file -- /usr/local/pkg/suricata/suricata_yaml_template.inc.

2. Scroll down to near the bottom of that file to line 394. That area of code looks like this:

###########################################################################
# Configure libhtp.
libhtp:
   default-config:
     {$http_hosts_default_policy}

   {$http_hosts_policy}

This is line 394:

{$http_hosts_default_policy}

Delete the leading spaces in that line such it backs up flush to the left-hand margin. Save the change. Then go to the GUI and edit the Suricata WAN interface. Just click the edit icon to open INTERFACE SETTINGS and then click Save. That will regenerate the suricata.yaml file. Now attempt to start Suricata on the interface.

[5cub4f1y]

That did it! It starts...

...and the log file is working... though there is a bunch of errors but I think that may have to do with Suricata rules?

[bmeeks]

Great! Thanks for patiently providing troubleshooting data and feedback. This looks like an error in the PHP code. It escaped my testing. Most likely because my favorite testing VMs for new versions already have an install of Suricata, so not starting from an empty slate each time. I will have to make it a point in the future to always test a green-field install as well as an upgrade.

There is a new Suricata binary release scheduled for October 8th from upstream. A few days after it is released, I will be updating the Suricata package on pfSense. I will include the fix for this bug in that update. Until then you are fine to run with the manual edit you made. When you update, a "fixed" file will get installed automatically. The actual fix will be inside another PHP file as that's where the typo originates. The file I had you modify is an easier temp workaround.

Lastly, those rule errors are expected if you are running some Snort rules. Suricata does not recognize all of the Snort rule options and keywords. It will just skip loading those rules.

[5cub4f1y]

Excellent! Thank you for your patience. Is there anyway to mark this thread as resolved or does this forum not work that way?