Snort Keeps Stopping

ultrajones

I recently upgraded my pfSense system to version 2.3.3-RELEASE-p1 and now Snort keeps stopping on its own about once every 2 days. I have to keep manually restarting the service. Anyone have any ideas how I can troubleshoot this? It appears to have failed during a rules update.

2017-03-23 12:05:57 Error php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] The Rules update has finished.
2017-03-23 12:05:56 Informational kernel pid 72883 (snort), uid 0: exited on signal 10
2017-03-23 12:05:56 Error php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Building new sid-msg.map file for WAN…

Regards,
Ultrajones

jeffhammett

I've had this happen from time to time as well, on multiple pfSense's running Snort.

If you're only running on one interface, you should be able to use the Service Watchdog pfSense package to restart Snort. I have had good luck with this.

If you're running Snort on more than one interface Service Watchdog won't work. I write the script below to email me for instances when I am running more than one Snort interface so at least I get a notification that things aren't right.

I then used the Cron pfSense package to schedule the script to run at certain times. Just make sure you have Notification info setup (I think its System->Advanced->Notifications)

#!/bin/tcsh

#set client and location
set client=`echo "Client Name"`

#set number of expected Snort processes that should be running
set expected=2

#check number of Snort processes that are running
set actual=`ps aux | pgrep snort | wc -l`

#if the above numbers don't match, send email to notification address
if ($expected != $actual) then
	echo "Expected $expected Snort processes, but $actual running, please login and verify" | /usr/local/bin/mail.php -s"$client - Possible IPS Process Failure"
	exit 1
endif

bimmerdriver

Snort is dying on my system also, every day or so. Here are the last log entries:

Mar 30 00:07:41 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] The Rules update has finished.
Mar 30 00:07:40 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Building new sid-msg.map file for WAN…
Mar 30 00:07:29 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: WAN …
Mar 30 00:07:08 kernel hn1: promiscuous mode disabled
Mar 30 00:07:08 kernel pid 74397 (snort), uid 0: exited on signal 11

There aren't any entries more recent, but the service is stopped.

coffeecup25

knock on wood - I had that problem but it hasn't reappeared in about a week. I changed my update to a non default time with the thinking that everyone hitting the update at the same time (default) might have made a difference. I also updated the package. I still check every morning though.

The issue had to do with an error in the update file that stopped everything - I think.

ultrajones

I just upgraded to an Netgate SG-2440 and Snort is still failing almost daily. Is there a way I can report this issue so it can be addressed?

Regards,
Ultrajones

u3c307

Why don't you just use Service watchdog package? It will start Snort automatically when fail.

bmeeks

@ultrajones:

I just upgraded to an Netgate SG-2440 and Snort is still failing almost daily. Is there a way I can report this issue so it can be addressed?

Regards,
Ultrajones

The problem needs to be repeatable in order to be diagnosed. You can help by doing some troubleshooting. I suspect one or more rule signatures are at fault for Snort failures. Since different users run different rules configurations, there is no reasonable way to test everything. I would start with a very minimal rule set and slowly expand it day-by-day to see if the unexpected halts occur. When they return, you can know the last rule set you added and troubleshoot further from there.

Not every user of Snort has this issue. A few do, but not many. I have Snort running on three firewall interfaces. I have a few of the ET-Open rule categories enabled on all three interfaces, and I use the Snort VRT IPS Policy "Balanced" on one of the interfaces. I have not had an unexpected Snort stop in over two years. In fact, the only time it has stopped is during the times I've upgrade the underlying pfSense operating system. The halt was part of the restart of the firewall.

You can enable verbose Snort logging on the GLOBAL SETTINGS tab. That may help shed some light on what's happening. Be prepared for a lot of log noise, though!

Bill

bmeeks

@u3c307:

Why don't you just use Service watchdog package? It will start Snort automatically when fail.

This is not always a wise move. The Service Watchdog package is not intelligent about Snort or Suricata. It does not understand multiple instances of either binary when you run the IDS/IPS on more than one interface. For example, if you have WAN and LAN both configured with Snort, there will be two completely separate Snort instances running during normal operation. Service Watchdog will only look for one. So WAN could fail but Service Watchdog would still be happy because it sees LAN still up.

The other issue is the speed with which Service Watchdog responds to perceived "failures". The instant it sees the process die, it restarts it using the shell script. The problem with Snort and Suricata is that the processes are "restarted" as part of the daily rules update (if a new rules package is downloaded and installed). If the Service Watchdog package happens to see Snort or Suricata down during the "restart" due to a rules update, it will start a new instance of Snort or Suricata on each interface (because it just calls the shell script which starts all interfaces). So now when the rules update completes and it restarts Snort or Suricata, you will have double instances running on each interface. Lots of weird stuff happens then as two processes on the same interface is uncharted territory!

Bill

Khampol

Hi,

After an update manual today, well snort refuse the start…. See this in LOG :

FATAL ERROR: The dynamic detection library "/usr/local/lib/snort_dynamicrules/server-webapp.so" version 1.0 compiled with dynamic engine library version 2.6 isn't compatible with the current dynamic engine library "/usr/local/lib/snort_dynamicengine/libsf_engine.so" version 3.0.

::) ::) ::)

bmeeks

@Khampol:

Hi,

After an update manual today, well snort refuse the start…. See this in LOG :

FATAL ERROR: The dynamic detection library "/usr/local/lib/snort_dynamicrules/server-webapp.so" version 1.0 compiled with dynamic engine library version 2.6 isn't compatible with the current dynamic engine library "/usr/local/lib/snort_dynamicengine/libsf_engine.so" version 3.0.

::) ::) ::)

Two things: (1) do you have the latest Snort package installed? (2) did you do a "remove and then reinstall" when updating the Snort package?

Sounds like you have a problem another user had. You have old versions of the precompiled shared-object rules libraries hanging around on your system. To remove them, do the following:

(1) Remove the Snort package
(2) Get to a CLI (command line) prompt on the firewall and delete any "snort" directories you find in the /usr/local/lib path.
(3) Install the Snort package again

The above steps will not cause you to lose any configuration data so long as "save settings" is enabled on the GLOBAL SETTINGS tab. That setting is "on" by default.

Bill