SNORT - Snort 2.9.2.3 pkg v. 2.2.1 process do not quit via update scripts or GUI

breusshe

Since installing the Snort 2.9.2.3 pkg v. 2.2.1, when the snort rules are updated at night, the command to kill existing snort processes fails. Also, when in the GUI, clicking the red 'X' boxes does not kill the processes. The GUI acts like it did, but a quick run of ps aux | grep snort shows otherwise. While the processes fail to stop, this does not stop additional snort processes from starting. So, after a few days of running snort and auto-updates going on, in my case, I have 2*{a few days} number of snort processes running (2 since I have two snort processes configured on my box).

The only fix I've found is to run 'kill -QUIT {pid}' on all snort processes and then start new snort processes once those have terminated. I actually have to use the -QUIT signal or they don't stop. No idea why, but 'kill {pid}' won't do it. Yes, I'm logged in as admin when I do this.

I'm wondering how the GUI and the rules update scripts go about restarting Snort so I can look at what is wrong with those scripts or does anyone know what the issue is, if not the scripts?

Thanks.

UPDATE:

'kill {pid}' works sometimes, other times it does not. However, 'kill -QUIT {pid}' always works. No idea why.
Rebooting pfSense causes four processes of Snort (I believe this is because of some, long running oddity where Snort will start at boot time, then stop, then start again).
Running 'ps aux | grep snort' does not always show the running Snort processes. However, using 'top' always works. No idea why.

rcfa

Noticed more or less the same here…
...worse, at least for a while, snort was running but the UI would claim otherwise. So hence IPv6 traffic was blocked and I was pulling my hair out, why despite a wide open FW rules, no traffic would flow...

Also, the GUI will at most show Snort active on three of four interfaces, even though ps shows it being active on all four.

Here you see the GUI showing it active on two interfaces, while ps shows it active on all four, with the process running twice on one of the interfaces...

Oh, and this is on 2.1-BETA0 (amd64) built on Wed Jun 20 12:19:46 EDT 2012 FreeBSD 8.3-RELEASE-p3
with the regular non-dev snort package.

![Screen Shot 2012-06-22 at 19.38.20.png](/public/imported_attachments/1/Screen Shot 2012-06-22 at 19.38.20.png)
![Screen Shot 2012-06-22 at 19.38.20.png_thumb](/public/imported_attachments/1/Screen Shot 2012-06-22 at 19.38.20.png_thumb)
![Screen Shot 2012-06-22 at 19.39.00.png](/public/imported_attachments/1/Screen Shot 2012-06-22 at 19.39.00.png)
![Screen Shot 2012-06-22 at 19.39.00.png_thumb](/public/imported_attachments/1/Screen Shot 2012-06-22 at 19.39.00.png_thumb)

dwood

2.0.1-RELEASE (amd64)
built on Mon Dec 12 18:16:13 EST 2011
FreeBSD 8.1-RELEASE-p6

Seeing same thing here with Snort 2.9.2.3 pkg v. 2.2.1, installed and running (after killing dependent packages and clean install of pfsense/snort). The GUI shows the two SNORT interfaces have stopped when requested, but they continue to run, blocking etc.

Executing ps aux | grep snort at the shell, shows two running process, with the GUI interfaces toggled off.
Executing kill -QUIT {pid} does stop the processes.

For now, I'm guessing any SNORT changes need to accompanied by a router reboot? Everytime there issues like this crop up, there's a few good lessons passed along in management from the shell :-) My official favourite new shell commands for relative newbies like myself… and thanks to all who provided them :-)

**to list all packages installed:
pkg_info

to delete a package (in this case, perl-5.12.3):
pkg_delete -f perl-5.12.3

to find all snort references:
find /* | grep snort

to find and remove snort references:
find /* | grep -i snort | xargs rm -rv

find processes with snort description:
ps aux | grep snort

kill same processes:
kill -QUIT {pid}**

Cheers,
Dennis.

krankykoder

I also see the same behavior.

I have snort only on one interface.

Every time snort restarts (auto update rules, manually) all that happens is a new instance of snort running. I have to kill/killall to get them to stop or reboot.

jimp

When it's still running, run:

ps uxawww | grep snort

And show the output. It's possible that the way it's run via the PBI wrapper the process check/test may be failing to catch it.

mschiek01

@jimp:

When it's still running, run:
ps uxawww | grep snort
And show the output. It's possible that the way it's run via the PBI wrapper the process check/test may be failing to catch it.

2.0.1-RELEASE (amd64)
Snort 2.9.2.3 pkg v. 2.2.1

root 4501 0.0 9.6 1117960 802636 ?? Ss 5:03PM 0:21.03 /usr/local/bin/snort -R 64038 -D -q -l /var/log/snort/snort_em064038 –pid-path /var/run --nolock-pidfile -G 64038 -c /usr/local/etc/snort/snort_64038_em0/snort.conf -i em0
root 6101 0.0 9.6 1117960 799584 ?? Ss 9:34PM 0:00.31 /usr/local/bin/snort -R 64038 -D -q -l /var/log/snort/snort_em064038 --pid-path /var/run --nolock-pidfile -G 64038 -c /usr/local/etc/snort/snort_64038_em0/snort.conf -i em0
root 24261 0.0 9.6 1117960 802800 ?? Ss 4:51PM 0:21.83 /usr/local/bin/snort -R 64038 -D -q -l /var/log/snort/snort_em064038 --pid-path /var/run --nolock-pidfile -G 64038 -c /usr/local/etc/snort/snort_64038_em0/snort.conf -i em0
root 45617 0.0 9.5 1111816 793256 ?? Ss 9:37PM 0:00.35 /usr/local/bin/snort -R 24899 -D -q -l /var/log/snort/snort_em124899 --pid-path /var/run --nolock-pidfile -G 24899 -c /usr/local/etc/snort/snort_24899_em1/snort.conf -i em1
root 55522 0.0 9.9 1126280 826816 ?? Ss 5:06PM 17:23.89 /usr/local/bin/snort -R 24899 -D -q -l /var/log/snort/snort_em124899 --pid-path /var/run --nolock-pidfile -G 24899 -c /usr/local/etc/snort/snort_24899_em1/snort.conf -i em1
root 55915 0.0 9.6 1115912 801000 ?? SNs 4:56PM 0:21.24 /usr/local/bin/snort -R 64038 -D -q -l /var/log/snort/snort_em064038 --pid-path /var/run --nolock-pidfile -G 64038 -c /usr/local/etc/snort/snort_64038_em0/snort.conf -i em0
Administrator 55857 0.0 0.0 9120 1452 0 S+ 9:39PM 0:00.00 grep snort

miles267

what is the fix for this? currently, I have to manually pgrep snort then pkill snort to kill all processes. after I update rules manually and snort restarts with only one instance.

breusshe

Okay, guys, this is sorta fixed. I found out what the problem is. You need to edit the file:

/usr/local/etc/rc.d/snort.sh

Near the bottom of the file, there is this case statement:

case $1 in
        start)
                rc_start
                ;;
        stop)
                rc_stop
                ;;
        restart)
                rc_start
                ;;
esac

Change it to this:

case $1 in
        start)
                rc_start
                ;;
        stop)
                rc_stop
                ;;
        restart)
                rc_stop
                rc_start
                ;;
esac

Essentially, the rc_stop function call is missing from the restart segment. Here is the only problem with this fix. This file is auto-generated whenever there is a change to the snort service. So, if you add/remove/edit any of your snort interfaces, you will need to add this back in. I'm looking for the script that auto-generates this file, but have not located it yet.

At least this fix will get you to a place where your Snort isn't failing to kill old iterations of your service.

Oh, one other thing, this only fixes the auto-restart that is done during rules updates.

jimp

Fixed that for ya:
https://github.com/bsdperimeter/pfsense-packages/commit/cd645a1b11544eda4f3db68ba49caaec8bbf973e

breusshe

Huh. I spoke too soon. While it is true the rc_stop needed to be added to the snort.sh file, the problem is still not fixed.

There is one other problem in the /usr/local/pkg/snort/snort.inc file. Line number 894 currently reads as:

if [ "`/bin/pgrep -nF {$g['varrun_path']}/snort_{$if_real}{$snort_uuid}.pid`" = "0" ]; then

It should be:

if [ "`/bin/pgrep -nF {$g['varrun_path']}/snort_{$if_real}{$snort_uuid}.pid`" != "0" ]; then

The reason is that this line is part of the rc_stop() function and is checking to see if there are any existing instances of Snort running. "0" simply means the command completed. However, if an active pid is found in the pid file, that pid number is returned. Therefore, "=" should be "!=" in order for the pkill command to be run, thus stopping any running instances. This also needs to be updated in the snort.sh file and can be done by either making some arbitrary change to your snort config, saving, changing back, then resaving or manually editing /usr/local/etc/rc.d/snort.sh so that in the rc_stop() function, each instance of the above if statement is updated to "!=". Here's an example:

if [ "`/bin/pgrep -nF /var/run/snort_re027549.pid`" = "0" ]; then

becomes:

if [ "`/bin/pgrep -nF /var/run/snort_re027549.pid`" != "0" ]; then

The pid file name will be different for each instance, so don't worry about matching the name.

If you manually edit the snort.sh file DO NOT make this change to the rc_start() function. It messes things up if you do.

miles267

@jimp:

Fixed that for ya:
https://github.com/bsdperimeter/pfsense-packages/commit/cd645a1b11544eda4f3db68ba49caaec8bbf973e

What does this mean? Do you mean that, although there isn't a new version of the snort package, you applied a fix to the problem where snort doesn't cleanup old instances during update?

breusshe

@miles267:

What does this mean? Do you mean that, although there isn't a new version of the snort package, you applied a fix to the problem where snort doesn't cleanup old instances during update?

Yes, he fixed the code in the repository. Snort.inc is a support file that is downloaded during the post-install of Snort. So, anyone installing/reinstalling Snort will have the fixed snort.inc. However, I just applied the fixed listed in this thread (there are two separate posts that make for the entire fix, look for the ones with smiley faces) to my pfSense server manually rather than waiting for the repository to get corrected. As of this posting, only the first part, the missing rc_stop function call, is fixed in the repository.

breusshe

Just realized all I needed to do was create a GitHub account and I could do the second part of the fix. So, I did. Just waiting for an admin to approve the change and pull it back into the master branch. Once that is done, this problem should be completely fixed.

https://github.com/bsdperimeter/pfsense-packages/pull/275

miles267

@breusshe:

@miles267:

What does this mean? Do you mean that, although there isn't a new version of the snort package, you applied a fix to the problem where snort doesn't cleanup old instances during update?

Yes, he fixed the code in the repository. Snort.inc is a support file that is downloaded during the post-install of Snort. So, anyone installing/reinstalling Snort will have the fixed snort.inc. However, I just applied the fixed listed in this thread (there are two separate posts that make for the entire fix, look for the ones with smiley faces) to my pfSense server manually rather than waiting for the repository to get corrected. As of this posting, only the first part, the missing rc_stop function call, is fixed in the repository.

Great! Thanks for investigating this issue. Unfortunately, I attempted the manual fix. While it appeared to work at first, I woke up only to once again find 4-6 instances of snort (I only have 2 interfaces) after the nightly update of definitions so it didn't work as I had hoped. By sounds of it, we shouldn't have to suffer very much longer.

fragged

Re-install Snort (I did remove + install from gui) and you should be ok. There's no need for the package version number to be bumped for these fixes to be applied.

eri--

Actually the fix was committed and the bump of the version was done for this specific issue.

miles267

@ermal:

Actually the fix was committed and the bump of the version was done for this specific issue.

Thank you SO MUCH Ermal. Am installing the latest SNORT now (2.9.2.3 pkg v. 2.2.2).

miles267

@fragged:

Re-install Snort (I did remove + install from gui) and you should be ok. There's no need for the package version number to be bumped for these fixes to be applied.

Sadly, I removed and re-installed snort from the packages list and the problem persists. After performing a manual update of snort definitions followed by a 'pgrep snort' command, it returns the following:

$ pgrep snort
26593
23394
43660
33218

However I only have snort installed on 2 interfaces. In other words, it's still not killing the original process. After pgrep i should only have (2) PIDs. I must continue to issue a 'pkill snort' and then restart the snort service to fix. manually.

AhnHEL

@miles267:

Sadly, I removed and re-installed snort from the packages list and the problem persists. After performing a manual update of snort definitions followed by a 'pgrep snort' command, it returns the following:

$ pgrep snort
26593
23394
43660
33218

However I only have snort installed on 2 interfaces. In other words, it's still not killing the original process. After pgrep i should only have (2) PIDs. I must continue to issue a 'pkill snort' and then restart the snort service to fix. manually.

I'm not seeing this at all, reinstalled from previous version of package and all is good now with start and restart of snort only showing one instance.

breusshe

@ermal:

Actually the fix was committed and the bump of the version was done for this specific issue.

Hey, Ermal;

Sorry to tell you this, but the changes that jimp and myself put into the repository are not there. Even jimp's is gone and that was there when I added my fix. As such, line 894 still needs the "!=" and rc_stop needs to be added at line 937. I've re-added them and requested a new merge to see if it takes this time:

https://github.com/bsdperimeter/pfsense-packages/pull/277

I figured this out by reinstalling the package and looking at the snort.sh file. It was missing the required changes.