SNORT - Snort 2.9.2.3 pkg v. 2.2.1 process do not quit via update scripts or GUI

mschiek01

When it's still running, run:
ps uxawww | grep snort
And show the output. It's possible that the way it's run via the PBI wrapper the process check/test may be failing to catch it.

2.0.1-RELEASE (amd64)
Snort 2.9.2.3 pkg v. 2.2.1

root 4501 0.0 9.6 1117960 802636 ?? Ss 5:03PM 0:21.03 /usr/local/bin/snort -R 64038 -D -q -l /var/log/snort/snort_em064038 –pid-path /var/run --nolock-pidfile -G 64038 -c /usr/local/etc/snort/snort_64038_em0/snort.conf -i em0
root 6101 0.0 9.6 1117960 799584 ?? Ss 9:34PM 0:00.31 /usr/local/bin/snort -R 64038 -D -q -l /var/log/snort/snort_em064038 --pid-path /var/run --nolock-pidfile -G 64038 -c /usr/local/etc/snort/snort_64038_em0/snort.conf -i em0
root 24261 0.0 9.6 1117960 802800 ?? Ss 4:51PM 0:21.83 /usr/local/bin/snort -R 64038 -D -q -l /var/log/snort/snort_em064038 --pid-path /var/run --nolock-pidfile -G 64038 -c /usr/local/etc/snort/snort_64038_em0/snort.conf -i em0
root 45617 0.0 9.5 1111816 793256 ?? Ss 9:37PM 0:00.35 /usr/local/bin/snort -R 24899 -D -q -l /var/log/snort/snort_em124899 --pid-path /var/run --nolock-pidfile -G 24899 -c /usr/local/etc/snort/snort_24899_em1/snort.conf -i em1
root 55522 0.0 9.9 1126280 826816 ?? Ss 5:06PM 17:23.89 /usr/local/bin/snort -R 24899 -D -q -l /var/log/snort/snort_em124899 --pid-path /var/run --nolock-pidfile -G 24899 -c /usr/local/etc/snort/snort_24899_em1/snort.conf -i em1
root 55915 0.0 9.6 1115912 801000 ?? SNs 4:56PM 0:21.24 /usr/local/bin/snort -R 64038 -D -q -l /var/log/snort/snort_em064038 --pid-path /var/run --nolock-pidfile -G 64038 -c /usr/local/etc/snort/snort_64038_em0/snort.conf -i em0
Administrator 55857 0.0 0.0 9120 1452 0 S+ 9:39PM 0:00.00 grep snort

miles267

what is the fix for this? currently, I have to manually pgrep snort then pkill snort to kill all processes. after I update rules manually and snort restarts with only one instance.

breusshe

Okay, guys, this is sorta fixed. I found out what the problem is. You need to edit the file:

/usr/local/etc/rc.d/snort.sh

Near the bottom of the file, there is this case statement:

case $1 in
        start)
                rc_start
                ;;
        stop)
                rc_stop
                ;;
        restart)
                rc_start
                ;;
esac

Change it to this:

case $1 in
        start)
                rc_start
                ;;
        stop)
                rc_stop
                ;;
        restart)
                rc_stop
                rc_start
                ;;
esac

Essentially, the rc_stop function call is missing from the restart segment. Here is the only problem with this fix. This file is auto-generated whenever there is a change to the snort service. So, if you add/remove/edit any of your snort interfaces, you will need to add this back in. I'm looking for the script that auto-generates this file, but have not located it yet.

At least this fix will get you to a place where your Snort isn't failing to kill old iterations of your service.

Oh, one other thing, this only fixes the auto-restart that is done during rules updates.

jimp

Fixed that for ya:
https://github.com/bsdperimeter/pfsense-packages/commit/cd645a1b11544eda4f3db68ba49caaec8bbf973e

breusshe

Huh. I spoke too soon. While it is true the rc_stop needed to be added to the snort.sh file, the problem is still not fixed.

There is one other problem in the /usr/local/pkg/snort/snort.inc file. Line number 894 currently reads as:

if [ "`/bin/pgrep -nF {$g['varrun_path']}/snort_{$if_real}{$snort_uuid}.pid`" = "0" ]; then

It should be:

if [ "`/bin/pgrep -nF {$g['varrun_path']}/snort_{$if_real}{$snort_uuid}.pid`" != "0" ]; then

The reason is that this line is part of the rc_stop() function and is checking to see if there are any existing instances of Snort running. "0" simply means the command completed. However, if an active pid is found in the pid file, that pid number is returned. Therefore, "=" should be "!=" in order for the pkill command to be run, thus stopping any running instances. This also needs to be updated in the snort.sh file and can be done by either making some arbitrary change to your snort config, saving, changing back, then resaving or manually editing /usr/local/etc/rc.d/snort.sh so that in the rc_stop() function, each instance of the above if statement is updated to "!=". Here's an example:

if [ "`/bin/pgrep -nF /var/run/snort_re027549.pid`" = "0" ]; then

becomes:

if [ "`/bin/pgrep -nF /var/run/snort_re027549.pid`" != "0" ]; then

The pid file name will be different for each instance, so don't worry about matching the name.

If you manually edit the snort.sh file DO NOT make this change to the rc_start() function. It messes things up if you do.

miles267

@jimp:

Fixed that for ya:
https://github.com/bsdperimeter/pfsense-packages/commit/cd645a1b11544eda4f3db68ba49caaec8bbf973e

What does this mean? Do you mean that, although there isn't a new version of the snort package, you applied a fix to the problem where snort doesn't cleanup old instances during update?

breusshe

@miles267:

What does this mean? Do you mean that, although there isn't a new version of the snort package, you applied a fix to the problem where snort doesn't cleanup old instances during update?

Yes, he fixed the code in the repository. Snort.inc is a support file that is downloaded during the post-install of Snort. So, anyone installing/reinstalling Snort will have the fixed snort.inc. However, I just applied the fixed listed in this thread (there are two separate posts that make for the entire fix, look for the ones with smiley faces) to my pfSense server manually rather than waiting for the repository to get corrected. As of this posting, only the first part, the missing rc_stop function call, is fixed in the repository.

breusshe

Just realized all I needed to do was create a GitHub account and I could do the second part of the fix. So, I did. Just waiting for an admin to approve the change and pull it back into the master branch. Once that is done, this problem should be completely fixed.

https://github.com/bsdperimeter/pfsense-packages/pull/275

miles267

@breusshe:

@miles267:

What does this mean? Do you mean that, although there isn't a new version of the snort package, you applied a fix to the problem where snort doesn't cleanup old instances during update?

Yes, he fixed the code in the repository. Snort.inc is a support file that is downloaded during the post-install of Snort. So, anyone installing/reinstalling Snort will have the fixed snort.inc. However, I just applied the fixed listed in this thread (there are two separate posts that make for the entire fix, look for the ones with smiley faces) to my pfSense server manually rather than waiting for the repository to get corrected. As of this posting, only the first part, the missing rc_stop function call, is fixed in the repository.

Great! Thanks for investigating this issue. Unfortunately, I attempted the manual fix. While it appeared to work at first, I woke up only to once again find 4-6 instances of snort (I only have 2 interfaces) after the nightly update of definitions so it didn't work as I had hoped. By sounds of it, we shouldn't have to suffer very much longer.

fragged

Re-install Snort (I did remove + install from gui) and you should be ok. There's no need for the package version number to be bumped for these fixes to be applied.

eri--

Actually the fix was committed and the bump of the version was done for this specific issue.

miles267

@ermal:

Actually the fix was committed and the bump of the version was done for this specific issue.

Thank you SO MUCH Ermal. Am installing the latest SNORT now (2.9.2.3 pkg v. 2.2.2).

miles267

@fragged:

Re-install Snort (I did remove + install from gui) and you should be ok. There's no need for the package version number to be bumped for these fixes to be applied.

Sadly, I removed and re-installed snort from the packages list and the problem persists. After performing a manual update of snort definitions followed by a 'pgrep snort' command, it returns the following:

$ pgrep snort
26593
23394
43660
33218

However I only have snort installed on 2 interfaces. In other words, it's still not killing the original process. After pgrep i should only have (2) PIDs. I must continue to issue a 'pkill snort' and then restart the snort service to fix. manually.

AhnHEL

@miles267:

Sadly, I removed and re-installed snort from the packages list and the problem persists. After performing a manual update of snort definitions followed by a 'pgrep snort' command, it returns the following:

$ pgrep snort
26593
23394
43660
33218

However I only have snort installed on 2 interfaces. In other words, it's still not killing the original process. After pgrep i should only have (2) PIDs. I must continue to issue a 'pkill snort' and then restart the snort service to fix. manually.

I'm not seeing this at all, reinstalled from previous version of package and all is good now with start and restart of snort only showing one instance.

breusshe

@ermal:

Actually the fix was committed and the bump of the version was done for this specific issue.

Hey, Ermal;

Sorry to tell you this, but the changes that jimp and myself put into the repository are not there. Even jimp's is gone and that was there when I added my fix. As such, line 894 still needs the "!=" and rc_stop needs to be added at line 937. I've re-added them and requested a new merge to see if it takes this time:

https://github.com/bsdperimeter/pfsense-packages/pull/277

I figured this out by reinstalling the package and looking at the snort.sh file. It was missing the required changes.

fragged

I reinstalled before the package version number was bumped and mine is running fine so far since last morning. I guess I got lucky with timing :)

Available: 2.9.2.3 pkg v. 2.2.2
Installed: 2.9.2.3 pkg v. 2.2.1

eri--

@breusshe:

@ermal:

Actually the fix was committed and the bump of the version was done for this specific issue.

Hey, Ermal;

Sorry to tell you this, but the changes that jimp and myself put into the repository are not there. Even jimp's is gone and that was there when I added my fix. As such, line 894 still needs the "!=" and rc_stop needs to be added at line 937. I've re-added them and requested a new merge to see if it takes this time:

https://github.com/bsdperimeter/pfsense-packages/pull/277

I figured this out by reinstalling the package and looking at the snort.sh file. It was missing the required changes.

Put a comment on the pull request but those were not the right fixes.
Especially the != 0 is wrong.

HOD

The latest version Snort 2.9.2.3 pkg v. 2.2.2 has the same error.

Could not remove pid file /var/run/snort_pppoe013406.pid: No such file or directory

Yesterday all works fine for me.

to reproduce: go to Services and klick on restart. I got twice snort pids. (Rule Update is the same…)

miles267

@HOD:

The latest version Snort 2.9.2.3 pkg v. 2.2.2 has the same error.
Could not remove pid file /var/run/snort_pppoe013406.pid: No such file or directory 
Yesterday all works fine for me.

to reproduce: go to Services and klick on restart. I got twice snort pids. (Rule Update is the same…)

Same issue here. Appears 2.9.2.3 v. 2.2.2 hasn't resolved the issue as we had hoped.

breusshe

@ermal:

Put a comment on the pull request but those were not the right fixes.
Especially the != 0 is wrong.

Then why is it this fixed my snort? Changing to != stopped my snort from not killing existing instances when the service is restarted. The "0" simply means the process exited without error. However, that if statement is looking for existing instances in the PID file. When pgrep finds a pid number in the pid file, it returns that number rather than 0. Therefore, it SHOULD be != or the script will not work. I suggest you try it, you'll see I'm right. Just issue that pgrep command as part of an echo statement at a command prompt and see what it returns when the PID file has a valid instance and when it does not. If an instance exists, you'll see the pid number, which is when you should run pkill to stop it, since we are looking to stop existing instances. If an instance does not exists, you will get 0. In that case, no running instances, so don't bother trying to stop them with pkill. Honestly, I did actually test this out before suggesting the fix.