SNORT - Snort 2.9.2.3 pkg v. 2.2.1 process do not quit via update scripts or GUI
-
Hey Breusshe!
Thx for your investigations. Did you also add the rc_stop statement in snort.inc before line 937? This generates working snort.sh for me. Logic bug or not - at leats it works. Stock installation gives no errors in log. You just get two instances.
Greets, Judex
-
Here is restart:
/usr/local/etc/rc.d(60): ./snort.sh restart inside rc_start() inside rc_stop() Spawning daemon child... My daemon child 45455 lives... Daemon parent exiting (0) Spawning daemon child... My daemon child 45983 lives... Daemon parent exiting (0)The inside rc_start() and inside rc_stop() are echo tags I put into the snort.sh file so I could see when pid file does not exist errors were occurring. My current iteration of snort.sh does not throw such errors any longer.
and here is pgrep showing how many are now running:
/usr/local/etc/rc.d(61): pgrep snort 45983 45455 -
Yeah, I put that in with my first round of corrections. But, after seeing Ermal's post and checking things out, I've revised my suggested fix. See the long post I put up a few minutes ago.
-
@ermal:
Install stock 2.2.2 and give me syslog errors on what is happening!
Or run snort.sh from ssh session manually specifying restart and post it here.I run "snort.sh restart" with stock snort.sh. Before one pid, after two pids:
[2.0.1-RELEASE][root@gatekeeper.me.local]/root(27): pgrep snort 57551 [2.0.1-RELEASE][root@gatekeeper.me.local]/root(28): /usr/local/etc/rc.d/snort.sh restart Spawning daemon child... My daemon child 11345 lives... Daemon parent exiting (0) [2.0.1-RELEASE][root@gatekeeper.me.local]/root(29): pgrep snort 11345 57551Sorry for the redundant information. Breusshe was faster…
-
No, that's great, Judex. It shows stock isn't working, but my is. Now the problem can be fixed.
-
Uninstalling vs 2.2.1 and installing 2.2.2 worked for me (just now). Snort can be stopped now, and pgrep snort shows only 1 instance per interface when started. Blocked IPs when the interface shows stopped issue also addressed.
Thx :-)
-
I've also found that I had two instances of snort running after the update.
I'm not sure if it happened immediately or not, but I noticed today that my RAM and SWAP usage was way up, so I checked top and noticed both the instances.
I also noticed that pressing the stop button in the services window did not stop both snort services, however, pressing stop on the specific interface did successfully stop one of them.
Not sure if this helps or not.
-
Okay, for the third time, I've posted fixes for the snort restart problem to the repository. Here is the pull request:
https://github.com/bsdperimeter/pfsense-packages/pull/278
Hopefully someone will actually try these changes out before stating they will not fix the problem and closing the pull request.
Here's to hope!
-
Ok fixed and found the issue why your suggestion of != 0 was seemingly working for you.
Check the new commit i made that fixes the issue/bug in the rc script. -
@ermal:
Ok fixed and found the issue why your suggestion of != 0 was seemingly working for you.
Check the new commit i made that fixes the issue/bug in the rc script.Thanks, Ermal. I re-installed snort and re-saved an instance to update the snort.sh file. Your solution works, with two minor hitches. See the below output from top:
1st Trial:
Mem: 950M Active, 107M Inact, 336M Wired, 4K Cache, 248M Buf, 2524M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 59839 root 2 44 0 928M 613M nanslp 0 0:00 0.00% snort 23711 root 2 44 0 616M 301M nanslp 0 0:00 0.00% snort Mem: 979M Active, 305M Inact, 339M Wired, 4K Cache, 248M Buf, 2294M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 59839 root 2 44 0 1056M 731M nanslp 0 1:27 0.00% snort 23711 root 2 44 0 738M 411M nanslp 1 0:29 0.00% snort Mem: 409M Active, 280M Inact, 334M Wired, 4K Cache, 248M Buf, 2893M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 23711 root 2 44 0 876M 543M nanslp 0 0:57 0.00% snort2nd Trial:
Mem: 945M Active, 112M Inact, 336M Wired, 4K Cache, 248M Buf, 2523M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 29717 root 2 44 0 928M 614M nanslp 2 0:04 20.65% snort 31324 root 2 45 0 616M 300M nanslp 2 0:02 70.61% snort Mem: 975M Active, 312M Inact, 339M Wired, 4K Cache, 248M Buf, 2290M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 29717 root 2 44 0 1050M 726M nanslp 2 1:54 10.89% snort 31324 root 2 44 0 746M 420M nanslp 0 0:55 10.25% snort Mem: 413M Active, 276M Inact, 334M Wired, 4K Cache, 248M Buf, 2894M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 31324 root 2 44 0 874M 542M nanslp 0 1:44 3.96% snort3rd trial:
Mem: 948M Active, 109M Inact, 336M Wired, 4K Cache, 248M Buf, 2523M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 44906 root 2 44 0 616M 300M nanslp 0 0:05 15.58% snort 43256 root 2 44 0 928M 614M nanslp 1 0:05 9.18% snort Mem: 985M Active, 317M Inact, 339M Wired, 4K Cache, 248M Buf, 2276M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 43256 root 2 44 0 1058M 734M nanslp 2 1:48 5.76% snort 44906 root 2 44 0 752M 426M nanslp 0 0:52 5.57% snort Mem: 414M Active, 294M Inact, 335M Wired, 4K Cache, 248M Buf, 2874M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 44906 root 2 44 0 894M 561M nanslp 0 1:39 0.68% snortThe above are three trial runs where I just kept issuing the command snort.sh restart, each trial only has three restarts. I then had to issue snort.sh stop because, as you can see in the 3rd run of each trial, one of my snort instances does not restart. That is when I began the next trial.
As you can see, the snort instances use more memory between each successive runs of the snort.sh script. That makes sense to me since -HUP doesn't kill the process, it only refreshes it, mapping changes to new segments of memory. The problem is that many folks use systems with limited resources. As each update to the snort rules happens, a little more memory is lost to HUP. Eventually, all the resources are used up. That means admins need to watch Snort to make sure it isn't hitting that limit. However, this might not be an issue since, as I mentioned, one of my snort instances will not survive the 3rd restart. It is a consistent behavior. On the 3rd restart of snort, an instance of snort does not recover. I then removed -HUP from snort.sh and modified the if-else statement it was in into just an if-statement. The contents of the else-statement were moved outside of the if-statement. Here is the modified code:
if [ $? = 0 ]; then /bin/pkill -F /var/run/snort_re027549.pid -a /usr/bin/logger -p daemon.info -i -t SnortStartup "Snort SOFT START For Block Only Rules(27549_re0)..." fi # Start snort and barnyard2 /bin/rm /var/run/snort_re027549.pid /usr/local/bin/snort -R 27549 -D -q -l /var/log/snort/snort_re027549 --pid-path /var/run --nolock-pidfile -G 27549 -c /usr/local/etc/snort/snort_27549_re0/snort.conf -i re0 /usr/bin/logger -p daemon.info -i -t SnortStartup "Snort START For Block Only Rules(27549_re0)..."I then started repeating my trials. Only got one trial, but it has six restarts in it. Both snort instances come back with each restart and memory usage is consistent across all restarts.
Removed -HUP, new trials:
Mem: 951M Active, 107M Inact, 336M Wired, 4K Cache, 248M Buf, 2523M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 15636 root 2 44 0 616M 300M nanslp 2 0:13 1.95% snort 52030 root 2 44 0 928M 614M nanslp 2 0:13 1.95% snort Mem: 946M Active, 111M Inact, 336M Wired, 4K Cache, 248M Buf, 2524M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12265 root 2 44 0 928M 614M nanslp 2 0:00 0.29% snort 41341 root 2 44 0 616M 300M nanslp 2 0:00 0.29% snort Mem: 949M Active, 109M Inact, 336M Wired, 4K Cache, 248M Buf, 2523M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 24100 root 2 47 0 616M 300M nanslp 2 0:01 92.82% snort 61007 root 2 44 0 928M 614M nanslp 0 0:01 14.65% snort Mem: 950M Active, 108M Inact, 336M Wired, 4K Cache, 248M Buf, 2523M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 42501 root 2 44 0 616M 300M nanslp 2 0:03 43.46% snort 12811 root 2 44 0 928M 614M nanslp 2 0:05 19.78% snort Mem: 952M Active, 106M Inact, 336M Wired, 4K Cache, 248M Buf, 2522M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 36328 root 2 45 0 616M 301M nanslp 2 0:01 71.44% snort 34615 root 2 44 0 928M 614M nanslp 1 0:03 18.95% snort Mem: 950M Active, 108M Inact, 336M Wired, 4K Cache, 248M Buf, 2523M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 17614 root 2 44 0 616M 300M nanslp 0 0:03 43.65% snort 15390 root 2 44 0 928M 614M nanslp 0 0:05 19.34% snortI'm suggesting that the -HUP be removed from snort.inc so it does not, at the very least, hinder performance for limited resource users. I know this means that when a snort restart happens, it will also mean the system isn't protected by snort for a few minutes. However, I believe snort dying after a restart will also be prevented with other users. I'm not saying at least one snort instance will die on the third restart, but I do assert that snort will die after some number of restarts and, for a given system, will be consistent. I've posted a patch for proposed patch to snort.inc, unless you can see a better solution than mine:
https://github.com/bsdperimeter/pfsense-packages/pull/281
Maybe a better solution would be to add a new setting to the global settings, a checkbox that allows the user to determine if they want to use -HUP or not. Then they can look at their environment and decide which is better for them: a steady creep of resource loss or not being unprotected for a few minutes. Depending on that setting, either the code as it stands in the repository is used, or something similar to the code I'm proposing.
-
I think that time is a factor here with your snort.sh.
A service like snort needs its time to prepare itself for releasing and allocating new resources from a HUP so you have to take that into consideration.Furthermore i would be expecting a system log associated with that.
-
@ermal:
I think that time is a factor here with your snort.sh.
A service like snort needs its time to prepare itself for releasing and allocating new resources from a HUP so you have to take that into consideration.Furthermore i would be expecting a system log associated with that.
How much time? will that result in the memory usage dropping back down closer to it's original levels?