Snort 2.9.4.6 Pkg v 2.5.9

bmeeks · Jul 17, 2013, 12:15 AM

I just updated Snort to package version 2.5.9 on my machine which is running 2.1-RC0.

I'm not sure if this bug is known or not, and I didn't see it in the thread.

With the new installation of the Snort package, I created a new, empty suppress list. Afterwards, clicking the suppress button in the alerts tab under the SID column did not work as expected. No suppression entry was generated in the suppression list and the line in the alert was not 'greyed out' to indicate the rule was being suppressed.

As a workaround, I manually created one suppression rule by copying and pasting a rule from the example on the suppression page and saved. Automatic suppression generation buttons worked fine after that.

@adam65535:

I remember seeing similar behavior and I am pretty sure it was with a blank whitelist too. I ended up editing the whitelist manually for some reason and at some point I noticed it was working again.

I found this bug. The fix will be in the next Snort Package update. The code I wrote blindly assumed any pre-existing Suppress List would not initially be "empty". That was a dumb assumption on my part. Should have thought about the scenario described here. I was testing for a "if not empty" list and updating it with the new entry, but if the list was empty my code was just falling through the "if statement" and doing nothing.

Sorry about that… :(
Bill

Supermule · Jul 17, 2013, 2:53 AM

No worries mate!! The Snort package has elevated itself a 1000 times since you began development on it!

Bloody good job!

gogol · Jul 20, 2013, 10:50 AM

Every time I do a reboot, no upgrade or so, just a reboot, I have 2 processes of Snort running:

root(2): ps uaxwwww | grep snort
root    8350  4.5 17.6 629508 364924  ??  Ss   12:24PM   0:53.61 /usr/pbi/snort-i386/bin/snort -R 54477 -E -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0
root   21658  2.3 17.9 634628 370092  ??  SNs  12:24PM   0:03.93 /usr/pbi/snort-i386/bin/snort -R 54477 -D -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0
root   31323  0.0  0.1  3468  1240   0  S+   12:26PM   0:00.00 grep snort

I have to kill both processes from the terminal and restart Snort from the webGUI.
The -E option has something to do with Windows logging. Where does that come from? Does this have something to do how this version was compiled?

Supermule · Jul 20, 2013, 10:51 AM

You have a cron job running??

bmeeks · Jul 20, 2013, 7:00 PM

@gogol:

Every time I do a reboot, no upgrade or so, just a reboot, I have 2 processes of Snort running:
root(2): ps uaxwwww | grep snort
root    8350  4.5 17.6 629508 364924  ??  Ss   12:24PM   0:53.61 /usr/pbi/snort-i386/bin/snort -R 54477 -E -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0
root   21658  2.3 17.9 634628 370092  ??  SNs  12:24PM   0:03.93 /usr/pbi/snort-i386/bin/snort -R 54477 -D -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0
root   31323  0.0  0.1  3468  1240   0  S+   12:26PM   0:00.00 grep snort
I have to kill both processes from the terminal and restart Snort from the webGUI.
The -E option has something to do with Windows logging. Where does that come from? Does this have something to do how this version was compiled?

The process with the -E argument is not being generated by the default package installation. Something else is launching that process. Have you looked in all the startup scripts for your firewall to see if something else is launching a Snort instance?

Bill

gogol · Jul 20, 2013, 8:33 PM

@bmeeks:

@gogol:
Every time I do a reboot, no upgrade or so, just a reboot, I have 2 processes of Snort running:
root(2): ps uaxwwww | grep snort
root    8350  4.5 17.6 629508 364924  ??  Ss   12:24PM   0:53.61 /usr/pbi/snort-i386/bin/snort -R 54477 -E -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0
root   21658  2.3 17.9 634628 370092  ??  SNs  12:24PM   0:03.93 /usr/pbi/snort-i386/bin/snort -R 54477 -D -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0
root   31323  0.0  0.1  3468  1240   0  S+   12:26PM   0:00.00 grep snort
I have to kill both processes from the terminal and restart Snort from the webGUI.
The -E option has something to do with Windows logging. Where does that come from? Does this have something to do how this version was compiled?
The process with the -E argument is not being generated by the default package installation. Something else is launching that process. Have you looked in all the startup scripts for your firewall to see if something else is launching a Snort instance?

Bill

Bill,

I also see this on my VM. During the boot process I sshd into my VM and the snort.sh startup script is also running twice:

ps auxwwww | grep snort
root   10800 78.0 17.9 636420 372192  v0- RL   10:15PM   0:15.15 /usr/pbi/snort-i386/bin/snort -R 54477 -D -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0
root   41460 20.0  1.8 240132 36832  ??  RN   10:15PM   0:00.48 /usr/pbi/snort-i386/bin/snort -R 54477 -D -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0
root   40567 17.0  0.1  3644  1576  ??  SN   10:15PM   0:00.00 /bin/sh /usr/local/etc/rc.d/snort.sh start
root   10447  0.0  0.1  3644  1500  v0- S    10:15PM   0:00.00 /bin/sh /usr/local/etc/rc.d/snort.sh start
root   42050  0.0  0.1  4696  2332   0  RV   10:15PM   0:00.00 grep snort (tcsh)

A few moments later:

ps auxwwww | grep snort
root   41460 89.0 50.9 2305540 1057752  ??  RN   10:15PM   1:43.48 /usr/pbi/snort-i386/bin/snort -R 54477 -D -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0
root   71694 62.0  4.7 361988 98480  ??  Rs   10:17PM   0:08.92 /usr/pbi/snort-i386/bin/snort -R 54477 -E -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0
root   40567  0.0  0.0  3644     0  ??  IWN  -         0:00.00 /bin/sh /usr/local/etc/rc.d/snort.sh start
root   74561  0.0  0.1  3468  1236   0  S+   10:17PM   0:00.00 grep snort

One snort.sh script has stopped and the -E argument appears

Another few moments later the other snort.sh script has ended and I am left with two snort processes.

I grepped into my machine with the following string: "/var/log/snort/snort_em054477" and only the snort startup script has this string.
What could be causing the snort startup script running twice?

bmeeks · Jul 21, 2013, 2:05 PM

@gogol:

I grepped into my machine with the following string: "/var/log/snort/snort_em054477" and only the snort startup script has this string.
What could be causing the snort startup script running twice?

I don't know what could be launching the startup script twice. Also not sure how that -E argument is getting on there either.

If you can, completely remove Snort from the firewall via System…Packages and click the X to delete the package. So long as you have the box checked on the Global Settings tab to save the configuration on de-install, then everything will come back OK. After removing Snort, and before installing again, reboot the firewall to see what happens in terms of something else trying to start Snort. If some other thing is attempting to launch it, then it should log some kind of error with the files removed.

Re-install Snort and see if that makes any difference. This is a strange problem. There were some issues a long time ago (as in about a year ago) with multiple instances of Snort getting started on reboot, but Ermal and I are pretty sure those are behind us with changes to the startup scripts. I have been doing a lot of startup/shutdown/reboot operations on my VMs while testing the next Snort update, and I have not noticed this problem.

Bill

gogol · Jul 21, 2013, 9:18 PM

@bmeeks:

@gogol:

I grepped into my machine with the following string: "/var/log/snort/snort_em054477" and only the snort startup script has this string.
What could be causing the snort startup script running twice?

I don't know what could be launching the startup script twice. Also not sure how that -E argument is getting on there either.

If you can, completely remove Snort from the firewall via System…Packages and click the X to delete the package. So long as you have the box checked on the Global Settings tab to save the configuration on de-install, then everything will come back OK. After removing Snort, and before installing again, reboot the firewall to see what happens in terms of something else trying to start Snort. If some other thing is attempting to launch it, then it should log some kind of error with the files removed.

Re-install Snort and see if that makes any difference. This is a strange problem. There were some issues a long time ago (as in about a year ago) with multiple instances of Snort getting started on reboot, but Ermal and I are pretty sure those are behind us with changes to the startup scripts. I have been doing a lot of startup/shutdown/reboot operations on my VMs while testing the next Snort update, and I have not noticed this problem.

It is becoming even more weird. This what I did.

Booted my VM where the two processes of Snort were running after a (re)boot
Killed both processes
Removed Snort package
Rebooted and there was no trace of Snort processes
Installed Snort package again and now this:

Jul 21 21:58:12	php: /pkg_mgr_install.php: Beginning package installation for snort .
Jul 21 21:59:01	php: /pkg_mgr_install.php: [Snort] Saved settings detected... rebuilding installation with saved settings...
Jul 21 21:59:01	php: /pkg_mgr_install.php: [Snort] Downloading and updating configured rule types...
Jul 21 21:59:02	php: /pkg_mgr_install.php: [Snort] There is a new set of Snort VRT rules posted. Downloading...
Jul 21 21:59:49	php: /pkg_mgr_install.php: [Snort] There is a new set of EmergingThreats rules posted. Downloading...
Jul 21 21:59:51	php: /pkg_mgr_install.php: [Snort] EmergingThreats rules file update downloaded successfully
Jul 21 21:59:54	php: /pkg_mgr_install.php: [Snort] The Rules update has finished.
Jul 21 21:59:54	php: /pkg_mgr_install.php: [Snort] Updating rules configuration for: WAN ...
Jul 21 21:59:56	php: /pkg_mgr_install.php: [Snort] Enabling any flowbit-required rules for: WAN...
Jul 21 21:59:56	php: /pkg_mgr_install.php: [Snort] Building new sig-msg.map file for WAN...
Jul 21 21:59:57	php: /pkg_mgr_install.php: [Snort] Finished rebuilding installation from saved settings...
Jul 21 21:59:57	php: /pkg_mgr_install.php: [Snort] Starting Snort using rebuilt configuration...
Jul 21 21:59:59	SnortStartup[37422]: Snort START for WAN(54477_em0)...
Jul 21 21:59:59	php: /pkg_mgr_install.php: [Snort] Package post-installation tasks completed...
Jul 21 21:59:59	check_reload_status: Syncing firewall
Jul 21 22:00:01	check_reload_status: Reloading filter
Jul 21 22:01:51	snort[22965]: Could not create configuration reload thread.

Googling for that error shows that this is in the Snort code.

I rebooted again without starting snort in the WebGUI gives this log:

Jul 21 22:08:17	SnortStartup[11448]: Snort START for WAN(54477_em0)...
Jul 21 22:08:19	login: login on ttyv0 as root
Jul 21 22:08:19	sshlockout[15602]: sshlockout/webConfigurator v3.0 starting up
Jul 21 22:08:24	check_reload_status: Syncing firewall
Jul 21 22:08:29	SnortStartup[33999]: Snort START for WAN(54477_em0)...
Jul 21 22:10:20	snort[75620]: Could not create configuration reload thread.
Jul 21 22:10:36	snort[83412]: Could not create configuration reload thread.

Now there is a trace of two Snort processes starting up, but exiting with that error.

Rebooted again without Snort starting from the webGUI (I thought that maybe it would become three processes ;))
But that gave me the same log entries as above.

Then I started Snort from the webGUI:

Jul 21 22:19:03	php: /snort/snort_interfaces.php: Toggle (snort starting) for WAN(WAN)...
Jul 21 22:19:03	php: /snort/snort_interfaces.php: [Snort] Updating rules configuration for: WAN ...
Jul 21 22:19:05	php: /snort/snort_interfaces.php: [Snort] Enabling any flowbit-required rules for: WAN...
Jul 21 22:19:05	php: /snort/snort_interfaces.php: [Snort] Building new sig-msg.map file for WAN...
Jul 21 22:19:06	php: /snort/snort_interfaces.php: [Snort] Snort START for WAN(em0)...
Jul 21 22:20:54	snort[89067]: Could not create configuration reload thread.

So now Snort doesn't even start anymore :(

Now I removed Snort without saving the settings and reinstalled the package and configured it again. Rebooted and bam: back where I started with 2 Snort processes and one with the -E argument.

What can I do more?

gogol · Jul 22, 2013, 8:09 PM

Followup:

I installed the latest 2.1 snapshot from scratch in a VM. Logged in the webGUI and reloaded the configuration file and rebooted the system.
It began installing the Snort package and booted up normally. I see Snort running on my interface and in services, but:

#ps auxwwww | grep snort
root    8542  0.0 70.2 1721044 1458396  ??  Ss    9:30PM   1:48.01 /usr/pbi/snort-i386/bin/snort -R 54137 -E -q -l /var/log/snort/snort_em054137 --pid-path /var/run --nolock-pidfile -G 54137 -c /usr/pbi/snort-i386/etc/snort/snort_54137_em0/snort.conf -i em0
root   87032  0.0  0.1  3468  1240   0  S+   10:01PM   0:00.00 grep snort

Again with the -E argument

Rebooted and again my problems with two snort.sh startup scripts running. I can't imagine that this is a problem in the <snortglobal>section

Edit: I just saw this in the snort.sh startup script:

#!/bin/sh
########
# This file was automatically generated
# by the pfSense service handler.
# Code added to protect from double starts on pfSense bootup
######## Begining of Main snort.sh

Can these double starts have anything to do with my problem?</snortglobal>

gogol · Jul 23, 2013, 8:43 AM

I am digging deeper and deeper.

An extract from system.log:

Jul 23 10:11:52 pfsensetest php: rc.start_packages: Restarting/Starting all packages.
Jul 23 10:11:53 pfsensetest php: rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Jul 23 10:11:53 pfsensetest php: rc.newwanip: Creating rrd update script
Jul 23 10:11:54 pfsensetest check_reload_status: Syncing firewall
Jul 23 10:11:55 pfsensetest php: rc.newwanip: pfSense package system has detected an ip change 192.168.67.161 ->  192.168.67.161 ... Restarting packages.
Jul 23 10:11:55 pfsensetest check_reload_status: Starting packages
Jul 23 10:11:55 pfsensetest check_reload_status: Reloading filter
Jul 23 10:11:57 pfsensetest php: rc.start_packages: Restarting/Starting all packages.
Jul 23 10:11:58 pfsensetest SnortStartup[14194]: Snort START for WAN(54137_em0)...
Jul 23 10:12:01 pfsensetest login: login on ttyv0 as root
Jul 23 10:12:01 pfsensetest sshlockout[32403]: sshlockout/webConfigurator v3.0 starting up
Jul 23 10:12:05 pfsensetest check_reload_status: Syncing firewall
Jul 23 10:12:08 pfsensetest sshd[38417]: Accepted keyboard-interactive/pam for admin from 192.168.1.3 port 56484 ssh2
Jul 23 10:12:10 pfsensetest SnortStartup[46525]: Snort START for WAN(54137_em0)...
Jul 23 10:13:49 pfsensetest kernel: em0: promiscuous mode enabled
Jul 23 10:13:53 pfsensetest snort[64616]: SMTP reload: Changing the file_depth requires a restart.
Jul 23 10:13:54 pfsensetest kernel: em0: promiscuous mode disabled
Jul 23 10:14:05 pfsensetest kernel: em0: promiscuous mode enabled

I thought that it might be possible that while a new WAN IP is detected and rc.start_packages restarts the packages Snort has not yet written a pid file and so was not detected already running, a new process would be started. So I disabled the interface that Snort was running on to prevent an automatic startup and manually start Snort.
I was very surprised to see Snort running after a reboot. At the Snort tab I see everything disabled and at Running Services I see Snort running, also when I look at "ps aux | grep snort".

I am very confused.

Supermule · Jul 23, 2013, 8:55 AM

Can you post the config file? OR reinstall a 2.1 VM with Snort without importing the config file?

gogol · Jul 23, 2013, 11:41 AM

Snort has serious timing issues at boot up on my systems.

I tried a lot of combinations with rulesets. It was even possible to start snort without rules, which in my opinion should not be possible.

On my VM with only IPS balanced, which takes about 15 seconds to start on 1 interface, it goes well with only 1 snort process running, but I see the -E argument which is weird.

root   49494 69.0  6.0 387540 124584  ??  Rs    1:11PM   0:04.96 /usr/pbi/snort-i386/bin/snort -R 4082 -E -q -l /var/log/snort/snort_em04082 --pid-path /var/run --nolock-pidfile -G 4082 -c /usr/pbi/snort-i386/etc/snort/snort_4082_em0/snort.conf -i em0

With only 1 ET ruleset enabled I can see only 1 process, but with the -D argument.

Another log snippet with only 1 ruleset enabled:

Jul 23 12:59:09 pfsensetest php: rc.start_packages: Restarting/Starting all packages.
Jul 23 12:59:10 pfsensetest php: rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Jul 23 12:59:10 pfsensetest php: rc.newwanip: Creating rrd update script
Jul 23 12:59:11 pfsensetest check_reload_status: Syncing firewall
Jul 23 12:59:12 pfsensetest php: rc.newwanip: pfSense package system has detected an ip change 192.168.67.161 ->  192.168.67.161 ... Restarting packages.
Jul 23 12:59:12 pfsensetest check_reload_status: Starting packages
Jul 23 12:59:12 pfsensetest check_reload_status: Reloading filter
Jul 23 12:59:14 pfsensetest php: rc.start_packages: Restarting/Starting all packages.
Jul 23 12:59:15 pfsensetest SnortStartup[19390]: Snort START for WAN(4082_em0)...
Jul 23 12:59:15 pfsensetest kernel: em0: promiscuous mode enabled
Jul 23 12:59:16 pfsensetest login: login on ttyv0 as root
Jul 23 12:59:16 pfsensetest sshlockout[31161]: sshlockout/webConfigurator v3.0 starting up
Jul 23 12:59:17 pfsensetest SnortStartup[45501]: Snort STOP for WAN(4082_em0)...
Jul 23 12:59:18 pfsensetest snort[22786]: *** Caught Term-Signal
Jul 23 12:59:18 pfsensetest kernel: em0: promiscuous mode disabled
Jul 23 12:59:23 pfsensetest SnortStartup[47764]: Snort START for WAN(4082_em0)...
Jul 23 12:59:23 pfsensetest kernel: em0: promiscuous mode enabled

Watch the Snort STOP which is missing in another log snippet I posted before.

To resume:

The boot process is interfering with Snort Startup in my opinion or the other way around.

rc.newwanip detects an ip change while there isn't one and triggers a restart packages while Snort is starting, which takes a while
check_reload_status is also Starting Packages

Sometimes there is the -E argument instead of the -D in the process.

@Supermule I did start from scratch and there is nothing special in my config. Snort in test mode gives no errors.

Supermule · Jul 23, 2013, 11:44 AM

Can you pls. make a FRESH install on 2.1 and NOT import the config? And see if it happens again?

gogol · Jul 23, 2013, 1:19 PM

I understand it is a long story but I reported a clean install of latest 2.1 snapshot and a removal of Snort without saving settings and configuring from scratch of few posts ago ;)

Supermule · Jul 23, 2013, 1:22 PM

Yes but its not the snort imported config but that of the firewall.

I understand that you installed from scratch and then imported the FW config afterwards.

gogol · Jul 24, 2013, 8:09 PM

Ok, for me it is definitely the check for the pid file that messes things up while booting and restarting packages twice as I understand it now. The pid file is written after snort has completely started up and that can take a while. So if the packages are restarted for the second time after a new WAN IP is detected the first snort process is still running without a pid file, so the snort.sh startup script thinks that snort is not running and doesn't stop the first process and just starts a new one. Hence two snort processes.

Is it possible for the snort.sh script to write a dummy pid file that will be replaced after snort has started up? Or another check?

bmeeks · Jul 25, 2013, 2:40 AM

@gogol:

Ok, for me it is definitely the check for the pid file that messes things up while booting and restarting packages twice as I understand it now. The pid file is written after snort has completely started up and that can take a while. So if the packages are restarted for the second time after a new WAN IP is detected the first snort process is still running without a pid file, so the snort.sh startup script thinks that snort is not running and doesn't stop the first process and just starts a new one. Hence two snort processes.

Is it possible for the snort.sh script to write a dummy pid file that will be replaced after snort has started up? Or another check?

Sorry to chime in late. I am working out of town until the weekend and can't look at this until then. I think you may be correct about the timing, but you can't really write a dummy file since that is in the hands of the OS. I have not seen this behavior, and apparently it is rare or I would expect lots of posts here with the same problem. My current theory is perhaps there is something weird in your particular situation.

Bill

dotOne · Jul 25, 2013, 7:12 AM

Unfortunately I have the same issue.
After a reboot snort is always started twice and sometimes even three times.
It's depending on the machine I run it on.

on an Atom D525 based machine it's alway 2 times and sometimes three
on a i5 3.3G machine it's most of the time only a single process.

It looks like a timing issue indeed.

gogol · Jul 25, 2013, 9:09 AM

@bmeeks:

@gogol:

Ok, for me it is definitely the check for the pid file that messes things up while booting and restarting packages twice as I understand it now. The pid file is written after snort has completely started up and that can take a while. So if the packages are restarted for the second time after a new WAN IP is detected the first snort process is still running without a pid file, so the snort.sh startup script thinks that snort is not running and doesn't stop the first process and just starts a new one. Hence two snort processes.

Is it possible for the snort.sh script to write a dummy pid file that will be replaced after snort has started up? Or another check?

Sorry to chime in late. I am working out of town until the weekend and can't look at this until then. I think you may be correct about the timing, but you can't really write a dummy file since that is in the hands of the OS. I have not seen this behavior, and apparently it is rare or I would expect lots of posts here with the same problem. My current theory is perhaps there is something weird in your particular situation.

Hi Bill,

I supposed you were away for some time, because I always get a rapid reply from you.
I have an Atom (N270) based system and a VM on a quad core i5(3,1Ghz) with 2 processors dedicated to the VM. Both have this issue.
This month /etc/rc.newwanip has changed on July 5th and as fas as my understanding of PHP goes I think that this caused it:


46	46	  if($g['booting'])
47	47	    exit;
48	48	  
 	49	 +/* NOTE: Check #2495 before being smart here */
 	50	 +interface_ipalias_cleanup($interface, "inet4");
 	51	 +
49	52	  function restart_packages() {
50	53	    global $oldip, $curwanip, $g;
51	54	  
...	...	 @@ -97,7 +100,6 @@ system_resolvconf_generate(true);
97	100	  /* write current WAN IP to file */
98	101	  file_put_contents("{$g['vardb_path']}/{$interface}_ip", $curwanip);
99	102	  
100	 	 -interface_ipalias_cleanup($interface, "inet4");
101	103	  link_interface_to_vips($interface, "update");
102	104	  
103	105	  unset($gre);

Going through my log files I have the double startup of snort process since a day later (not exactly sure).

eri-- · Jul 25, 2013, 1:09 PM

Yeah that's something i am working on.
Its not the final solution.