Snort 2.9.4.6 Pkg v 2.5.9
-
You have a cron job running??
-
Every time I do a reboot, no upgrade or so, just a reboot, I have 2 processes of Snort running:
root(2): ps uaxwwww | grep snort root 8350 4.5 17.6 629508 364924 ?? Ss 12:24PM 0:53.61 /usr/pbi/snort-i386/bin/snort -R 54477 -E -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0 root 21658 2.3 17.9 634628 370092 ?? SNs 12:24PM 0:03.93 /usr/pbi/snort-i386/bin/snort -R 54477 -D -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0 root 31323 0.0 0.1 3468 1240 0 S+ 12:26PM 0:00.00 grep snortI have to kill both processes from the terminal and restart Snort from the webGUI.
The -E option has something to do with Windows logging. Where does that come from? Does this have something to do how this version was compiled?The process with the -E argument is not being generated by the default package installation. Something else is launching that process. Have you looked in all the startup scripts for your firewall to see if something else is launching a Snort instance?
Bill
-
Every time I do a reboot, no upgrade or so, just a reboot, I have 2 processes of Snort running:
root(2): ps uaxwwww | grep snort root 8350 4.5 17.6 629508 364924 ?? Ss 12:24PM 0:53.61 /usr/pbi/snort-i386/bin/snort -R 54477 -E -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0 root 21658 2.3 17.9 634628 370092 ?? SNs 12:24PM 0:03.93 /usr/pbi/snort-i386/bin/snort -R 54477 -D -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0 root 31323 0.0 0.1 3468 1240 0 S+ 12:26PM 0:00.00 grep snortI have to kill both processes from the terminal and restart Snort from the webGUI.
The -E option has something to do with Windows logging. Where does that come from? Does this have something to do how this version was compiled?The process with the -E argument is not being generated by the default package installation. Something else is launching that process. Have you looked in all the startup scripts for your firewall to see if something else is launching a Snort instance?
Bill
Bill,
I also see this on my VM. During the boot process I sshd into my VM and the snort.sh startup script is also running twice:
ps auxwwww | grep snort root 10800 78.0 17.9 636420 372192 v0- RL 10:15PM 0:15.15 /usr/pbi/snort-i386/bin/snort -R 54477 -D -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0 root 41460 20.0 1.8 240132 36832 ?? RN 10:15PM 0:00.48 /usr/pbi/snort-i386/bin/snort -R 54477 -D -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0 root 40567 17.0 0.1 3644 1576 ?? SN 10:15PM 0:00.00 /bin/sh /usr/local/etc/rc.d/snort.sh start root 10447 0.0 0.1 3644 1500 v0- S 10:15PM 0:00.00 /bin/sh /usr/local/etc/rc.d/snort.sh start root 42050 0.0 0.1 4696 2332 0 RV 10:15PM 0:00.00 grep snort (tcsh)A few moments later:
ps auxwwww | grep snort root 41460 89.0 50.9 2305540 1057752 ?? RN 10:15PM 1:43.48 /usr/pbi/snort-i386/bin/snort -R 54477 -D -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0 root 71694 62.0 4.7 361988 98480 ?? Rs 10:17PM 0:08.92 /usr/pbi/snort-i386/bin/snort -R 54477 -E -q -l /var/log/snort/snort_em054477 --pid-path /var/run --nolock-pidfile -G 54477 -c /usr/pbi/snort-i386/etc/snort/snort_54477_em0/snort.conf -i em0 root 40567 0.0 0.0 3644 0 ?? IWN - 0:00.00 /bin/sh /usr/local/etc/rc.d/snort.sh start root 74561 0.0 0.1 3468 1236 0 S+ 10:17PM 0:00.00 grep snortOne snort.sh script has stopped and the -E argument appears
Another few moments later the other snort.sh script has ended and I am left with two snort processes.
I grepped into my machine with the following string: "/var/log/snort/snort_em054477" and only the snort startup script has this string.
What could be causing the snort startup script running twice? -
I grepped into my machine with the following string: "/var/log/snort/snort_em054477" and only the snort startup script has this string.
What could be causing the snort startup script running twice?I don't know what could be launching the startup script twice. Also not sure how that -E argument is getting on there either.
If you can, completely remove Snort from the firewall via System…Packages and click the X to delete the package. So long as you have the box checked on the Global Settings tab to save the configuration on de-install, then everything will come back OK. After removing Snort, and before installing again, reboot the firewall to see what happens in terms of something else trying to start Snort. If some other thing is attempting to launch it, then it should log some kind of error with the files removed.
Re-install Snort and see if that makes any difference. This is a strange problem. There were some issues a long time ago (as in about a year ago) with multiple instances of Snort getting started on reboot, but Ermal and I are pretty sure those are behind us with changes to the startup scripts. I have been doing a lot of startup/shutdown/reboot operations on my VMs while testing the next Snort update, and I have not noticed this problem.
Bill
-
I grepped into my machine with the following string: "/var/log/snort/snort_em054477" and only the snort startup script has this string.
What could be causing the snort startup script running twice?I don't know what could be launching the startup script twice. Also not sure how that -E argument is getting on there either.
If you can, completely remove Snort from the firewall via System…Packages and click the X to delete the package. So long as you have the box checked on the Global Settings tab to save the configuration on de-install, then everything will come back OK. After removing Snort, and before installing again, reboot the firewall to see what happens in terms of something else trying to start Snort. If some other thing is attempting to launch it, then it should log some kind of error with the files removed.
Re-install Snort and see if that makes any difference. This is a strange problem. There were some issues a long time ago (as in about a year ago) with multiple instances of Snort getting started on reboot, but Ermal and I are pretty sure those are behind us with changes to the startup scripts. I have been doing a lot of startup/shutdown/reboot operations on my VMs while testing the next Snort update, and I have not noticed this problem.
It is becoming even more weird. This what I did.
-
Booted my VM where the two processes of Snort were running after a (re)boot
-
Killed both processes
-
Removed Snort package
-
Rebooted and there was no trace of Snort processes
-
Installed Snort package again and now this:
Jul 21 21:58:12 php: /pkg_mgr_install.php: Beginning package installation for snort . Jul 21 21:59:01 php: /pkg_mgr_install.php: [Snort] Saved settings detected... rebuilding installation with saved settings... Jul 21 21:59:01 php: /pkg_mgr_install.php: [Snort] Downloading and updating configured rule types... Jul 21 21:59:02 php: /pkg_mgr_install.php: [Snort] There is a new set of Snort VRT rules posted. Downloading... Jul 21 21:59:49 php: /pkg_mgr_install.php: [Snort] There is a new set of EmergingThreats rules posted. Downloading... Jul 21 21:59:51 php: /pkg_mgr_install.php: [Snort] EmergingThreats rules file update downloaded successfully Jul 21 21:59:54 php: /pkg_mgr_install.php: [Snort] The Rules update has finished. Jul 21 21:59:54 php: /pkg_mgr_install.php: [Snort] Updating rules configuration for: WAN ... Jul 21 21:59:56 php: /pkg_mgr_install.php: [Snort] Enabling any flowbit-required rules for: WAN... Jul 21 21:59:56 php: /pkg_mgr_install.php: [Snort] Building new sig-msg.map file for WAN... Jul 21 21:59:57 php: /pkg_mgr_install.php: [Snort] Finished rebuilding installation from saved settings... Jul 21 21:59:57 php: /pkg_mgr_install.php: [Snort] Starting Snort using rebuilt configuration... Jul 21 21:59:59 SnortStartup[37422]: Snort START for WAN(54477_em0)... Jul 21 21:59:59 php: /pkg_mgr_install.php: [Snort] Package post-installation tasks completed... Jul 21 21:59:59 check_reload_status: Syncing firewall Jul 21 22:00:01 check_reload_status: Reloading filter Jul 21 22:01:51 snort[22965]: Could not create configuration reload thread.Googling for that error shows that this is in the Snort code.
I rebooted again without starting snort in the WebGUI gives this log:
Jul 21 22:08:17 SnortStartup[11448]: Snort START for WAN(54477_em0)... Jul 21 22:08:19 login: login on ttyv0 as root Jul 21 22:08:19 sshlockout[15602]: sshlockout/webConfigurator v3.0 starting up Jul 21 22:08:24 check_reload_status: Syncing firewall Jul 21 22:08:29 SnortStartup[33999]: Snort START for WAN(54477_em0)... Jul 21 22:10:20 snort[75620]: Could not create configuration reload thread. Jul 21 22:10:36 snort[83412]: Could not create configuration reload thread.Now there is a trace of two Snort processes starting up, but exiting with that error.
Rebooted again without Snort starting from the webGUI (I thought that maybe it would become three processes ;))
But that gave me the same log entries as above.Then I started Snort from the webGUI:
Jul 21 22:19:03 php: /snort/snort_interfaces.php: Toggle (snort starting) for WAN(WAN)... Jul 21 22:19:03 php: /snort/snort_interfaces.php: [Snort] Updating rules configuration for: WAN ... Jul 21 22:19:05 php: /snort/snort_interfaces.php: [Snort] Enabling any flowbit-required rules for: WAN... Jul 21 22:19:05 php: /snort/snort_interfaces.php: [Snort] Building new sig-msg.map file for WAN... Jul 21 22:19:06 php: /snort/snort_interfaces.php: [Snort] Snort START for WAN(em0)... Jul 21 22:20:54 snort[89067]: Could not create configuration reload thread.So now Snort doesn't even start anymore :(
Now I removed Snort without saving the settings and reinstalled the package and configured it again. Rebooted and bam: back where I started with 2 Snort processes and one with the -E argument.
What can I do more?
-
-
Followup:
I installed the latest 2.1 snapshot from scratch in a VM. Logged in the webGUI and reloaded the configuration file and rebooted the system.
It began installing the Snort package and booted up normally. I see Snort running on my interface and in services, but:#ps auxwwww | grep snort root 8542 0.0 70.2 1721044 1458396 ?? Ss 9:30PM 1:48.01 /usr/pbi/snort-i386/bin/snort -R 54137 -E -q -l /var/log/snort/snort_em054137 --pid-path /var/run --nolock-pidfile -G 54137 -c /usr/pbi/snort-i386/etc/snort/snort_54137_em0/snort.conf -i em0 root 87032 0.0 0.1 3468 1240 0 S+ 10:01PM 0:00.00 grep snortAgain with the -E argument
Rebooted and again my problems with two snort.sh startup scripts running. I can't imagine that this is a problem in the <snortglobal>section
Edit: I just saw this in the snort.sh startup script:
#!/bin/sh ######## # This file was automatically generated # by the pfSense service handler. # Code added to protect from double starts on pfSense bootup ######## Begining of Main snort.shCan these double starts have anything to do with my problem?</snortglobal>
-
I am digging deeper and deeper.
An extract from system.log:
Jul 23 10:11:52 pfsensetest php: rc.start_packages: Restarting/Starting all packages. Jul 23 10:11:53 pfsensetest php: rc.newwanip: Resyncing OpenVPN instances for interface WAN. Jul 23 10:11:53 pfsensetest php: rc.newwanip: Creating rrd update script Jul 23 10:11:54 pfsensetest check_reload_status: Syncing firewall Jul 23 10:11:55 pfsensetest php: rc.newwanip: pfSense package system has detected an ip change 192.168.67.161 -> 192.168.67.161 ... Restarting packages. Jul 23 10:11:55 pfsensetest check_reload_status: Starting packages Jul 23 10:11:55 pfsensetest check_reload_status: Reloading filter Jul 23 10:11:57 pfsensetest php: rc.start_packages: Restarting/Starting all packages. Jul 23 10:11:58 pfsensetest SnortStartup[14194]: Snort START for WAN(54137_em0)... Jul 23 10:12:01 pfsensetest login: login on ttyv0 as root Jul 23 10:12:01 pfsensetest sshlockout[32403]: sshlockout/webConfigurator v3.0 starting up Jul 23 10:12:05 pfsensetest check_reload_status: Syncing firewall Jul 23 10:12:08 pfsensetest sshd[38417]: Accepted keyboard-interactive/pam for admin from 192.168.1.3 port 56484 ssh2 Jul 23 10:12:10 pfsensetest SnortStartup[46525]: Snort START for WAN(54137_em0)... Jul 23 10:13:49 pfsensetest kernel: em0: promiscuous mode enabled Jul 23 10:13:53 pfsensetest snort[64616]: SMTP reload: Changing the file_depth requires a restart. Jul 23 10:13:54 pfsensetest kernel: em0: promiscuous mode disabled Jul 23 10:14:05 pfsensetest kernel: em0: promiscuous mode enabledI thought that it might be possible that while a new WAN IP is detected and rc.start_packages restarts the packages Snort has not yet written a pid file and so was not detected already running, a new process would be started. So I disabled the interface that Snort was running on to prevent an automatic startup and manually start Snort.
I was very surprised to see Snort running after a reboot. At the Snort tab I see everything disabled and at Running Services I see Snort running, also when I look at "ps aux | grep snort".I am very confused.
-
Can you post the config file? OR reinstall a 2.1 VM with Snort without importing the config file?
-
Snort has serious timing issues at boot up on my systems.
I tried a lot of combinations with rulesets. It was even possible to start snort without rules, which in my opinion should not be possible.
On my VM with only IPS balanced, which takes about 15 seconds to start on 1 interface, it goes well with only 1 snort process running, but I see the -E argument which is weird.
root 49494 69.0 6.0 387540 124584 ?? Rs 1:11PM 0:04.96 /usr/pbi/snort-i386/bin/snort -R 4082 -E -q -l /var/log/snort/snort_em04082 --pid-path /var/run --nolock-pidfile -G 4082 -c /usr/pbi/snort-i386/etc/snort/snort_4082_em0/snort.conf -i em0With only 1 ET ruleset enabled I can see only 1 process, but with the -D argument.
Another log snippet with only 1 ruleset enabled:
Jul 23 12:59:09 pfsensetest php: rc.start_packages: Restarting/Starting all packages. Jul 23 12:59:10 pfsensetest php: rc.newwanip: Resyncing OpenVPN instances for interface WAN. Jul 23 12:59:10 pfsensetest php: rc.newwanip: Creating rrd update script Jul 23 12:59:11 pfsensetest check_reload_status: Syncing firewall Jul 23 12:59:12 pfsensetest php: rc.newwanip: pfSense package system has detected an ip change 192.168.67.161 -> 192.168.67.161 ... Restarting packages. Jul 23 12:59:12 pfsensetest check_reload_status: Starting packages Jul 23 12:59:12 pfsensetest check_reload_status: Reloading filter Jul 23 12:59:14 pfsensetest php: rc.start_packages: Restarting/Starting all packages. Jul 23 12:59:15 pfsensetest SnortStartup[19390]: Snort START for WAN(4082_em0)... Jul 23 12:59:15 pfsensetest kernel: em0: promiscuous mode enabled Jul 23 12:59:16 pfsensetest login: login on ttyv0 as root Jul 23 12:59:16 pfsensetest sshlockout[31161]: sshlockout/webConfigurator v3.0 starting up Jul 23 12:59:17 pfsensetest SnortStartup[45501]: Snort STOP for WAN(4082_em0)... Jul 23 12:59:18 pfsensetest snort[22786]: *** Caught Term-Signal Jul 23 12:59:18 pfsensetest kernel: em0: promiscuous mode disabled Jul 23 12:59:23 pfsensetest SnortStartup[47764]: Snort START for WAN(4082_em0)... Jul 23 12:59:23 pfsensetest kernel: em0: promiscuous mode enabledWatch the Snort STOP which is missing in another log snippet I posted before.
To resume:
The boot process is interfering with Snort Startup in my opinion or the other way around.
-
rc.newwanip detects an ip change while there isn't one and triggers a restart packages while Snort is starting, which takes a while
-
check_reload_status is also Starting Packages
Sometimes there is the -E argument instead of the -D in the process.
@Supermule I did start from scratch and there is nothing special in my config. Snort in test mode gives no errors.
-
-
Can you pls. make a FRESH install on 2.1 and NOT import the config? And see if it happens again?
-
I understand it is a long story but I reported a clean install of latest 2.1 snapshot and a removal of Snort without saving settings and configuring from scratch of few posts ago ;)
-
Yes but its not the snort imported config but that of the firewall.
I understand that you installed from scratch and then imported the FW config afterwards.
-
Ok, for me it is definitely the check for the pid file that messes things up while booting and restarting packages twice as I understand it now. The pid file is written after snort has completely started up and that can take a while. So if the packages are restarted for the second time after a new WAN IP is detected the first snort process is still running without a pid file, so the snort.sh startup script thinks that snort is not running and doesn't stop the first process and just starts a new one. Hence two snort processes.
Is it possible for the snort.sh script to write a dummy pid file that will be replaced after snort has started up? Or another check?
-
Ok, for me it is definitely the check for the pid file that messes things up while booting and restarting packages twice as I understand it now. The pid file is written after snort has completely started up and that can take a while. So if the packages are restarted for the second time after a new WAN IP is detected the first snort process is still running without a pid file, so the snort.sh startup script thinks that snort is not running and doesn't stop the first process and just starts a new one. Hence two snort processes.
Is it possible for the snort.sh script to write a dummy pid file that will be replaced after snort has started up? Or another check?
Sorry to chime in late. I am working out of town until the weekend and can't look at this until then. I think you may be correct about the timing, but you can't really write a dummy file since that is in the hands of the OS. I have not seen this behavior, and apparently it is rare or I would expect lots of posts here with the same problem. My current theory is perhaps there is something weird in your particular situation.
Bill
-
Unfortunately I have the same issue.
After a reboot snort is always started twice and sometimes even three times.
It's depending on the machine I run it on.on an Atom D525 based machine it's alway 2 times and sometimes three
on a i5 3.3G machine it's most of the time only a single process.It looks like a timing issue indeed.
-
Ok, for me it is definitely the check for the pid file that messes things up while booting and restarting packages twice as I understand it now. The pid file is written after snort has completely started up and that can take a while. So if the packages are restarted for the second time after a new WAN IP is detected the first snort process is still running without a pid file, so the snort.sh startup script thinks that snort is not running and doesn't stop the first process and just starts a new one. Hence two snort processes.
Is it possible for the snort.sh script to write a dummy pid file that will be replaced after snort has started up? Or another check?
Sorry to chime in late. I am working out of town until the weekend and can't look at this until then. I think you may be correct about the timing, but you can't really write a dummy file since that is in the hands of the OS. I have not seen this behavior, and apparently it is rare or I would expect lots of posts here with the same problem. My current theory is perhaps there is something weird in your particular situation.
Hi Bill,
I supposed you were away for some time, because I always get a rapid reply from you.
I have an Atom (N270) based system and a VM on a quad core i5(3,1Ghz) with 2 processors dedicated to the VM. Both have this issue.
This month /etc/rc.newwanip has changed on July 5th and as fas as my understanding of PHP goes I think that this caused it:46 46 if($g['booting']) 47 47 exit; 48 48 49 +/* NOTE: Check #2495 before being smart here */ 50 +interface_ipalias_cleanup($interface, "inet4"); 51 + 49 52 function restart_packages() { 50 53 global $oldip, $curwanip, $g; 51 54 ... ... @@ -97,7 +100,6 @@ system_resolvconf_generate(true); 97 100 /* write current WAN IP to file */ 98 101 file_put_contents("{$g['vardb_path']}/{$interface}_ip", $curwanip); 99 102 100 -interface_ipalias_cleanup($interface, "inet4"); 101 103 link_interface_to_vips($interface, "update"); 102 104 103 105 unset($gre);Going through my log files I have the double startup of snort process since a day later (not exactly sure).
-
Yeah that's something i am working on.
Its not the final solution. -
It is good to read that this is acknowledged. Thanks. In the meantime I kill the snort processes after boot and restart them again manually.
-
It is good to read that this is acknowledged. Thanks. In the meantime I kill the snort processes after boot and restart them again manually.
I am back home and read the response from Ermal. If the problem started on July 5 and seems to coincide with a change being made in the pfSense core, I will hold off making any Snort adjustments.
Bill
-
I repro multiple snort processes on the latest 2.1 builds as well. What additional information can I provide to help debug this issue and get a solution? Thanks!
