Snort 2.9.4.1 pkg v. 2.5.5 Issue(s)

[Cino]

@bmeeks:

@Cino:

Anyone else having this issue? IPs are removed from the block list after 5 minutes when the cron job is run. I've check the snort2c table and they aren't there anymore. Any ideas?

I have not timed mine, but I think the blocks are lasting for an hour.  That's what I have mine set for.  How is the time on your firewall synchronized?  Does it have a NTP source to sync with, and is it holding the correct time?

Bill

It syncs with a local NTP server in my time zone and if its not available then it syncs with couple from pool.ntp.org. The time is holding as far as I can tell…

[AhnHEL]

@bmeeks:

Also take a look in the /conf/config.xml file in the [cron] section to see what is listed there (and how many times).

Looks normal enough, no duplicates.

 <cron><minute>0</minute>
			<hour>*</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/nice -n20 newsyslog 
		 <minute>1,31</minute>
			<hour>0-5</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/nice -n20 adjkerntz -a 
		 <minute>1</minute>
			<hour>3</hour>
			<mday>1</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/nice -n20 /etc/rc.update_bogons.sh 
		 <minute>*/60</minute>
			<hour>*</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 sshlockout 
		 <minute>1</minute>
			<hour>1</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/nice -n20 /etc/rc.dyndns.update 
		 <minute>*/60</minute>
			<hour>*</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot 
		 <minute>30</minute>
			<hour>12</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/nice -n20 /etc/rc.update_urltables 
		 <minute>*/5</minute>
			<hour>*</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_cron_misc.inc 
		 <minute>*/5</minute>
			<hour>*</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/nice -n20 /usr/local/sbin/expiretable -t 3600 snort2c 
		 <minute>3</minute>
			<hour>0</hour>
			<mday>*/1</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_for_rule_updates.php >> /tmp/snort_update.log</cron> 

I'll report back what I see tonight when it updates again just after midnight

[Gradius]

@bmeeks:

You seem to have something weird going on in that install.  You are clicking the "X" icon to completely remove the Snort package on the Installed Packages tab, and then going to the Available Packages tab and installing it again, correct?

That's correct.

@bmeeks:

That error you are posting indicates an incomplete uninstall/reinstall process.  Those files (with -example- in the filenames) are fixed up by the full package installation process.  The fact you keep seeing this error means either that process is not happening, or is not running to conclusion.

One thing to try – click the "X" to totally remove Snort.  Then go to the command line and issue this command to completely remove any remaining traces of Snort:

rm -rf /usr/pbi/snort-i386

Then go to the Available Packages tab and install it fresh.

I'll do manual uninstall, but is really weird this is happening, since I did that a week or two ago (manually uninstalled).

[Gradius]

This is all I have after uninstall by X:

/usr/local/etc/snort/snort_59419_pppoe0/rules/snort.rules
/usr/local/etc/snort/snort_59419_pppoe0/snort.conf
/usr/local/etc/snort/snort.conf
/usr/local/etc/snort/rules/snort_attack-responses.rules
/usr/local/etc/snort/rules/snort_backdoor.rules
/usr/local/etc/snort/rules/snort_bad-traffic.rules
/usr/local/etc/snort/rules/snort_blacklist.rules
/usr/local/etc/snort/rules/snort_botnet-cnc.rules
/usr/local/etc/snort/rules/snort_chat.rules
/usr/local/etc/snort/rules/snort_content-replace.rules
/usr/local/etc/snort/rules/snort_ddos.rules
/usr/local/etc/snort/rules/snort_deleted.rules
/usr/local/etc/snort/rules/snort_dns.rules
/usr/local/etc/snort/rules/snort_dos.rules
/usr/local/etc/snort/rules/snort_experimental.rules
/usr/local/etc/snort/rules/snort_exploit.rules
/usr/local/etc/snort/rules/snort_file-identify.rules
/usr/local/etc/snort/rules/snort_file-office.rules
/usr/local/etc/snort/rules/snort_file-other.rules
/usr/local/etc/snort/rules/snort_file-pdf.rules
/usr/local/etc/snort/rules/snort_finger.rules
/usr/local/etc/snort/rules/snort_ftp.rules
/usr/local/etc/snort/rules/snort_icmp-info.rules
/usr/local/etc/snort/rules/snort_icmp.rules
/usr/local/etc/snort/rules/snort_imap.rules
/usr/local/etc/snort/rules/snort_indicator-compromise.rules
/usr/local/etc/snort/rules/snort_indicator-obfuscation.rules
/usr/local/etc/snort/rules/snort_info.rules
/usr/local/etc/snort/rules/snort_local.rules
/usr/local/etc/snort/rules/snort_misc.rules
/usr/local/etc/snort/rules/snort_multimedia.rules
/usr/local/etc/snort/rules/snort_mysql.rules
/usr/local/etc/snort/rules/snort_netbios.rules
/usr/local/etc/snort/rules/snort_nntp.rules
/usr/local/etc/snort/rules/snort_oracle.rules
/usr/local/etc/snort/rules/snort_other-ids.rules
/usr/local/etc/snort/rules/snort_p2p.rules
/usr/local/etc/snort/rules/snort_phishing-spam.rules
/usr/local/etc/snort/rules/snort_policy-multimedia.rules
/usr/local/etc/snort/rules/snort_policy-other.rules
/usr/local/etc/snort/rules/snort_policy-social.rules
/usr/local/etc/snort/rules/snort_policy.rules
/usr/local/etc/snort/rules/snort_pop2.rules
/usr/local/etc/snort/rules/snort_pop3.rules
/usr/local/etc/snort/rules/snort_pua-p2p.rules
/usr/local/etc/snort/rules/snort_pua-toolbars.rules
/usr/local/etc/snort/rules/snort_rpc.rules
/usr/local/etc/snort/rules/snort_rservices.rules
/usr/local/etc/snort/rules/snort_scada.rules
/usr/local/etc/snort/rules/snort_scan.rules
/usr/local/etc/snort/rules/snort_server-mail.rules
/usr/local/etc/snort/rules/snort_shellcode.rules
/usr/local/etc/snort/rules/snort_smtp.rules
/usr/local/etc/snort/rules/snort_snmp.rules
/usr/local/etc/snort/rules/snort_specific-threats.rules
/usr/local/etc/snort/rules/snort_spyware-put.rules
/usr/local/etc/snort/rules/snort_sql.rules
/usr/local/etc/snort/rules/snort_telnet.rules
/usr/local/etc/snort/rules/snort_tftp.rules
/usr/local/etc/snort/rules/snort_virus.rules
/usr/local/etc/snort/rules/snort_voip.rules
/usr/local/etc/snort/rules/snort_web-activex.rules
/usr/local/etc/snort/rules/snort_web-attacks.rules
/usr/local/etc/snort/rules/snort_web-cgi.rules
/usr/local/etc/snort/rules/snort_web-client.rules
/usr/local/etc/snort/rules/snort_web-coldfusion.rules
/usr/local/etc/snort/rules/snort_x11.rules
/usr/local/etc/snort/rules/snort_web-frontpage.rules
/usr/local/etc/snort/rules/snort_web-iis.rules
/usr/local/etc/snort/rules/snort_web-misc.rules
/usr/local/etc/snort/rules/snort_web-php.rules
/usr/local/etc/snort/rules/snort_bad-traffic.so.rules
/usr/local/etc/snort/rules/snort_chat.so.rules
/usr/local/etc/snort/rules/snort_dos.so.rules
/usr/local/etc/snort/rules/snort_exploit.so.rules
/usr/local/etc/snort/rules/snort_icmp.so.rules
/usr/local/etc/snort/rules/snort_imap.so.rules
/usr/local/etc/snort/rules/snort_misc.so.rules
/usr/local/etc/snort/rules/snort_multimedia.so.rules
/usr/local/etc/snort/rules/snort_netbios.so.rules
/usr/local/etc/snort/rules/snort_nntp.so.rules
/usr/local/etc/snort/rules/snort_p2p.so.rules
/usr/local/etc/snort/rules/snort_smtp.so.rules
/usr/local/etc/snort/rules/snort_snmp.so.rules
/usr/local/etc/snort/rules/snort_specific-threats.so.rules
/usr/local/etc/snort/rules/snort_web-activex.so.rules
/usr/local/etc/snort/rules/snort_web-client.so.rules
/usr/local/etc/snort/rules/snort_web-iis.so.rules
/usr/local/etc/snort/rules/snort_web-misc.so.rules
/usr/local/etc/snort/rules/snort-2.9.0-open.txt
/usr/local/etc/snort/rules/snort_app-detect.rules
/usr/local/etc/snort/rules/snort_browser-chrome.rules
/usr/local/etc/snort/rules/snort_browser-firefox.rules
/usr/local/etc/snort/rules/snort_browser-ie.rules
/usr/local/etc/snort/rules/snort_browser-other.rules
/usr/local/etc/snort/rules/snort_browser-webkit.rules
/usr/local/etc/snort/rules/snort_exploit-kit.rules
/usr/local/etc/snort/rules/snort_file-executable.rules
/usr/local/etc/snort/rules/snort_file-flash.rules
/usr/local/etc/snort/rules/snort_file-image.rules
/usr/local/etc/snort/rules/snort_file-multimedia.rules
/usr/local/etc/snort/rules/snort_malware-backdoor.rules
/usr/local/etc/snort/rules/snort_malware-cnc.rules
/usr/local/etc/snort/rules/snort_malware-other.rules
/usr/local/etc/snort/rules/snort_malware-tools.rules
/usr/local/etc/snort/rules/snort_browser-plugins.rules
/usr/local/etc/snort/rules/snort_indicator-shellcode.rules
/usr/local/etc/snort/rules/snort_os-linux.rules
/usr/local/etc/snort/rules/snort_os-other.rules
/usr/local/etc/snort/rules/snort_os-solaris.rules
/usr/local/etc/snort/rules/snort_os-windows.rules
/usr/local/etc/snort/rules/snort_policy-spam.rules
/usr/local/etc/snort/rules/snort_protocol-finger.rules
/usr/local/etc/snort/rules/snort_protocol-ftp.rules
/usr/local/etc/snort/rules/snort_protocol-icmp.rules
/usr/local/etc/snort/rules/snort_protocol-imap.rules
/usr/local/etc/snort/rules/snort_protocol-pop.rules
/usr/local/etc/snort/rules/snort_protocol-services.rules
/usr/local/etc/snort/rules/snort_protocol-voip.rules
/usr/local/etc/snort/rules/snort_pua-adware.rules
/usr/local/etc/snort/rules/snort_pua-other.rules
/usr/local/etc/snort/rules/snort_server-apache.rules
/usr/local/etc/snort/rules/snort_server-iis.rules
/usr/local/etc/snort/rules/snort_server-mssql.rules
/usr/local/etc/snort/rules/snort_server-mysql.rules
/usr/local/etc/snort/rules/snort_server-oracle.rules
/usr/local/etc/snort/rules/snort_server-other.rules
/usr/local/etc/snort/rules/snort_server-webapp.rules
/usr/local/etc/snort/rules/snort-2.9.0-open-nogpl.txt

[bmeeks]

@Gradius:

This is all I have after uninstall by X:

/usr/local/etc/snort/snort_59419_pppoe0/rules/snort.rules
/usr/local/etc/snort/snort_59419_pppoe0/snort.conf
/usr/local/etc/snort/snort.conf
/usr/local/etc/snort/rules/snort_attack-responses.rules
/usr/local/etc/snort/rules/snort_backdoor.rules
/usr/local/etc/snort/rules/snort_bad-traffic.rules
/usr/local/etc/snort/rules/snort_blacklist.rules
/usr/local/etc/snort/rules/snort_botnet-cnc.rules
/usr/local/etc/snort/rules/snort_chat.rules
/usr/local/etc/snort/rules/snort_content-replace.rules
/usr/local/etc/snort/rules/snort_ddos.rules
/usr/local/etc/snort/rules/snort_deleted.rules
/usr/local/etc/snort/rules/snort_dns.rules
/usr/local/etc/snort/rules/snort_dos.rules
/usr/local/etc/snort/rules/snort_experimental.rules
/usr/local/etc/snort/rules/snort_exploit.rules
/usr/local/etc/snort/rules/snort_file-identify.rules
/usr/local/etc/snort/rules/snort_file-office.rules
/usr/local/etc/snort/rules/snort_file-other.rules
/usr/local/etc/snort/rules/snort_file-pdf.rules
/usr/local/etc/snort/rules/snort_finger.rules
/usr/local/etc/snort/rules/snort_ftp.rules
/usr/local/etc/snort/rules/snort_icmp-info.rules
/usr/local/etc/snort/rules/snort_icmp.rules
/usr/local/etc/snort/rules/snort_imap.rules
/usr/local/etc/snort/rules/snort_indicator-compromise.rules
/usr/local/etc/snort/rules/snort_indicator-obfuscation.rules
/usr/local/etc/snort/rules/snort_info.rules
/usr/local/etc/snort/rules/snort_local.rules
/usr/local/etc/snort/rules/snort_misc.rules
/usr/local/etc/snort/rules/snort_multimedia.rules
/usr/local/etc/snort/rules/snort_mysql.rules
/usr/local/etc/snort/rules/snort_netbios.rules
/usr/local/etc/snort/rules/snort_nntp.rules
/usr/local/etc/snort/rules/snort_oracle.rules
/usr/local/etc/snort/rules/snort_other-ids.rules
/usr/local/etc/snort/rules/snort_p2p.rules
/usr/local/etc/snort/rules/snort_phishing-spam.rules
/usr/local/etc/snort/rules/snort_policy-multimedia.rules
/usr/local/etc/snort/rules/snort_policy-other.rules
/usr/local/etc/snort/rules/snort_policy-social.rules
/usr/local/etc/snort/rules/snort_policy.rules
/usr/local/etc/snort/rules/snort_pop2.rules
/usr/local/etc/snort/rules/snort_pop3.rules
/usr/local/etc/snort/rules/snort_pua-p2p.rules
/usr/local/etc/snort/rules/snort_pua-toolbars.rules
/usr/local/etc/snort/rules/snort_rpc.rules
/usr/local/etc/snort/rules/snort_rservices.rules
/usr/local/etc/snort/rules/snort_scada.rules
/usr/local/etc/snort/rules/snort_scan.rules
/usr/local/etc/snort/rules/snort_server-mail.rules
/usr/local/etc/snort/rules/snort_shellcode.rules
/usr/local/etc/snort/rules/snort_smtp.rules
/usr/local/etc/snort/rules/snort_snmp.rules
/usr/local/etc/snort/rules/snort_specific-threats.rules
/usr/local/etc/snort/rules/snort_spyware-put.rules
/usr/local/etc/snort/rules/snort_sql.rules
/usr/local/etc/snort/rules/snort_telnet.rules
/usr/local/etc/snort/rules/snort_tftp.rules
/usr/local/etc/snort/rules/snort_virus.rules
/usr/local/etc/snort/rules/snort_voip.rules
/usr/local/etc/snort/rules/snort_web-activex.rules
/usr/local/etc/snort/rules/snort_web-attacks.rules
/usr/local/etc/snort/rules/snort_web-cgi.rules
/usr/local/etc/snort/rules/snort_web-client.rules
/usr/local/etc/snort/rules/snort_web-coldfusion.rules
/usr/local/etc/snort/rules/snort_x11.rules
/usr/local/etc/snort/rules/snort_web-frontpage.rules
/usr/local/etc/snort/rules/snort_web-iis.rules
/usr/local/etc/snort/rules/snort_web-misc.rules
/usr/local/etc/snort/rules/snort_web-php.rules
/usr/local/etc/snort/rules/snort_bad-traffic.so.rules
/usr/local/etc/snort/rules/snort_chat.so.rules
/usr/local/etc/snort/rules/snort_dos.so.rules
/usr/local/etc/snort/rules/snort_exploit.so.rules
/usr/local/etc/snort/rules/snort_icmp.so.rules
/usr/local/etc/snort/rules/snort_imap.so.rules
/usr/local/etc/snort/rules/snort_misc.so.rules
/usr/local/etc/snort/rules/snort_multimedia.so.rules
/usr/local/etc/snort/rules/snort_netbios.so.rules
/usr/local/etc/snort/rules/snort_nntp.so.rules
/usr/local/etc/snort/rules/snort_p2p.so.rules
/usr/local/etc/snort/rules/snort_smtp.so.rules
/usr/local/etc/snort/rules/snort_snmp.so.rules
/usr/local/etc/snort/rules/snort_specific-threats.so.rules
/usr/local/etc/snort/rules/snort_web-activex.so.rules
/usr/local/etc/snort/rules/snort_web-client.so.rules
/usr/local/etc/snort/rules/snort_web-iis.so.rules
/usr/local/etc/snort/rules/snort_web-misc.so.rules
/usr/local/etc/snort/rules/snort-2.9.0-open.txt
/usr/local/etc/snort/rules/snort_app-detect.rules
/usr/local/etc/snort/rules/snort_browser-chrome.rules
/usr/local/etc/snort/rules/snort_browser-firefox.rules
/usr/local/etc/snort/rules/snort_browser-ie.rules
/usr/local/etc/snort/rules/snort_browser-other.rules
/usr/local/etc/snort/rules/snort_browser-webkit.rules
/usr/local/etc/snort/rules/snort_exploit-kit.rules
/usr/local/etc/snort/rules/snort_file-executable.rules
/usr/local/etc/snort/rules/snort_file-flash.rules
/usr/local/etc/snort/rules/snort_file-image.rules
/usr/local/etc/snort/rules/snort_file-multimedia.rules
/usr/local/etc/snort/rules/snort_malware-backdoor.rules
/usr/local/etc/snort/rules/snort_malware-cnc.rules
/usr/local/etc/snort/rules/snort_malware-other.rules
/usr/local/etc/snort/rules/snort_malware-tools.rules
/usr/local/etc/snort/rules/snort_browser-plugins.rules
/usr/local/etc/snort/rules/snort_indicator-shellcode.rules
/usr/local/etc/snort/rules/snort_os-linux.rules
/usr/local/etc/snort/rules/snort_os-other.rules
/usr/local/etc/snort/rules/snort_os-solaris.rules
/usr/local/etc/snort/rules/snort_os-windows.rules
/usr/local/etc/snort/rules/snort_policy-spam.rules
/usr/local/etc/snort/rules/snort_protocol-finger.rules
/usr/local/etc/snort/rules/snort_protocol-ftp.rules
/usr/local/etc/snort/rules/snort_protocol-icmp.rules
/usr/local/etc/snort/rules/snort_protocol-imap.rules
/usr/local/etc/snort/rules/snort_protocol-pop.rules
/usr/local/etc/snort/rules/snort_protocol-services.rules
/usr/local/etc/snort/rules/snort_protocol-voip.rules
/usr/local/etc/snort/rules/snort_pua-adware.rules
/usr/local/etc/snort/rules/snort_pua-other.rules
/usr/local/etc/snort/rules/snort_server-apache.rules
/usr/local/etc/snort/rules/snort_server-iis.rules
/usr/local/etc/snort/rules/snort_server-mssql.rules
/usr/local/etc/snort/rules/snort_server-mysql.rules
/usr/local/etc/snort/rules/snort_server-oracle.rules
/usr/local/etc/snort/rules/snort_server-other.rules
/usr/local/etc/snort/rules/snort_server-webapp.rules
/usr/local/etc/snort/rules/snort-2.9.0-open-nogpl.txt

None of that should be there after a Delete.  Get to the command line (either locally or via SSH) and execute the following commands –

rm -rf /usr/local/etc/snort
rm -rf /usr/local/lib/snort

If you can, reboot the firewall after the commands above to be 100% sure no other Snort process is out there running.  If you are confident no Snort processes remain, you can skip the reboot.

Reinstall Snort again and let's see if things work better.

Bill

[fragged]

I have Snort set to never remove blocked IP's, but they still get removed and not all IP's from alerts are put on blocked at all. It almost seems like Snort stops working after running for a while. The process is still running and alerts are made, but nothing is blocked and IP's already blocked are removed. Last time I noticed this was just now at ~21:50 and Snort was last restarted at 12:00 after rules update.

I'm on pfSense:

2.1-BETA1 (amd64)
built on Thu Apr 4 12:39:50 EDT 2013
FreeBSD 8.3-RELEASE-p7

[bmeeks]

@fragged:

I have Snort set to never remove blocked IP's, but they still get removed and not all IP's from alerts are put on blocked at all. It almost seems like Snort stops working after running for a while. The process is still running and alerts are made, but nothing is blocked and IP's already blocked are removed. Last time I noticed this was just now at ~21:50 and Snort was last restarted at 12:00 after rules update.

I'm on pfSense:

2.1-BETA1 (amd64)
built on Thu Apr 4 12:39:50 EDT 2013
FreeBSD 8.3-RELEASE-p7

Are all the IPs you think should be blocked, but aren't, located outside HOME_NET?  In other words, are they all IP addresses that are NOT in your automatic whitelist (the $HOME_NET variable).

Second question is do you have Snort on that interface set to block SRC, DST or BOTH?  That setting can affect what you see getting blocked.

I don't have a good answer yet for the IPs getting removed from the block table if you have the automatic removal disabled.  Look in the file /etc/crontab to see if there is a line that reads similar to this one:

/usr/bin/nice -n20 /usr/local/sbin/expiretable -t {# of seconds} snort2c

If you see such a line, then it means the cron job did not get cleaned up/deleted.

Bill

[fragged]

I'm not sure if this is by design or not, but applying firewall rules (filter reload?) will erase Snort blocked list.

[bmeeks]

@fragged:

I'm not sure if this is by design or not, but applying firewall rules (filter reload?) will erase Snort blocked list.

That does make sense.  I had not thought of that.  Snort actually uses some of the same pf tables to perform its inline blocking (using a third-party output plugin for Snort called Spoink).  So when the firewall rules refresh, I would assume the tables are cleared.

Once the Spoink plugin writes the IP address from an alert into the table, it does not check on it anymore.  It's a "post and forget" type of operation from Snort's point of view.  There is a completely separate cron task that fires based on your configured time, and this task is the one that actually clears the pf table where the blocks were inserted.  The Spoink plugin only fires on the initial alert from Snort.  It does not do any kind of "state keeping" whereby it periodically scans the Snort log or anything like that.  It just intercepts that initial alert, then posts the required IP address in the pf table for blocking.

[Supermule]

Snort was killed again after rules update!

Apr 18 00:05:44 php: : The Rules update has finished.
Apr 18 00:05:44 php: : Snort has restarted with your new set of rules…
Apr 18 00:05:42 SnortStartup[800]: Snort SOFT START For Internet(36256_em0)…
Apr 18 00:05:42 snort[27180]: Could not remove pid file /var/run/snort_em036256.pid: No such file or directory
Apr 18 00:05:42 snort[27180]: Could not remove pid file /var/run/snort_em036256.pid: No such file or directory
Apr 18 00:05:41 kernel: em0: promiscuous mode disabled
Apr 18 00:05:41 snort[27180]: *** Caught Term-Signal
Apr 18 00:05:41 snort[27180]: *** Caught Term-Signal
Apr 18 00:05:40 SnortStartup[63434]: Snort STOP For Internet(36256_em0)…
Apr 18 00:05:40 php: : Building new sig-msg.map file for WAN...
Apr 18 00:05:38 php: : Resolving and auto-enabling any flowbit-required rules for WAN...
Apr 18 00:05:36 php: : Updating rules configuration for: WAN ...
Apr 18 00:05:36 php: : EmergingThreats rules file update downloaded succsesfully
Apr 18 00:05:33 php: : There is a new set of EmergingThreats rules posted. Downloading...
Apr 18 00:05:33 php: : Snort GPLv2 Community Rules file update downloaded succsesfully
Apr 18 00:05:31 php: : There is a new set of Snort GPLv2 Community Rules posted. Downloading...
Apr 18 00:05:30 php: : Snort VRT rules are up to date...
Apr 18 00:05:30 php: : Snort MD5 Attempts: 2
Apr 17 00:03:07 php: : The Rules update has finished.
Apr 17 00:03:07 php: : Emerging Threat rules are up to date...
Apr 17 00:03:07 php: : Snort GPLv2 Community Rules are up to date...
Apr 17 00:03:06 php: : Snort VRT rules are up to date...
Apr 17 00:03:06 php: : Snort MD5 Attempts: 1

And then manually started

pr 18 12:00:44 php: /status_services.php: The command '/usr/local/etc/rc.d/snort.sh stop' returned exit code '1', the output was ''
Apr 18 12:00:42 SnortStartup[51714]: Snort STOP For Internet(36256_em0)…

Another FW got killed. Same error.

Apr 18 00:03:41 php: : The Rules update has finished.
Apr 18 00:03:41 php: : Snort has restarted with your new set of rules...
Apr 18 00:03:39 SnortStartup[4528]: Snort SOFT START For Internet(9626_em0)…
Apr 18 00:03:39 snort[54662]: Could not remove pid file /var/run/snort_em09626.pid: No such file or directory
Apr 18 00:03:39 snort[54662]: Could not remove pid file /var/run/snort_em09626.pid: No such file or directory
Apr 18 00:03:38 kernel: em0: promiscuous mode disabled
Apr 18 00:03:38 snort[54662]: *** Caught Term-Signal
Apr 18 00:03:38 snort[54662]: *** Caught Term-Signal
Apr 18 00:03:37 SnortStartup[3264]: Snort STOP For Internet(9626_em0)…
Apr 18 00:03:37 php: : Building new sig-msg.map file for WAN...
Apr 18 00:03:35 php: : Resolving and auto-enabling any flowbit-required rules for WAN...
Apr 18 00:03:33 php: : Updating rules configuration for: WAN ...
Apr 18 00:03:32 php: : EmergingThreats rules file update downloaded succsesfully
Apr 18 00:03:29 php: : There is a new set of EmergingThreats rules posted. Downloading...
Apr 18 00:03:29 php: : Snort GPLv2 Community Rules file update downloaded succsesfully
Apr 18 00:03:27 php: : There is a new set of Snort GPLv2 Community Rules posted. Downloading...
Apr 18 00:03:26 php: : Snort VRT rules are up to date...
Apr 18 00:03:26 php: : Snort MD5 Attempts: 1

[gogol]

Confirmed:

Apr 17 12:06:12	php: : The Rules update has finished.
Apr 17 12:06:12	php: : Snort has restarted with your new set of rules...Apr 17 12:06:10	kernel: em0: promiscuous mode enabled
Apr 17 12:06:10	SnortStartup[65474]: Snort START For WAN(54477_em0)...
Apr 17 12:05:40	snort[71072]: Could not remove pid file /var/run/snort_em054477.pid: No such file or directory
Apr 17 12:05:40	kernel: em0: promiscuous mode disabled
Apr 17 12:05:40	snort[71072]: *** Caught Term-Signal
Apr 17 12:05:39	SnortStartup[63404]: Snort STOP For WAN(54477_em0)...
Apr 17 12:05:38	php: : Building new sig-msg.map file for WAN...
Apr 17 12:05:30	php: : Resolving and auto-enabling any flowbit-required rules for WAN...
Apr 17 12:05:22	php: : Updating rules configuration for: WAN ...
Apr 17 12:05:21	php: : EmergingThreats rules file update downloaded succsesfully
Apr 17 12:05:12	php: : There is a new set of EmergingThreats rules posted. Downloading...
Apr 17 12:05:12	php: : Snort VRT rules are up to date...
Apr 17 12:05:12	php: : Snort MD5 Attempts: 2

This one looks OK, but the next one does not:

Apr 18 12:07:19	php: : The Rules update has finished.
Apr 18 12:07:18	php: : Building new sig-msg.map file for WAN...
Apr 18 12:07:09	php: : Resolving and auto-enabling any flowbit-required rules for WAN...
Apr 18 12:07:02	php: : Updating rules configuration for: WAN ...
Apr 18 12:07:01	kernel: em0: promiscuous mode disabled
Apr 18 12:07:01	kernel: pid 65246 (snort), uid 0: exited on signal 4
Apr 18 12:06:52	php: : EmergingThreats rules file update downloaded succsesfully
Apr 18 12:06:50	php: : There is a new set of EmergingThreats rules posted. Downloading...
Apr 18 12:06:49	php: : Snort Rules Attempts: 1
Apr 18 12:06:10	php: : There is a new set of Snort VRT rules posted. Downloading...
Apr 18 12:06:10	php: : Snort MD5 Attempts: 3

It happens when there is a new Snort ruleset, not when the ET rules are updated.

[bmeeks]

Supermule and gogol:

Thank you for your detailed reports with the logs.  I think I spotted a clue.  I will need to investigate it a bit, but the clue I see is in Supermule's log entries.  Notice it says "Snort SOFT START".  That is not what I expected.  The rules update procedure asks for a complete shutdown and restart of Snort by calling the /usr/local/etc/rc.d/snort.sh script with the restart argument.  In the shell script, that causes Snort to be stopped and then restarted.  The STOP part is happening, but the restart is getting hosed, and I need to see why.

The logic in the shell script tries to be smart on starting Snort.  If it sees the process is already running, it issues a SIGHUP command to the running process signalling Snort to reload its rules and other configuration settings.  If there is no running Snort process detected, it just cold-starts Snort.

So here is what I think is going on.  The STOP signal is caught by the running Snort process and it begins the shutdown process.  However, the START command is issued before STOP has completetly finished cleaning up the PID files.  So START sees the not-yet-cleaned up PID file and thinks Snort is still running.  Thinking there is a running Snort process, it just issues the SIGHUP soft restart instead of the hard START.  That clue comes from the "SOFT START" part of the log entry.  That line only gets printed when the script tries the SIGHUP approach.

I need to test out my theory in my VM world a bit, and this may take me a few days.  In the interim, try the fix I posted earlier of adding a sleep 3 command between the rc_stop() and rc_start() calls in the shell script.

Bill

[Supermule]

Thanks Bill!

Looking forward to see what you come up with.

[bmeeks]

@Supermule:

Thanks Bill!

Looking forward to see what you come up with.

If you want to try a quick experiment, follow the instructions here:  http://forum.pfsense.org/index.php/topic,61001.msg330160.html#msg330160

Try fiddling with the sleep time.  Maybe try a little longer.  If my theory is correct, it's a timing issue.

Bill

[gogol]

Ok, I added the sleep 3 timer and will wait and see.
Snort is not very consistent with restarting, because I also have a SOFT RESTART in my logs:

Apr 16 12:04:13	php: : The Rules update has finished.
Apr 16 12:04:13	php: : Snort has restarted with your new set of rules...
Apr 16 12:04:11	SnortStartup[77296]: Snort SOFT START For WAN(54477_em0)...
Apr 16 12:04:10	snort[38532]: Could not remove pid file /var/run/snort_em054477.pid: No such file or directory
Apr 16 12:04:09	snort[38532]: *** Caught Term-Signal
Apr 16 12:04:08	SnortStartup[75228]: Snort STOP For WAN(54477_em0)...
Apr 16 12:04:08	php: : Building new sig-msg.map file for WAN...
Apr 16 12:03:59	php: : Resolving and auto-enabling any flowbit-required rules for WAN...
Apr 16 12:03:52	php: : Updating rules configuration for: WAN ...
Apr 16 12:03:51	php: : EmergingThreats rules file update downloaded succsesfully
Apr 16 12:03:48	php: : There is a new set of EmergingThreats rules posted. Downloading...
Apr 16 12:03:48	php: : Snort VRT rules are up to date...
Apr 16 12:03:48	php: : Snort MD5 Attempts: 1

I have an N270 Atom processor and I know it takes some time for Snort to startup, so maybe you are right that it might be a timing issue for slower processors.
Thanks again.

[Supermule]

I have done that. Waiting update but restarte Snort since I edited the file.

Apr 18 14:45:17 php: /status_services.php: The command '/usr/local/etc/rc.d/snort.sh stop' returned exit code '1', the output was ''
Apr 18 14:45:15 SnortStartup[52848]: Snort STOP For Internet(36256_em0)…
Apr 18 14:45:15 php: /status_services.php: The command '/usr/local/etc/rc.d/snort.sh stop' returned exit code '1', the output was ''
Apr 18 14:45:14 snort[43724]: Could not remove pid file /var/run/snort_em036256.pid: No such file or directory
Apr 18 14:45:14 snort[43724]: Could not remove pid file /var/run/snort_em036256.pid: No such file or directory
Apr 18 14:45:14 kernel: em0: promiscuous mode disabled
Apr 18 14:45:14 snort[43724]: *** Caught Term-Signal
Apr 18 14:45:14 snort[43724]: *** Caught Term-Signal
Apr 18 14:45:13 SnortStartup[51967]: Snort STOP For Internet(36256_em0)…

Done it on only one of the FW's so that I can compare.

[bmeeks]

@Supermule:

I have done that. Waiting update but restarte Snort since I edited the file.

….

Done it on only one of the FW's so that I can compare.

How about making this change instead of simply adding the sleep command ??

In the same file, find this section:

case $1 in
	start)
		rc_start
		;;
	stop)
		rc_stop
		;;
	restart)
		rc_stop
		rc_start
		;;
esac

and change it to read as follows:

case $1 in
	start)
		rc_start
		;;
	stop)
		rc_stop
		;;
	restart)
		rc_start
		;;
esac

Report back after the next update and let me know if it helps.  This will allow the restart to just use the SOFT START command and not attempt to STOP Snort.

Bill

[bmeeks]

@gogol:

Ok, I added the sleep 3 timer and will wait and see.
Snort is not very consistent with restarting, because I also have a SOFT RESTART in my logs:

…

I have an N270 Atom processor and I know it takes some time for Snort to startup, so maybe you are right that it might be a timing issue for slower processors.
Thanks again.

You can also try the fix I posted a bit earlier in this reply:  http://forum.pfsense.org/index.php/topic,60994.msg330447.html#msg330447

Actually, this may work better than adding the sleep command.

Bill

[Supermule]

Done :)

@bmeeks:

@Supermule:

I have done that. Waiting update but restarte Snort since I edited the file.

….

Done it on only one of the FW's so that I can compare.

How about making this change instead of simply adding the sleep command ??

In the same file, find this section:

case $1 in
	start)
		rc_start
		;;
	stop)
		rc_stop
		;;
	restart)
		rc_stop
		rc_start
		;;
esac

and change it to read as follows:

case $1 in
	start)
		rc_start
		;;
	stop)
		rc_stop
		;;
	restart)
		rc_start
		;;
esac

Report back after the next update and let me know if it helps.  This will allow the restart to just use the SOFT START command and not attempt to STOP Snort.

Bill

[bmeeks]

And to flesh out my theory a bit more, SOFT START simply causes a running Snort process to refresh its configuration.  It will not start a process that is no longer running.  So continuing from my explanation earlier, during the restart process following a rules update, the STOP command stops the running process.  The START command, though, sometimes erroneously sees the PID file not yet removed and thinks Snort is running.  So instead of a hard start of a new Snort process, it just tries to refresh the one that is in the process of shutting down.  Therefore at the end of the line, Snort is left stopped.  This is why you guys can manually start Snort with no problems.  There is no real "problem" like bad rules or something, it's simply that the process was stopped for the rules update and not restarted (because of the SOFT START snafu).  At least that is my theory.

I did add the STOP command prior to the START command in the RESTART part of the shell script.  The old version had the rc_start() call only.

Bill