Snort 2.9.4.1 pkg v. 2.5.5 Issue(s)

Supermule

Thanks Bill!

Looking forward to see what you come up with.

bmeeks

Thanks Bill!

Looking forward to see what you come up with.

If you want to try a quick experiment, follow the instructions here: http://forum.pfsense.org/index.php/topic,61001.msg330160.html#msg330160

Try fiddling with the sleep time. Maybe try a little longer. If my theory is correct, it's a timing issue.

Bill

gogol

Ok, I added the sleep 3 timer and will wait and see.
Snort is not very consistent with restarting, because I also have a SOFT RESTART in my logs:


Apr 16 12:04:13	php: : The Rules update has finished.
Apr 16 12:04:13	php: : Snort has restarted with your new set of rules...
Apr 16 12:04:11	SnortStartup[77296]: Snort SOFT START For WAN(54477_em0)...
Apr 16 12:04:10	snort[38532]: Could not remove pid file /var/run/snort_em054477.pid: No such file or directory
Apr 16 12:04:09	snort[38532]: *** Caught Term-Signal
Apr 16 12:04:08	SnortStartup[75228]: Snort STOP For WAN(54477_em0)...
Apr 16 12:04:08	php: : Building new sig-msg.map file for WAN...
Apr 16 12:03:59	php: : Resolving and auto-enabling any flowbit-required rules for WAN...
Apr 16 12:03:52	php: : Updating rules configuration for: WAN ...
Apr 16 12:03:51	php: : EmergingThreats rules file update downloaded succsesfully
Apr 16 12:03:48	php: : There is a new set of EmergingThreats rules posted. Downloading...
Apr 16 12:03:48	php: : Snort VRT rules are up to date...
Apr 16 12:03:48	php: : Snort MD5 Attempts: 1

I have an N270 Atom processor and I know it takes some time for Snort to startup, so maybe you are right that it might be a timing issue for slower processors.
Thanks again.

Supermule

I have done that. Waiting update but restarte Snort since I edited the file.

Apr 18 14:45:17 php: /status_services.php: The command '/usr/local/etc/rc.d/snort.sh stop' returned exit code '1', the output was ''
Apr 18 14:45:15 SnortStartup[52848]: Snort STOP For Internet(36256_em0)…
Apr 18 14:45:15 php: /status_services.php: The command '/usr/local/etc/rc.d/snort.sh stop' returned exit code '1', the output was ''
Apr 18 14:45:14 snort[43724]: Could not remove pid file /var/run/snort_em036256.pid: No such file or directory
Apr 18 14:45:14 snort[43724]: Could not remove pid file /var/run/snort_em036256.pid: No such file or directory
Apr 18 14:45:14 kernel: em0: promiscuous mode disabled
Apr 18 14:45:14 snort[43724]: *** Caught Term-Signal
Apr 18 14:45:14 snort[43724]: *** Caught Term-Signal
Apr 18 14:45:13 SnortStartup[51967]: Snort STOP For Internet(36256_em0)…

Done it on only one of the FW's so that I can compare.

bmeeks

@Supermule:

I have done that. Waiting update but restarte Snort since I edited the file.

….

Done it on only one of the FW's so that I can compare.

How about making this change instead of simply adding the sleep command ??

In the same file, find this section:

case $1 in
	start)
		rc_start
		;;
	stop)
		rc_stop
		;;
	restart)
		rc_stop
		rc_start
		;;
esac

and change it to read as follows:

case $1 in
	start)
		rc_start
		;;
	stop)
		rc_stop
		;;
	restart)
		rc_start
		;;
esac

Report back after the next update and let me know if it helps. This will allow the restart to just use the SOFT START command and not attempt to STOP Snort.

Bill

bmeeks

@gogol:

Ok, I added the sleep 3 timer and will wait and see.
Snort is not very consistent with restarting, because I also have a SOFT RESTART in my logs:

…

I have an N270 Atom processor and I know it takes some time for Snort to startup, so maybe you are right that it might be a timing issue for slower processors.
Thanks again.

You can also try the fix I posted a bit earlier in this reply: http://forum.pfsense.org/index.php/topic,60994.msg330447.html#msg330447

Actually, this may work better than adding the sleep command.

Bill

Supermule

Done :)

@bmeeks:

@Supermule:

I have done that. Waiting update but restarte Snort since I edited the file.

….

Done it on only one of the FW's so that I can compare.

How about making this change instead of simply adding the sleep command ??

In the same file, find this section:
case $1 in
	start)
		rc_start
		;;
	stop)
		rc_stop
		;;
	restart)
		rc_stop
		rc_start
		;;
esac
and change it to read as follows:
case $1 in
	start)
		rc_start
		;;
	stop)
		rc_stop
		;;
	restart)
		rc_start
		;;
esac
Report back after the next update and let me know if it helps. This will allow the restart to just use the SOFT START command and not attempt to STOP Snort.

Bill

bmeeks

And to flesh out my theory a bit more, SOFT START simply causes a running Snort process to refresh its configuration. It will not start a process that is no longer running. So continuing from my explanation earlier, during the restart process following a rules update, the STOP command stops the running process. The START command, though, sometimes erroneously sees the PID file not yet removed and thinks Snort is running. So instead of a hard start of a new Snort process, it just tries to refresh the one that is in the process of shutting down. Therefore at the end of the line, Snort is left stopped. This is why you guys can manually start Snort with no problems. There is no real "problem" like bad rules or something, it's simply that the process was stopped for the rules update and not restarted (because of the SOFT START snafu). At least that is my theory.

I did add the STOP command prior to the START command in the RESTART part of the shell script. The old version had the rc_start() call only.

Bill

bmeeks

If this fixes it, what I want to do is make the change in the rules update code itself and not alter the shell script. Ermal and I agree that the proper sequence for "restart" in the shell script should be "STOP and then START". This is the standard methodolgy. I can instead alter the rules update PHP code so that it calls the shell script with just the START argument. That will emulate the old behavior of attempting a SOFT START on a running process.

Bill

Supermule

Allright. Will report back tomorrow when rules have been updated. :)

bmeeks

@Supermule:

Allright. Will report back tomorrow when rules have been updated. :)

Thanks. Hopefully you can report back "good news"… :D

I have not seen this behavior on my system, nor in testing. But if my theory is correct, this is most likely a load and hardware capability related issue. So lightly loaded hardware (few rules and/or lots of CPU horsepower) can get Snort stopped and cleaned up quick enough for things to progress as expected. But heavily loaded hardware (lots of rules and/or medium to low CPU horsepower) might get caught up in the timing issue I described.

That's my theory for now. Getting feedback from you guys actively having the issue will help me nail it down.

Supermule

This is my test box that got killed last night under updating the rules.

NO load what so ever on the box and running server class hardware with 4GB of RAM and 2 CPU's.

wan_rules.jpg_thumb

PPTP_rules.jpg_thumb

systeminfo.jpg_thumb

bmeeks

@Supermule:

This is my test box that got killed last night under updating the rules.

NO load what so ever on the box and running server class hardware with 4GB of RAM and 2 CPU's.

I've done a bit more research on this and I think I'm on to the cause. Turns out that the pkill command does not wait for the "kill" to finish before returning, so this means the STOP process may well not have finished before the START part of the script gets called. This leads to the condition I described earlier. Snort has a history of being a bit unreliable for starting/stopping on pfSense. I've got some ideas on how to make the whole thing more robust, but I will have to run them by Ermal for his concurrence. So it may be a few days before a fix is posted.

If I am right in my theory on the cause, then the last suggested workaround I posted should fix it in the interim.

Bill

Supermule

judex

I had those start issues after updates and added "sleep 3" to the start script yesterday. Since then snort started fine after an update where all 3 ruletypes were new and also after an update where no new rules got downloaded.

Looks good!

Greets, Judex

bmeeks

@judex:

I had those start issues after updates and added "sleep 3" to the start script yesterday. Since then snort started fine after an update where all 3 ruletypes were new and also after an update where no new rules got downloaded.

Looks good!

Greets, Judex

Good. Glad that seems to be a good interim solution. Different users may have to try slightly longer sleep times depending on how long Snort typically takes to shutdown on their hardware. As I said in a previous post, this would depend on how robust the hardware is and how many rules are active. I know in my case with Snort running on three interfaces on an Atom 330 box, it takes several seconds for Snort to shutdown. It takes much longer for it to start up. That's due to my lower-end hardware and the rules I normally run.

Bill

Supermule

The fix didnt work. Both boxes killed despite the difference in the .sh file.

Box 1 with the modded file:

Apr 19 00:06:06 kernel: pid 49942 (snort), uid 0: exited on signal 11
Apr 19 00:06:06 kernel: em0: promiscuous mode disabled
Apr 19 00:06:05 snort[49942]: FATAL ERROR: Flowbits: flowbits name is limited to any alphanumeric string including .-_
Apr 19 00:06:05 snort[49942]: FATAL ERROR: Flowbits: flowbits name is limited to any alphanumeric string including .-_
Apr 19 00:06:01 php: : The Rules update has finished.
Apr 19 00:06:01 php: : Snort has restarted with your new set of rules…
Apr 19 00:05:59 SnortStartup[33057]: Snort SOFT START For Internet(36256_em0)…
Apr 19 00:05:58 php: : Building new sig-msg.map file for WAN...
Apr 19 00:05:56 php: : Resolving and auto-enabling any flowbit-required rules for WAN...
Apr 19 00:05:54 php: : Updating rules configuration for: WAN ...
Apr 19 00:05:49 php: : EmergingThreats rules file update downloaded succsesfully
Apr 19 00:05:46 php: : There is a new set of EmergingThreats rules posted. Downloading...
Apr 19 00:05:46 php: : Snort GPLv2 Community Rules file update downloaded succsesfully
Apr 19 00:05:45 php: : There is a new set of Snort GPLv2 Community Rules posted. Downloading...
Apr 19 00:05:44 php: : Snort Rules Attempts: 1
Apr 19 00:04:51 php: : There is a new set of Snort VRT rules posted. Downloading...
Apr 19 00:04:51 php: : Snort MD5 Attempts: 2

Box 2 nothing changed:

Apr 19 00:09:23 php: : The Rules update has finished.
Apr 19 00:09:23 php: : Snort has restarted with your new set of rules...
Apr 19 00:09:21 SnortStartup[60130]: Snort SOFT START For Internet(9626_em0)…
Apr 19 00:09:21 snort[7492]: Could not remove pid file /var/run/snort_em09626.pid: No such file or directory
Apr 19 00:09:21 snort[7492]: Could not remove pid file /var/run/snort_em09626.pid: No such file or directory
Apr 19 00:09:20 kernel: em0: promiscuous mode disabled
Apr 19 00:09:20 snort[7492]: *** Caught Term-Signal
Apr 19 00:09:20 snort[7492]: *** Caught Term-Signal
Apr 19 00:09:19 SnortStartup[58451]: Snort STOP For Internet(9626_em0)…
Apr 19 00:09:19 php: : Building new sig-msg.map file for WAN...
Apr 19 00:09:17 php: : Resolving and auto-enabling any flowbit-required rules for WAN...
Apr 19 00:09:15 php: : Updating rules configuration for: WAN ...
Apr 19 00:09:13 php: : EmergingThreats rules file update downloaded succsesfully
Apr 19 00:09:11 php: : There is a new set of EmergingThreats rules posted. Downloading...
Apr 19 00:09:10 php: : Snort GPLv2 Community Rules file update downloaded succsesfully
Apr 19 00:09:09 php: : There is a new set of Snort GPLv2 Community Rules posted. Downloading...
Apr 19 00:09:08 php: : Failed Rules Filesize: 251
Apr 19 00:09:08 php: : Snort VRT rules file download failed...
Apr 19 00:09:08 php: : Snort Rules Attempts: 5
Apr 19 00:06:04 php: : There is a new set of Snort VRT rules posted. Downloading...
Apr 19 00:06:04 php: : Snort MD5 Attempts: 3

gogol

I am still testing because I am not sure about things.
Be careful about changes in /usr/local/etc/rc.d/snort.sh because the file can be recreated when you change something and your changes are lost.
I had this prior to a manual update, so I had an update with the old /usr/local/etc/rc.d/snort.sh file:

/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_for_rule_updates.php >> /tmp/snort_update.log

and snort crashed:

Apr 19 10:46:56	php: : The Rules update has finished.
Apr 19 10:46:56	php: : Snort has restarted with your new set of rules...
Apr 19 10:46:54	kernel: pid 86248 (snort), uid 0: exited on signal 11
Apr 19 10:46:54	SnortStartup[30864]: Snort SOFT START For WAN(54477_em0)...
Apr 19 10:46:53	snort[86248]: Could not remove pid file /var/run/snort_em054477.pid: No such file or directory
Apr 19 10:46:53	kernel: em0: promiscuous mode disabled
Apr 19 10:46:53	snort[86248]: *** Caught Term-Signal
Apr 19 10:46:52	SnortStartup[29535]: Snort STOP For WAN(54477_em0)...
Apr 19 10:46:51	php: : Building new sig-msg.map file for WAN...
Apr 19 10:46:42	php: : Resolving and auto-enabling any flowbit-required rules for WAN...
Apr 19 10:46:35	php: : Updating rules configuration for: WAN ...
Apr 19 10:46:25	php: : EmergingThreats rules file update downloaded succsesfully
Apr 19 10:46:20	php: : There is a new set of EmergingThreats rules posted. Downloading...
Apr 19 10:46:20	php: : Snort Rules Attempts: 1
Apr 19 10:45:59	php: : There is a new set of Snort VRT rules posted. Downloading...
Apr 19 10:45:59	php: : Snort MD5 Attempts: 1

I did this manual update because I did not want to wait for an automatic update. BTW: changing the interval in the Global Settings of Snort did not change anything in /etc/crontab. The interval there stays at 12 hours. Another bug?

@Supermule In my opinion has that Flowbits Fatal Error on box 1 nothing to do with the rules update problem. More with a syntax error in a flowbits rule. Did you change anything? And I see that on box 2 there is an update of the VRT rules within 15 minutes from box 1, that is why the update fails (at least if you use identical OINK codes).
And why do you use the GPLv2 Community rules as they are a subset of the VRT rules? I am saying this to rule out possible extra problems.

Supermule

Didnt change a thing compared to running it on 2.0.2.

I use identical OINK codes. I guess thats not good?
I am not a paid subscriber, so the GPLv2 rules I believe is not part of the VRT ruleset?

bmeeks

@Supermule:

Didnt change a thing compared to running it on 2.0.2.

I use identical OINK codes. I guess thats not good?
I am not a paid subscriber, so the GPLv2 rules I believe is not part of the VRT ruleset?

Gogol is correct about the 15-minute window with Snort VRT. It is a limit imposed by them to limit traffic I guess.

As for the snort.sh script getting overwritten, it is true that several actions in the GUI will cause the generation of a new snort.sh file. For example, adding or deleting an interface, or enabling or disabling Barnyard2 on an interface, are all GUI edits that will cause a new snort.sh file to get created. Not saying you did that, but just go back and check that the snort.sh file still contains the mod you made.

Oh, and Gogol is also correct about the Flowbits syntax error and subsequent FATAL ERROR on startup. That's not related to the issue we've been chasing. There is either a typo in somebody's text rule that came down in one of the updates, or your downloaded file got corrupted somehow. I lean toward option #1 with maybe a typo in the rule.

My production firewall got updates at midnight U.S. Eastern last night for both VRT and ET and restarted with no problems. I run Snort on three interfaces. So the fact this is not affecting everyone makes it hard to nail down for sure. I still think I'm on the right track, though.
Bill