Snort 2.9.4.1 pkg v. 2.5.5 Issue(s)

bmeeks

Ok, I added the sleep 3 timer and will wait and see.
Snort is not very consistent with restarting, because I also have a SOFT RESTART in my logs:

…

I have an N270 Atom processor and I know it takes some time for Snort to startup, so maybe you are right that it might be a timing issue for slower processors.
Thanks again.

You can also try the fix I posted a bit earlier in this reply: http://forum.pfsense.org/index.php/topic,60994.msg330447.html#msg330447

Actually, this may work better than adding the sleep command.

Bill

Supermule

Done :)

@bmeeks:

@Supermule:

I have done that. Waiting update but restarte Snort since I edited the file.

….

Done it on only one of the FW's so that I can compare.

How about making this change instead of simply adding the sleep command ??

In the same file, find this section:
case $1 in
	start)
		rc_start
		;;
	stop)
		rc_stop
		;;
	restart)
		rc_stop
		rc_start
		;;
esac
and change it to read as follows:
case $1 in
	start)
		rc_start
		;;
	stop)
		rc_stop
		;;
	restart)
		rc_start
		;;
esac
Report back after the next update and let me know if it helps. This will allow the restart to just use the SOFT START command and not attempt to STOP Snort.

Bill

bmeeks

And to flesh out my theory a bit more, SOFT START simply causes a running Snort process to refresh its configuration. It will not start a process that is no longer running. So continuing from my explanation earlier, during the restart process following a rules update, the STOP command stops the running process. The START command, though, sometimes erroneously sees the PID file not yet removed and thinks Snort is running. So instead of a hard start of a new Snort process, it just tries to refresh the one that is in the process of shutting down. Therefore at the end of the line, Snort is left stopped. This is why you guys can manually start Snort with no problems. There is no real "problem" like bad rules or something, it's simply that the process was stopped for the rules update and not restarted (because of the SOFT START snafu). At least that is my theory.

I did add the STOP command prior to the START command in the RESTART part of the shell script. The old version had the rc_start() call only.

Bill

bmeeks

If this fixes it, what I want to do is make the change in the rules update code itself and not alter the shell script. Ermal and I agree that the proper sequence for "restart" in the shell script should be "STOP and then START". This is the standard methodolgy. I can instead alter the rules update PHP code so that it calls the shell script with just the START argument. That will emulate the old behavior of attempting a SOFT START on a running process.

Bill

Supermule

Allright. Will report back tomorrow when rules have been updated. :)

bmeeks

@Supermule:

Allright. Will report back tomorrow when rules have been updated. :)

Thanks. Hopefully you can report back "good news"… :D

I have not seen this behavior on my system, nor in testing. But if my theory is correct, this is most likely a load and hardware capability related issue. So lightly loaded hardware (few rules and/or lots of CPU horsepower) can get Snort stopped and cleaned up quick enough for things to progress as expected. But heavily loaded hardware (lots of rules and/or medium to low CPU horsepower) might get caught up in the timing issue I described.

That's my theory for now. Getting feedback from you guys actively having the issue will help me nail it down.

Supermule

This is my test box that got killed last night under updating the rules.

NO load what so ever on the box and running server class hardware with 4GB of RAM and 2 CPU's.

wan_rules.jpg_thumb

PPTP_rules.jpg_thumb

systeminfo.jpg_thumb

bmeeks

@Supermule:

This is my test box that got killed last night under updating the rules.

NO load what so ever on the box and running server class hardware with 4GB of RAM and 2 CPU's.

I've done a bit more research on this and I think I'm on to the cause. Turns out that the pkill command does not wait for the "kill" to finish before returning, so this means the STOP process may well not have finished before the START part of the script gets called. This leads to the condition I described earlier. Snort has a history of being a bit unreliable for starting/stopping on pfSense. I've got some ideas on how to make the whole thing more robust, but I will have to run them by Ermal for his concurrence. So it may be a few days before a fix is posted.

If I am right in my theory on the cause, then the last suggested workaround I posted should fix it in the interim.

Bill

Supermule

judex

I had those start issues after updates and added "sleep 3" to the start script yesterday. Since then snort started fine after an update where all 3 ruletypes were new and also after an update where no new rules got downloaded.

Looks good!

Greets, Judex

bmeeks

@judex:

I had those start issues after updates and added "sleep 3" to the start script yesterday. Since then snort started fine after an update where all 3 ruletypes were new and also after an update where no new rules got downloaded.

Looks good!

Greets, Judex

Good. Glad that seems to be a good interim solution. Different users may have to try slightly longer sleep times depending on how long Snort typically takes to shutdown on their hardware. As I said in a previous post, this would depend on how robust the hardware is and how many rules are active. I know in my case with Snort running on three interfaces on an Atom 330 box, it takes several seconds for Snort to shutdown. It takes much longer for it to start up. That's due to my lower-end hardware and the rules I normally run.

Bill

Supermule

The fix didnt work. Both boxes killed despite the difference in the .sh file.

Box 1 with the modded file:

Apr 19 00:06:06 kernel: pid 49942 (snort), uid 0: exited on signal 11
Apr 19 00:06:06 kernel: em0: promiscuous mode disabled
Apr 19 00:06:05 snort[49942]: FATAL ERROR: Flowbits: flowbits name is limited to any alphanumeric string including .-_
Apr 19 00:06:05 snort[49942]: FATAL ERROR: Flowbits: flowbits name is limited to any alphanumeric string including .-_
Apr 19 00:06:01 php: : The Rules update has finished.
Apr 19 00:06:01 php: : Snort has restarted with your new set of rules…
Apr 19 00:05:59 SnortStartup[33057]: Snort SOFT START For Internet(36256_em0)…
Apr 19 00:05:58 php: : Building new sig-msg.map file for WAN...
Apr 19 00:05:56 php: : Resolving and auto-enabling any flowbit-required rules for WAN...
Apr 19 00:05:54 php: : Updating rules configuration for: WAN ...
Apr 19 00:05:49 php: : EmergingThreats rules file update downloaded succsesfully
Apr 19 00:05:46 php: : There is a new set of EmergingThreats rules posted. Downloading...
Apr 19 00:05:46 php: : Snort GPLv2 Community Rules file update downloaded succsesfully
Apr 19 00:05:45 php: : There is a new set of Snort GPLv2 Community Rules posted. Downloading...
Apr 19 00:05:44 php: : Snort Rules Attempts: 1
Apr 19 00:04:51 php: : There is a new set of Snort VRT rules posted. Downloading...
Apr 19 00:04:51 php: : Snort MD5 Attempts: 2

Box 2 nothing changed:

Apr 19 00:09:23 php: : The Rules update has finished.
Apr 19 00:09:23 php: : Snort has restarted with your new set of rules...
Apr 19 00:09:21 SnortStartup[60130]: Snort SOFT START For Internet(9626_em0)…
Apr 19 00:09:21 snort[7492]: Could not remove pid file /var/run/snort_em09626.pid: No such file or directory
Apr 19 00:09:21 snort[7492]: Could not remove pid file /var/run/snort_em09626.pid: No such file or directory
Apr 19 00:09:20 kernel: em0: promiscuous mode disabled
Apr 19 00:09:20 snort[7492]: *** Caught Term-Signal
Apr 19 00:09:20 snort[7492]: *** Caught Term-Signal
Apr 19 00:09:19 SnortStartup[58451]: Snort STOP For Internet(9626_em0)…
Apr 19 00:09:19 php: : Building new sig-msg.map file for WAN...
Apr 19 00:09:17 php: : Resolving and auto-enabling any flowbit-required rules for WAN...
Apr 19 00:09:15 php: : Updating rules configuration for: WAN ...
Apr 19 00:09:13 php: : EmergingThreats rules file update downloaded succsesfully
Apr 19 00:09:11 php: : There is a new set of EmergingThreats rules posted. Downloading...
Apr 19 00:09:10 php: : Snort GPLv2 Community Rules file update downloaded succsesfully
Apr 19 00:09:09 php: : There is a new set of Snort GPLv2 Community Rules posted. Downloading...
Apr 19 00:09:08 php: : Failed Rules Filesize: 251
Apr 19 00:09:08 php: : Snort VRT rules file download failed...
Apr 19 00:09:08 php: : Snort Rules Attempts: 5
Apr 19 00:06:04 php: : There is a new set of Snort VRT rules posted. Downloading...
Apr 19 00:06:04 php: : Snort MD5 Attempts: 3

gogol

I am still testing because I am not sure about things.
Be careful about changes in /usr/local/etc/rc.d/snort.sh because the file can be recreated when you change something and your changes are lost.
I had this prior to a manual update, so I had an update with the old /usr/local/etc/rc.d/snort.sh file:

/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_for_rule_updates.php >> /tmp/snort_update.log

and snort crashed:

Apr 19 10:46:56	php: : The Rules update has finished.
Apr 19 10:46:56	php: : Snort has restarted with your new set of rules...
Apr 19 10:46:54	kernel: pid 86248 (snort), uid 0: exited on signal 11
Apr 19 10:46:54	SnortStartup[30864]: Snort SOFT START For WAN(54477_em0)...
Apr 19 10:46:53	snort[86248]: Could not remove pid file /var/run/snort_em054477.pid: No such file or directory
Apr 19 10:46:53	kernel: em0: promiscuous mode disabled
Apr 19 10:46:53	snort[86248]: *** Caught Term-Signal
Apr 19 10:46:52	SnortStartup[29535]: Snort STOP For WAN(54477_em0)...
Apr 19 10:46:51	php: : Building new sig-msg.map file for WAN...
Apr 19 10:46:42	php: : Resolving and auto-enabling any flowbit-required rules for WAN...
Apr 19 10:46:35	php: : Updating rules configuration for: WAN ...
Apr 19 10:46:25	php: : EmergingThreats rules file update downloaded succsesfully
Apr 19 10:46:20	php: : There is a new set of EmergingThreats rules posted. Downloading...
Apr 19 10:46:20	php: : Snort Rules Attempts: 1
Apr 19 10:45:59	php: : There is a new set of Snort VRT rules posted. Downloading...
Apr 19 10:45:59	php: : Snort MD5 Attempts: 1

I did this manual update because I did not want to wait for an automatic update. BTW: changing the interval in the Global Settings of Snort did not change anything in /etc/crontab. The interval there stays at 12 hours. Another bug?

@Supermule In my opinion has that Flowbits Fatal Error on box 1 nothing to do with the rules update problem. More with a syntax error in a flowbits rule. Did you change anything? And I see that on box 2 there is an update of the VRT rules within 15 minutes from box 1, that is why the update fails (at least if you use identical OINK codes).
And why do you use the GPLv2 Community rules as they are a subset of the VRT rules? I am saying this to rule out possible extra problems.

Supermule

Didnt change a thing compared to running it on 2.0.2.

I use identical OINK codes. I guess thats not good?
I am not a paid subscriber, so the GPLv2 rules I believe is not part of the VRT ruleset?

bmeeks

@Supermule:

Didnt change a thing compared to running it on 2.0.2.

I use identical OINK codes. I guess thats not good?
I am not a paid subscriber, so the GPLv2 rules I believe is not part of the VRT ruleset?

Gogol is correct about the 15-minute window with Snort VRT. It is a limit imposed by them to limit traffic I guess.

As for the snort.sh script getting overwritten, it is true that several actions in the GUI will cause the generation of a new snort.sh file. For example, adding or deleting an interface, or enabling or disabling Barnyard2 on an interface, are all GUI edits that will cause a new snort.sh file to get created. Not saying you did that, but just go back and check that the snort.sh file still contains the mod you made.

Oh, and Gogol is also correct about the Flowbits syntax error and subsequent FATAL ERROR on startup. That's not related to the issue we've been chasing. There is either a typo in somebody's text rule that came down in one of the updates, or your downloaded file got corrupted somehow. I lean toward option #1 with maybe a typo in the rule.

My production firewall got updates at midnight U.S. Eastern last night for both VRT and ET and restarted with no problems. I run Snort on three interfaces. So the fact this is not affecting everyone makes it hard to nail down for sure. I still think I'm on the right track, though.
Bill

Supermule

I will send you info on your mail. Then you can see it "live" :)

bmeeks

@Supermule:

I will send you info on your mail. Then you can see it "live" :)

OK. I have to be away from my PC for a while. It's currently 9:00 AM where I am, and my wife has me a list of things to take care of before this evening when some family arrives for a weekend visit… ;D

Bill

sronsen

I updated to pfSense 2.0.3 last evening and Snort was updated automatically as well, despite the fact that Snort updates have been problematic for me for several years and, if given a choice, I would never have selected upgrading both Snort and pfSense at the same time.

My issue is the inability to start Snort. Attempting to start from the command-line gives this message: "/libexec/ld-elf.so.1: /usr/local/lib/libdnet.1: unsupported file layout". Nothing related to snort is being logged in the system log. The new update logs shows that the rules updates were successful. I have removed and reinstalled Snort several times, including manual removals of Snort files and reboots before re-installs. Nothing seems to help.

Any suggestions?

Supermule

Just send you the link and its a 5mn video :)

kilthro

Bill,
I am going to put my posts here so you are not having to go between two topics.. Ok going back into the file today, I see that the sleep command was gone from my edit yesterday. Now, the only thing I can think of is I made a change to the gui and it overwrote the file back at its defaults. This is most likely what caused the fail this morning when I updated my rules manually. So I added the sleep command back and then deleted the md5 files. With snort running I clicked the update rules and it went through successfully without leaving snort in the stopped phase.

Right now auto updates are turned off. So I went into the gui and turned them back on daily. I then rechecked the snort sh file and the sleep command is gone again. So my tests on that command not working were skewed since I didnt realize those changes would get removed if I touched the gui.

So since the manual test was successful, I am going to enable the auto updates, then edit the file with the sleep command and see if it works just fine.