Snort 2.9.4.1 pkg v. 2.5.5 Issue(s)
-
And to flesh out my theory a bit more, SOFT START simply causes a running Snort process to refresh its configuration. It will not start a process that is no longer running. So continuing from my explanation earlier, during the restart process following a rules update, the STOP command stops the running process. The START command, though, sometimes erroneously sees the PID file not yet removed and thinks Snort is running. So instead of a hard start of a new Snort process, it just tries to refresh the one that is in the process of shutting down. Therefore at the end of the line, Snort is left stopped. This is why you guys can manually start Snort with no problems. There is no real "problem" like bad rules or something, it's simply that the process was stopped for the rules update and not restarted (because of the SOFT START snafu). At least that is my theory.
I did add the STOP command prior to the START command in the RESTART part of the shell script. The old version had the rc_start() call only.
Bill
-
If this fixes it, what I want to do is make the change in the rules update code itself and not alter the shell script. Ermal and I agree that the proper sequence for "restart" in the shell script should be "STOP and then START". This is the standard methodolgy. I can instead alter the rules update PHP code so that it calls the shell script with just the START argument. That will emulate the old behavior of attempting a SOFT START on a running process.
Bill
-
Allright. Will report back tomorrow when rules have been updated. :)
-
Allright. Will report back tomorrow when rules have been updated. :)
Thanks. Hopefully you can report back "good news"… :D
I have not seen this behavior on my system, nor in testing. But if my theory is correct, this is most likely a load and hardware capability related issue. So lightly loaded hardware (few rules and/or lots of CPU horsepower) can get Snort stopped and cleaned up quick enough for things to progress as expected. But heavily loaded hardware (lots of rules and/or medium to low CPU horsepower) might get caught up in the timing issue I described.
That's my theory for now. Getting feedback from you guys actively having the issue will help me nail it down.
-
This is my test box that got killed last night under updating the rules.
NO load what so ever on the box and running server class hardware with 4GB of RAM and 2 CPU's.
-
This is my test box that got killed last night under updating the rules.
NO load what so ever on the box and running server class hardware with 4GB of RAM and 2 CPU's.
I've done a bit more research on this and I think I'm on to the cause. Turns out that the pkill command does not wait for the "kill" to finish before returning, so this means the STOP process may well not have finished before the START part of the script gets called. This leads to the condition I described earlier. Snort has a history of being a bit unreliable for starting/stopping on pfSense. I've got some ideas on how to make the whole thing more robust, but I will have to run them by Ermal for his concurrence. So it may be a few days before a fix is posted.
If I am right in my theory on the cause, then the last suggested workaround I posted should fix it in the interim.
Bill
-
-
I had those start issues after updates and added "sleep 3" to the start script yesterday. Since then snort started fine after an update where all 3 ruletypes were new and also after an update where no new rules got downloaded.
Looks good!
Greets, Judex
-
I had those start issues after updates and added "sleep 3" to the start script yesterday. Since then snort started fine after an update where all 3 ruletypes were new and also after an update where no new rules got downloaded.
Looks good!
Greets, Judex
Good. Glad that seems to be a good interim solution. Different users may have to try slightly longer sleep times depending on how long Snort typically takes to shutdown on their hardware. As I said in a previous post, this would depend on how robust the hardware is and how many rules are active. I know in my case with Snort running on three interfaces on an Atom 330 box, it takes several seconds for Snort to shutdown. It takes much longer for it to start up. That's due to my lower-end hardware and the rules I normally run.
Bill
-
The fix didnt work. Both boxes killed despite the difference in the .sh file.
Box 1 with the modded file:
Apr 19 00:06:06 kernel: pid 49942 (snort), uid 0: exited on signal 11
Apr 19 00:06:06 kernel: em0: promiscuous mode disabled
Apr 19 00:06:05 snort[49942]: FATAL ERROR: Flowbits: flowbits name is limited to any alphanumeric string including .-_
Apr 19 00:06:05 snort[49942]: FATAL ERROR: Flowbits: flowbits name is limited to any alphanumeric string including .-_
Apr 19 00:06:01 php: : The Rules update has finished.
Apr 19 00:06:01 php: : Snort has restarted with your new set of rules…
Apr 19 00:05:59 SnortStartup[33057]: Snort SOFT START For Internet(36256_em0)…
Apr 19 00:05:58 php: : Building new sig-msg.map file for WAN...
Apr 19 00:05:56 php: : Resolving and auto-enabling any flowbit-required rules for WAN...
Apr 19 00:05:54 php: : Updating rules configuration for: WAN ...
Apr 19 00:05:49 php: : EmergingThreats rules file update downloaded succsesfully
Apr 19 00:05:46 php: : There is a new set of EmergingThreats rules posted. Downloading...
Apr 19 00:05:46 php: : Snort GPLv2 Community Rules file update downloaded succsesfully
Apr 19 00:05:45 php: : There is a new set of Snort GPLv2 Community Rules posted. Downloading...
Apr 19 00:05:44 php: : Snort Rules Attempts: 1
Apr 19 00:04:51 php: : There is a new set of Snort VRT rules posted. Downloading...
Apr 19 00:04:51 php: : Snort MD5 Attempts: 2Box 2 nothing changed:
Apr 19 00:09:23 php: : The Rules update has finished.
Apr 19 00:09:23 php: : Snort has restarted with your new set of rules...
Apr 19 00:09:21 SnortStartup[60130]: Snort SOFT START For Internet(9626_em0)…
Apr 19 00:09:21 snort[7492]: Could not remove pid file /var/run/snort_em09626.pid: No such file or directory
Apr 19 00:09:21 snort[7492]: Could not remove pid file /var/run/snort_em09626.pid: No such file or directory
Apr 19 00:09:20 kernel: em0: promiscuous mode disabled
Apr 19 00:09:20 snort[7492]: *** Caught Term-Signal
Apr 19 00:09:20 snort[7492]: *** Caught Term-Signal
Apr 19 00:09:19 SnortStartup[58451]: Snort STOP For Internet(9626_em0)…
Apr 19 00:09:19 php: : Building new sig-msg.map file for WAN...
Apr 19 00:09:17 php: : Resolving and auto-enabling any flowbit-required rules for WAN...
Apr 19 00:09:15 php: : Updating rules configuration for: WAN ...
Apr 19 00:09:13 php: : EmergingThreats rules file update downloaded succsesfully
Apr 19 00:09:11 php: : There is a new set of EmergingThreats rules posted. Downloading...
Apr 19 00:09:10 php: : Snort GPLv2 Community Rules file update downloaded succsesfully
Apr 19 00:09:09 php: : There is a new set of Snort GPLv2 Community Rules posted. Downloading...
Apr 19 00:09:08 php: : Failed Rules Filesize: 251
Apr 19 00:09:08 php: : Snort VRT rules file download failed...
Apr 19 00:09:08 php: : Snort Rules Attempts: 5
Apr 19 00:06:04 php: : There is a new set of Snort VRT rules posted. Downloading...
Apr 19 00:06:04 php: : Snort MD5 Attempts: 3 -
I am still testing because I am not sure about things.
Be careful about changes in /usr/local/etc/rc.d/snort.sh because the file can be recreated when you change something and your changes are lost.
I had this prior to a manual update, so I had an update with the old /usr/local/etc/rc.d/snort.sh file:/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_for_rule_updates.php >> /tmp/snort_update.logand snort crashed:
Apr 19 10:46:56 php: : The Rules update has finished. Apr 19 10:46:56 php: : Snort has restarted with your new set of rules... Apr 19 10:46:54 kernel: pid 86248 (snort), uid 0: exited on signal 11 Apr 19 10:46:54 SnortStartup[30864]: Snort SOFT START For WAN(54477_em0)... Apr 19 10:46:53 snort[86248]: Could not remove pid file /var/run/snort_em054477.pid: No such file or directory Apr 19 10:46:53 kernel: em0: promiscuous mode disabled Apr 19 10:46:53 snort[86248]: *** Caught Term-Signal Apr 19 10:46:52 SnortStartup[29535]: Snort STOP For WAN(54477_em0)... Apr 19 10:46:51 php: : Building new sig-msg.map file for WAN... Apr 19 10:46:42 php: : Resolving and auto-enabling any flowbit-required rules for WAN... Apr 19 10:46:35 php: : Updating rules configuration for: WAN ... Apr 19 10:46:25 php: : EmergingThreats rules file update downloaded succsesfully Apr 19 10:46:20 php: : There is a new set of EmergingThreats rules posted. Downloading... Apr 19 10:46:20 php: : Snort Rules Attempts: 1 Apr 19 10:45:59 php: : There is a new set of Snort VRT rules posted. Downloading... Apr 19 10:45:59 php: : Snort MD5 Attempts: 1I did this manual update because I did not want to wait for an automatic update. BTW: changing the interval in the Global Settings of Snort did not change anything in /etc/crontab. The interval there stays at 12 hours. Another bug?
@Supermule In my opinion has that Flowbits Fatal Error on box 1 nothing to do with the rules update problem. More with a syntax error in a flowbits rule. Did you change anything? And I see that on box 2 there is an update of the VRT rules within 15 minutes from box 1, that is why the update fails (at least if you use identical OINK codes).
And why do you use the GPLv2 Community rules as they are a subset of the VRT rules? I am saying this to rule out possible extra problems. -
Didnt change a thing compared to running it on 2.0.2.
I use identical OINK codes. I guess thats not good?
I am not a paid subscriber, so the GPLv2 rules I believe is not part of the VRT ruleset? -
Didnt change a thing compared to running it on 2.0.2.
I use identical OINK codes. I guess thats not good?
I am not a paid subscriber, so the GPLv2 rules I believe is not part of the VRT ruleset?Gogol is correct about the 15-minute window with Snort VRT. It is a limit imposed by them to limit traffic I guess.
As for the snort.sh script getting overwritten, it is true that several actions in the GUI will cause the generation of a new snort.sh file. For example, adding or deleting an interface, or enabling or disabling Barnyard2 on an interface, are all GUI edits that will cause a new snort.sh file to get created. Not saying you did that, but just go back and check that the snort.sh file still contains the mod you made.
Oh, and Gogol is also correct about the Flowbits syntax error and subsequent FATAL ERROR on startup. That's not related to the issue we've been chasing. There is either a typo in somebody's text rule that came down in one of the updates, or your downloaded file got corrupted somehow. I lean toward option #1 with maybe a typo in the rule.
My production firewall got updates at midnight U.S. Eastern last night for both VRT and ET and restarted with no problems. I run Snort on three interfaces. So the fact this is not affecting everyone makes it hard to nail down for sure. I still think I'm on the right track, though.
Bill -
I will send you info on your mail. Then you can see it "live" :)
-
I will send you info on your mail. Then you can see it "live" :)
OK. I have to be away from my PC for a while. It's currently 9:00 AM where I am, and my wife has me a list of things to take care of before this evening when some family arrives for a weekend visit… ;D
Bill
-
I updated to pfSense 2.0.3 last evening and Snort was updated automatically as well, despite the fact that Snort updates have been problematic for me for several years and, if given a choice, I would never have selected upgrading both Snort and pfSense at the same time.
My issue is the inability to start Snort. Attempting to start from the command-line gives this message: "/libexec/ld-elf.so.1: /usr/local/lib/libdnet.1: unsupported file layout". Nothing related to snort is being logged in the system log. The new update logs shows that the rules updates were successful. I have removed and reinstalled Snort several times, including manual removals of Snort files and reboots before re-installs. Nothing seems to help.
Any suggestions?
-
Just send you the link and its a 5mn video :)
-
Bill,
I am going to put my posts here so you are not having to go between two topics.. Ok going back into the file today, I see that the sleep command was gone from my edit yesterday. Now, the only thing I can think of is I made a change to the gui and it overwrote the file back at its defaults. This is most likely what caused the fail this morning when I updated my rules manually. So I added the sleep command back and then deleted the md5 files. With snort running I clicked the update rules and it went through successfully without leaving snort in the stopped phase.Right now auto updates are turned off. So I went into the gui and turned them back on daily. I then rechecked the snort sh file and the sleep command is gone again. So my tests on that command not working were skewed since I didnt realize those changes would get removed if I touched the gui.
So since the manual test was successful, I am going to enable the auto updates, then edit the file with the sleep command and see if it works just fine.
-
Bill,
I am going to put my posts here so you are not having to go between two topics.. Ok going back into the file today, I see that the sleep command was gone from my edit yesterday. Now, the only thing I can think of is I made a change to the gui and it overwrote the file back at its defaults. This is most likely what caused the fail this morning when I updated my rules manually. So I added the sleep command back and then deleted the md5 files. With snort running I clicked the update rules and it went through successfully without leaving snort in the stopped phase.Right now auto updates are turned off. So I went into the gui and turned them back on daily. I then rechecked the snort sh file and the sleep command is gone again. So my tests on that command not working were skewed since I didnt realize those changes would get removed if I touched the gui.
So since the manual test was successful, I am going to enable the auto updates, then edit the file with the sleep command and see if it works just fine.
Yeah, I posted in another thread (or maybe this one, I'm losing track… :D) about how certain changes in the GUI will cause the snort.sh shell script to be recreated. This is necessary, for instance, should someone add or delete an interface, or enable or disable Barnyard2 on an interface. Other configuration changes can also trigger the recreation of snort.sh.
I'm working on a permanent fix now.
Bill
-
I updated to pfSense 2.0.3 last evening and Snort was updated automatically as well, despite the fact that Snort updates have been problematic for me for several years and, if given a choice, I would never have selected upgrading both Snort and pfSense at the same time.
My issue is the inability to start Snort. Attempting to start from the command-line gives this message: "/libexec/ld-elf.so.1: /usr/local/lib/libdnet.1: unsupported file layout". Nothing related to snort is being logged in the system log. The new update logs shows that the rules updates were successful. I have removed and reinstalled Snort several times, including manual removals of Snort files and reboots before re-installs. Nothing seems to help.
Any suggestions?
This error message almost always means you have mixed 32-bit and 64-bit libraries on the system. These "unsupported layout" errors have happened before for many other packages besides just Snort, and each time it's caused by having a mix of 32-bit and 64-bit stuff on a system. In particular this error can happen when 64-bit libs wind up on a 32-bit box. I can't tell you how this might have happened, but I'm pretty sure that's what is wrong now.
You didn't mention in your post whether you were using the 32-bit or 64-bit version of pfSense 2.0.3. From the error, I'm guessing maybe 32-bit pfSense and you have somehow picked up some 64-bit libraries along the way. Did you by chance have any other packages installed besides Snort?
Here is my suggestion for you. Go to the Installed Packages tab and remove Snort by clicking the X icon.
Get command line access to the box and issue the three commands below.
rm -rf /usr/local/etc/snort rm -rf /usr/local/lib/snort rm -rf /usr/local/etc/snortIf possible, now reboot the firewall.
When it comes back up, install Snort again from the Available Packages tab.I still can't guarantee you this will fix it, though. With a mix of 32-bit and 64-bit stuff, I hate to say it but you might be better saving off the configuration and rebuilding the box from a fresh ISO image. Search the forums here for the "unsupported file layout" message and you should find several other posts about it. See what was suggested to those folks for a fix.
Bill
