Snort segfaulting on startup w/ http pre-processors enabled
-
Strangely I'm seeing this "exited on signal 11" error on only one of my 3 boxes. They're all on the same, latest snap. I wasn't going to say anything about it until I was able to download the free, registered Snort rules for 2.9.4.1. Don't want to get into trying to troubleshoot an issue that is possibly caused by the missing Snort.org ruleset and will possibly resolve itself.
-
Glad you are trying to help.
I have only the ET rules. Snort exits on signal 11 within 10 minutes, sometimes immediately or within a minute when experimenting with other settings, with http_inspect enabled and even no categories selected.
Snort also crashes with other preprocessors, like only imap, and no http_inspect.I have run configuration test (snort -T) with preprocessors enabled and snort.conf was validated successfully (if you like I can post the output).
The only way I can run snort now, is with categories: botcc, ciarmy, drop, shield, rbi, tor (thus only looking for IP addresses) and enabled portscan detection.
-
I have a 2.1-BETA virtual machine, but it is a few weeks old. Haven't checked it lately to be sure of its age. Been working on my 2.0.2 machines for the latest update testing.
I can try an update to my 2.1-BETA machine and see if I can duplicate the issue. May be a few days before I can post back, though. Are all of you having this problem with the latest 2.1-BETA running the 64-bit image? My VM is a 32-bit machine, so I may need to build me a 64-bit VM as well for testing.
Bill
-
I have a 2.1-BETA virtual machine, but it is a few weeks old. Haven't checked it lately to be sure of its age. Been working on my 2.0.2 machines for the latest update testing.
I can try an update to my 2.1-BETA machine and see if I can duplicate the issue. May be a few days before I can post back, though. Are all of you having this problem with the latest 2.1-BETA running the 64-bit image? My VM is a 32-bit machine, so I may need to build me a 64-bit VM as well for testing.
Bill
I just did that.
I have a virtual machine ( Parallels desktop 8 ) with imported configuration file and same setup as my main pfSense box where I have the problems. Although this virtual machine is inside my LAN and has other traffic than on my WAN it runs fine with snort HTTP_inspect preprocessor enabled.
But that is what others have reported: no problems!!
My main box:
2.1-BETA1 (i386)
built on Wed Mar 27 07:29:08 EDT 2013
snort 2.9.4.1 v 2.5.4Thanks for your efforts. I am now going to try to run snort on another adapter on my main machine.
-
OK. Could be something going on with an underlying driver somewhere. Strange that http_inspect would trigger it, but weird things can happen in the software world.
Bill
-
Tested on rl0 interface, which I configured the same as em0 (disabled), so with other driver, but I get another signal 11 exit.
As far as I can see Snort does not log anything anywhere.See also this: http://redmine.pfsense.org/issues/2581
-
I saw this behavior today myself with a 2.1-BETA AMD64 snapshot (March 28 build) in VMware on a virtual machine I was testing. The VM was running with no rules enabled (I had not selected any yet). Only the http_inspect preprocessor and three other preprocessors were enabled. It ran fine for a few minutes, and then next time I looked, the Snort process had exited with Signal 11. Nothing else in the logs. I was actually busy on a different VM when the 64-bit VM crashed, so I did not see the crash. I just noticed it a bit later when I checked the machine again.
I have no idea why it crashed, but at least I've seen it happen myself. Have yet to see this on a 32-bit build on 2.0.2 either on real hardware or a VM. I will check this out over the weekend and see if I can narrow it down to a 2.1 issue or a 64-bit issue. One thing is for sure, a VM can't have "real hardware failure"; so the Signal 11 has got be a software problem in the binary. Of course it could be in the Snort code itself (not likely since that is shared across platforms), it could be in the 2.1 kernel code (8.3 versus 8.1), or it could be a 32-bit vs 64-bit thing… ???
Bill
-
I have no idea why it crashed, but at least I've seen it happen myself. Have yet to see this on a 32-bit build on 2.0.2 either on real hardware or a VM. I will check this out over the weekend and see if I can narrow it down to a 2.1 issue or a 64-bit issue. One thing is for sure, a VM can't have "real hardware failure"; so the Signal 11 has got be a software problem in the binary. Of course it could be in the Snort code itself (not likely since that is shared across platforms), it could be in the 2.1 kernel code (8.3 versus 8.1), or it could be a 32-bit vs 64-bit thing… ???
I am running 32 bit (i386) and you saw proof on 64bit. I am not a software engineer, but that suggests it is not a 32bit nor a 64 bit thing, or am I wrong?
-
I am running 32 bit (i386) and you saw proof on 64bit. I am not a software engineer, but that suggests it is not a 32bit nor a 64 bit thing, or am I wrong?
What is the max run time you've seen with Snort 2.9.4.1 without a Signal 11? I've had my test VM up for several hours thus far with no sign of trouble. It's also running an assortment of Snort VRT and ET rules just for kicks. Granted it's not getting a lot of traffic to inspect, but neither was the 64-bit VM that crashed on me earlier. I will continue to run the VMs through the weekend to see if anything turns up.
Bill
-
What is the max run time you've seen with Snort 2.9.4.1 without a Signal 11? I've had my test VM up for several hours thus far with no sign of trouble. It's also running an assortment of Snort VRT and ET rules just for kicks. Granted it's not getting a lot of traffic to inspect, but neither was the 64-bit VM that crashed on me earlier. I will continue to run the VMs through the weekend to see if anything turns up.
The run time varied between several seconds and about 10 minutes. I don't know if some traffic caused a signal 11 exit, because I never saw an alert.
I started from scratch and an empty suppress list and I can remember from previous version that I got a lot of these messages almost immediately:
(http_inspect) NO CONTENT-LENGTH OR TRANSFER-ENCODING IN HTTP RESPONSE 120:3:1 -
After some investigation it seems that /usr/local/etc/snort/my_snort_sensor/preproc_rules/ was totally empty.
After copying decoder.rules, preprocessor.rules, sensitive-data.rules from /usr/pbi/snort-i386/etc/snort/preproc_rules/ to /usr/local/etc/snort/my_snort_sensor/preproc_rules/ I can now see all those alerts I have to suppress ;)Maybe that was the culprit and I will see if snort keeps running now. Fingers crossed!
-
As far as I can see now snort runs fine again.
But why are those missing files (see my previous post) not copied to the right directory?Can someone have a look at this in snort.inc (start at line 1901). It is a guess because I have no programming skills:
/* create a few directories and ensure the sample files are in place */ $snort_dirs = array( $snortdir, $snortcfgdir, "{$snortcfgdir}/rules", "{$snortlogdir}/snort_{$if_real}{$snort_uuid}", "{$snortlogdir}/snort_{$if_real}{$snort_uuid}/barnyard2", "{$snortcfgdir}/preproc_rules", "dynamicrules" => "/usr/local/lib/snort/dynamicrules", "dynamicengine" => "/usr/local/lib/snort/dynamicengine", "dynamicpreprocessor" => "{$snortcfgdir}/dynamicpreprocessor" ); foreach ($snort_dirs as $dir) { if (!is_dir($dir)) safe_mkdir($dir); } $snort_files = array("gen-msg.map", "classification.config", "reference.config", "sid-msg.map", "unicode.map", "threshold.conf", "preproc_rules/preprocessor.rules", "preproc_rules/decoder.rules", "preproc_rules/sensitive-data.rules" ); foreach ($snort_files as $file) { if (file_exists("{$snortdir}/{$file}")) @copy("{$snortdir}/{$file}", "{$snortcfgdir}/{$file}");
-
After some investigation it seems that /usr/local/etc/snort/my_snort_sensor/preproc_rules/ was totally empty.
After copying decoder.rules, preprocessor.rules, sensitive-data.rules from /usr/pbi/snort-i386/etc/snort/preproc_rules/ to /usr/local/etc/snort/my_snort_sensor/preproc_rules/ I can now see all those alerts I have to suppress ;)Maybe that was the culprit and I will see if snort keeps running now. Fingers crossed!
I am pretty sure you found the culprit! I was logging on to post my own findings when I read your post above. I can reliably reproduce the "Signal 11" crash in Snort by running with the preproc_rules directory empty. The problem is a quirk in the PBI package on 2.1-BETA (both 32-bit and 64-bit share the same issue). I will work on putting together a fix and get it submitted. Read on below if you are interested in the details.
The new PBI package process in 2.1-BETA places certain "default configuration files" for Snort in /usr/pbi/snort-{arch}/etc where {arch} is either i386 or amd64. It then creates symbolic links to these files in the default regular install path in /usr/local/etc/snort. The problem appears to be that the PBI process incorrectly names the symbolic links in /usr/local/etc/snort/preproc_rules with "sample" as the ending part of the file names. So when the setup code for configuring a Snort interface tries to find the files by their actual names, it does not find them and defaults to the old auto-generation behavior in Snort. The old auto-generation behavior for preprocessor rules appears to have a problem with the http_inspect preprocessor.
Until I can get a patch figured out and submitted for the PBI stuff, here is a quick workaround for anyone having this problem. By the way, you should NOT be seeing this crash IF you are running the Snort VRT rules. They come with the preprocessor rules packaged in them, and the rules get copied to the proper folder on a ruleset update.
WORKAROUND –
1. Locate the correct directory in /usr/local/etc/snort for your affected interface.
2. Copy the rules files (*.rules) in /usr/pbi/snort-{arch}/etc/preproc_rules to the preproc_rules folder underneath the interface directory.Bill
-
Great, thanks for confirming.
I am sure that the snort package will get the attention it needs. ;)
All is well now and I even have set it up with logging to mysql database. -
I did not routinely see the Signal 11 issue because I am a Snort VRT paid subscriber and run the latest 2.9.4.1 rule set. The preprocessor rules are bundled in there and get unpacked and copied for me. That's also why not all 2.1-BETA users were seeing the problem. It is totally rule set dependent.
Bill
-
What a great find!!!