Snort segfaulting on startup w/ http pre-processors enabled



  • Hey all,

    I'm running a version of 2.1 beta on two pfsense boxes and am running into a weird issue. If I have the http preprocessor enabled on snort the system will segfault (11) as seen below.

    
    Jan 25 14:27:12 	kernel: pid 24539 (snort), uid 0: exited on signal 11
    Jan 25 14:27:12 	snort[24539]: Commencing packet processing (pid=24539)
    Jan 25 14:27:12 	snort[24539]: Commencing packet processing (pid=24539)
    Jan 25 14:27:12 	snort[24539]: --== Initialization Complete ==--
    Jan 25 14:27:12 	snort[24539]: --== Initialization Complete ==--
    
    

    here is my version of snort (the latest stable installed)
    Services: Snort 2.9.2.3 pkg v. 2.5.3

    
    [2.1-BETA0][admin@xxx]/root(1): snort -V
    
       ,,_     -*> Snort! <*-
      o"  )~   Version 2.9.2.3 IPv6 GRE (Build 205) FreeBSD
       ''''    By Martin Roesch & The Snort Team: http://www.snort.org/snort/snort-team
               Copyright (C) 1998-2012 Sourcefire, Inc., et al.
               Using libpcap version 1.0.0
               Using PCRE version: 8.32 2012-11-30
               Using ZLIB version: 1.2.3
    
    [2.1-BETA0][admin@xxx]/root(2):
    
    

    Any thoughts? Extra command etc. I'm kinda stumped on this one.

    If I don't have the pre-processor enabled snort will start fine (albeit without useful rules), and will record information.



  • Which exact snapshot of the 2.1-BETA?  I have Snort running on a 2.1-BETA from Tuesday, January 22nd, with no issues and the http_inspect preprocessor is enabled.  I am running this version in a VMware virtual machine.  I can try an update this weekend to see if it breaks my Snort on the VM.

    Have you tried the standard uninstall and then install process for the Snort package?  Don't just click the PKG icon to re-install, click the "X" to remove the package completely, then install it again.



  • I am currently running:

    2.1-BETA0 (amd64)
    built on Wed Aug 15 08:43:33 EDT 2012
    FreeBSD 8.3-RELEASE-p4



  • Also yes, i've removed snort and added it back. More options but still breaks in the same way.


  • Banned

    Update your snapshot…



  • @Supermule:

    Update your snapshot…

    Supermule, that was already on the table. I've looked over the update documentation and have drawn up a plan. It seems to be strait forward. Use the updater inside the web gui/cli and point it at a newer version of the beta from http://snapshots.pfsense.org/

    I was advised to look over any kind of change logs to make sure I wasn't gonna brick the system when I did the update. Any advise on where I can find them. I know its a lot to go though but I'd rather be safe then sorry.


  • Banned

    Just do it…..you are on a very old snapshot and a lot of things have changed.

    And the package is bumped to 2.5.4



  • Do you have an explanation why this is happening AFTER I upgraded to latest snapshot of today and latest snort package (2.9.4.1 v 2.5.4)?
    It worked well before.



  • @gogol:

    Do you have an explanation why this is happening AFTER I upgraded to latest snapshot of today and latest snort package (2.9.4.1 v 2.5.4)?
    It worked well before.

    Do you have any Shared Object (SO) rules enabled?  Will Snort run with ALL rules disabled, but the http_inspect preprocessor enabled?  Will it run WITHOUT the http_inspect preprocessor enabled, but a normal selection of rules?  These questions will help guide further troubleshooting.

    Bill



  • Strangely I'm seeing this "exited on signal 11" error on only one of my 3 boxes.  They're all on the same, latest snap.  I wasn't going to say anything about it until I was able to download the free, registered Snort rules for 2.9.4.1.  Don't want to get into trying to troubleshoot an issue that is possibly caused by the missing Snort.org ruleset and will possibly resolve itself.



  • Glad you are trying to help.

    I have only the ET rules. Snort exits on signal 11 within 10 minutes, sometimes immediately or within a minute when experimenting with other settings, with http_inspect enabled and even no categories selected.
    Snort also crashes with other preprocessors, like only imap, and no http_inspect.

    I have run configuration test (snort -T) with preprocessors enabled and snort.conf was validated successfully (if you like I can post the output).

    The only way I can run snort now, is with categories: botcc, ciarmy, drop, shield, rbi, tor (thus only looking for IP addresses) and enabled portscan detection.



  • I have a 2.1-BETA virtual machine, but it is a few weeks old.  Haven't checked it lately to be sure of its age.  Been working on my 2.0.2 machines for the latest update testing.

    I can try an update to my 2.1-BETA machine and see if I can duplicate the issue.  May be a few days before I can post back, though.  Are all of you having this problem with the latest 2.1-BETA running the 64-bit image?  My VM is a 32-bit machine, so I may need to build me a 64-bit VM as well for testing.

    Bill



  • @bmeeks:

    I have a 2.1-BETA virtual machine, but it is a few weeks old.  Haven't checked it lately to be sure of its age.  Been working on my 2.0.2 machines for the latest update testing.

    I can try an update to my 2.1-BETA machine and see if I can duplicate the issue.  May be a few days before I can post back, though.  Are all of you having this problem with the latest 2.1-BETA running the 64-bit image?  My VM is a 32-bit machine, so I may need to build me a 64-bit VM as well for testing.

    Bill

    I just did that.
    I have a virtual machine ( Parallels desktop 8 ) with imported configuration file and same setup as my main pfSense box where I have the problems. Although this virtual machine is inside my LAN and has other traffic than on my WAN it runs fine with snort HTTP_inspect preprocessor enabled.
    But that is what others have reported: no problems!!
    My main box:
    2.1-BETA1 (i386)
    built on Wed Mar 27 07:29:08 EDT 2013
    snort 2.9.4.1 v 2.5.4

    Thanks for your efforts. I am now going to try to run snort on another adapter on my main machine.



  • OK.  Could be something going on with an underlying driver somewhere.  Strange that http_inspect would trigger it, but weird things can happen in the software world.

    Bill



  • Tested on rl0 interface, which I configured the same as em0 (disabled), so with other driver, but I get another signal 11 exit.
    As far as I can see Snort does not log anything anywhere.

    See also this: http://redmine.pfsense.org/issues/2581



  • I saw this behavior today myself with a 2.1-BETA AMD64 snapshot (March 28 build) in VMware on a virtual machine I was testing.  The VM was running with no rules enabled (I had not selected any yet).  Only the http_inspect preprocessor and three other preprocessors were enabled.  It ran fine for a few minutes, and then next time I looked, the Snort process had exited with Signal 11.  Nothing else in the logs.  I was actually busy on a different VM when the 64-bit VM crashed, so I did not see the crash.  I just noticed it a bit later when I checked the machine again.

    I have no idea why it crashed, but at least I've seen it happen myself.  Have yet to see this on a 32-bit build on 2.0.2 either on real hardware or a VM.  I will check this out over the weekend and see if I can narrow it down to a 2.1 issue or a 64-bit issue.  One thing is for sure, a VM can't have "real hardware failure"; so the Signal 11 has got be a software problem in the binary.  Of course it could be in the Snort code itself (not likely since that is shared across platforms), it could be in the 2.1 kernel code (8.3 versus 8.1), or it could be a 32-bit vs 64-bit thing… ???

    Bill



  • @bmeeks:

    I have no idea why it crashed, but at least I've seen it happen myself.  Have yet to see this on a 32-bit build on 2.0.2 either on real hardware or a VM.  I will check this out over the weekend and see if I can narrow it down to a 2.1 issue or a 64-bit issue.  One thing is for sure, a VM can't have "real hardware failure"; so the Signal 11 has got be a software problem in the binary.  Of course it could be in the Snort code itself (not likely since that is shared across platforms), it could be in the 2.1 kernel code (8.3 versus 8.1), or it could be a 32-bit vs 64-bit thing… ???

    I am running 32 bit (i386) and you saw proof on 64bit. I am not a software engineer, but that suggests it is not a 32bit nor a 64 bit thing, or am I wrong?



  • @gogol:

    I am running 32 bit (i386) and you saw proof on 64bit. I am not a software engineer, but that suggests it is not a 32bit nor a 64 bit thing, or am I wrong?

    What is the max run time you've seen with Snort 2.9.4.1 without a Signal 11?  I've had my test VM up for several hours thus far with no sign of trouble.  It's also running an assortment of Snort VRT and ET rules just for kicks.  Granted it's not getting a lot of traffic to inspect, but neither was the 64-bit VM that crashed on me earlier.  I will continue to run the VMs through the weekend to see if anything turns up.

    Bill



  • @bmeeks:

    What is the max run time you've seen with Snort 2.9.4.1 without a Signal 11?  I've had my test VM up for several hours thus far with no sign of trouble.  It's also running an assortment of Snort VRT and ET rules just for kicks.  Granted it's not getting a lot of traffic to inspect, but neither was the 64-bit VM that crashed on me earlier.  I will continue to run the VMs through the weekend to see if anything turns up.

    The run time varied between several seconds and about 10 minutes. I don't know if some traffic caused a signal 11 exit, because I never saw an alert.
    I started from scratch and an empty suppress list and I can remember from previous version that I got a lot of these messages almost immediately:
    (http_inspect) NO CONTENT-LENGTH OR TRANSFER-ENCODING IN HTTP RESPONSE  120:3:1



  • After some investigation it seems that /usr/local/etc/snort/my_snort_sensor/preproc_rules/ was totally empty.
    After copying decoder.rules, preprocessor.rules, sensitive-data.rules from /usr/pbi/snort-i386/etc/snort/preproc_rules/ to /usr/local/etc/snort/my_snort_sensor/preproc_rules/ I can now see all those alerts I have to suppress ;)

    Maybe that was the culprit and I will see if snort keeps running now. Fingers crossed!



  • As far as I can see now snort runs fine again.
    But why are those missing files (see my previous post) not copied to the right directory?

    Can someone have a look at this in snort.inc (start at line 1901). It is a guess because I have no programming skills:

    /* create a few directories and ensure the sample files are in place */
    	$snort_dirs = array( $snortdir, $snortcfgdir, "{$snortcfgdir}/rules",
    		"{$snortlogdir}/snort_{$if_real}{$snort_uuid}",
    		"{$snortlogdir}/snort_{$if_real}{$snort_uuid}/barnyard2", 
    		"{$snortcfgdir}/preproc_rules", 
    		"dynamicrules" => "/usr/local/lib/snort/dynamicrules",
    		"dynamicengine" => "/usr/local/lib/snort/dynamicengine",
    		"dynamicpreprocessor" => "{$snortcfgdir}/dynamicpreprocessor"
    	    );
    	foreach ($snort_dirs as $dir) {
    		if (!is_dir($dir))
    			safe_mkdir($dir);
    	}
    
    	$snort_files = array("gen-msg.map", "classification.config", "reference.config",
    			"sid-msg.map", "unicode.map", "threshold.conf", "preproc_rules/preprocessor.rules",
    			"preproc_rules/decoder.rules", "preproc_rules/sensitive-data.rules"
    		);
    	foreach ($snort_files as $file) {
    		if (file_exists("{$snortdir}/{$file}"))
    			@copy("{$snortdir}/{$file}", "{$snortcfgdir}/{$file}");
    


  • @gogol:

    After some investigation it seems that /usr/local/etc/snort/my_snort_sensor/preproc_rules/ was totally empty.
    After copying decoder.rules, preprocessor.rules, sensitive-data.rules from /usr/pbi/snort-i386/etc/snort/preproc_rules/ to /usr/local/etc/snort/my_snort_sensor/preproc_rules/ I can now see all those alerts I have to suppress ;)

    Maybe that was the culprit and I will see if snort keeps running now. Fingers crossed!

    I am pretty sure you found the culprit!  I was logging on to post my own findings when I read your post above.  I can reliably reproduce the "Signal 11" crash in Snort by running with the preproc_rules directory empty.  The problem is a quirk in the PBI package on 2.1-BETA (both 32-bit and 64-bit share the same issue).  I will work on putting together a fix and get it submitted.  Read on below if you are interested in the details.

    The new PBI package process in 2.1-BETA places certain "default configuration files" for Snort in /usr/pbi/snort-{arch}/etc where {arch} is either i386 or amd64.  It then creates symbolic links to these files in the default regular install path in /usr/local/etc/snort.  The problem appears to be that the PBI process incorrectly names the symbolic links in /usr/local/etc/snort/preproc_rules with "sample" as the ending part of the file names.  So when the setup code for configuring a Snort interface tries to find the files by their actual names, it does not find them and defaults to the old auto-generation behavior in Snort.  The old auto-generation behavior for preprocessor rules appears to have a problem with the http_inspect preprocessor.

    Until I can get a patch figured out and submitted for the PBI stuff, here is a quick workaround for anyone having this problem.  By the way, you should NOT be seeing this crash IF you are running the Snort VRT rules.  They come with the preprocessor rules packaged in them, and the rules get copied to the proper folder on a ruleset update.

    WORKAROUND –

    1.  Locate the correct directory in /usr/local/etc/snort for your affected interface.
    2.  Copy the rules files (*.rules) in /usr/pbi/snort-{arch}/etc/preproc_rules to the preproc_rules folder underneath the interface directory.

    Bill



  • Great, thanks for confirming.
    I am sure that the snort package will get the attention it needs. ;)
    All is well now and I even have set it up with logging to mysql database.



  • I did not routinely see the Signal 11 issue because I am a Snort VRT paid subscriber and run the latest 2.9.4.1 rule set.  The preprocessor rules are bundled in there and get unpacked and copied for me.  That's also why not all 2.1-BETA users were seeing the problem.  It is totally rule set dependent.

    Bill


  • Banned

    What a great find!!!


Locked