Snort segfaulting on startup w/ http pre-processors enabled
-
Hey all,
I'm running a version of 2.1 beta on two pfsense boxes and am running into a weird issue. If I have the http preprocessor enabled on snort the system will segfault (11) as seen below.
Jan 25 14:27:12 kernel: pid 24539 (snort), uid 0: exited on signal 11 Jan 25 14:27:12 snort[24539]: Commencing packet processing (pid=24539) Jan 25 14:27:12 snort[24539]: Commencing packet processing (pid=24539) Jan 25 14:27:12 snort[24539]: --== Initialization Complete ==-- Jan 25 14:27:12 snort[24539]: --== Initialization Complete ==--
here is my version of snort (the latest stable installed)
Services: Snort 2.9.2.3 pkg v. 2.5.3[2.1-BETA0][admin@xxx]/root(1): snort -V ,,_ -*> Snort! <*- o" )~ Version 2.9.2.3 IPv6 GRE (Build 205) FreeBSD '''' By Martin Roesch & The Snort Team: http://www.snort.org/snort/snort-team Copyright (C) 1998-2012 Sourcefire, Inc., et al. Using libpcap version 1.0.0 Using PCRE version: 8.32 2012-11-30 Using ZLIB version: 1.2.3 [2.1-BETA0][admin@xxx]/root(2):
Any thoughts? Extra command etc. I'm kinda stumped on this one.
If I don't have the pre-processor enabled snort will start fine (albeit without useful rules), and will record information.
-
Which exact snapshot of the 2.1-BETA? I have Snort running on a 2.1-BETA from Tuesday, January 22nd, with no issues and the http_inspect preprocessor is enabled. I am running this version in a VMware virtual machine. I can try an update this weekend to see if it breaks my Snort on the VM.
Have you tried the standard uninstall and then install process for the Snort package? Don't just click the PKG icon to re-install, click the "X" to remove the package completely, then install it again.
-
I am currently running:
2.1-BETA0 (amd64)
built on Wed Aug 15 08:43:33 EDT 2012
FreeBSD 8.3-RELEASE-p4 -
Also yes, i've removed snort and added it back. More options but still breaks in the same way.
-
Update your snapshot…
-
Update your snapshot…
Supermule, that was already on the table. I've looked over the update documentation and have drawn up a plan. It seems to be strait forward. Use the updater inside the web gui/cli and point it at a newer version of the beta from http://snapshots.pfsense.org/
I was advised to look over any kind of change logs to make sure I wasn't gonna brick the system when I did the update. Any advise on where I can find them. I know its a lot to go though but I'd rather be safe then sorry.
-
Just do it…..you are on a very old snapshot and a lot of things have changed.
And the package is bumped to 2.5.4
-
Do you have an explanation why this is happening AFTER I upgraded to latest snapshot of today and latest snort package (2.9.4.1 v 2.5.4)?
It worked well before. -
Do you have an explanation why this is happening AFTER I upgraded to latest snapshot of today and latest snort package (2.9.4.1 v 2.5.4)?
It worked well before.Do you have any Shared Object (SO) rules enabled? Will Snort run with ALL rules disabled, but the http_inspect preprocessor enabled? Will it run WITHOUT the http_inspect preprocessor enabled, but a normal selection of rules? These questions will help guide further troubleshooting.
Bill
-
Strangely I'm seeing this "exited on signal 11" error on only one of my 3 boxes. They're all on the same, latest snap. I wasn't going to say anything about it until I was able to download the free, registered Snort rules for 2.9.4.1. Don't want to get into trying to troubleshoot an issue that is possibly caused by the missing Snort.org ruleset and will possibly resolve itself.
-
Glad you are trying to help.
I have only the ET rules. Snort exits on signal 11 within 10 minutes, sometimes immediately or within a minute when experimenting with other settings, with http_inspect enabled and even no categories selected.
Snort also crashes with other preprocessors, like only imap, and no http_inspect.I have run configuration test (snort -T) with preprocessors enabled and snort.conf was validated successfully (if you like I can post the output).
The only way I can run snort now, is with categories: botcc, ciarmy, drop, shield, rbi, tor (thus only looking for IP addresses) and enabled portscan detection.
-
I have a 2.1-BETA virtual machine, but it is a few weeks old. Haven't checked it lately to be sure of its age. Been working on my 2.0.2 machines for the latest update testing.
I can try an update to my 2.1-BETA machine and see if I can duplicate the issue. May be a few days before I can post back, though. Are all of you having this problem with the latest 2.1-BETA running the 64-bit image? My VM is a 32-bit machine, so I may need to build me a 64-bit VM as well for testing.
Bill
-
I have a 2.1-BETA virtual machine, but it is a few weeks old. Haven't checked it lately to be sure of its age. Been working on my 2.0.2 machines for the latest update testing.
I can try an update to my 2.1-BETA machine and see if I can duplicate the issue. May be a few days before I can post back, though. Are all of you having this problem with the latest 2.1-BETA running the 64-bit image? My VM is a 32-bit machine, so I may need to build me a 64-bit VM as well for testing.
Bill
I just did that.
I have a virtual machine ( Parallels desktop 8 ) with imported configuration file and same setup as my main pfSense box where I have the problems. Although this virtual machine is inside my LAN and has other traffic than on my WAN it runs fine with snort HTTP_inspect preprocessor enabled.
But that is what others have reported: no problems!!
My main box:
2.1-BETA1 (i386)
built on Wed Mar 27 07:29:08 EDT 2013
snort 2.9.4.1 v 2.5.4Thanks for your efforts. I am now going to try to run snort on another adapter on my main machine.
-
OK. Could be something going on with an underlying driver somewhere. Strange that http_inspect would trigger it, but weird things can happen in the software world.
Bill
-
Tested on rl0 interface, which I configured the same as em0 (disabled), so with other driver, but I get another signal 11 exit.
As far as I can see Snort does not log anything anywhere.See also this: http://redmine.pfsense.org/issues/2581
-
I saw this behavior today myself with a 2.1-BETA AMD64 snapshot (March 28 build) in VMware on a virtual machine I was testing. The VM was running with no rules enabled (I had not selected any yet). Only the http_inspect preprocessor and three other preprocessors were enabled. It ran fine for a few minutes, and then next time I looked, the Snort process had exited with Signal 11. Nothing else in the logs. I was actually busy on a different VM when the 64-bit VM crashed, so I did not see the crash. I just noticed it a bit later when I checked the machine again.
I have no idea why it crashed, but at least I've seen it happen myself. Have yet to see this on a 32-bit build on 2.0.2 either on real hardware or a VM. I will check this out over the weekend and see if I can narrow it down to a 2.1 issue or a 64-bit issue. One thing is for sure, a VM can't have "real hardware failure"; so the Signal 11 has got be a software problem in the binary. Of course it could be in the Snort code itself (not likely since that is shared across platforms), it could be in the 2.1 kernel code (8.3 versus 8.1), or it could be a 32-bit vs 64-bit thing… ???
Bill
-
I have no idea why it crashed, but at least I've seen it happen myself. Have yet to see this on a 32-bit build on 2.0.2 either on real hardware or a VM. I will check this out over the weekend and see if I can narrow it down to a 2.1 issue or a 64-bit issue. One thing is for sure, a VM can't have "real hardware failure"; so the Signal 11 has got be a software problem in the binary. Of course it could be in the Snort code itself (not likely since that is shared across platforms), it could be in the 2.1 kernel code (8.3 versus 8.1), or it could be a 32-bit vs 64-bit thing… ???
I am running 32 bit (i386) and you saw proof on 64bit. I am not a software engineer, but that suggests it is not a 32bit nor a 64 bit thing, or am I wrong?
-
I am running 32 bit (i386) and you saw proof on 64bit. I am not a software engineer, but that suggests it is not a 32bit nor a 64 bit thing, or am I wrong?
What is the max run time you've seen with Snort 2.9.4.1 without a Signal 11? I've had my test VM up for several hours thus far with no sign of trouble. It's also running an assortment of Snort VRT and ET rules just for kicks. Granted it's not getting a lot of traffic to inspect, but neither was the 64-bit VM that crashed on me earlier. I will continue to run the VMs through the weekend to see if anything turns up.
Bill
-
What is the max run time you've seen with Snort 2.9.4.1 without a Signal 11? I've had my test VM up for several hours thus far with no sign of trouble. It's also running an assortment of Snort VRT and ET rules just for kicks. Granted it's not getting a lot of traffic to inspect, but neither was the 64-bit VM that crashed on me earlier. I will continue to run the VMs through the weekend to see if anything turns up.
The run time varied between several seconds and about 10 minutes. I don't know if some traffic caused a signal 11 exit, because I never saw an alert.
I started from scratch and an empty suppress list and I can remember from previous version that I got a lot of these messages almost immediately:
(http_inspect) NO CONTENT-LENGTH OR TRANSFER-ENCODING IN HTTP RESPONSE 120:3:1 -
After some investigation it seems that /usr/local/etc/snort/my_snort_sensor/preproc_rules/ was totally empty.
After copying decoder.rules, preprocessor.rules, sensitive-data.rules from /usr/pbi/snort-i386/etc/snort/preproc_rules/ to /usr/local/etc/snort/my_snort_sensor/preproc_rules/ I can now see all those alerts I have to suppress ;)Maybe that was the culprit and I will see if snort keeps running now. Fingers crossed!