Snort crashed roughly shortly after startup
-
Hello!
Just set up snort to mimic a setup very close to a buddy of mine. However, on my end, Snort crashed anywhere from 90 seconds to 10 minutes after startup with the following cryptic message:
kernel: pid 37362 (snort), uid 0: exited on signal 11Screenshot of logs:
All my memory allocation settings are at default; search method is AC-BNFA-NQ (though AC and AC-BNFA has had the same results).
Hardware specs are: Atom D525 dual core; 4 GB of RAM, 50 GB SSD.
-
Can you provide the pfSense version you are currently running? This one may be tough to diagnose if that is all there is in the system log. Make sure you never use anything but AC-BNFA or AC-BNFA-NQ for the pattern matcher setting. The other settings can cause memory to be eaten at a fierce rate (particularly the AC setting).
Bill
-
Update - looks like things are working better after disabling hardware checksum offload (this is on 2.2.2 btw). Memory utilization is about 66% of my 4 gigs.
Currently I have it set for AC-STD; with 4 gigs of RAM and the D525 - would that be the best option? I also have all the memory settings in the Preprocessors and Flow tab set to default - keep it that way?
-
Update - looks like things are working better after disabling hardware checksum offload (this is on 2.2.2 btw). Memory utilization is about 66% of my 4 gigs.
Currently I have it set for AC-STD; with 4 gigs of RAM and the D525 - would that be the best option? I also have all the memory settings in the Preprocessors and Flow tab set to default - keep it that way?
No, I recommend no settings other than AC-BNFA or AC-BNFA-NQ. Several folks have tried the others and seem to inevitably run into trouble. Also, some testing by users here and elsewhere has shown there really is no noticeable difference in performance, only more memory used (and in unpredictable ways that can lead to crashes).
Bill
-
Alas - my happiness was shortlived; snort crapped out 15 minutes and 8 seconds after being launched after disabling hardware checksum offload. I have reverted to AC-BNFA; will report shortly.
Edit: changing to AC-BNFA had no effect; snort died after 16 minutes this time.
-
The only guess I have at this point is some rule is being triggered that kills Snort. For troubleshooting, you can try disabling all of your current rule categories and enable them one-by-one, let each one run a few hours, then rinse and repeat until you get the crash. The last category enabled would likely then contain the "crashing" rule.
First I would just let Snort run with no extra rules enabled to see if it is stable. You likely have something else going on in your system, though. There are many Snort users on pfSense, and most have a ton of rules enabled and they are not reporting crashes like this.
Bill
-
The only thing I can think of, maybe, because of the timing, is the NUT package for my UPS. I have it modified where it will be able to text msg me if my setup goes on battery power - however the modification causes the app to crash every 15 minutes so I have a crontab setup to restart the process for NUT every 15 minutes…
Let me see what happens if I simply disable all the rules.
-
I disabled all rules - every single one; restarted snort - it died about a minute later:
May 21 11:39:07 snort[39235]: Commencing packet processing (pid=39235) May 21 11:40:00 snort[39235]: *** Opening /var/log/snort/snort_em021983/app-stats.log.1432233600 for output May 21 11:40:07 kernel: pid 39235 (snort), uid 0: exited on signal 11 May 21 11:40:07 kernel: em0: promiscuous mode disabled May 21 11:45:00 upsmon[95418]: Signal 15: exiting May 21 11:45:00 upsd[92641]: mainloop: Interrupted system call May 21 11:45:00 upsd[92641]: Signal 15: exitingthe UPSMON/UPSD processes are for NUT; as you can see, though, that one died 5 minutes after snort. I'll try to disable NUT and see what happens.
-
Nope - disabling NUT and all the modified packages did not do the trick :'(
I am at a loss now :(
-
If you have the OpenAppID enabled, try to disable that and see if that helps..
-
If you have the OpenAppID enabled, try to disable that and see if that helps..
No bueno :(
-
If you run this command from the shell does it show any duplicate PIDS for Snort. It should only have one process per Snort Interface…
So maybe its not starting due to a existing Snort process...ps aux | grep snort
You can kill the pid with kill -9 <pid #="">You can also, completely remove Snort and Re-install it.... Just make sure to "unclick" Keep settings in the "Global" Tab to clear all existing settings...</pid>
-
:sigh:
I just re-installed snort (after unchecking the 'keep settings' box) - still crapped out about 2 minutes later :(
I guess I am not destined to have snort after all :\
-
My only theory at this point is maybe the libpcap library is having issues with your NIC driver. That's just a guess, though. There are lots of Snort users on pfSense with no issues, so I know the code is fundamentally sound. That's not to say it may not have issues with some hardware, though.
What brand of NIC is in use on your firewall? Also, can you post the output of this command run from the firewall console –
snort -VThat command should print some version information and then exit. I'm particularly interested in what it shows for the pcap library version.
Bill
-
snort -V output:
,,_ -*> Snort! <*- o" )~ Version 2.9.7.2 GRE (Build 177) FreeBSD '''' By Martin Roesch & The Snort Team: http://www.snort.org/contact#team Copyright (C) 2014 Cisco and/or its affiliates. All rights reserved. Copyright (C) 1998-2013 Sourcefire, Inc., et al. Using libpcap version 1.6.2 Using PCRE version: 8.35 2014-04-04 Using ZLIB version: 1.2.8The NICs I have are Intel 82574L; this is on a Supermicro MBD-X7SPE-H-D525-O motherboard (embedded dual core Atom D525).
-
Well, nothing unusual or unexpected in the version information. It's all like it should be.
This is especially vexing since it appears to die even with no rules selected. It's also weird that the death seems to happen on radically different time intervals. You reported a 2-minute run time and then an almost 16-minute run time.
Have you run an extensive test on your system RAM to rule out a potential memory problem?
Bill
-
Well, nothing unusual or unexpected in the version information. It's all like it should be.
This is especially vexing since it appears to die even with no rules selected. It's also weird that the death seems to happen on radically different time intervals. You reported a 2-minute run time and then an almost 16-minute run time.
Have you run an extensive test on your system RAM to rule out a potential memory problem?
Bill
I haven't done that, no; didn't really see a reason to do that considering the box has been up and running for roughly 3 years now with about 4 days of total downtime… I have pfBlockerNG and OpenVPN running on it without any issues if that means anything. I suppose I can test the RAM but I'd rather not take the box down if I can help it.
-
I use the OpenVPN, Snort and apcuspd packages on my production box with no issues. I only mentioned a RAM test because Snort can use a lot of RAM and might be the only application "tickling" a particular bank of high RAM (just a guess, though).
Do you have any messages in your system log about packages restarting? You don't mention it, so I assume you don't, but are you using the Service Watchdog package with Snort?
Bill
-
The only other package that restarts, as I mentioned in earlier posts, is NUT - but that is per design due to the customization that I've set up. The package restarts every 15 minutes through a cron job.
I do not have Service Watchdog installed; I'd imagine it will wreak havoc with the snort package :(
-
Swapped RAM for kicks - same thing :'(
