Snort crashed roughly shortly after startup
-
The only guess I have at this point is some rule is being triggered that kills Snort. For troubleshooting, you can try disabling all of your current rule categories and enable them one-by-one, let each one run a few hours, then rinse and repeat until you get the crash. The last category enabled would likely then contain the "crashing" rule.
First I would just let Snort run with no extra rules enabled to see if it is stable. You likely have something else going on in your system, though. There are many Snort users on pfSense, and most have a ton of rules enabled and they are not reporting crashes like this.
Bill
-
The only thing I can think of, maybe, because of the timing, is the NUT package for my UPS. I have it modified where it will be able to text msg me if my setup goes on battery power - however the modification causes the app to crash every 15 minutes so I have a crontab setup to restart the process for NUT every 15 minutes…
Let me see what happens if I simply disable all the rules.
-
I disabled all rules - every single one; restarted snort - it died about a minute later:
May 21 11:39:07 snort[39235]: Commencing packet processing (pid=39235) May 21 11:40:00 snort[39235]: *** Opening /var/log/snort/snort_em021983/app-stats.log.1432233600 for output May 21 11:40:07 kernel: pid 39235 (snort), uid 0: exited on signal 11 May 21 11:40:07 kernel: em0: promiscuous mode disabled May 21 11:45:00 upsmon[95418]: Signal 15: exiting May 21 11:45:00 upsd[92641]: mainloop: Interrupted system call May 21 11:45:00 upsd[92641]: Signal 15: exitingthe UPSMON/UPSD processes are for NUT; as you can see, though, that one died 5 minutes after snort. I'll try to disable NUT and see what happens.
-
Nope - disabling NUT and all the modified packages did not do the trick :'(
I am at a loss now :(
-
If you have the OpenAppID enabled, try to disable that and see if that helps..
-
If you have the OpenAppID enabled, try to disable that and see if that helps..
No bueno :(
-
If you run this command from the shell does it show any duplicate PIDS for Snort. It should only have one process per Snort Interface…
So maybe its not starting due to a existing Snort process...ps aux | grep snort
You can kill the pid with kill -9 <pid #="">You can also, completely remove Snort and Re-install it.... Just make sure to "unclick" Keep settings in the "Global" Tab to clear all existing settings...</pid>
-
:sigh:
I just re-installed snort (after unchecking the 'keep settings' box) - still crapped out about 2 minutes later :(
I guess I am not destined to have snort after all :\
-
My only theory at this point is maybe the libpcap library is having issues with your NIC driver. That's just a guess, though. There are lots of Snort users on pfSense with no issues, so I know the code is fundamentally sound. That's not to say it may not have issues with some hardware, though.
What brand of NIC is in use on your firewall? Also, can you post the output of this command run from the firewall console –
snort -VThat command should print some version information and then exit. I'm particularly interested in what it shows for the pcap library version.
Bill
-
snort -V output:
,,_ -*> Snort! <*- o" )~ Version 2.9.7.2 GRE (Build 177) FreeBSD '''' By Martin Roesch & The Snort Team: http://www.snort.org/contact#team Copyright (C) 2014 Cisco and/or its affiliates. All rights reserved. Copyright (C) 1998-2013 Sourcefire, Inc., et al. Using libpcap version 1.6.2 Using PCRE version: 8.35 2014-04-04 Using ZLIB version: 1.2.8The NICs I have are Intel 82574L; this is on a Supermicro MBD-X7SPE-H-D525-O motherboard (embedded dual core Atom D525).
-
Well, nothing unusual or unexpected in the version information. It's all like it should be.
This is especially vexing since it appears to die even with no rules selected. It's also weird that the death seems to happen on radically different time intervals. You reported a 2-minute run time and then an almost 16-minute run time.
Have you run an extensive test on your system RAM to rule out a potential memory problem?
Bill
-
Well, nothing unusual or unexpected in the version information. It's all like it should be.
This is especially vexing since it appears to die even with no rules selected. It's also weird that the death seems to happen on radically different time intervals. You reported a 2-minute run time and then an almost 16-minute run time.
Have you run an extensive test on your system RAM to rule out a potential memory problem?
Bill
I haven't done that, no; didn't really see a reason to do that considering the box has been up and running for roughly 3 years now with about 4 days of total downtime… I have pfBlockerNG and OpenVPN running on it without any issues if that means anything. I suppose I can test the RAM but I'd rather not take the box down if I can help it.
-
I use the OpenVPN, Snort and apcuspd packages on my production box with no issues. I only mentioned a RAM test because Snort can use a lot of RAM and might be the only application "tickling" a particular bank of high RAM (just a guess, though).
Do you have any messages in your system log about packages restarting? You don't mention it, so I assume you don't, but are you using the Service Watchdog package with Snort?
Bill
-
The only other package that restarts, as I mentioned in earlier posts, is NUT - but that is per design due to the customization that I've set up. The package restarts every 15 minutes through a cron job.
I do not have Service Watchdog installed; I'd imagine it will wreak havoc with the snort package :(
-
Swapped RAM for kicks - same thing :'(
-
The RAM guess was a longshot. You are correct that Snort and the Service Watchdog package don't play well together.
I am really stumped by your problem. Have you tried the Suricata package? It will do essentially the same thing as Snort. The look and feel of the two packages are identical. They share tons of the same PHP code for the GUI.
Bill
-
The RAM guess was a longshot. You are correct that Snort and the Service Watchdog package don't play well together.
I am really stumped by your problem. Have you tried the Suricata package? It will do essentially the same thing as Snort. The look and feel of the two packages are identical. They share tons of the same PHP code for the GUI.
Bill
Hrm… I guess I can give that a shot; see what happens. I'll try and report back.
-
Update: Odd things are happening…!
So I decided to install Suricata to try it out. After installing it, I meant to start configuring it but accidentally clicked on snort as opposed to suricata from the services menu. Imagine my surprise when I noticed that snort was actually running...! I tried looking at system logs but considering the log gets rotated quickly, I was not able to find when it started.
I decided to enable categories/rules and let it run - 20 minutes later, it was still running. I then decided to uninstall suricata figuring snort fixed itself however about 10 minutes after uninstalling, snort crapped out!
So - I re-enabled snort, but it yet again crapped out about 5 minutes later. This got me thinking - I installed Suricata but did not configure it (just as last time), enabled snort and - it has been running for the past two hours. I've then uninstalled Suricata - but snort has not crashed as of yet. Will let it run and see how it behaves.
Insanely odd behavior!
-
Perhaps my happiness was pre-emptive: I enabled AppID, restarted snort and now the interface died relatively quickly.
I will try the suricata trick.
edit: did not work… let's see what happens if OpenAppID is off...
edit2: nope, disabling OpenAppID did not work either.
Back to square one - how is it that things worked fine for several hours and once OpenAppID was enabled, broke, and not that it is disabled, still broken? I guess suricata was just a coincidence.
Edit 3: Decided to disable openappID, reinstall suricata and then enable snort interface - works! Enabled openappID - works! Forced update (cleared MD5 hashes) - works! So far it has been running for the past 2 hours with no issues... but Suricata is still installed (albeit not running).
-
Welps - with openappid, snort crapped out about 2 hours after being fired up. Will try a lengthy test with AppID off.
