Snort down for a couple of days for no obvious reason. Up now. Any idea why?
-
Snort was down for a couple of days … WAN was not active. Had I not looked, it might have been down for longer. (yellow x)
It went down for no obvious reason. I've been using it for a couple of years. Restart had no effect. Update packages had no effect. A reboot might have fixed it. I followed the reboot with a restart click on the interface and it came back to life.
Any ideas what happened?
edit: a look at the log stated it failed shortly after midnight 1-11. Perhaps after an update?
-
Could have died on an update. There is also a segfault bug that was fixed in the stream preprocessor in the newest 2.9.9.0 version recently released by the Snort team. Right now pfSense (and FreeBSD ports last time I checked) are still using 2.9.8.3 which has the stream processor bug. If something caused that bug to be "hit", that could account for the Snort daemon dying.
I've posted in a few other threads that I will update the pfSense Snort package as soon as the FreeBSD port is updated by the maintainer there.
Bill
-
thanks.
-
it's down again. playing around with fixing it from my end. something is broken in snort / pfsense. Going to turn it off for a couple of days.
edit.: did manual force update. It's back up.
Been having problems streamimg Hulu very recently. Probably no connection with snort broken and Hulu stuttering, but only a coincidence. Home network is 150/20 (actual 180/25) and light demand, wired to access point.
Just changed update interval to 4 days. It went down again after midnight update a couple of days ago. I have annual subscription so 4 day old rules shouldn't be end of the world.
-
I really don't know what your problem might be. I have been running the Snort package for years. I've been running the 2.9.8.3 version without issues since it was released. I use the Snort VRT rules and the Emerging Threats Open rules. For my LAN I use the Snort VRT "Balanced" IPS Policy rules along with a handful of the ET malware and trojan rules. On the WAN I run a couple of the ET CIARMY and similar rules just to generate some log noise so I know Snort is there.
I have not had a Snort stoppage in months. I think I had maybe two last year, if that many. My rules update runs every 12 hours at 1:30 AM and 1:30 PM US Eastern Time.
I'm not denying folks might be having Snort stopping unexpectedly, but that is something I have not been able to reproduce either in my virtual machine testing environment nor on my personal firewall. If it matters, my firewall obviously sees very light traffic since it is a home network. The only other packages I have loaded are OpenVPN Client and NUT (for my UPS).
Have you examined the system log carefully to see what other messages are logged around the times you see Snort die? For example, is your WAN IP address perhaps getting refreshed via a DHCP renew? That process will restart packages, and if it issues multiple Snort restart commands in very quick succession this can cause the deamon to die.
Bill
-
bill,
it appears to die right after update, which is set for midnight and is automatic. This has happened twice. Force update appeared to work fine. This is new as it has worked fine for a couple of years. The first failure was timed with my first comment above. I may have applied an update to snort, but I am not sure of the timing and the 1st incident. Snort is still up today. I will force update until a new version is released.
I'm also on a home network and in all actuality probably don't need it as my only open ports are for OpenVPN, and each server is highly protected with certificates. I think of snort as a form of antivirus so intend to keep using it. My home is battered daily with snort attempts and pfBlockerNG is battered even more by a large factor. Snort is hundreds a day, pfBlockerNG CAN BE hundreds per hour. I'm selective about snort suppression - usually by IP address and mostly akamai being unblocked.
Also, what is the yellow snort icon? Red is off and green is on. Yellow is ??.
The Hulu issue was only coincidental and may be / probably is unrelated although it mostly works great when snort isn't acting up. Again, probably unrelated.
-
This has happened to me when one of the rules is bad…for example, the openappid file_transfer rule is broken, it is missing the ";)" at the end of the rules line so if I select that rule snort will stop but it does complain in the logs about it. I have manually fixed the line in the rules but when it updates it will get the bad rule again and stop. Have you checked your logs?
-
Also, I would definitely avoid running the update at midnight. Way to many things going on at that time that can interfere.
-
Also, I would definitely avoid running the update at midnight. Way to many things going on at that time that can interfere.
I'll try it with a change in update time although it's always midnight every hour somewhere isn't it?
Edit: just did forced update. Interface said complete. Yellow icon came back. Going to disable snort on WAN for now.
-
Uhm, I meant the midnight local time and the local cronjobs (though, running update exactly on the hour is more likely to hit update server issues as well).
-
Uhm, I meant the midnight local time and the local cronjobs (though, running update exactly on the hour is more likely to hit update server issues as well).
Thanks. I got it, just having fun. Changed to 00:45.
Even weirder, I stopped WAN interface but checked alerts a few minutes later. Snort WAN Disabled but blocks and alerts still collecting. I thought they would stop. Cleared all saved blocks and alerts but they still came on coming. Started interface again and immediately showed green. Looks odd to me.
-
Uhm, I meant the midnight local time and the local cronjobs (though, running update exactly on the hour is more likely to hit update server issues as well).
Thanks. I got it, just having fun. Changed to 00:45.
Even weirder, I stopped WAN interface but checked alerts a few minutes later. Snort WAN Disabled but blocks and alerts still collecting. I thought they would stop. Cleared all saved blocks and alerts but they still came on coming. Started interface again and immediately showed green. Looks odd to me.
That sounds like you have two Snort processes running on the same interface. This can randomly happen if several "restart all packages" commands get executed by pfSense for various reasons (DHCP renew on WAN is one way). To see, check the output of this command:
ps -ax |snortIf you have Snort stopped, then you should see no processes listed. I'm betting you will see one. If you do, kill it by PID or else reboot the firewall if that is easier.
Bill
-
bill,
thanks for your attention on this. I will reboot tomorrow when nobody's on it. Easy and fast.
-
reboot did not help. Uninstall / reinstall appears to work. Green icon after an auto-update, not yellow.
Upon uninstall, one message was 'removed old package update leftovers' or something like that. I wonder if it was cluttered and that caused it to get flaky? I wonder this also because I performed an update to the package and the problems started. Perhaps the current version was interacting with some ancient leftover flotsam?
Again, thanks for the replies above.
Edit:
Spoke too soon. Yellow icon back after suppressing an entry that blocked simple downloads on port 80.
Removing package until new version released for next pfSense update.
FYI to developer: simple router - only pfBlockerNG and OpenVPN client download packages loaded. J1900 based with 120GB SSD and 8GB RAM. Intel ports.
-
Played around with it and figured it out - I think. Problems with reliability are consistent with snort openapppid feature beings turned on and off. I really don't know what it does but I thought I would turn it on a few days ago. A package update coincided with it being turn on, more or less. When on, problems developed. When off, snort worked. After turning it back on, problems came back. Checked off in snort now and snort works as always before these issues developed.
openappid appears to be a new feature. I don't understand it and no documentation exists that tells a non-expert how to use it or even what it does. Problems are gone with it off.
-
The OpenAppID feaure was added by the Snort VRT about 2 or 3 years ago if I recall correctly. Shortly after it was introduced I incorporated support for configuring it within the pfSense Snort package. However, there is much more to using OpenAppID than simply checking the box in the GUI. You must create your own custom rules to actually implement Application ID inspection. There are a critical set of OpenAppID stems that come from the Snort VRT via the updates, but they are not all that you need to actually implement OpenAppID. So if you enabled the feature without also creating the necessary custom rules for traffic inspection, it is actually doing nothing.
There have been several reports of errors within the OpenAppID stems that are packaged in the Snort VRT signature updates. Unfortunately with Snort, when it encounters any kind of syntax error in rules or other items it is loading, it will error out and quit. Suricata will log an error, but then skip the errant rule and continue loading the others. So what is likely happening with OpenAppID enabled is Snort hits one of those random errors that seem to get into the OpenAppID stems update and quits. Because Snort is so terribly chatty and fills the system log with essentially every action it takes when you enable normal logging, the pfSense package always starts Snort with the "quiet" switch to cut down on all the log noise as Snort starts. You can disable this feature and turn on the verbose logging by toggling a parameter on the GLOBAL SETTINGS tab.
Here is how I think this might be happening to you. Enabling the OpenAppID preprocessor will cause Snort to load that piece of code and to download the OpenAppID stem updates along with the regular VRT rules update. Snort will then start to load and process the updated files. If OpenAppID is enabled, and the OpenAppID stem files have any errors in them, Snort will log the error and die. The error will only show up in the system log on pfSense if you have turned on verbose Snort logging (that GLOBAL SETTINGS parameter I mentioned earlier). So if Snort encounters an error in the rules or OpenAppID updates, it will just seemingly die for no reason when the "quiet" switch is enabled. As I mentioned, using the "quiet" switch is the default on pfSense otherwise you get several hundred lines of Snort start-up text in the system log.
Bill
