Snort 2.9.2.3 pkg v. 2.5.4 killed by World of Tanks

Fesoj

Hi,

Snort crashes on one of by boxes (guest access in a hotel) quite frequently. It seems to be related to one of the patrons playing World of Tanks, i.e. Snort never crashes when World of Tanks sites do not show up in sarg reports for any given day. I am running Snort 2.9.2.3 pkg v. 2.5.4 and the crashes are independent from the "block offenders" settings. So far I have not found anything relevant in the log files.

Has anybody seen this as well?

Sorry for asking here, because if it is Snort itself, then this would be the wrong place to ask this type of question.

bmeeks

Couple of possibilities:

(1) Make sure you have the latest binary package and the GUI components. Easiest way to make 100% sure is to remove the package completely and then reinstall it. Click the "X" to remove, then go to the "Available Packages" tab and install fresh. DO NOT simply click the XML icon on the "Installed Packages" tab to do a reinstall. That is not always dependable and can result in a mixed bag of files and problems starting and/or running. After the reinstall completes, go to the "Updates" tab in Snort and force a download of new rules.

(2) My guess is a particular rule is firing and that rule is crashing Snort. In the past, prior to some changes that were made in the GUI components, there could be issues with the classification.config file not being correct for a given rule set if you are using both ET and VRT rule packages. A rule that referenced a classification that was not in the file would generate a crash. This was fixed in changes starting with 2.5.3. There are also potentially some preprocessor issues that could be at fault, but more often preprocessor issues cause startup failures and not crashes after running succesfully for a while. The last round of changes scans for disabled preprocessor and rule interdependencies and disables rules that have rule options dependent upon particular disabled preprocessors.

Fesoj

I am always running the latest components && I cannot (yet) trace it back to a single rule. If the person in question is online, Snort crashes with a rate of roughly once every 6 hours (and he is always online and playing WoT). So it could also be a stability issue. Maybe a pcap trace gives more info….

bmeeks

Hmm…that is weird. Any chance of correlating a crash time with the last alert in the logs? By that I mean can you perhaps isolate the crash to a specific time window that would correlate with an ALERTS tab entry? Just grasping at straws here, though.

Still, even matching up times would not necessarily find the errant rule (if that's what the cause is), because Snort could well crash before writing the offending traffic to the logs.

I would assume that over a period of 6 hours the user would hit the same rules over and over playing the same game, so I'm now inclined to maybe think it's not an errant rule. Do you see memory utilization creeping up over the 6 hour window?

Add some performance logging parameters as follows to the Advanced Configuration Passthrough box at the bottom of the Interfaces window in the Snort GUI.

config profile_rules: print 50, sort total_ticks, filename LAN_rules_perf.txt append
config profile_preprocs: sort avg_ticks, filename LAN_preprocs_perf.txt append

This will write some stats out to the /var/log/snort directory. Restart Snort and let it run for several hours while this user plays his game. Manually stop and restart Snort again and examine the data. This will show you which rules are being hit and how much time is being spent processing them. Same for the preprocessors. This may lead to some insight as far as what rules are being triggered most often. Note that Snort will only write this log data upon a successful shutdown (hence the need to stop and restart). If you let it run all the way to a crash, it probably won't update the performance log files.

Fesoj

If the box would be in my office it would be easier to analyze. I'll keep an eye on the issue. I just prepared a virtual machine with pfsense, so I can study the behavior on my desk–--but that implies I have to play WoT...

Fesoj

Hi think I found s.th:

snort[10544]: S5: Session exceeded configured max bytes to queue 1048576 using 1048775 bytes (client queue). 192.168.x.x 49684 –> x.x.x.x 38922 (0) : LWstate 0xe LWFlags 0x406007
snort[10544]: S5: Session exceeded configured max bytes to queue 1048576 using 1049416 bytes (client queue). 192.168.x.x 55461 –> x.x.x.x 80 (0) : LWstate 0xe LWFlags 0x406007
snort[10544]: S5: Session exceeded configured max bytes to queue 1048576 using 1049694 bytes (client queue). 192.168.x.x 52025 –> x.x.x.x 80 (0) : LWstate 0xf LWFlags 0x6007

I'll update the Stream5 parameters and then I'll wait for the "power surfer"…

My problem seems to be related to a similar problem discussed here some time ago: http://forum.pfsense.org/index.php?topic=25748.0;prev_next=prev

bmeeks

You could possibly be running out of memory, but I would expect some more self-evident errors if that were the case. This stream5 message is more of a "notification" than a "warning" message. However, you can get rid of it by bumping up the memcap value substantially (assuming you have sufficient RAM in the box). The trick is to get a high enough memcap number coupled with sufficiently large max_queued_bytes values. The magic number is determined by your Internet pipe size.

One other potential cause for this according to some posts I found from the Snort developers, is asymmetric traffic through Snort such that it does not see all of the 3-way handshake for a session. If I understood that post correctly, in a situation like that Snort misses the "close" of the session and does not reset the stream5 counters properly.

For me, with relatively slow 12 megabit cable service, I have memcap at 33 MB to squash all my stream5 messages. Another user commented in a recent post he had to take his up to 128 megabytes, I believe, to squash the notifications. He had a very high speed connection with either cable or maybe it was Verizon FIOS (fiber to the home).

Bill

Fesoj

Hi Bill,

this all makes sense to me && since under normal conditions less than 20% of the memory gets used I multiplied the stream5 values by 8. If I understood you correctly, the increased values would mean a higher theshold for the warning messages, but the underlying problem does not get solved.

This particular patron will show up again on Friday, then I'll see whether this is possibly related to the Snort crashes.

bmeeks

Correct, that is the way I understand the memcap and stream5 settings. Essentially you need enough memcap buffer to "keep up with" your incoming packet rate. The faster your pipe, the more memcap buffer you need to avoid the pruning messages.

Still not sure that this is source of your crash, but worth a shot to investigate.

Bill

Fesoj

… I am not either. The log files have been cleaned, port morrioring has been set up: I'll be prepared for Friday ;)

Fesoj

Bill,

I got the data from last weekend, and, as you suggested, there is more than the stream5 issue. I need a couple of days to look at the details, read a bit, and check some things, but then I'll report.

Supermule

Sounds good since IDS/IPS is a VITAL part of any firewall and currently Pfsense is having issues with a vital package that does NOT work as intended!

Fesoj

Well, after dozens of more crashes I am now inclined to say that the current Snort version has a basic stability problem. It is not only that "WoT" crashes but other sources, especially sites that stream media can also bring Snort down.

I always see something like this:

Mar 17 14:07:06	SnortStartup[54733]: Snort STOP For LAN side(55183_em1)...
Mar 17 14:03:27	kernel: em1: promiscuous mode disabled
Mar 17 14:03:27	kernel: pid 35907 (snort), uid 0: exited on signal 11

There are casual S5 messages, but basically there is no hint on what is going on (using the default log settings). Just to check that it is not related to a specific box (Atom based), I set up an 8GB machine (using a 4 core AMD processor) and got the same behavior.

I cannot follow most of the other Snort related reported problems here, except that Snort crashes under heavy load. If only 1 or two users are active, my Snort installation seems to be fine.

Supermule

I see this when streaming via VPN pn PFsense…

Snort suddenly stops without reason

php: /status_services.php: The command '/usr/local/etc/rc.d/snort.sh stop' returned exit code '1', the output was ''
Mar 17 21:26:14 SnortStartup[24393]: Snort STOP For Internet(36256_em0)…

bmeeks

A load-based bug is going to perhaps be difficult to find. Are you up to building a separate FreeBSD 8.1 virtual machine (or actual box) and run the traffic through it in just pure sniffing mode? Take pfSense out of the mix and maybe just do alerting on a clean FreeBSD 8.1 kernel. If that crashes, then try the 8.2 kernel. Other options are the Snort 2.9.4.1 binary on the 8.1 or 8.2 kernels. There are compilation instructions for Snort on FreeBSD posted on the Snort.org web site (in the documentation section).

Bill

Fesoj

Yes, exactly. Get the "independent" sniffing mode working and then see what can be done for pfSense.

eri--

I updated the binary to snort 2.4.9.1 so try with that.

Fesoj

I'll wait a couple of hours such that the repository is udpated && then I'll update.

dwood

Not sure if this is a snort binary update timing issue or not. The package installation fails on amd64, 2.0.2 This from the install packages window

Beginning package installation for snort…
Downloading package configuration file... done.
Saving updated package information... done.
Downloading snort and its dependencies...
Checking for package installation...
Downloading http://files.pfsense.org/packages/amd64/8/All/mysql-client-5.1.68.tbz ... could not download from there or http://ftp2.FreeBSD.org/pub/FreeBSD/ports/amd64/packages-8.1-release/All/mysql-client-5.1.68.tbz.
of mysql-client-5.1.68 failed!

Installation aborted.Backing up libraries..

eri--

Heh that was fast.
The binaries are being built they will be uploaded shortly.