Snort 2.9.4.6 Pkg v 2.5.9

bmeeks

So I bought me a dual gigabit NIC. I now have WAN, LAN and the Guest LAN on three physical NICs as opposed to VLANs. I still have the same problem with multiple snort processes all on the same interface. I am clueless. Please help!

At start up


[2.1-RC2][admin@sense.home]/root(2): ps -ax | grep snort
33426  ??  IWN    0:00.00 /bin/sh /usr/local/etc/rc.d/snort.sh start
34178  ??  DNL    6:45.45 /usr/pbi/snort-amd64/bin/snort -R 62688 -D -q -l /var/log/snort/snort_em062688 --pid-path /var/run --nolock-pidfile -G 62688 -c /usr/pbi/snort-amd64/etc/snort/snort_62688_em0/snort.conf -i em0
62838  ??  IWN    0:00.00 /bin/sh /usr/local/etc/rc.d/snort.sh start
64327  ??  DNL    6:25.72 /usr/pbi/snort-amd64/bin/snort -R 62688 -D -q -l /var/log/snort/snort_em062688 --pid-path /var/run --nolock-pidfile -G 62688 -c /usr/pbi/snort-amd64/etc/snort/snort_62688_em0/snort.conf -i em0
54725  v0- IW     0:00.00 /bin/sh /usr/local/etc/rc.d/snort.sh start
57256  v0- R      6:32.50 /usr/pbi/snort-amd64/bin/snort -R 62688 -D -q -l /var/log/snort/snort_em062688 --pid-path /var/run --nolock-pidfile -G 62688 -c /usr/pbi/snort-amd64/etc/snort/snort_62688_em0/snort.conf -i em0
48672   0  DL+    0:00.00 grep snort
[2.1-RC2][admin@sense.home]/root(3):

After executing killing these processes and manually restarting…


[2.1-RC2][admin@sense.home]/root(5): ps -ax | grep snort
  715  ??  Ss     0:00.08 /usr/pbi/snort-amd64/bin/snort -R 22796 -D -q -l /var/log/snort/snort_re122796 --pid-path /var/run --nolock-pidfile -G 22796 -c /usr/pbi/snort-amd64/etc/snort/snort_22796_re1/snort.conf -i re1
51274  ??  Ss     0:04.80 /usr/pbi/snort-amd64/bin/snort -R 62688 -D -q -l /var/log/snort/snort_em062688 --pid-path /var/run --nolock-pidfile -G 62688 -c /usr/pbi/snort-amd64/etc/snort/snort_62688_em0/snort.conf -i em0
73396  ??  Ss     0:00.66 /usr/pbi/snort-amd64/bin/snort -R 26667 -D -q -l /var/log/snort/snort_re026667 --pid-path /var/run --nolock-pidfile -G 26667 -c /usr/pbi/snort-amd64/etc/snort/snort_26667_re0/snort.conf -i re0
68790   0  S+     0:00.00 grep snort
[2.1-RC2][admin@sense.home]/root(6):

I'm still looking into this. Thus far I can't reproduce the problem in my VMware test setup, so it's still a bit of a mystery as to the root cause. I am not throwing in the towel yet, though.

Bill

pfSenseRocks

Thanks Bill! How can I help?

pfSenseRocks

Attaching a few screenshots of my snort configuration.

SNortGlobal.PNG_thumb

SnortIf.PNG_thumb

SnortLANCat.PNG_thumb

SnortWANCat.PNG_thumb

SnortWANSettings.PNG_thumb

Supermule

Hi Bill

I run a bunch of VLANS's and dont have this issue.

@bmeeks:

@Cino:

Bill,

For a test to grab clean logs of the issue, I bounced my cable modem.. Hope this helps and enjoying labor day… I know I will be

Stephen

Thanks! These logs sure do help. I'm thinking VLANs are somehow the culprit. I don't have any defined on my systems, and I do not see the multiple processes. So far, the folks who are seeing multiple processes (too many processes, actually), all seem to have VLANs defined on their Snort interfaces. I'm taking that as a good indicator of where to start looking… ;)

Bill

bmeeks

@Supermule:

Hi Bill

I run a bunch of VLANS's and dont have this issue.

Thanks for the feedback Brian. This a peculiar bug that does not seem to be easily reproduced. For the folks that have it, they are reporting it is 100% reproducible on their systems. For other systems…??

Bill

bmeeks

@pfSenseRocks:

Thanks Bill! How can I help?

Thanks for the screenshots. I also sent you a PM asking for a little more information if you can share it.

Bill

pfSenseRocks

Done! Let me know when you receive it. Also, let me know if I can provide any other information.

bmeeks

@pfSenseRocks:

Done! Let me know when you receive it. Also, let me know if I can provide any other information.

I have it. Thanks. As I mentioned in my reply e-mail, I will be busy until the weekend and can take a look then.

Bill

pfSenseRocks

Any luck, Bill?

bmeeks

@pfSenseRocks:

Any luck, Bill?

Not yet. I can't reproduce the problem in my test environment. Does this only happen on a reboot for you, or does it also happen with the auto-rule updates in Snort?

EDIT: Never mind on the question. I looked back and see you provided the answer several posts back. You said it happens usually on restarts after the Snort rule updates.

Bill

pfSenseRocks

When I had VLANs going, the repro was 100% at reboot time and every 12 hours (on a successful rule update). Now, that I have moved away from VLANs and spent some money retro fitting my laptop with additional Ethernet ports, the multiple snort processes issue only repros on restart. (I am going to jinx myself for making that claim).

bmeeks

@pfSenseRocks:

When I had VLANs going, the repro was 100% at reboot time and every 12 hours (on a successful rule update). Now, that I have moved away from VLANs and spent some money retro fitting my laptop with additional Ethernet ports, the multiple snort processes issue only repros on restart. (I am going to jinx myself for making that claim).

Thanks for the clarification. I originally thought VLANs were at the root, but some other folks (Supermule, for one) have VLANs and don't have this issue. I will keep digging.

I have also been working this past weekend on getting the new Snort 2.9.5.3 binary going. I have a package built in my test environment that works. I want to get that new binary out by the end of this month and also update the package PHP code to 2.6.0. The new package code fixes a number of small bugs and adds multi-engine configurations for HTTP_INSPECT, Stream5 and Frag3.

Bill

Supermule

Thanks is a poor word for what you have done to Snort and the comuunity Bill!

fragged

I don't have any VLAN's set up.
Running 2.1-RELEASE now

and I have issues with multiple Snort instances running every now and then. Last night there was only one instance of the Snort process, but since last upgrade there's another one running again. It was fine for a while after the upgrade, then I saw a extra instance running -> killall snort -> launch Snort again and it was fine for a few update cycles (maybe no actual update happened?) and now there are two instances running again.


[2.1-RELEASE][admin@pfsense.localdomain]/root(6): ps -ax | grep snort
55531  ??  SNs    0:47.02 /usr/pbi/snort-amd64/bin/snort -R 2226 -D -q -l /var/log/snort/snort_em02226 --pid-path /var/run --nolock-pidfile -G 2226 -c /usr/pbi/snort-amd64/etc/snort/snort_2226_em0/snort.conf -i em0
56242  ??  SNs    0:46.66 /usr/pbi/snort-amd64/bin/snort -R 2226 -D -q -l /var/log/snort/snort_em02226 --pid-path /var/run --nolock-pidfile -G 2226 -c /usr/pbi/snort-amd64/etc/snort/snort_2226_em0/snort.conf -i em0

The update log looks a bit odd?


Starting rules update...  Time: 2013-09-17 02:15:01
	Downloading Snort VRT md5 file...
Starting rules update...  Time: 2013-09-17 02:15:01
	Downloading Snort VRT md5 file...
Starting rules update...  Time: 2013-09-17 02:15:01
	Downloading Snort VRT md5 file...
	Checking Snort VRT md5 file...
	Snort VRT rules are up to date.
	Downloading EmergingThreats md5 file...
	Checking Snort VRT md5 file...
	Snort VRT rules are up to date.
	Downloading EmergingThreats md5 file...
	Checking EmergingThreats md5.
	Emerging Threats rules are up to date.
The Rules update has finished.  Time: 2013-09-17 02:15:03

	Checking Snort VRT md5 file...
	Snort VRT rules are up to date.
	Downloading EmergingThreats md5 file...
	Checking EmergingThreats md5.
	Emerging Threats rules are up to date.
The Rules update has finished.  Time: 2013-09-17 02:15:03

	Checking EmergingThreats md5.
	Emerging Threats rules are up to date.
The Rules update has finished.  Time: 2013-09-17 02:15:04

Starting rules update...  Time: 2013-09-17 14:15:01
	Downloading Snort VRT md5 file...
Starting rules update...  Time: 2013-09-17 14:15:01
	Downloading Snort VRT md5 file...
	Snort VRT md5 download failed.
	Server returned error code ''.
	Server error message was 'Failed to create file /usr/pbi/snort-amd64/etc/snort/tmp/snort_rules_up/snortrules-snapshot-2946.tar.gz.md5'
	Snort VRT rules will not be updated.
	Downloading EmergingThreats md5 file...
	EmergingThreats md5 file download failed.  Server returned error code ''.
	The error text is 'Failed to create file /usr/pbi/snort-amd64/etc/snort/tmp/snort_rules_up/emerging.rules.tar.gz.md5'
	EmergingThreats rules will not be updated.
Starting rules update...  Time: 2013-09-17 14:15:01
	Downloading Snort VRT md5 file...
The Rules update has finished.  Time: 2013-09-17 14:15:01

	Checking Snort VRT md5 file...
	There is a new set of Snort VRT rules posted. Downloading...
	Checking Snort VRT md5 file...
	Snort VRT rules are up to date.
	Downloading EmergingThreats md5 file...
	Checking EmergingThreats md5.
	There is a new set of EmergingThreats rules posted. Downloading...
	Done downloading EmergingThreats rules file.
	Extracting and installing EmergingThreats.org rules...
	Installation of EmergingThreats.org rules completed.
	Copying new config and map files...
	Updating rules configuration for: WAN ...
	Restarting Snort to activate the new set of rules...
	Snort has restarted with your new set of rules.
The Rules update has finished.  Time: 2013-09-17 14:15:21

	Done downloading rules file.
	Downloading EmergingThreats md5 file...
	EmergingThreats md5 file download failed.  Server returned error code ''.
	The error text is 'Failed to create file /usr/pbi/snort-amd64/etc/snort/tmp/snort_rules_up/emerging.rules.tar.gz.md5'
	EmergingThreats rules will not be updated.
	Copying new config and map files...
	Updating rules configuration for: WAN ...
	Restarting Snort to activate the new set of rules...
	Snort has restarted with your new set of rules.
The Rules update has finished.  Time: 2013-09-17 14:15:57

I don't have syslog since 15th as syslog seems to have crashed soon after the upgrade to -RELEASE :F


Sep 15 13:23:22 	syslogd: exiting on signal 15

bmeeks

@fragged:

I don't have any VIP's set up.
Running 2.1-RELEASE now

and I have issues with multiple Snort instances running every now and then. Last night there was only one instance of the Snort process, but since last upgrade there's another one running again. It was fine for a while after the upgrade, then I saw a extra instance running -> killall snort -> launch Snort again and it was fine for a few update cycles (maybe no actual update happened?) and now there are two instances running again.


[2.1-RELEASE][admin@pfsense.localdomain]/root(6): ps -ax | grep snort
55531  ??  SNs    0:47.02 /usr/pbi/snort-amd64/bin/snort -R 2226 -D -q -l /var/log/snort/snort_em02226 --pid-path /var/run --nolock-pidfile -G 2226 -c /usr/pbi/snort-amd64/etc/snort/snort_2226_em0/snort.conf -i em0
56242  ??  SNs    0:46.66 /usr/pbi/snort-amd64/bin/snort -R 2226 -D -q -l /var/log/snort/snort_em02226 --pid-path /var/run --nolock-pidfile -G 2226 -c /usr/pbi/snort-amd64/etc/snort/snort_2226_em0/snort.conf -i em0

The update log looks a bit odd?


Starting rules update...  Time: 2013-09-17 02:15:01
	Downloading Snort VRT md5 file...
Starting rules update...  Time: 2013-09-17 02:15:01
	Downloading Snort VRT md5 file...
Starting rules update...  Time: 2013-09-17 02:15:01
	Downloading Snort VRT md5 file...
	Checking Snort VRT md5 file...
	Snort VRT rules are up to date.
	Downloading EmergingThreats md5 file...
	Checking Snort VRT md5 file...
	Snort VRT rules are up to date.
	Downloading EmergingThreats md5 file...
	Checking EmergingThreats md5.
	Emerging Threats rules are up to date.
The Rules update has finished.  Time: 2013-09-17 02:15:03

	Checking Snort VRT md5 file...
	Snort VRT rules are up to date.
	Downloading EmergingThreats md5 file...
	Checking EmergingThreats md5.
	Emerging Threats rules are up to date.
The Rules update has finished.  Time: 2013-09-17 02:15:03

	Checking EmergingThreats md5.
	Emerging Threats rules are up to date.
The Rules update has finished.  Time: 2013-09-17 02:15:04

Starting rules update...  Time: 2013-09-17 14:15:01
	Downloading Snort VRT md5 file...
Starting rules update...  Time: 2013-09-17 14:15:01
	Downloading Snort VRT md5 file...
	Snort VRT md5 download failed.
	Server returned error code ''.
	Server error message was 'Failed to create file /usr/pbi/snort-amd64/etc/snort/tmp/snort_rules_up/snortrules-snapshot-2946.tar.gz.md5'
	Snort VRT rules will not be updated.
	Downloading EmergingThreats md5 file...
	EmergingThreats md5 file download failed.  Server returned error code ''.
	The error text is 'Failed to create file /usr/pbi/snort-amd64/etc/snort/tmp/snort_rules_up/emerging.rules.tar.gz.md5'
	EmergingThreats rules will not be updated.
Starting rules update...  Time: 2013-09-17 14:15:01
	Downloading Snort VRT md5 file...
The Rules update has finished.  Time: 2013-09-17 14:15:01

	Checking Snort VRT md5 file...
	There is a new set of Snort VRT rules posted. Downloading...
	Checking Snort VRT md5 file...
	Snort VRT rules are up to date.
	Downloading EmergingThreats md5 file...
	Checking EmergingThreats md5.
	There is a new set of EmergingThreats rules posted. Downloading...
	Done downloading EmergingThreats rules file.
	Extracting and installing EmergingThreats.org rules...
	Installation of EmergingThreats.org rules completed.
	Copying new config and map files...
	Updating rules configuration for: WAN ...
	Restarting Snort to activate the new set of rules...
	Snort has restarted with your new set of rules.
The Rules update has finished.  Time: 2013-09-17 14:15:21

	Done downloading rules file.
	Downloading EmergingThreats md5 file...
	EmergingThreats md5 file download failed.  Server returned error code ''.
	The error text is 'Failed to create file /usr/pbi/snort-amd64/etc/snort/tmp/snort_rules_up/emerging.rules.tar.gz.md5'
	EmergingThreats rules will not be updated.
	Copying new config and map files...
	Updating rules configuration for: WAN ...
	Restarting Snort to activate the new set of rules...
	Snort has restarted with your new set of rules.
The Rules update has finished.  Time: 2013-09-17 14:15:57

I don't have syslog since 15th as syslog seems to have crashed soon after the upgrade to -RELEASE :F


Sep 15 13:23:22 	syslogd: exiting on signal 15

This is quite strange. The updates are failing (and the log looks weird) I assume because two processes are trying to do the same thing to the same file at the same time. The two processes are colliding during the MD5 file write, for one example.

Do you have any VLANs defined on an interface?

Bill

fragged

No VLAN's. I seem to have mistaken VIP's and VLAN's in my post.

maex

Hi,

Got the same problem on an AMD fx6300 - 6core. snort ALWAYS starts 3 instances at the same time.
I think it should be easily reproducible when you enable the maximum of rules and preprocessors. That way snort needs masses of ram (around 2GB when loaded) and takes minutes to load (almost 4GB during loading).

r, max

bmeeks

@maex:

Hi,

Got the same problem on an AMD fx6300 - 6core. snort ALWAYS starts 3 instances at the same time.
I think it should be easily reproducible when you enable the maximum of rules and preprocessors. That way snort needs masses of ram (around 2GB when loaded) and takes minutes to load (almost 4GB during loading).

r, max

Thanks for the suggestion. I will try this (loading up a Snort instance) in a VM and see if I can reproduce the multiple starts problem. Thus far I have not been able to, but I have not tried enabling all the rules at once.

When you say the "maximum of rules and preprocessors", which rule families (Snort VRT, Emerging Threats, GPLv2 Community) are you running?

Bill

pfSenseRocks

I run all families.

priller

@fragged:

I don't have any VLAN's set up.

Running 2.1-RELEASE now

and I have issues with multiple Snort instances running every now and then.

Not much to add other than I've seen this too. The initial appearance was of a memory leak. Then I notice that there were multiple Snort instances running. I restart Snort and memory utilization goes back to normal (dup processes killed).

It does seem to happen after a rules update. Now that I know what to look for, I'll track more closely.

I do not have any VLAN's.

I only run a very minimal set of rules. Just "Connectivity" and 10 ET categories.