Snort 2.9.4.6 Pkg v 2.5.9
-
Hi Bill
I run a bunch of VLANS's and dont have this issue.
Bill,
For a test to grab clean logs of the issue, I bounced my cable modem.. Hope this helps and enjoying labor day… I know I will be
Stephen
Thanks! These logs sure do help. I'm thinking VLANs are somehow the culprit. I don't have any defined on my systems, and I do not see the multiple processes. So far, the folks who are seeing multiple processes (too many processes, actually), all seem to have VLANs defined on their Snort interfaces. I'm taking that as a good indicator of where to start looking… ;)
Bill
-
Hi Bill
I run a bunch of VLANS's and dont have this issue.
Thanks for the feedback Brian. This a peculiar bug that does not seem to be easily reproduced. For the folks that have it, they are reporting it is 100% reproducible on their systems. For other systems…??
Bill
-
Thanks Bill! How can I help?
Thanks for the screenshots. I also sent you a PM asking for a little more information if you can share it.
Bill
-
Done! Let me know when you receive it. Also, let me know if I can provide any other information.
-
Done! Let me know when you receive it. Also, let me know if I can provide any other information.
I have it. Thanks. As I mentioned in my reply e-mail, I will be busy until the weekend and can take a look then.
Bill
-
Any luck, Bill?
-
Any luck, Bill?
Not yet. I can't reproduce the problem in my test environment. Does this only happen on a reboot for you, or does it also happen with the auto-rule updates in Snort?
EDIT: Never mind on the question. I looked back and see you provided the answer several posts back. You said it happens usually on restarts after the Snort rule updates.
Bill
-
When I had VLANs going, the repro was 100% at reboot time and every 12 hours (on a successful rule update). Now, that I have moved away from VLANs and spent some money retro fitting my laptop with additional Ethernet ports, the multiple snort processes issue only repros on restart. (I am going to jinx myself for making that claim).
-
When I had VLANs going, the repro was 100% at reboot time and every 12 hours (on a successful rule update). Now, that I have moved away from VLANs and spent some money retro fitting my laptop with additional Ethernet ports, the multiple snort processes issue only repros on restart. (I am going to jinx myself for making that claim).
Thanks for the clarification. I originally thought VLANs were at the root, but some other folks (Supermule, for one) have VLANs and don't have this issue. I will keep digging.
I have also been working this past weekend on getting the new Snort 2.9.5.3 binary going. I have a package built in my test environment that works. I want to get that new binary out by the end of this month and also update the package PHP code to 2.6.0. The new package code fixes a number of small bugs and adds multi-engine configurations for HTTP_INSPECT, Stream5 and Frag3.
Bill
-
Thanks is a poor word for what you have done to Snort and the comuunity Bill!
-
- I don't have any VLAN's set up.
- Running 2.1-RELEASE now
and I have issues with multiple Snort instances running every now and then. Last night there was only one instance of the Snort process, but since last upgrade there's another one running again. It was fine for a while after the upgrade, then I saw a extra instance running -> killall snort -> launch Snort again and it was fine for a few update cycles (maybe no actual update happened?) and now there are two instances running again.
[2.1-RELEASE][admin@pfsense.localdomain]/root(6): ps -ax | grep snort 55531 ?? SNs 0:47.02 /usr/pbi/snort-amd64/bin/snort -R 2226 -D -q -l /var/log/snort/snort_em02226 --pid-path /var/run --nolock-pidfile -G 2226 -c /usr/pbi/snort-amd64/etc/snort/snort_2226_em0/snort.conf -i em0 56242 ?? SNs 0:46.66 /usr/pbi/snort-amd64/bin/snort -R 2226 -D -q -l /var/log/snort/snort_em02226 --pid-path /var/run --nolock-pidfile -G 2226 -c /usr/pbi/snort-amd64/etc/snort/snort_2226_em0/snort.conf -i em0The update log looks a bit odd?
Starting rules update... Time: 2013-09-17 02:15:01 Downloading Snort VRT md5 file... Starting rules update... Time: 2013-09-17 02:15:01 Downloading Snort VRT md5 file... Starting rules update... Time: 2013-09-17 02:15:01 Downloading Snort VRT md5 file... Checking Snort VRT md5 file... Snort VRT rules are up to date. Downloading EmergingThreats md5 file... Checking Snort VRT md5 file... Snort VRT rules are up to date. Downloading EmergingThreats md5 file... Checking EmergingThreats md5. Emerging Threats rules are up to date. The Rules update has finished. Time: 2013-09-17 02:15:03 Checking Snort VRT md5 file... Snort VRT rules are up to date. Downloading EmergingThreats md5 file... Checking EmergingThreats md5. Emerging Threats rules are up to date. The Rules update has finished. Time: 2013-09-17 02:15:03 Checking EmergingThreats md5. Emerging Threats rules are up to date. The Rules update has finished. Time: 2013-09-17 02:15:04 Starting rules update... Time: 2013-09-17 14:15:01 Downloading Snort VRT md5 file... Starting rules update... Time: 2013-09-17 14:15:01 Downloading Snort VRT md5 file... Snort VRT md5 download failed. Server returned error code ''. Server error message was 'Failed to create file /usr/pbi/snort-amd64/etc/snort/tmp/snort_rules_up/snortrules-snapshot-2946.tar.gz.md5' Snort VRT rules will not be updated. Downloading EmergingThreats md5 file... EmergingThreats md5 file download failed. Server returned error code ''. The error text is 'Failed to create file /usr/pbi/snort-amd64/etc/snort/tmp/snort_rules_up/emerging.rules.tar.gz.md5' EmergingThreats rules will not be updated. Starting rules update... Time: 2013-09-17 14:15:01 Downloading Snort VRT md5 file... The Rules update has finished. Time: 2013-09-17 14:15:01 Checking Snort VRT md5 file... There is a new set of Snort VRT rules posted. Downloading... Checking Snort VRT md5 file... Snort VRT rules are up to date. Downloading EmergingThreats md5 file... Checking EmergingThreats md5. There is a new set of EmergingThreats rules posted. Downloading... Done downloading EmergingThreats rules file. Extracting and installing EmergingThreats.org rules... Installation of EmergingThreats.org rules completed. Copying new config and map files... Updating rules configuration for: WAN ... Restarting Snort to activate the new set of rules... Snort has restarted with your new set of rules. The Rules update has finished. Time: 2013-09-17 14:15:21 Done downloading rules file. Downloading EmergingThreats md5 file... EmergingThreats md5 file download failed. Server returned error code ''. The error text is 'Failed to create file /usr/pbi/snort-amd64/etc/snort/tmp/snort_rules_up/emerging.rules.tar.gz.md5' EmergingThreats rules will not be updated. Copying new config and map files... Updating rules configuration for: WAN ... Restarting Snort to activate the new set of rules... Snort has restarted with your new set of rules. The Rules update has finished. Time: 2013-09-17 14:15:57I don't have syslog since 15th as syslog seems to have crashed soon after the upgrade to -RELEASE :F
Sep 15 13:23:22 syslogd: exiting on signal 15 -
- I don't have any VIP's set up.
- Running 2.1-RELEASE now
and I have issues with multiple Snort instances running every now and then. Last night there was only one instance of the Snort process, but since last upgrade there's another one running again. It was fine for a while after the upgrade, then I saw a extra instance running -> killall snort -> launch Snort again and it was fine for a few update cycles (maybe no actual update happened?) and now there are two instances running again.
[2.1-RELEASE][admin@pfsense.localdomain]/root(6): ps -ax | grep snort 55531 ?? SNs 0:47.02 /usr/pbi/snort-amd64/bin/snort -R 2226 -D -q -l /var/log/snort/snort_em02226 --pid-path /var/run --nolock-pidfile -G 2226 -c /usr/pbi/snort-amd64/etc/snort/snort_2226_em0/snort.conf -i em0 56242 ?? SNs 0:46.66 /usr/pbi/snort-amd64/bin/snort -R 2226 -D -q -l /var/log/snort/snort_em02226 --pid-path /var/run --nolock-pidfile -G 2226 -c /usr/pbi/snort-amd64/etc/snort/snort_2226_em0/snort.conf -i em0The update log looks a bit odd?
Starting rules update... Time: 2013-09-17 02:15:01 Downloading Snort VRT md5 file... Starting rules update... Time: 2013-09-17 02:15:01 Downloading Snort VRT md5 file... Starting rules update... Time: 2013-09-17 02:15:01 Downloading Snort VRT md5 file... Checking Snort VRT md5 file... Snort VRT rules are up to date. Downloading EmergingThreats md5 file... Checking Snort VRT md5 file... Snort VRT rules are up to date. Downloading EmergingThreats md5 file... Checking EmergingThreats md5. Emerging Threats rules are up to date. The Rules update has finished. Time: 2013-09-17 02:15:03 Checking Snort VRT md5 file... Snort VRT rules are up to date. Downloading EmergingThreats md5 file... Checking EmergingThreats md5. Emerging Threats rules are up to date. The Rules update has finished. Time: 2013-09-17 02:15:03 Checking EmergingThreats md5. Emerging Threats rules are up to date. The Rules update has finished. Time: 2013-09-17 02:15:04 Starting rules update... Time: 2013-09-17 14:15:01 Downloading Snort VRT md5 file... Starting rules update... Time: 2013-09-17 14:15:01 Downloading Snort VRT md5 file... Snort VRT md5 download failed. Server returned error code ''. Server error message was 'Failed to create file /usr/pbi/snort-amd64/etc/snort/tmp/snort_rules_up/snortrules-snapshot-2946.tar.gz.md5' Snort VRT rules will not be updated. Downloading EmergingThreats md5 file... EmergingThreats md5 file download failed. Server returned error code ''. The error text is 'Failed to create file /usr/pbi/snort-amd64/etc/snort/tmp/snort_rules_up/emerging.rules.tar.gz.md5' EmergingThreats rules will not be updated. Starting rules update... Time: 2013-09-17 14:15:01 Downloading Snort VRT md5 file... The Rules update has finished. Time: 2013-09-17 14:15:01 Checking Snort VRT md5 file... There is a new set of Snort VRT rules posted. Downloading... Checking Snort VRT md5 file... Snort VRT rules are up to date. Downloading EmergingThreats md5 file... Checking EmergingThreats md5. There is a new set of EmergingThreats rules posted. Downloading... Done downloading EmergingThreats rules file. Extracting and installing EmergingThreats.org rules... Installation of EmergingThreats.org rules completed. Copying new config and map files... Updating rules configuration for: WAN ... Restarting Snort to activate the new set of rules... Snort has restarted with your new set of rules. The Rules update has finished. Time: 2013-09-17 14:15:21 Done downloading rules file. Downloading EmergingThreats md5 file... EmergingThreats md5 file download failed. Server returned error code ''. The error text is 'Failed to create file /usr/pbi/snort-amd64/etc/snort/tmp/snort_rules_up/emerging.rules.tar.gz.md5' EmergingThreats rules will not be updated. Copying new config and map files... Updating rules configuration for: WAN ... Restarting Snort to activate the new set of rules... Snort has restarted with your new set of rules. The Rules update has finished. Time: 2013-09-17 14:15:57I don't have syslog since 15th as syslog seems to have crashed soon after the upgrade to -RELEASE :F
Sep 15 13:23:22 syslogd: exiting on signal 15This is quite strange. The updates are failing (and the log looks weird) I assume because two processes are trying to do the same thing to the same file at the same time. The two processes are colliding during the MD5 file write, for one example.
Do you have any VLANs defined on an interface?
Bill
-
No VLAN's. I seem to have mistaken VIP's and VLAN's in my post.
-
Hi,
Got the same problem on an AMD fx6300 - 6core. snort ALWAYS starts 3 instances at the same time.
I think it should be easily reproducible when you enable the maximum of rules and preprocessors. That way snort needs masses of ram (around 2GB when loaded) and takes minutes to load (almost 4GB during loading).r, max
-
Hi,
Got the same problem on an AMD fx6300 - 6core. snort ALWAYS starts 3 instances at the same time.
I think it should be easily reproducible when you enable the maximum of rules and preprocessors. That way snort needs masses of ram (around 2GB when loaded) and takes minutes to load (almost 4GB during loading).r, max
Thanks for the suggestion. I will try this (loading up a Snort instance) in a VM and see if I can reproduce the multiple starts problem. Thus far I have not been able to, but I have not tried enabling all the rules at once.
When you say the "maximum of rules and preprocessors", which rule families (Snort VRT, Emerging Threats, GPLv2 Community) are you running?
Bill
-
I run all families.
-
- I don't have any VLAN's set up.
- Running 2.1-RELEASE now
and I have issues with multiple Snort instances running every now and then.
Not much to add other than I've seen this too. The initial appearance was of a memory leak. Then I notice that there were multiple Snort instances running. I restart Snort and memory utilization goes back to normal (dup processes killed).
It does seem to happen after a rules update. Now that I know what to look for, I'll track more closely.
I do not have any VLAN's.
I only run a very minimal set of rules. Just "Connectivity" and 10 ET categories.
-
I have same problems as Bill found out in my other thread… :(
I have 2 2.1 pfsenses one is static other is pppoe.
On both OpenVPN and squid+havp and thats it.On one there are 2 processes and on other 4.
Stopping snort, killing other live processes and then start snort seems to fix it. -
Hopefully I have some good news on the multiple Snort processes after a reboot front. Using the suggestions here of trying a "fully loaded" configuration with everything enabled, I was finally able to consistently get two duplicate Snort processes running with each reboot of a VM. The VM had Snort configured on only one interface (the WAN), and thus should have had only one running Snort process.
The problem, I believe, is really the fault of pfSense itself. However I am still tracking this hypothesis down. I'm thinking now the problem is in the rc.newwanip script.
The good news is I think I've found a workaround to use within the snort.sh shell script that is called to start Snort on a reboot. I need to test another day or two to be sure the workaround does not break something else.
Oddly enough, there was code in the existing snort.sh script that should avoid the multiple processes on reboot. But I found out during some research that the utility I was using (pgrep) has a long-standing known bug where it only matches the first 15 characters of a process name. I modified the arguments supplied to pgrep to shorten the match string and it started working correctly.
As others have surmised, the trigger for the multiple processes is a suitably loaded Snort configuration such that it takes a long time to start up. That in turn means there is a delay in creating the PID file in /var/run. In the start routine, the shell script tries to find a running process and will send it a SIGHUP to restart instead of starting a completely new one. It tries to find a running process by first looking for a PID file, and if that fails, it uses pgrep to see if a Snort process with the same arguments is running. On a slowly starting Snort setup, the PID file may not be there yet because it is written at the end of the startup. And with the 15-character bug I mentioned earlier, pgrep was not finding the running process either. Now I've been able to get pgrep to find the running process.
Of course the real problem here is pfSense itself sending multiple package restart commands on a reboot. Snort is actually getting called to start twice during the reboot cycle by two different pfSense init processes.
Bill
-
Awsome news and it totally makes sense!
If you need test machine, I will be more than glad to install new package/.sh to my firewall.
Regards,
M
