Snort 2.9.4.6 Pkg v 2.5.9

bmeeks

Hopefully I have some good news on the multiple Snort processes after a reboot front. Using the suggestions here of trying a "fully loaded" configuration with everything enabled, I was finally able to consistently get two duplicate Snort processes running with each reboot of a VM. The VM had Snort configured on only one interface (the WAN), and thus should have had only one running Snort process.

The problem, I believe, is really the fault of pfSense itself. However I am still tracking this hypothesis down. I'm thinking now the problem is in the rc.newwanip script.

The good news is I think I've found a workaround to use within the snort.sh shell script that is called to start Snort on a reboot. I need to test another day or two to be sure the workaround does not break something else.

Oddly enough, there was code in the existing snort.sh script that should avoid the multiple processes on reboot. But I found out during some research that the utility I was using (pgrep) has a long-standing known bug where it only matches the first 15 characters of a process name. I modified the arguments supplied to pgrep to shorten the match string and it started working correctly.

As others have surmised, the trigger for the multiple processes is a suitably loaded Snort configuration such that it takes a long time to start up. That in turn means there is a delay in creating the PID file in /var/run. In the start routine, the shell script tries to find a running process and will send it a SIGHUP to restart instead of starting a completely new one. It tries to find a running process by first looking for a PID file, and if that fails, it uses pgrep to see if a Snort process with the same arguments is running. On a slowly starting Snort setup, the PID file may not be there yet because it is written at the end of the startup. And with the 15-character bug I mentioned earlier, pgrep was not finding the running process either. Now I've been able to get pgrep to find the running process.

Of course the real problem here is pfSense itself sending multiple package restart commands on a reboot. Snort is actually getting called to start twice during the reboot cycle by two different pfSense init processes.

Bill

maverick_slo

Awsome news and it totally makes sense!

If you need test machine, I will be more than glad to install new package/.sh to my firewall.

Regards,
M

Cino

@maverick_slo:

Awsome news and it totally makes sense!

If you need test machine, I will be more than glad to install new package/.sh to my firewall.

Regards,
M

I wouldn't mind testing it either before its deployed to the masses

bmeeks

@Cino:

@maverick_slo:

Awsome news and it totally makes sense!

If you need test machine, I will be more than glad to install new package/.sh to my firewall.

Regards,
M

I wouldn't mind testing it either before its deployed to the masses

Thanks guys for volunteering to test. Let me do some more testing of my own for a day or two, then I will post the file here for others having the multiple processes issue to test. The fix will consist of a new snort.inc file that will go in /usr/local/pkg/snort. This file contains the subroutine that creates the snort.sh rc.d script on each reboot and when saving changes in the Snort GUI.

Bill

pfSenseRocks

Bill, this is great news! I appreciate all your help, hard work and diligence. Let me know if and how I can help.

bmeeks

Guys:

Attached to this post is a test fix for the multiple Snort processes that get started on a reboot on pfSense. This problem seems most acute on 2.1 systems. This is a test file. It should do no harm, but use at your own risk!

Use WinSCP, SCP or a similar process to copy this file to the firewall and place it in /usr/local/pkg/snort.

First rename the existing snort.inc file to snort.inc.orig. This will allow you to easily rollback if something goes wrong.

Second, rename the new file from snort.inc.txt to just snort.inc

Third, go into Snort and click the Interfaces tab and then select any interface. Click the Save button to force a generation of a new /usr/local/etc/rc.d/snort.sh file.

Fourth, try a reboot and see if the multiple Snort processes are history.

IMPORTANT – The file is in UNIX format (LF only with no CR/LF). Do not edit it on a Windows machine! When you transfer it, send it over in binary format to preserve the line endings. I had to name it with a *.txt extension in order to attach it to this post.

Bill

snort.inc.txt

pfSenseRocks

At the risk of jinxing it, we have a winner! 2 reboots and counting…

bmeeks

@pfSenseRocks:

At the risk of jinxing it, we have a winner! 2 reboots and counting…

Good news! Let me know if the multiple processes reappear. I will wait to hear from some of the other folks who had the problem and volunteered to test. If it works for everyone with no side effects, then I will submit a Pull Request on GitHub for the pfSense developers.

Bill

pfSenseRocks

I have rebooted about 7 times and I haven't been able to reproduce the issue. I have even snort running on 3 interfaces and I get exactly one process per interface.

Thank you so much for fixing this.

turker

Thank you so much.
The problem has been solved.

Atom D2500
4GB RAM
Pfsense 2.1

WAN+LAN+VLAN running smoothly.

Supermule

Does the name SNORT KING sound rude Bill? :D

maverick_slo

Morning all!

Rebooted 3 times no more snort issues.
Running smooth with AC-STD now.

Thanks man!

gogol

Thanks Bill, finally.

I booted a VM and have only 1 snort process now but still with the -E argument. Can you explain a bit what you changed or what was happening?

I am on 2.1 RELEASE.

/root(1): ps auxwwww | grep snort
root   57580  0.0 12.6 527956 260680  ??  Ss    1:13PM   0:11.61 /usr/pbi/snort-i386/bin/snort -R 4082 -E -q -l /var/log/snort/snort_em04082 --pid-path /var/run --nolock-pidfile -G 4082 -c /usr/pbi/snort-i386/etc/snort/snort_4082_em0/snort.conf -i em0
root   77957  0.0  0.1  3468  1240   0  S+    1:14PM   0:00.00 grep snort

bmeeks

Thanks for the testing and quick feedback guys. I will bundle this fix up with a few other bug hotfixes I've been holding and submit a GitHub Pull Request for the pfSense Core Team to review and hopefully approve and merge. The new package version will be 2.6.0 when it gets pushed to the Packages Repository. It will still use the 2.9.4.6 Snort binary.

I have a fully functional package based on the Snort 2.9.5.3 binary with a lot of new GUI features in the works. One of these new features is support for multi-engine configurations for the frag3, stream5_tcp and http_inspect preprocessors. There are also some new GUI configurable options for the SDF Sensitive Data preprocessor and a few others. I still need to do a bit more testing with this version before I release it.

Bill

bmeeks

@gogol:

Thanks Bill, finally.

I booted a VM and have only 1 snort process now but still with the -E argument. Can you explain a bit what you changed or what was happening?

I am on 2.1 RELEASE.
/root(1): ps auxwwww | grep snort
root   57580  0.0 12.6 527956 260680  ??  Ss    1:13PM   0:11.61 /usr/pbi/snort-i386/bin/snort -R 4082 -E -q -l /var/log/snort/snort_em04082 --pid-path /var/run --nolock-pidfile -G 4082 -c /usr/pbi/snort-i386/etc/snort/snort_4082_em0/snort.conf -i em0
root   77957  0.0  0.1  3468  1240   0  S+    1:14PM   0:00.00 grep snort

I don't know about the -E argument. What I changed in the code was to shorten the match argument list supplied to the pgrep utility. The shell script, when called with the start argument, first tries to see if there is any running Snort process for the interface. If it finds one, it sends that instance a SIGHUP (soft restart) signal instead of starting a totally new process. This prevents multiple Snort instances on the same interface.

The code described above has always been there, but was not working 100% due to a bug in the behavior of pgrep. I was using pgrep -xf [the complete command line]. The "-xf" argument tells pgrep to match exactly the entire command line list of arguments. Turns out that pgrep will silently fail to match very long arguments lists when given the "-x" exact-match parameter. Snort has a very long command line argument list. So the hunt for a running process on a given interface silently failed, and a second process would get started. This only happened on a reboot with a slow starting Snort configuration. The shell script also looks for a PID file in /var/run, and if one is found uses the PID in there to send the SIGHUP to the correct process. During a reboot, the PID file is generally not present; so the shell script falls back to the pgrep method to find the correct process. This problem is compounded by the fact pfSense itself launches two processes that each issue a restart all packages command. So during the reboot the Snort shell script gets called twice a few hundred milliseconds apart.

The fix was to change the pgrep parameters and match argument for both the stop and start routines in the shell script. The new code uses pgrep -f snort -R uuid where uuid is the unique number identifying the interface.

fragged

Should this fix multiple Snort processes being left over after rules updates? With my setup Snort takes a while to fully load all rules. For me Snort always launches correctly on boot up, but there might be multiple processes running after a few days of uptime.

bmeeks

@fragged:

Should this fix multiple Snort processes being left over after rules updates? With my setup Snort takes a while to fully load all rules. For me Snort always launches correctly on boot up, but there might be multiple processes running after a few days of uptime.

Yes, it should help with that as well. The same pgrep bug was in the STOP part of the script, also. Just make sure that any existing "extra processes" are killed manually, or else just reboot the box.

Bill

AhnHEL

Thank you for the fix Bill, when will this go Gold? v2.6.0?

bmeeks

@AhnHEL:

Thank you for the fix Bill, when will this go Gold? v2.6.0?

The code update has been submitted via a GitHub Pull Request. Here is a link to the request:

https://github.com/pfsense/pfsense-packages/pull/514

There are a few other fixes and new features included with the multiple processes bug fix.

The changes need to be reviewed, approved and merged by the pfSense Core Team before the new version shows up in the Package Repository.

Bill

shinzo

Hello Bill, was wondering if u fixed the typo in the snort.inc. It was "{$stream5_no_reassemble_async}$stream5_dont_store_lg_pkts" on line 3137, it was talked about on page 5 so like 100 years ago. :)

UPDATE

Nevermind just saw it was updated thanks again!