Snort Disabled After Enabling "emerging-web_server.rules"
-
Hi all,
I try to troubleshoot my own issues, and I usually do reasonably well. However, this issue is one that needs some fresh ideas to achieve a resolution.
–------------------------------------------------------------------------------------------------------------------------------
Here is the issue:
1. Snort will fail (become disabled) within minutes (usually NOT immediately) after enabling any combination of the following two rules under the "WAN Categories" tab:
a. emerging-web_server.rules
b. emerging-web_specific_apps.rulesAll other combinations of rules (ET Open Rules, Snort Text Rules, Snort SO Rules) have no effect on the issue– any combination I've tried will neither mitigate nor trigger the Snort failure. It took some time, but I finally narrowed down the issue to those two rules.
Conditions and Environment:
I run pfSense 2.15 on bare metal: Dell ATX mini tower desktop, 3 GB ram, AMD 64 processor, plenty of HDD space, and I added an Intel dual NIC (server class). I've configured Snort to defaults (mostly except increased the RAM allotment for a variety of settings for the sake of headroom), and use the emerging-compromised-ips.txt file option. I maintain a short suppress list (4 entries for frequent false positives). Everything is stable and it will run for weeks at a time so long as the two rules in question are unchecked (disabled).
I intentionally run pfSense in a double NAT configuration. One router faces the internet, and behind that router is pfSense (also in NAT configuration) on a different IP class (192.68.x.x and 10.x.x.x). I successfully configured port forwarding for RDP from the internet. VPN works well, and all other services function as normal. Before anyone criticizes this concept as "sacrilege", take a quick read through this article: https://www.grc.com/nat/nat.htm
Hardware tests from a bootable CD come clean: memory, processor, board, etc.
I've tried upgrading as well as a clean install for different versions of pfSense (2.13, 2.14, 2.15): no change in behavior of the issue.
For what it's worth, this issue has persisted through pfSense versions 2.13, 2.14, and now 2.15.
My config file is standard (only change implemented was to increase available memory).
–------------------------------------------------------------------------------------------------------------------------------
So, what is it about these two ET rules that will disable Snort within minutes?
I'll be happy to respond to questions about config and logs. I consider myself still new to pfSense, so please be clear about such requests. Thank you– sincerely so-- for your consideration and ideas!
-
Hi all,
I try to troubleshoot my own issues, and I usually do reasonably well. However, this issue is one that needs some fresh ideas to achieve a resolution.
–------------------------------------------------------------------------------------------------------------------------------
Here is the issue:
1. Snort will fail (become disabled) within minutes (usually NOT immediately) after enabling any combination of the following two rules under the "WAN Categories" tab:
a. emerging-web_server.rules
b. emerging-web_specific_apps.rulesAll other combinations of rules (ET Open Rules, Snort Text Rules, Snort SO Rules) have no effect on the issue– any combination I've tried will neither mitigate nor trigger the Snort failure. It took some time, but I finally narrowed down the issue to those two rules.
Conditions and Environment:
I run pfSense 2.15 on bare metal: Dell ATX mini tower desktop, 3 GB ram, AMD 64 processor, plenty of HDD space, and I added an Intel dual NIC (server class). I've configured Snort to defaults (mostly except increased the RAM allotment for a variety of settings for the sake of headroom), and use the emerging-compromised-ips.txt file option. I maintain a short suppress list (4 entries for frequent false positives). Everything is stable and it will run for weeks at a time so long as the two rules in question are unchecked (disabled).
I intentionally run pfSense in a double NAT configuration. One router faces the internet, and behind that router is pfSense (also in NAT configuration) on a different IP class (192.68.x.x and 10.x.x.x). I successfully configured port forwarding for RDP from the internet. VPN works well, and all other services function as normal. Before anyone criticizes this concept as "sacrilege", take a quick read through this article: https://www.grc.com/nat/nat.htm
Hardware tests from a bootable CD come clean: memory, processor, board, etc.
I've tried upgrading as well as a clean install for different versions of pfSense (2.13, 2.14, 2.15): no change in behavior of the issue.
For what it's worth, this issue has persisted through pfSense versions 2.13, 2.14, and now 2.15.
My config file is standard (only change implemented was to increase available memory).
–------------------------------------------------------------------------------------------------------------------------------
So, what is it about these two ET rules that will disable Snort within minutes?
I'll be happy to respond to questions about config and logs. I consider myself still new to pfSense, so please be clear about such requests. Thank you– sincerely so-- for your consideration and ideas!
There very well could be a problem with some of the content matching elements in one or more of the individual rules in those packages. It would not be unheard of for some particular content match to trigger a fault in the Snort binary. Unfortunately, if that is the case here, it's something the Snort VRT and/or Emerging Threats guys would have to fix. My personal guess would be an issue with one of the rules in those sets (or maybe two different rules with some similar content) that triggers a segfault or other failure when a matching packet comes in.
If you have some time and want to experiment further, I would start using just one of those categories and disabling all the rules and then enable them just one at the time. You can do this on the RULES tab. There is an icon for disabling all the rules in the currently selected category. So choose the category you are testing with in the drop-down, and then click the icon to disable all of its rules. Tooltip text will pop up when you hover over the icons to explain what they do. Doing this one by one could identify the specific rule that triggers the fault.
EDIT: another perhaps slightly faster way to do the same thing is to make note of the default enabled rules in the category, and then start disabling those one at the time. The "default enabled" rules will have a red X icon beside them. Default disabled rules will have a pale red X icon beside them. To disable a default enabled rule, click the X icon. It will change color to indicate it has been manually disabled by the user. Click APPLY after doing this to rebuild the rules and soft-start Snort. Give it some time to see if it still crashes, then rinse and repeat. You should discover the offending rule with this method, but it will take a little time.
Bill
-
Bill,
What you mention as a potential cause seems logical. I will carry this question over to the Snort forum and see if anyone considers a response after attempting your suggestion.
As for deducing the offending rule(s), I have done much of what you said already, but I have done so to the "WAN Categories" section, and NOT to the "WAN Rules" section. I used the "half-way there" method: use half of the rulesets on, and half off. Wait for the issue to appear (Snort becoming disabled). Try again, using half of the remaining rulesets each time. I finally narrowed it down to the 2 rulesets mentioned already. However, I did not consider that interactions between rulesets (on the "WAN Categories" tab) might be causing the issue as well as any single ruleset. Therefore, the number of permutations to test is time consuming given that each test will not produce a Snort failure immediately.
After reading your reply carefully, I realize that you are referring to the configuration of individual categories of rules (individual rule signature IDs) on the "WAN Rules" tab for each of the ruleset categories. And my post was referring only to the ET Open Rules found on the "WAN Categories" tab. I had not realized that each rule within a WAN Category set of rules could be configured individually! I overlooked this by taking note of the following warning on that page: "WARNING: You should not disable flowbit rules! Add Suppress List entries for them instead by clicking here." And so?… I never did disable those rules. Funny how the brain works at times and misses the elephant in the front yard.
Your recommendation is to essentially do what I have already done on the WAN Categories page. This process will take us another level of granularity further to define the offending rule.
Thank you for your time and contribution to this forum, Bill.
-
Bill,
What you mention as a potential cause seems logical. I will carry this question over to the Snort forum and see if anyone considers a response after attempting your suggestion.
As for deducing the offending rule(s), I have done much of what you said already, but I have done so to the "WAN Categories" section, and NOT to the "WAN Rules" section. I used the "half-way there" method: use half of the rulesets on, and half off. Wait for the issue to appear (Snort becoming disabled). Try again, using half of the remaining rulesets each time. I finally narrowed it down to the 2 rulesets mentioned already. However, I did not consider that interactions between rulesets (on the "WAN Categories" tab) might be causing the issue as well as any single ruleset. Therefore, the number of permutations to test is time consuming given that each test will not produce a Snort failure immediately.
After reading your reply carefully, I realize that you are referring to the configuration of individual categories of rules (individual rule signature IDs) on the "WAN Rules" tab for each of the ruleset categories. And my post was referring only to the ET Open Rules found on the "WAN Categories" tab. I had not realized that each rule within a WAN Category set of rules could be configured individually! I overlooked this by taking note of the following warning on that page: "WARNING: You should not disable flowbit rules! Add Suppress List entries for them instead by clicking here." And so?… I never did disable those rules. Funny how the brain works at times and misses the elephant in the front yard.
Your recommendation is to essentially do what I have already done on the WAN Categories page. This process will take us another level of granularity further to define the offending rule.
Thank you for your time and contribution to this forum, Bill.
Yes, the CATEGORIES tab controls families of related rules while the RULES tab allow customization of individual rules within a specific category. The rule set vendors ship their categories with some rules enabled by default and some disabled. They make the choices depending on a number of factors. Some rules are disabled when the exploit they were looking for is considered widely "fixed". If some particular admin still has some of machines not patch for that "widely fixed" flaw, then he can re-enable that particular rule in his setup.
You will also find some categories where pretty much every single rule within it is default disabled. The RULES tab will show you at a glance which rules within a category are default enabled or default disabled "out of the box". Admins can make their own forced adjustments by clicking the icons at the far left of each displayed rule on the RULES tab.
EDIT: about those "flowbit-rules"…they are special and generally should never be disabled. That is the only category on the RULES tab carrying that expectation, though. It's open season on the other categories. It's a bit much to explain here, but there are some good references scattered around on Google describing what flowbits are and that will help understand why I recommend never disabling flowbit rules. Suppress them instead.
Bill
-
With 5382 default-enabled rules in the category emerging-web_specific_apps.rules and 388 default-enabled rules in the category emerging-web_server.rules (the two categories that shut Snort down when activated), this could take a while to whittle down!
Thank you for the warning regarding the flowbit rules. I now understand them to allow rules to generically track the state of an application protocol and so, the auto-defined feature for flowbit rules is of great benefit give the sheer number of them.
Thanks again for your time. Very helpful. Now to walk half-way to 5382… :o
-
BTW, about autoflowbit rules… Every time I want to disable a rule, I first search through the list of autoflowbit rules to make sure the rule I want to disable is not on it.
Would it be possible to make autoflowbit rules ‘always enabled’, so that if one accidently tries to disable an autoflowbit rule, it would just stay enabled?
Thanks.
-
1. Snort will fail (become disabled) within minutes (usually NOT immediately) after enabling any combination of the following two rules under the "WAN Categories" tab:
a. emerging-web_server.rules
b. emerging-web_specific_apps.rulesI don't normally enable those "categories", but I tried to enable them on my Box
With the Following Error:
Fatal error: Allowed memory size of 201326592 bytes exhausted (tried to allocate 71 bytes) in /usr/local/pkg/snort/snort.inc on line 976
Maybe this will help shed some light on this issue? If I only enable the First Cat, it didn't report the error. Running just the First Category doesn't seem to trip out Snort thou.
That part of the code just seems to be where its reading the rules into an array.
-
@G.D.:
BTW, about autoflowbit rules… Every time I want to disable a rule, I first search through the list of autoflowbit rules to make sure the rule I want to disable is not on it.
Would it be possible to make autoflowbit rules ‘always enabled’, so that if one accidently tries to disable an autoflowbit rule, it would just stay enabled?
Thanks.
No, that would not be very easy to implement in code. Not saying impossible, but hard.
The thing about flowbit rules is that 99% of them do not produce alerts. They just set bits that other rules may read. Look sometimes at the list of flowbit rules by clicking that VIEW button beside the Auto-Flowbits option on the CATEGORIES tab. Notice how pretty much all of the flowbit rules will have a flowbit value that includes the keyword "no-alert". That means the rule will set the corresponding flowbits, but not cause an alert. There are just a few exceptions to the "no-alert" rule, and I think most of those are in the Emerging Threats set. The VRT rules generally always include the "no-alert" option with flowbit rules.
So the purpose of the long-winded explanation above is to say there is usually no reason to disable a flowbit rule for fear of a false alert: most of them have the "no-alert" keyword in them.
Bill
-
1. Snort will fail (become disabled) within minutes (usually NOT immediately) after enabling any combination of the following two rules under the "WAN Categories" tab:
a. emerging-web_server.rules
b. emerging-web_specific_apps.rulesI don't normally enable those "categories", but I tried to enable them on my Box
With the Following Error:
Fatal error: Allowed memory size of 201326592 bytes exhausted (tried to allocate 71 bytes) in /usr/local/pkg/snort/snort.inc on line 976
Maybe this will help shed some light on this issue? If I only enable the First Cat, it didn't report the error. Running just the First Category doesn't seem to trip out Snort thou.
That part of the code just seems to be where its reading the rules into an array.
Yep, this helps. That would be the code that is parsing the rules category file and pulling the pieces (GID, SID, MSG, FLOWBITS, etc.) into a large in-memory array for further processing. I will take a look at this and see if I can figure out what's happening. Could be the PCRE I'm using to parse the rules is blowing up on something, or there may be a rule in there that is malformed.
Bill
-
Guys:
I tried to reproduce this in my test virtual machines and was unable. I tried both the ET-OPEN and ET-PRO versions of these two rules. Now my test environment has very limited network traffic and really nothing to trigger alerts in either of these rule categories, but I was hoping I might reproduce BBcan177's out-of-memory error. I could not. However, I am testing using a slightly newer Snort package (the update I'm working on). It has been bumped up to a 256MB memory limit for PHP where the current production package has a 192MB memory limit for PHP.
For those of you having the problem, try this for me and report back.
Go to Diagnostics…Edit File and browse to and open this file:
/usr/local/pkg/snort/snort.incFind this line near the top of that file:
// Snort GUI needs some extra PHP memory space to manipulate large rules arrays ini_set("memory_limit", "192M");and change it to read like this:
// Snort GUI needs some extra PHP memory space to manipulate large rules arrays ini_set("memory_limit", "256M");Save the change and then see if you still get the out-of-memory error.
Bill
-
I've been away for a couple of days– sorry for the intermission.
I haven't been able to reproduce the memory error that BBcan177 is experiencing. I am running a (mostly) default install of pfSense and Snort on bare metal with the exception of the fact that I increased the memory of many parameters to offer headroom... to prevent out of memory errors. Just in case.
Per bill Meek's request, I added the following lines to my snort.inc file:
// Snort GUI needs some extra PHP memory space to manipulate large rules arrays
ini_set("memory_limit", "256M");Since I was NOT experiencing this specific issue, I didn't expect any behavior to change.
Additionally, I have not had the time to whittle down the specific rule(s) responsible for my issue. There are, after all, over 5000 emerging-web_specific_apps.rules to work with.
I do have a few updates to add to the log:
Rather surprisingly (and somewhat disconcertingly), after several months of slowly working with this error on my own involving the two rule sets (emerging-web_specific_apps.rules and emerging-web_server.rules) which causes Snort to enter a 'Disabled' state within minutes to hours of enabling either or both of those two rule sets, I can no longer reproduce the error at all. In fact, turning ON those two rule sets will no longer cause Snort to become disabled as it once did. This new behavior was observed within approximately 24 hours of these posts to the pfSense forum. And I've made no changes to the system. None.
Perhaps Bill or someone on the Snort/pfSense team read this post and fixed the issue in a rules update? I can't determine any other cause for the change in behavior. This is an exceptional coincidence in my many years of tinkering with PCs. In fact, instead of considering the issue closed and moving ahead, I am even more curious now to determine what is happening "under the hood".
Another bit of info to consider: once I had determined that one or both of the rule sets in question would disable Snort, I of course verified this hypothesis by working with all permutations of those rule sets to verify that the problem was reproducible before posting to this forum.
BBcan177: short of re-installing pfSense or reverting to a default configuration, have you tried to run a hardware test to ensure that your memory and other components are in working order? I have found bad memory in a number of computers over the years; bad memory can cause very strange behavior and sometimes the problem is consistent and reproducible, and other times it is not.
Thank you all for your assistance and input.
