Snort 4.1.6_7 crash report / Problems force-disabling rules

jsnl

Installing a new firewall, but having problems with suppression lists and force-disabling rules. I haven't had this issue on any other firewall.

When I click the red X in Alerts under gid:sid (the one to force-disable a rule) it attempts to work properly - it reloads the page, pops up the green "wait 15 seconds" prompt, and the X turns yellow. However 15 seconds later (or on the next page reload) the X goes back to red and the rule is not disabled.

This was from a fresh install of 23.01. I've even tried importing suppression lists and configs from other working boxes, but nothing fixes it.

On investigating the problem and going into the rules display on my WAN, I got a crash.

Crash report begins.  Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #0 plus-RELENG_23_01-n256037-6e914874a5e: Fri Feb 10 20:30:29 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/obj/amd64/VDZvZksF/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBS

Crash report details:


[29-Mar-2023 07:40:15 America/New_York] PHP Fatal error:  preg_replace(): Cannot use output buffering in output buffering display handlers in /usr/local/www/csrf/csrf-magic.php on line 159

They may be unrelated, but this problem is getting stranger and stranger. I've tried uninstalling, clearing config, and reinstalling the package. I've even tried hard-copying the config xml from another matching box that works. Nothing makes it work. This problem exists for all interfaces (WAN, LAN) and changing suppression lists doesn't help.

Thoughts?

bmeeks

That error is not being generated by the Snort package directly. It is coming from the pfSense web session code that is responsible for cross-site request forgery protection. That's what csrf means.

I think you have some other issue within the pfSense web GUI, and it is manifesting when attempting to perform the actions in Snort.

jsnl

Interesting. I went ahead and did a fresh install, and didn't do the plus upgrade before setting things up. Snort isn't reporting any blocked traffic. I went into the rules again, and the individual rules display fine but once again viewing all active rules crashed the box.

Crash report begins.  Anonymous machine information:

amd64
12.3-STABLE
FreeBSD 12.3-STABLE RELENG_2_6_0-n226742-1285d6d205f pfSense

Crash report details:

PHP Errors:
[29-Mar-2023 12:29:57 US/Michigan] PHP Fatal error:  Allowed memory size of 402653184 bytes exhausted (tried to allocate 168780416 bytes) in /usr/local/www/csrf/csrf-magic.php on line 159



No FreeBSD crash data found.

Is anyone else noticing this if you go to view all active rules? I have 16G of RAM and it's not close to being full. I have /var and /tmp on ramdisks each with 1G space, and I'm still not over 25% utilization.

bmeeks

@jsnl:
This issue of PHP running out of space when trying to view log files comes up all the time. It has nothing to do with how much RAM is in your box. It is a consequence of the amount of RAM the PHP process reserves for running applications.

For very large files, the PHP code first attempts to read the file contents into a string variable in memory, then once that is done it writes the string out all at once to the browser. Very large text files will exceed the amount of available PHP memory. This most often happens when users let their log files grow uncontrolled. In your case, it would appear you have enabled a ton of rules and thus get the error when attemtping to view them all.

Because web-based PHP systems don't really maintain a "state" with the browser allowing easy two-way communication, it is not an easy task to read a file a few lines at the time and send it to the web client in small chunks. That's just not how PHP is designed. It expects to dump out the web page in one fell swoop and be done with the session. There are some hacks to work around this a bit, but they introduce other problems.

So, TLDR, there is no easy solution to this problem.

jsnl

@bmeeks Understood. That makes perfect sense, thanks.

I uninstalled and reinstalled the Snort package (again). I'm getting blocking now. Waiting for some inappropriate blocking to test the issue from the first post. Once I get a successful test I'm going to do the plus upgrade and see what happens.

jsnl

So without plus, I was able to successfully block rules.

I upgraded to plus. Now I have the same issue as before - I can't force-disable rules. It does all of the statusing as if it's doing something, but the rule doesn't stay blocked.

All rules that were blocked before the plus upgrade are still blocked (statused yellow) and don't revert.

How can I troubleshoot this further?

bmeeks

@jsnl said in Snort 4.1.6_7 crash report / Problems force-disabling rules:

So without plus, I was able to successfully block rules.

I upgraded to plus. Now I have the same issue as before - I can't force-disable rules. It does all of the statusing as if it's doing something, but the rule doesn't stay blocked.

All rules that were blocked before the plus upgrade are still blocked (statused yellow) and don't revert.

How can I troubleshoot this further?

Have you tried simply refreshing the page? Click on the tab at the top to reload/refresh the page. So ALERTS, RULES, etc., depending on where you are in the Snort tabs menu.

jsnl

@bmeeks Yes I have. The state resets.

I went looking for the rule (119:2 in this case) which I found out was a preprocessor rule. I went into WAN Rules and into preprocessor, and I can manually disable 119:2 and hit apply and it sticks. So it's something about the way the link is working from the "alerts" page. It triggers the green message and orange X, but the rule doesn't block (or stay blocked longer than 15 seconds).

booshwa

@jsnl Did anything ever come of this? I've had the snort force-disable problem since 23.01. Checked that there weren't two processes and even killing the active one with no resolve, there doesn't seem to be a lot of conversation about it either https://www.reddit.com/r/PFSENSE/comments/11w6c51/snort_forcedisable_no_longer_working/.

The problem exists separate from the crash report mentioned above and seems to be isolated to pfsense+.

Version: 23.05-RELEASE
System: Netgate 5100

jsnl

@booshwa Unfortunately no. I still have the issue on certain boxes, and haven't been able to find a way around it.

Fortunately manually disabling the rule in "rules" works. But it's just a terrible way to go about it when there's an infinitely easier way that's supposed to be available to us. I often find myself completely turned around looking through all of the subsets to find the rule I want to disable, and I can't display all rules and search for it because it takes too much memory and crashes. (Which should be another reported bug...)

bmeeks

I've verified the issue with saving "user force-disabled" rule states on the ALERTS tab in Snort. In fact, a couple of things on this page seem to be broken with the move to PHP 8.1 back early in 2023. Don't really know how I missed it during my testing back then as I migrated the package code over to PHP 8.1.

I did not find a Redmine ticket for the problem. No Redmine issue is likely how it has continued to slip by for so long. I will submit a Redmine ticket for the problems and work on getting them corrected.

FYI, I do not have a Reddit account and do not normally check in there to read any posts. If you have issues with Snort's operation and suspect a bug, please open a pfSense Redmine ticket here; https://redmine.pfsense.org/projects/pfsense. This is the official bug tracking system for pfSense and its associated packages.

Edit: I've created the following Redmine Issue to track the resolution of this bug: https://redmine.pfsense.org/issues/14832.

bmeeks

The pull request containing the fix for the user force-disabling rules issue on the ALERTS tab has been posted and is awaiting review and merging by the Netgate developer team. You can follow the progress here: https://github.com/pfsense/FreeBSD-ports/pull/1300.

booshwa

@bmeeks Awesome, thanks for taking time to look into this!

jsnl

@bmeeks Yes, thank you! Much appreciated.

jsnl

@bmeeks Verified that it's working perfectly on 4.1.6_11.

Thanks again!