Suricata 7.0.2_2 Cannot use output buffering in output buffering display handlers in Unknown on line 0

shabsta

After upgrading pfsense from version 2.6.0 > 2.7.0 > 2.7.2 I am no longer able to load the Suricata block page, below is the crash report generated
PHP ERROR: Type: 1, File: /usr/local/www/suricata/suricata_blocked.php, Line: 428, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 46239744 bytes) @ 2023-12-14 11:08:54

The above happens when I increase memory value of /etc/inc/config.inc to 2048 as per below
ini_set("memory_limit", "2048M");

I did manage to get the block list page to load once or twice without the error and during that time there were roughly 400 hosts blocked, how ever it appears that Suricata is storing historical value for longer for hosts that are blocked.

Any suggestions on the above?

shabsta

This morning I upgraded to 7.0.2_3 hoping that it would resolve the issues reported above where I cannot access the block list page.
After the upgraded completed I was able to access the blocks page and set the number of records to display to a few hundred whilst suricata was reloading.

After a minute or two I can no longer access the block page with the error below, how ever the alerts page shows fine and without issue.

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 4096 bytes) in /usr/local/www/suricata/suricata_blocked.php on line 344 PHP ERROR: Type: 1, File: /usr/local/www/suricata/suricata_blocked.php, Line: 344, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 4096 bytes)

Suricata is configured to only block hosts for 15 minutes and this keeps the log file small enough to be able to view blocks page without errors, prior to upgrade as per my previous post.

bmeeks

@shabsta said in Suricata 7.0.2_2 Cannot use output buffering in output buffering display handlers in Unknown on line 0:

This morning I upgraded to 7.0.2_3 hoping that it would resolve the issues reported above where I cannot access the block list page.
After the upgraded completed I was able to access the blocks page and set the number of records to display to a few hundred whilst suricata was reloading.

After a minute or two I can no longer access the block page with the error below, how ever the alerts page shows fine and without issue.
Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 4096 bytes) in /usr/local/www/suricata/suricata_blocked.php on line 344 PHP ERROR: Type: 1, File: /usr/local/www/suricata/suricata_blocked.php, Line: 344, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 4096 bytes)
Suricata is configured to only block hosts for 15 minutes and this keeps the log file small enough to be able to view blocks page without errors, prior to upgrade as per my previous post.

My gosh! How many blocks per minute are you processing on your firewall?

That error is self-evident. The block.log file (and it previously rotated copies) are together too large to fit into the restricted PHP string memory space. This is a hard limit set by PHP on pfSense. It does not matter how much RAM you have in the firewall, PHP is only allocated up to set amount. Reading verly text log files and sending them out as HTML with PHP does not work well. You will see this error with a number of packages and even from pfSense itself if you attempt to view a large enough text file.

The only solution is to reduce your log retention settings to keep the block log sizes down and to not keep as many rotated copies.

shabsta

@bmeeks I have taken a look at the Log Settings page and compared them to the pfsense documentation and confirm that the log settings are default with no modifications to them.
I appreciate that we may be blocking and logging many blocked IP's and connection attempts per second seem high, but that doesnt address the issue of why this worked prior to upgrade.
Any pointers of what I can try change would be appreciated at this time.

Based on memory, the number of times that an IP is blocked and displayed on the suricata blocked page does seem substantially longer than what I recall seeing on the system when version 2.6.0 was installed.

bmeeks

@shabsta said in Suricata 7.0.2_2 Cannot use output buffering in output buffering display handlers in Unknown on line 0:

I appreciate that we may be blocking and logging many blocked IP's and connection attempts per second seem high, but that doesnt address the issue of why this worked prior to upgrade.
Any pointers of what I can try change would be appreciated at this time.

Your block.log file is too large to fit into available PHP memory as a string. The only reason it would have worked before is the file was smaller. You need to account for the rotated files as well.

Look in /var/log/suricata/suricata_xxxyyyy on the firewall and check the file sizes of the block.log and all other rotated block.log files listed there. The rotated files will have a UNIX timestamp appended onto them. I know what I am talking about because I am the creator/maintiner for the package. It is not maintained by Netgate personnel.

The PHP code reads the entire file (including rotated files) into a PHP string in RAM and then writes that out to the HTML client. That means the entire file must fit into the available PHP memory space. That is limited by pfSense to 536,870,912 bytes. If the combined file sizes exceed that limit, the PHP error you are seeing will be thrown.

SteveITS

@shabsta Check the Suricata logs page:

bmeeks

@SteveITS said in Suricata 7.0.2_2 Cannot use output buffering in output buffering display handlers in Unknown on line 0:

@shabsta Check the Suricata logs page:

There are also settings for individual log files, including the block.log file that is most likely the culprit for the OP.

shabsta

Thank you to both of you, will review those settings and revert back.

shabsta

Thank you for your assistance within this thread and kindly accept my apologies for not responding sooner.

I was able to get the Blocked page to load after deleting a log from the server and have been monitoring the status of this page over the last few days.

Bill, would be possible to limit the number of times an IP is listed on the block page to say 10 or 20 results per connection as this may resolve the issue.
If it is possible then excuse me for asking what is already there and could you point me in the right direction of how to limit how much data is returned.

shabsta

As you can see below, one particular IP has attempted to connect over 100 times and is output on the block page, and at this time there are 501 blocked IP's.
If say each blocked IP attempted to connect to a server within our environment 500 times then the blocked page quickly becomes too much for php to handle

shabsta

Edit, Im unable to add the logs as the Post content was flagged as spam by Akismet.com keeps rejecting my post

bmeeks

As I mentioned previously, you will need to reduce the number of rotated blocks.log files you keep on hand. The code reads all of those files from disk into RAM, then correlates them with the current alerts.log file. The purpose is to show how many times a given IP has been blocked. This was done because in the past users wanted to have more "history" available for a blocked IP.

I recommend to users that they enable the option on the GLOBAL SETTINGS tab to automatically remove blocked IPs after an interval, and my suggested interval is 1 hour (or 3 hours max). There is no point in keeping an IP in the block table forever in my view. If Suricata blocked it once, it will block it the next time it attempts to connect. Let that table automatically clear itself out every hour or every 3 hours and you won't have the out-of-memory issue.

The auto-clear routine will not remove IP addresses that are seeing active traffic. It will only remove an IP that has been in the table for the interval chosen AND that IP has not seen any traffic during that interval. For example, if you select 1-hour as the automatic clear interval, then it will only remove IPs from the block table that have not been the source of any traffic for at least the past hour. If an IP is continuing connection attempts, then it will not be removed by the auto-clear routine.