Does this look like a possible buffer overflow attempt?
-
I normally avoid taking valuable time away from busy people, but here goes...
I just logged on my pfSense Plus installation in order to see why network performance was too slow, and I got this:Crash report begins. Anonymous machine information: amd64 14.0-CURRENT FreeBSD 14.0-CURRENT #0 plus-RELENG_23_01-n256037-6e914874a5e: Fri Feb 10 20:30:29 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/obj/amd64/VDZvZksF/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBS Crash report details: PHP Errors: [26-Mar-2023 05:40:40 Europe/Athens] PHP Fatal error: Uncaught ValueError: date_create_from_format(): Argument #2 ($datetime) must not contain any null bytes in /usr/local/www/widgets/widgets/suricata_alerts.widget.php:183 Stack trace: #0 /usr/local/www/widgets/widgets/suricata_alerts.widget.php(183): date_create_from_format('m/d/Y-H:i:s.u', '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...') #1 /usr/local/www/widgets/widgets/suricata_alerts.widget.php(78): suricata_widget_get_alerts() #2 {main} thrown in /usr/local/www/widgets/widgets/suricata_alerts.widget.php on line 183 [26-Mar-2023 06:06:40 Europe/Athens] PHP Fatal error: Uncaught ValueError: date_create_from_format(): Argument #2 ($datetime) must not contain any null bytes in /usr/local/www/widgets/widgets/suricata_alerts.widget.php:183 Stack trace: #0 /usr/local/www/widgets/widgets/suricata_alerts.widget.php(183): date_create_from_format('m/d/Y-H:i:s.u', '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...') #1 /usr/local/www/widgets/widgets/suricata_alerts.widget.php(78): suricata_widget_get_alerts() #2 {main} thrown in /usr/local/www/widgets/widgets/suricata_alerts.widget.php on line 183 [26-Mar-2023 06:10:40 Europe/Athens] PHP Fatal error: Uncaught ValueError: date_create_from_format(): Argument #2 ($datetime) must not contain any null bytes in /usr/local/www/widgets/widgets/suricata_alerts.widget.php:183 Stack trace: #0 /usr/local/www/widgets/widgets/suricata_alerts.widget.php(183): date_create_from_format('m/d/Y-H:i:s.u', '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...') #1 /usr/local/www/widgets/widgets/suricata_alerts.widget.php(78): suricata_widget_get_alerts() #2 {main} thrown in /usr/local/www/widgets/widgets/suricata_alerts.widget.php on line 183 No FreeBSD crash data found.I don't even know where to begin troubleshooting this.
-
I ask about a possible buffer overflow attempt because of the null chars (
\x00). -
Update: After rebooting pfSense, everything was back to normal, eventually... Immediately after rebooting, CPE gateway/router/VoIP combo went down, too. I think I don't need to say that it keeps no logs whatsoever.
-
I know that at least this already exists, but I had no crashes or problems prior to this.
-
Got it again:
-
So, I hope this is the final UPDATE:
I went to view the logs, and no logs existed. It said that either they were not enabled, or that the files didn't exist.
I then went to 'Logs Management' tab, where also no logs existed. Upon saving the existing configuration, without changing anything, log files magically appeared.
I'm thinking, a bug, maybe?
When logs reach maximum size and are deleted, new files are not immediately created?
IDK. -
Are you by chance using a RAM disk? If not, what hardware do you have (a spinning hard drive or some type of eMMC device)?
Check your pfSense system log to see if anything was recorded there relative to running out of space.
To answer your question about log rotation, the Suricata PHP package will attempt to rotate the logs (and delete those older than the minimum retention age) and then perform a "touch" operation to create new empty logs. However, the Suricata binary does not natively offer a log rotation option by size. It only offers rotation on a fixed time interval (and does not even allow that for all the logs). So, the PHP package code has to shuffle the physical log files and then send the Suricata binary daemon a SIGHUP telling it to restart logging. Sometimes the binary seems to ignore that request. Usually when this happens, the old log files continue to grow and grow until the disk space is consumed.
The original error you see there is a result of the code on the ALERTS tab (or in the Dashboard Widget) reading a corrupted alerts log file. Zeros are never expected there as Suricata will not write such an entry normally. Only a corrupted file read would result in that.
-
@bmeeks Hi! Not using RAM disk, I have an HDD, but its monitored by TrueNAS, not running out of space.
-
Somewhere here is the reason this happens:
$buf = ""; while (($buf = fgets($fd)) !== FALSE) { $fields = array(); $tmp = array(); // Parse alert log entry to find the parts we want to display $fields[0] = substr($buf, 0, strpos($buf, ' ')); // The regular expression match below returns an array as follows: // [2] => GID, [3] => SID, [4] => REV, [5] => MSG, [6] => CLASSIFICATION, [7] = PRIORITY preg_match('/\[\*{2}\]\s\[((\d+):(\d+):(\d+))\]\s(.*)\[\*{2}\]\s\[Classification:\s(.*)\]\s\[Priority:\s(\d+)\]\s/', $buf, $tmp); $fields['gid'] = trim($tmp[2]); $fields['sid'] = trim($tmp[3]); $fields['rev'] = trim($tmp[4]); $fields['msg'] = trim($tmp[5]); $fields['class'] = trim($tmp[6]); $fields['priority'] = trim($tmp[7]); // The regular expression match below looks for the PROTO, SRC and DST fields // and returns an array as follows: // [1] = PROTO, [2] => SRC:SPORT [3] => DST:DPORT if (preg_match('/\{(.*)\}\s(.*)\s->\s(.*)/', $buf, $tmp)) { // Get SRC $fields['src'] = trim(substr($tmp[2], 0, strrpos($tmp[2], ':'))); if (is_ipaddrv6($fields['src'])) $fields['src'] = inet_ntop(inet_pton($fields['src'])); // Get SPORT $fields['sport'] = trim(substr($tmp[2], strrpos($tmp[2], ':') + 1)); // Get DST $fields['dst'] = trim(substr($tmp[3], 0, strrpos($tmp[3], ':'))); if (is_ipaddrv6($fields['dst'])) $fields['dst'] = inet_ntop(inet_pton($fields['dst'])); // Get DPORT $fields['dport'] = trim(substr($tmp[3], strrpos($tmp[3], ':') + 1)); } else { // If no PROTO and IP ADDR, then this is a DECODER EVENT $fields['src'] = gettext("Decoder Event"); $fields['sport'] = ""; $fields['dst'] = ""; $fields['dport'] = ""; } // Create a DateTime object from the event timestamp that // we can use to easily manipulate output formats. if (($event_tm = date_create_from_format("m/d/Y-H:i:s.u", $fields[0])) !== FALSE) { $suricata_alerts[$counter]['timestamp'] = strval(date_timestamp_get($event_tm)); $suricata_alerts[$counter]['timeonly'] = date_format($event_tm, "H:i:s"); $suricata_alerts[$counter]['dateonly'] = date_format($event_tm, "M d"); } else { // For some reason the event timestamp was invalid, // set some default empty values for the fields. $suricata_alerts[$counter]['timestamp'] = 0; $suricata_alerts[$counter]['timeonly'] = ' '; $suricata_alerts[$counter]['dateonly'] = ' '; } -
@NightlyShark
I'm thinking the root cause here is a corrupted alerts log file. The PHP code is parsing the alerts log file in/var/log/suricata/suricata_xxxx/to grab the individual fields and store them into a multidimensional array.The "time" field is the first one parsed, and it happens with this line of code:
// Parse alert log entry to find the parts we want to display $fields[0] = substr($buf, 0, strpos($buf, ' '));Then later on in the code, this line executes to translate the parsed text into a PHP
dateobject for later formatting.// Create a DateTime object from the event timestamp that // we can use to easily manipulate output formats. if (($event_tm = date_create_from_format("m/d/Y-H:i:s.u", $fields[0])) !== FALSE) {The line above is triggering the error because for some reason the text field is empty. That should never happen. Can you post the contents of the alerts log when this is happening?
-
@bmeeks I am sorry for the delay, being an enthusiast puts a limit on how much time I can devote to my server... I disabled the widget and saw no other errors. I will re-enable it (just for testing) and will pull any logs on the first available chance.