pfSense Suricata Crashes on Malformed Block List Entry

micah

Subject:
Suricata is crashing my pfSense box. It looks like it is trying to parse an intentionally malformed blocklist entry. I can't get to the blocklist to identify it and it periodically crashes my box.

System Details:
Operating System: pfSense 2.7.0
Suricata Version: 6.0.13

Crash Details:

PHP Errors:
[18-Oct-2023 23:50:07 UTC] PHP Fatal error:  Uncaught ValueError: date_create_from_format(): Argument #2 ($datetime) must not contain any null bytes in /usr/local/www/suricata/suricata_blocked.php:326
Stack trace:
#0 /usr/local/www/suricata/suricata_blocked.php(326): date_create_from_format('m/d/Y-H:i:s.u', '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...')
#1 {main}
  thrown in /usr/local/www/suricata/suricata_blocked.php on line 326

It looks like this is an intentionally malformed connection packet. I also think we should probably not be crashing, but handling this error in a safer manner. It seems very unsafe to 500 because someone sends a malformed packet. How can I safely patch this behavior?

I can attach the full PHP crash dump if needed, but I think there’s enough info here to get started.

bmeeks

I am the volunteer package creater/maintainer for the Suricata package on pfSense. The package is not maintained by the Netgate team.

This is not an RCE. It is simply a result of a corrupt text file on your box. The data in that file is writtten by the custom Suricata blocking plugin used with Suricata on pfSense.

The PHP code simply parses the text log file into a series of space-separated fields and then reads each field into an array for display on the BLOCKS tab. You have an entry that has an invalid value of some type in the event time field for the block entry. It is not intentionally malformed nor is there any way for a remote user to cause this problem. It can only happen directly on the local file system from a process with root priviledges. The PHP error will only cause a problem on the BLOCKS tab of Suricata. It will not impact any other running piece of the firewall nor will it even impact the currently running Suricata daemon.

Here is the relevant PHP code --

/*************** FORMAT for file -- BLOCK -- **************************************************************************/
/* Line format: timestamp  action [**] [gid:sid:rev] msg [**] [Classification: class] [Priority: pri] {proto} ip:port */
/*              0          1            2   3   4    5                         6                 7     8      9  10   */
/**********************************************************************************************************************/
$buf = "";
while (($buf = fgets($fd)) !== FALSE) {
	$fields = array();
	$tmp = array();
	/***************************************************************/
	/* Parse block log entry to find the parts we want to display. */
	/* We parse out all the fields even though we currently use    */
	/* just a few of them.                                         */
	/***************************************************************/
	// Field 0 is the event timestamp
	$fields['time'] = substr($buf, 0, strpos($buf, '  '));

	// Create a DateTime object from the event timestamp that
	// we can use to easily manipulate output formats.
	$event_tm = date_create_from_format("m/d/Y-H:i:s.u", $fields['time']);

The code is not expecting any NULL entries from the log file. However, there have been a few random reports of this error both on the ALERTS and BLOCKS tabs. I can add an additional test and disregard any lines pulled from the file that might contain a NULL field.

micah

@bmeeks Great! Thank you for your feedback here. It had me worried about the risk, but this makes me feel better.

You are right that page doesn't load at all for me. When I purge my blocklist, that page loads for a brief amount of time. But, usually, within a day or so, I start getting that error again. Our pfSense box is our edge device, so we get all manner of malformed web request attacks. I just suspected that it was one of those.

The odd part here is that I get a crash report and these errors get reported daily - even on days that I don't log in or go to that page. So, I suspect there must be something that is trying to render that page in the background. Otherwise, I don't know how it would generate those errors.

bmeeks

@micah said in pfSense Suricata Crashes on Malformed Block List Entry:

The odd part here is that I get a crash report and these errors get reported daily - even on days that I don't log in or go to that page. So, I suspect there must be something that is trying to render that page in the background. Otherwise, I don't know how it would generate those errors.

The PHP error would only be generated when that tab was loaded by a user logged into the firewall. Are you the only user, or might someone else be logging in and viewing that tab?

The other potential source of that error would be a corrupted alerts log file for the interface. The alerts log is read by both the code on the ALERTS tab in the package and by the Suricata Dashboard Widget. The Dashboard Widget, if installed, would be reading through the alerts log each time the home page was loaded in pfSense.

I would love to see a log file with this actual problem. I have never encountered this during any testing I've done. The next time you see the error, post the current block.log file back here if you can. I want to see if it is actually corrupt as I suspect. Another possibility is a corrupt rotated block.log file. Rotated files will have a timestamp suffix on their filename. These logs will be under /var/log/suricata/suricata_xxxx/ where the xxxx is the physical interface name and a UUID.

micah

@bmeeks I am the only web admin user in our system. So, it's just me managing our pfSense box. I am using that dashboard widget though, so that is probably the source.

I received this error just this morning. I'll be back to my office in a couple of hours. I'll grab a copy of the block.log and send it over. Do you want a copy of the php error log too?

bmeeks

@micah said in pfSense Suricata Crashes on Malformed Block List Entry:

Do you want a copy of the php error log too?

No, no need to have the PHP error. I know what that one is and it's just PHP complaining about a NULL value for an argument. The point where the PHP error is happening is the very last line of code in the snippet I posted. The real issue to identify is what the log file contains and how it came to have an invalid line.

Seeing if the bad line is the first thing in the file or if it is located farther into the file will hopefully give me some theory to work with on how the file came to contain the invalid value.

bmeeks

The fix for this issue is coming with the next Suricata package update to 7.0.2. The pull request for review and merging by the Netgate developer team is posted here: https://github.com/pfsense/FreeBSD-ports/pull/1313.

micah

@bmeeks Sorry that I haven't had a chance to reply. I tested, and it fixed everything! Thank you very much!

bmeeks

@micah said in pfSense Suricata Crashes on Malformed Block List Entry:

@bmeeks Sorry that I haven't had a chance to reply. I tested, and it fixed everything! Thank you very much!

You're welcome. Thank you for following up with the confirmation.

I never did figure out how the blank lines got in the file, so I just fixed the code reading in the log file to detect and skip blank lines.

One theory is maybe it happens during log rotation, but that's just a guess.

micah

@bmeeks that would make sense. It does take a few hours to appear again after I purge the block list.

micah

Hi @bmeeks,

Just a heads up. I just logged in to check my pfSense today and it appears to still be crashing. That's so odd. It seems like it worked just fine for a few days after your patch.

bitslammer

I'm seeing the same thing. Seems to happen after a rules updatte.

Dec 20 00:39:27 kernel pid 95704 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Dec 20 00:36:05 kernel pid 91608 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Dec 20 00:30:50 php-cgi 11190 [Suricata] The Rules update has finished.

I've also seen a couple entries in the Suricata log about running out of memory which has never happened. I'm running a Netgate 3100 with a pretty small ruleset and have plenty of memory.

bmeeks

@micah said in pfSense Suricata Crashes on Malformed Block List Entry:

Hi @bmeeks,

Just a heads up. I just logged in to check my pfSense today and it appears to still be crashing. That's so odd. It seems like it worked just fine for a few days after your patch.

Crashing with the exact same PHP error message or something else?

bmeeks

@bitslammer said in pfSense Suricata Crashes on Malformed Block List Entry:

I'm seeing the same thing. Seems to happen after a rules updatte.

Dec 20 00:39:27 kernel pid 95704 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Dec 20 00:36:05 kernel pid 91608 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Dec 20 00:30:50 php-cgi 11190 [Suricata] The Rules update has finished.

I've also seen a couple entries in the Suricata log about running out of memory which has never happened. I'm running a Netgate 3100 with a pretty small ruleset and have plenty of memory.

I found two more bugs yesterday in the custom Legacy Blocking Module we use in Suricata on pfSense. Found those while looking into the reported Hyperscan fatal error exit bug being discussed in another very long thread in this sub-forum.

The additional bugs I found yesterday can most definitely lead to the Signal 11 segfault errors you are experiencing. They come into play when accessing the Pass List. They also appear to have been impacting the Hyperscan library functionality, too.

I am waiting on final confirmation from the Hyperscan bug testers running a test build I sent them that my fixes yesterday actually worked. They are letting their machines run for some time to be sure there is no latent crash. If the fixes have indeed solved the issue, then I will submit a pull request to the Netgate developer team with a package update.

bitslammer

@bmeeks Not sure. I will try and do more digging the next time. It's intermittent.

bitslammer

@bmeeks

Was able to grab more logs with the most recent crash. I did not see any errors in the system or PHP logs.

[108473 - Suricata-Main] 2023-12-22 00:31:02 Notice: detect: rule reload complete
[108473 - Suricata-Main] 2023-12-23 00:30:21 Notice: detect: rule reload starting
[108473 - Suricata-Main] 2023-12-23 00:30:21 Info: conf-yaml-loader: Configuration node 'filetype' redefined.
[108473 - Suricata-Main] 2023-12-23 00:30:21 Error: detect-within: previous keyword has a fast_pattern:only; set. Can't have relative keywords around a fast_pattern only content
[108473 - Suricata-Main] 2023-12-23 00:30:21 Error: detect: error parsing signature "alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"MALWARE-OTHER Win.Trojan.Zeus Spam 2013 dated zip/exe HTTP Response - potential malware download"; flow:to_client,established; content:"-2013.zip|0D 0A|"; fast_pattern:only; content:"-2013.zip|0D 0A|"; http_header; content:"-"; within:1; distance:-14; http_header; file_data; content:"-2013.exe"; content:"-"; within:1; distance:-14; metadata:impact_flag red, policy balanced-ips drop, policy max-detect-ips drop, policy security-ips drop, ruleset community, service http; reference:url,www.virustotal.com/en/file/2eff3ee6ac7f5bf85e4ebcbe51974d0708cef666581ef1385c628233614b22c0/analysis/; classtype:trojan-activity; sid:26470; rev:2;)" from file /usr/local/etc/suricata/suricata_3659_mvneta1/rules/suricata.rules at line 163
[108473 - Suricata-Main] 2023-12-23 00:30:27 Info: detect: 2 rule files processed. 16950 rules successfully loaded, 1 rules failed
[108473 - Suricata-Main] 2023-12-23 00:30:27 Info: threshold-config: Threshold config parsed: 0 rule(s) found
[108473 - Suricata-Main] 2023-12-23 00:30:28 Info: detect: 16950 signatures processed. 131 are IP-only rules, 2837 are inspecting packet payload, 13856 inspect application layer, 7 are decoder event only
[108473 - Suricata-Main] 2023-12-23 00:30:28 Warning: detect-flowbits: flowbit 'et.http.PK' is checked but not set. Checked in 2019835 and 1 other sigs
[108473 - Suricata-Main] 2023-12-23 00:30:36 Error: mpm-ac: Error allocating memory

bmeeks

@bitslammer said in pfSense Suricata Crashes on Malformed Block List Entry:

[108473 - Suricata-Main] 2023-12-23 00:30:36 Error: mpm-ac: Error allocating memory

This line tells you the problem. You do not have enough free RAM in the box to run the number of rules you have enabled with Suricata 7.x and its increased memory requirements for TCP stream memcap and reassembly memcap.

It's time to replace your hardware or else abandon attempting to run Suricata on it. That is 32-bit ARM hardware, so Hyperscan won't work at all.

You can try rather drastically reducing your enabled rules to see if that helps. I see you have 16,950 rules enabled. With Suricata 7.x and its increased memory requirements, that's pushing the ragged edge - especially if you have any other packages running as well.

bitslammer

@bmeeks I'm guessing the rules that I enabled have grown over time so I'll try to trim them. Oddly this doesn't happen every time which you'd kind of expect. It's a Netgate 3100 so it looks like I need to look at some other options since this isn't upgradeable. Thanks for the quick reply adn happy holidays.

bmeeks

@bitslammer said in pfSense Suricata Crashes on Malformed Block List Entry:

@bmeeks I'm guessing the rules that I enabled have grown over time so I'll try to trim them. Oddly this doesn't happen every time which you'd kind of expect. It's a Netgate 3100 so it looks like I need to look at some other options since this isn't upgradeable. Thanks for the quick reply adn happy holidays.

Several memory parameters have new increased minimums in Suricata 7.x. You are probably seeing the impact of those on the SG-3100. Same issue exists for SG-1100 users, too. 4GB is the new minimum, and even that might get cramped with lots of rules (more than 15,000).

If I were spec'ing a box today for someone who wanted to run IDS/IPS (and most folks want to run pfBlockerNG with DNSBL, too), then I would set 8 GB as a new minimum RAM requirement. The new default ZFS install will also chew up much more RAM than the old UFS setup.

Edit: also looking once again at your log snippet post, I see it seemed to be updating the rules as I see a "rule reload" message. RAM usage will increase during rule swaps, especially if "live rule swap" is enabled.