Suricata Update 4.0.13_9 PHP Warnings

Veldkornet

16/11/2018 -- 00:25:21 - <Error> - [ERRCODE: SC_ERR_CONF_YAML_ERROR(242)] - Failed to parse configuration file at line 399: did not find expected key

At line 399 though is the following (personality: IDS):

###########################################################################
# Configure libhtp.
libhtp:
   default-config:
          personality: IDS
     request-body-limit: 4096
     double-decode-path: no
     double-decode-query: no
     uri-include-all: no

Update:
Fixed those extra spaces in front on "personality". I know yaml's are fussy, so I put it in line with the request-body-limit.

Weird that it was like that in the first place though...

However, I see that removing these spaces fixes the issue, and I can start suricata. Although now during my testing, the spaces came back once or twice, and I had to remove them again for it to start.

Also, the log is now littered with errors. Not sure if it's related, or if it's just the first time I've noticed it...

16/11/2018 -- 00:52:31 - <Notice> -- This is Suricata version 4.0.6 RELEASE
16/11/2018 -- 00:52:31 - <Info> -- CPUs/cores online: 4
16/11/2018 -- 00:52:31 - <Info> -- HTTP memcap: 67108864
16/11/2018 -- 00:52:31 - <Notice> -- using flow hash instead of active packets
16/11/2018 -- 00:52:31 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - Signature combines packet specific matches (like dsize, flags, ttl) with stream / state matching by matching on app layer proto (like using http_* keywords).
16/11/2018 -- 00:52:31 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $HOME_NET any -> $EXTERNAL_NET 1024:65535 (msg:"MALWARE-CNC Win.Trojan.Fakeavlock variant outbound connection"; flow:to_server,established; dsize:267<>276; content:"User-Agent|3A| Mozilla/5.0 (Windows|3B| U|3B| MSIE 9.0|3B| Windows NT 9.0|3B| en-US)|0D 0A|"; fast_pattern:only; http_header; urilen:159; pcre:"/\x2f[A-F0-9]{158}/U"; metadata:impact_flag red, policy balanced-ips drop, policy security-ips drop, ruleset community, service http; reference:url,www.virustotal.com/file/c49f7dbc036ad0a86df02cbbde00cb3b3fbd651d82f6c9c5a98170644374f64f/analysis/; classtype:trojan-activity; sid:25675; rev:7;)" from file /usr/local/etc/suricata/suricata_8011_igb1/rules/suricata.rules at line 52
16/11/2018 -- 00:52:31 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - previous keyword has a fast_pattern:only; set. Can't have relative keywords around a fast_pattern only content
16/11/2018 -- 00:52:32 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"MALWARE-OTHER Win.Trojan.Zeus Spam 2013 dated zip/exe HTTP Response - potential malware download"; flow:to_client,established; content:"-2013.zip|0D 0A|"; fast_pattern:only; content:"-2013.zip|0D 0A|"; http_header; content:"-"; within:1; distance:-14; http_header; file_data; content:"-2013.exe"; content:"-"; within:1; distance:-14; metadata:impact_flag red, policy balanced-ips drop, policy security-ips drop, ruleset community, service http; reference:url,www.virustotal.com/en/file/2eff3ee6ac7f5bf85e4ebcbe51974d0708cef666581ef1385c628233614b22c0/analysis/; classtype:trojan-activity; sid:26470; rev:1;)" from file /usr/local/etc/suricata/suricata_8011_igb1/rules/suricata.rules at line 73
16/11/2018 -- 00:52:32 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - previous keyword has a fast_pattern:only; set. Can't have relative keywords around a fast_pattern only content

And a whole lot more tha tthe forum doesn't let me post because it thinks it's spam.

bmeeks

@veldkornet said in Suricata Update 4.0.13_9 PHP Warnings:

@bmeeks
16/11/2018 -- 00:25:21 - <Error> - [ERRCODE: SC_ERR_CONF_YAML_ERROR(242)] - Failed to parse configuration file at line 399: did not find expected key
At line 399 though is the following (personality: IDS):
###########################################################################
# Configure libhtp.
libhtp:
   default-config:
          personality: IDS
     request-body-limit: 4096
     double-decode-path: no
     double-decode-query: no
     uri-include-all: no
Update:
Fixed those extra spaces in front on "personality". I know yaml's are fussy, so I put it in line with the request-body-limit.

Weird that it was like that in the first place though...

I think you have or had corruption in part of your config.xml file related to Suricata. This item (libhtp) is the same thing that was causing the PHP error message you originally reported. This information is stored in the config.xml file as an array, and that array was not existent in your file when it really should have been there. All my fix did was initialize that array if it was missing before attempting to use it. This YAML config error happening with the same item is the connecting link.

Look carefully on the FLOW/STREAM tab at the configuration engine settings for HTTP. Something is amiss in your setup. Making manual edits to the suricata.yaml file is pointless as that file is overwritten by the GUI code each time you start Suricata from the INTERFACES tab.

The Invalid Signature errors are expected when using the Snort rules with Suricata. See several of my other posts in this forum in recent threads. Suricata does not understand and thus can't process several Snort rule keywords.

A Former User

I have this exact problem with absolutely no errors being logged with 4.0.13_11. I'm on different hardware as well.

bmeeks

@rawrsquad said in Suricata Update 4.0.13_9 PHP Warnings:

I have this exact problem with absolutely no errors being logged with 4.0.13_11. I'm on different hardware as well.

Which exact problem do you mean? Is Suricata failing to start with no error, or did you see the PHP error mentioned in the very first post of this thread?

If Suricata is failing to start but logging no error, then run the command I gave in this post, but you will need to adjust the random GUID number and physical interface in the sub-directory path to match the one on your firewall. It will not be the same, but you can easily find yours by looking at the sub-directory names under /usr/local/etc/suricata. See if you also get a suricata.yaml file syntax error.

Veldkornet

My problem on line 399 is back after updating to pfSense 4.3.3_p1.

Except this time, it’s changing back after I try start the service via the interface...

Any ideas?

bmeeks

@veldkornet said in Suricata Update 4.0.13_9 PHP Warnings:

My problem on line 399 is back after updating to pfSense 4.3.3_p1.

Except this time, it’s changing back after I try start the service via the interface...

Any ideas?

You have corruption within the Suricata section of your firewall's config.xml file. Specifically it looks like some funky data exists for the HTTP stream engine. Go to the FLOW/STREAM tab and scroll down to the HTTP engine section. Click to edit any entries there and see if you either get an error or see bogus data (like extra spaces or something).

Veldkornet

@bmeeks said in Suricata Update 4.0.13_9 PHP Warnings:

@veldkornet said in Suricata Update 4.0.13_9 PHP Warnings:

My problem on line 399 is back after updating to pfSense 4.3.3_p1.

Except this time, it’s changing back after I try start the service via the interface...

Any ideas?

You have corruption within the Suricata section of your firewall's config.xml file. Specifically it looks like some funky data exists for the HTTP stream engine. Go to the FLOW/STREAM tab and scroll down to the HTTP engine section. Click to edit any entries there and see if you either get an error or see bogus data (like extra spaces or something).

So I found that when I click on the “App Parsers” tab, I get the following error:

Fatal error: Uncaught Error: Cannot use string offset as an array in /usr/local/www/suricata/suricata_app_parsers.php:79 Stack trace: #0 {main} thrown in /usr/local/www/suricata/suricata_app_parsers.php on line 79 PHP ERROR: Type: 1, File: /usr/local/www/suricata/suricata_app_parsers.php, Line: 79, Message: Uncaught Error: Cannot use string offset as an array in /usr/local/www/suricata/suricata_app_parsers.php:79 Stack trace: #0 {main} thrown

Could this be causing the issue?

bmeeks

Yes, that is another symptom of something being amiss within your config.xml file for the firewall. Specifically within the <installedpackages><suricata></suricata></installedpackages> section. The last bug fix I posted for Suricata was supposed to fix that issue (an uninitialized array), but for some reason it's not working in your case. That's why I suspect something amiss within your configuration file.

Does the APP PARSERS tab display? If so, please post a screenshot of that tab.

Veldkornet

@bmeeks said in Suricata Update 4.0.13_9 PHP Warnings:

Yes, that is another symptom of something being amiss within your config.xml file for the firewall. Specifically within the <installedpackages><suricata></suricata></installedpackages> section. The last bug fix I posted for Suricata was supposed to fix that issue (an uninitialized array), but for some reason it's not working in your case. That's why I suspect something amiss within your configuration file.

Does the APP PARSERS tab display? If so, please post a screenshot of that tab.

Nope, it doesn’t display. As soon as you click the tab, it throws that error on the screen. White screen, nothing else visible except that error

bmeeks

@veldkornet said in Suricata Update 4.0.13_9 PHP Warnings:

@bmeeks said in Suricata Update 4.0.13_9 PHP Warnings:

Yes, that is another symptom of something being amiss within your config.xml file for the firewall. Specifically within the <installedpackages><suricata></suricata></installedpackages> section. The last bug fix I posted for Suricata was supposed to fix that issue (an uninitialized array), but for some reason it's not working in your case. That's why I suspect something amiss within your configuration file.

Does the APP PARSERS tab display? If so, please post a screenshot of that tab.

Nope, it doesn’t display. As soon as you click the tab, it throws that error on the screen. White screen, nothing else visible except that error

Sorry to be late replying. I have several other appointments today that are keeping me away from my PC.

You have two choices for fixing this. One is to send me your config.xml file privately and let me see what is really wrong within the Suricata package section. You can just send me everything between the <installedpackages><suricata></suricata> section of the file. The other choice is to completely delete the Suricata interface that is giving you this problem and create it again from scratch. This second option means you will have to rebuild your configuration in terms of enabled rules and such. If you have not done a ton of customization on the interface, this may be the fastest approach.

Your problem seems to be that somehow a key piece of the default Suricata configuration has gotten removed from your firewall. This "bad data" is now permanently in place in your configuration, so simply removing the package and reinstalling it is not going to fix it so long as you continue to let the reinstall process restore the old configuration (which has the bad data). Hence my suggestion to delete the Suricata interface entirely and start over. Deleting the interface should clean up that section of the configuration and restore the missing default pieces.

Veldkornet

Thanks, I recreated the interface and that fixed it indeed!

bmeeks

@veldkornet said in Suricata Update 4.0.13_9 PHP Warnings:

Thanks, I recreated the interface and that fixed it indeed!

Glad it is fixed for you. Thank you for the feedback.