Suricata Update 4.0.13_9 PHP Warnings

Veldkornet

My Suricata did an update a few days back, and beside from not showing up in the menu's anymore (although I can still access the pages directly), the update gives the following error, same if I try to re-install the package:

PHP ERROR: Type: 1, File: /usr/local/pkg/suricata/suricata_migrate_config.php, Line: 241, Message: Uncaught Error: Cannot create references to/from string offsets in /usr/local/pkg/suricata/suricata_migrate_config.php:241
Stack trace:
#0 /usr/local/pkg/suricata/suricata_post_install.php(185): include()
#1 /etc/inc/pkg-utils.inc(768) : eval()'d code(1): include_once('/usr/local/pkg/...')
#2 /etc/inc/pkg-utils.inc(768): eval()
#3 /etc/inc/pkg-utils.inc(854): eval_once('include_once(&quot;/...')
#4 /etc/rc.packages(74): install_package_xml('suricata')
#5 {main}

Any ideas?

bmeeks

This error is coming from the recent update to PHP 7.2 in pfSense 2.4.4. I will need to send a fix to the pfSense team for them to merge into production. Your configuration must be missing some of the default settings for a few Suricata items in order for this error to be triggered. What previous version of Suricata were you running? Or if you are not sure what the version was, when did you last update Suricata before this most recent update?

Veldkornet

I don’t recall what the previous version was, but I did install it a few weeks ago after 2.4.4 was out and installed, and everything was running fine at the time.

I often check for package updates though, so I assume it would be the version that was pushed out before this.

bmeeks

The pull request containing the fix for this error is posted here. After it is reviewed, approved and merged by the pfSense team, a new Suricata package version 4.0.13_10 will appear.

Veldkornet

After I updated to 4.0.13_11, it was all working fine, items are now all back in the services menus etc, but then I noticed today that it wasn’t running.... if I tried to start it, it just stopped again with no errors in the log.

So I rebooted my pfSense, and the following showed up in the startup log:

20:28:54 PHP ERROR: Type: 1, File: /usr/local/pkg/suricata/suricata_migrate_config.php, Line: 241, Message: Uncaught Error: Cannot create references to/from string offsets in /usr/local/pkg/suricata/suricata_migrate_config.php:241
Stack trace:
#0 /usr/local/pkg/suricata/suricata_post_install.php(185): include()
#1 /etc/inc/pkg-utils.inc(768) : eval()'d code(1): include_once('/usr/local/pkg/...')
#2 /etc/inc/pkg-utils.inc(768): eval()
#3 /etc/inc/pkg-utils.inc(854): eval_once('include_once("/...')
#4 /etc/rc.packages(74): install_package_xml('suricata')
#5 {main}
thrown

bmeeks

@veldkornet said in Suricata Update 4.0.13_9 PHP Warnings:

After I updated to 4.0.13_11, it was all working fine, items are now all back in the services menus etc, but then I noticed today that it wasn’t running.... if I tried to start it, it just stopped again with no errors in the log.

So I rebooted my pfSense, and the following showed up in the startup log:
20:28:54 PHP ERROR: Type: 1, File: /usr/local/pkg/suricata/suricata_migrate_config.php, Line: 241, Message: Uncaught Error: Cannot create references to/from string offsets in /usr/local/pkg/suricata/suricata_migrate_config.php:241
Stack trace:
#0 /usr/local/pkg/suricata/suricata_post_install.php(185): include()
#1 /etc/inc/pkg-utils.inc(768) : eval()'d code(1): include_once('/usr/local/pkg/...')
#2 /etc/inc/pkg-utils.inc(768): eval()
#3 /etc/inc/pkg-utils.inc(854): eval_once('include_once("/...')
#4 /etc/rc.packages(74): install_package_xml('suricata')
#5 {main}
thrown

Can you double-check the date and timestamp for those error messages? That's the error that was fixed in the last package update (4.0.13._11). Maybe you are seeing the error messages logged by the old package version. Another reason I say this is that the ONLY time the suricata_migrate_config.php file is loaded and run is during a package installation. You can't get an error from that file unless the package is being installed/updated again. That does not happen with just a plain-vanilla reboot of the firewall.

Next question for you is what kind of hardware are you using. Is it by chance a Netgate SG-3100 appliance?

Finally, look in the suricata.log for the interface and see what's in there. You can find that on the LOGS VIEW tab.

Veldkornet

@bmeeks said in Suricata Update 4.0.13_9 PHP Warnings:

Can you double-check the date and timestamp for those error messages? That's the error that was fixed in the last package update (4.0.13._11). Maybe you are seeing the error messages logged by the old package version. Another reason I say this is that the ONLY time the suricata_migrate_config.php file is loaded and run is during a package installation. You can't get an error from that file unless the package is being installed/updated again. That does not happen with just a plain-vanilla reboot of the firewall.

Next question for you is what kind of hardware are you using. Is it by chance a Netgate SG-3100 appliance?

Finally, look in the suricata.log for the interface and see what's in there. You can find that on the LOGS VIEW tab.

I think you’re probably right wrt being the old error. Weird that it showed up in my boot log though....

I’m using a PCEngines APU2C4.

I’ve checked the suricata.log but it’s empty... check via the interface and via terminal.

bmeeks

@veldkornet
Go to a command-line prompt and execute this command and post back the results if you can --

/usr/local/bin/suricata -V

That should print out some version information and exit. See if any error messages appear. If you have an empty suricata.log file for the interface, that would indicate perhaps some needed dependency packages are missing.

Veldkornet

@bmeeks said in Suricata Update 4.0.13_9 PHP Warnings:

@veldkornet
Go to a command-line prompt and execute this command and post back the results if you can --
/usr/local/bin/suricata -V
That should print out some version information and exit. See if any error messages appear. If you have an empty suricata.log file for the interface, that would indicate perhaps some needed dependency packages are missing.

This is Suricata version 4.0.6 RELEASE

bmeeks

That's the expected output. Now try this to start Suricata from the shell prompt --

/usr/local/etc/rc.d/suricata.sh start

Run that from the command line and give it a few seconds to startup. That should start Suricata on all of your configured interfaces. You should also see some data show up in the suricata.log file for each configured interface.

If it fails to start, you need to carefully examine the firewall system log and the suricata.log for the interface to see if any messages were logged. It would be extraordinarily rare for Suricata to fail in startup but not log a single message anywhere.

Veldkornet

It prints out this:

ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.26/mach/CORE
32-bit compatibility ldconfig path:

But nothing in the suricata.log

In the system.log, this is all that it shows:

Nov 15 22:43:13	SuricataStartup	5237	Suricata START for WAN(8011_igb1)...

bmeeks

Well, at this point the easiest next step is to remove the Suricata package and reinstall it. You won't lose any settings.

Go to the Package Manager screen and click the trashcan icon next to Suricata to remove the Suricata package. When that completes, click the Available Packages tab and find the Suricata package and install it again. Be sure to stay on that screen until the entire process completes. It will log the package installation progress on the screen.

Veldkornet

I tried a reinstall, same result.

Then I tried a remove, the new install... same result.

Really weird....

bmeeks

Please excuse the following questions, but I have to ask them. Your issue is way out of the ordinary, so I need to get a feel for your experience level.

How are you determining that Suricata is not running? What are you basing that on?

Did you go in and configure an interface to use Suricata and save those changes?

What rule sets (vendor rule archives) did you configure for download (Snort VRT and/or Emering Threats)?

When you go to the LOGS VIEW tab, what interface are you selecting in the INTERFACE drop-down? Is it the same interface where you have Suricata configured?

Do you have a /var/log/suricata directory tree? Does the root account have full permissions in that tree?

Veldkornet

@bmeeks said in Suricata Update 4.0.13_9 PHP Warnings:

Please excuse the following questions, but I have to ask them. Your issue is way out of the ordinary, so I need to get a feel for your experience level.

Like I said, it was working... Then I noticed that it was stopped, and I couldn't start it up again.

How are you determining that Suricata is not running? What are you basing that on?

Did you go in and configure an interface to use Suricata and save those changes?

What rule sets (vendor rule archives) did you configure for download (Snort VRT and/or Emering Threats)?

Note that the only reason they are up to date is because they updated when I re-installed.

When you go to the LOGS VIEW tab, what interface are you selecting in the INTERFACE drop-down? Is it the same interface where you have Suricata configured?

Do you have a /var/log/suricata directory tree? Does the root account have full permissions in that tree?

As extra info, I deleted all of the logs and tried to start suricata again. The suricata.log is re-created, but it's still blank.

bmeeks

I've never seen this type of problem before. It appears the Suricata process never even attempts to start up. But the fact you can run it from the command line (at least to print out the version information) would indicate the basic binary is there and installed OK.

The suricata.log file is now blanked by the shell script when Suricata is started (actually, just before the Suricata binary is called, the log is blanked out). This is to prevent the log from growing unabated with each restart. So the empty log file is sort of expected, but only if the binary never even attempts to start. The very first line in that log file during startup should be the version string information. The fact even that is not there indicates to me the binary never gets called. Very weird indeed.

Are you configured to use Legacy Mode blocking or Inline IPS Mode blocking? If Inline IPS Mode, try switching to Legacy Mode. I kind of doubt that's going to make a difference, though. At the very least I would expect to see something printed to the suricata.log file during the startup attempt.

Try this from the command line to actually launch a Suricata session. See what it prints out. To exit the process (if it starts), do a CTRL-C.

/usr/local/bin/suricata -i igb1 -c /usr/local/etc/suricata/suricata_18011_igb1/suricata.yaml

The above command will launch a Suricata instance using the configuration information in the YAML file given. It should start up and print various stuff to the output screen and then seem to freeze. At that point you can CTRL-C to break out. Post back here with anything you think might be helpful in troubleshooting that you see in the output.

Veldkornet

@bmeeks

16/11/2018 -- 00:25:21 - <Error> - [ERRCODE: SC_ERR_CONF_YAML_ERROR(242)] - Failed to parse configuration file at line 399: did not find expected key

At line 399 though is the following (personality: IDS):

###########################################################################
# Configure libhtp.
libhtp:
   default-config:
          personality: IDS
     request-body-limit: 4096
     double-decode-path: no
     double-decode-query: no
     uri-include-all: no

Update:
Fixed those extra spaces in front on "personality". I know yaml's are fussy, so I put it in line with the request-body-limit.

Weird that it was like that in the first place though...

However, I see that removing these spaces fixes the issue, and I can start suricata. Although now during my testing, the spaces came back once or twice, and I had to remove them again for it to start.

Also, the log is now littered with errors. Not sure if it's related, or if it's just the first time I've noticed it...

16/11/2018 -- 00:52:31 - <Notice> -- This is Suricata version 4.0.6 RELEASE
16/11/2018 -- 00:52:31 - <Info> -- CPUs/cores online: 4
16/11/2018 -- 00:52:31 - <Info> -- HTTP memcap: 67108864
16/11/2018 -- 00:52:31 - <Notice> -- using flow hash instead of active packets
16/11/2018 -- 00:52:31 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - Signature combines packet specific matches (like dsize, flags, ttl) with stream / state matching by matching on app layer proto (like using http_* keywords).
16/11/2018 -- 00:52:31 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $HOME_NET any -> $EXTERNAL_NET 1024:65535 (msg:"MALWARE-CNC Win.Trojan.Fakeavlock variant outbound connection"; flow:to_server,established; dsize:267<>276; content:"User-Agent|3A| Mozilla/5.0 (Windows|3B| U|3B| MSIE 9.0|3B| Windows NT 9.0|3B| en-US)|0D 0A|"; fast_pattern:only; http_header; urilen:159; pcre:"/\x2f[A-F0-9]{158}/U"; metadata:impact_flag red, policy balanced-ips drop, policy security-ips drop, ruleset community, service http; reference:url,www.virustotal.com/file/c49f7dbc036ad0a86df02cbbde00cb3b3fbd651d82f6c9c5a98170644374f64f/analysis/; classtype:trojan-activity; sid:25675; rev:7;)" from file /usr/local/etc/suricata/suricata_8011_igb1/rules/suricata.rules at line 52
16/11/2018 -- 00:52:31 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - previous keyword has a fast_pattern:only; set. Can't have relative keywords around a fast_pattern only content
16/11/2018 -- 00:52:32 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"MALWARE-OTHER Win.Trojan.Zeus Spam 2013 dated zip/exe HTTP Response - potential malware download"; flow:to_client,established; content:"-2013.zip|0D 0A|"; fast_pattern:only; content:"-2013.zip|0D 0A|"; http_header; content:"-"; within:1; distance:-14; http_header; file_data; content:"-2013.exe"; content:"-"; within:1; distance:-14; metadata:impact_flag red, policy balanced-ips drop, policy security-ips drop, ruleset community, service http; reference:url,www.virustotal.com/en/file/2eff3ee6ac7f5bf85e4ebcbe51974d0708cef666581ef1385c628233614b22c0/analysis/; classtype:trojan-activity; sid:26470; rev:1;)" from file /usr/local/etc/suricata/suricata_8011_igb1/rules/suricata.rules at line 73
16/11/2018 -- 00:52:32 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - previous keyword has a fast_pattern:only; set. Can't have relative keywords around a fast_pattern only content

And a whole lot more tha tthe forum doesn't let me post because it thinks it's spam.

bmeeks

@veldkornet said in Suricata Update 4.0.13_9 PHP Warnings:

@bmeeks
16/11/2018 -- 00:25:21 - <Error> - [ERRCODE: SC_ERR_CONF_YAML_ERROR(242)] - Failed to parse configuration file at line 399: did not find expected key
At line 399 though is the following (personality: IDS):
###########################################################################
# Configure libhtp.
libhtp:
   default-config:
          personality: IDS
     request-body-limit: 4096
     double-decode-path: no
     double-decode-query: no
     uri-include-all: no
Update:
Fixed those extra spaces in front on "personality". I know yaml's are fussy, so I put it in line with the request-body-limit.

Weird that it was like that in the first place though...

I think you have or had corruption in part of your config.xml file related to Suricata. This item (libhtp) is the same thing that was causing the PHP error message you originally reported. This information is stored in the config.xml file as an array, and that array was not existent in your file when it really should have been there. All my fix did was initialize that array if it was missing before attempting to use it. This YAML config error happening with the same item is the connecting link.

Look carefully on the FLOW/STREAM tab at the configuration engine settings for HTTP. Something is amiss in your setup. Making manual edits to the suricata.yaml file is pointless as that file is overwritten by the GUI code each time you start Suricata from the INTERFACES tab.

The Invalid Signature errors are expected when using the Snort rules with Suricata. See several of my other posts in this forum in recent threads. Suricata does not understand and thus can't process several Snort rule keywords.

I have this exact problem with absolutely no errors being logged with 4.0.13_11. I'm on different hardware as well.

bmeeks

@rawrsquad said in Suricata Update 4.0.13_9 PHP Warnings:

I have this exact problem with absolutely no errors being logged with 4.0.13_11. I'm on different hardware as well.

Which exact problem do you mean? Is Suricata failing to start with no error, or did you see the PHP error mentioned in the very first post of this thread?

If Suricata is failing to start but logging no error, then run the command I gave in this post, but you will need to adjust the random GUID number and physical interface in the sub-directory path to match the one on your firewall. It will not be the same, but you can easily find yours by looking at the sub-directory names under /usr/local/etc/suricata. See if you also get a suricata.yaml file syntax error.

Veldkornet

My problem on line 399 is back after updating to pfSense 4.3.3_p1.

Except this time, it’s changing back after I try start the service via the interface...

Any ideas?

bmeeks

@veldkornet said in Suricata Update 4.0.13_9 PHP Warnings:

My problem on line 399 is back after updating to pfSense 4.3.3_p1.

Except this time, it’s changing back after I try start the service via the interface...

Any ideas?

You have corruption within the Suricata section of your firewall's config.xml file. Specifically it looks like some funky data exists for the HTTP stream engine. Go to the FLOW/STREAM tab and scroll down to the HTTP engine section. Click to edit any entries there and see if you either get an error or see bogus data (like extra spaces or something).

Veldkornet

@bmeeks said in Suricata Update 4.0.13_9 PHP Warnings:

@veldkornet said in Suricata Update 4.0.13_9 PHP Warnings:

My problem on line 399 is back after updating to pfSense 4.3.3_p1.

Except this time, it’s changing back after I try start the service via the interface...

Any ideas?

You have corruption within the Suricata section of your firewall's config.xml file. Specifically it looks like some funky data exists for the HTTP stream engine. Go to the FLOW/STREAM tab and scroll down to the HTTP engine section. Click to edit any entries there and see if you either get an error or see bogus data (like extra spaces or something).

So I found that when I click on the “App Parsers” tab, I get the following error:

Fatal error: Uncaught Error: Cannot use string offset as an array in /usr/local/www/suricata/suricata_app_parsers.php:79 Stack trace: #0 {main} thrown in /usr/local/www/suricata/suricata_app_parsers.php on line 79 PHP ERROR: Type: 1, File: /usr/local/www/suricata/suricata_app_parsers.php, Line: 79, Message: Uncaught Error: Cannot use string offset as an array in /usr/local/www/suricata/suricata_app_parsers.php:79 Stack trace: #0 {main} thrown

Could this be causing the issue?

bmeeks

Yes, that is another symptom of something being amiss within your config.xml file for the firewall. Specifically within the <installedpackages><suricata></suricata></installedpackages> section. The last bug fix I posted for Suricata was supposed to fix that issue (an uninitialized array), but for some reason it's not working in your case. That's why I suspect something amiss within your configuration file.

Does the APP PARSERS tab display? If so, please post a screenshot of that tab.

Veldkornet

@bmeeks said in Suricata Update 4.0.13_9 PHP Warnings:

Yes, that is another symptom of something being amiss within your config.xml file for the firewall. Specifically within the <installedpackages><suricata></suricata></installedpackages> section. The last bug fix I posted for Suricata was supposed to fix that issue (an uninitialized array), but for some reason it's not working in your case. That's why I suspect something amiss within your configuration file.

Does the APP PARSERS tab display? If so, please post a screenshot of that tab.

Nope, it doesn’t display. As soon as you click the tab, it throws that error on the screen. White screen, nothing else visible except that error

bmeeks

@veldkornet said in Suricata Update 4.0.13_9 PHP Warnings:

@bmeeks said in Suricata Update 4.0.13_9 PHP Warnings:

Yes, that is another symptom of something being amiss within your config.xml file for the firewall. Specifically within the <installedpackages><suricata></suricata></installedpackages> section. The last bug fix I posted for Suricata was supposed to fix that issue (an uninitialized array), but for some reason it's not working in your case. That's why I suspect something amiss within your configuration file.

Does the APP PARSERS tab display? If so, please post a screenshot of that tab.

Nope, it doesn’t display. As soon as you click the tab, it throws that error on the screen. White screen, nothing else visible except that error

Sorry to be late replying. I have several other appointments today that are keeping me away from my PC.

You have two choices for fixing this. One is to send me your config.xml file privately and let me see what is really wrong within the Suricata package section. You can just send me everything between the <installedpackages><suricata></suricata> section of the file. The other choice is to completely delete the Suricata interface that is giving you this problem and create it again from scratch. This second option means you will have to rebuild your configuration in terms of enabled rules and such. If you have not done a ton of customization on the interface, this may be the fastest approach.

Your problem seems to be that somehow a key piece of the default Suricata configuration has gotten removed from your firewall. This "bad data" is now permanently in place in your configuration, so simply removing the package and reinstalling it is not going to fix it so long as you continue to let the reinstall process restore the old configuration (which has the bad data). Hence my suggestion to delete the Suricata interface entirely and start over. Deleting the interface should clean up that section of the configuration and restore the missing default pieces.

Veldkornet

Thanks, I recreated the interface and that fixed it indeed!

bmeeks

@veldkornet said in Suricata Update 4.0.13_9 PHP Warnings:

Thanks, I recreated the interface and that fixed it indeed!

Glad it is fixed for you. Thank you for the feedback.