Suricata process dying due to hyperscan problem

bmeeks

@JonathanLee said in Suricata process dying due to hyperscan problem:

@bmeeks Happy Holidays!!! I wanted to message you to say that.

Thank you. Same to you and your family.

bmeeks

@masons said in Suricata process dying due to hyperscan problem:

It looks like file sizes differ. I'm running pfSense 2.7.2 CE in this VM.

-rw-r--r--  1 root wheel  1421072 Jun 30 08:16 /usr/local/lib/libhs_runtime.a
lrwxr-xr-x  1 root wheel       18 Jun 30 08:17 /usr/local/lib/libhs_runtime.so -> libhs_runtime.so.5
lrwxr-xr-x  1 root wheel       22 Jun 30 08:17 /usr/local/lib/libhs_runtime.so.5 -> libhs_runtime.so.5.4.0
-rwxr-xr-x  1 root wheel  1007944 Jun 30 08:17 /usr/local/lib/libhs_runtime.so.5.4.0
-rw-r--r--  1 root wheel 12452892 Jun 30 08:17 /usr/local/lib/libhs.a
lrwxr-xr-x  1 root wheel       10 Jun 30 08:17 /usr/local/lib/libhs.so -> libhs.so.5
lrwxr-xr-x  1 root wheel       14 Jun 30 08:17 /usr/local/lib/libhs.so.5 -> libhs.so.5.4.0
-rwxr-xr-x  1 root wheel  4518952 Jun 30 08:17 /usr/local/lib/libhs.so.5.4.0

Your file dates are weird. June 30 for 2.7.2 files is too early. Mine show December 16 dates, which coincides with the day I updated this particular VM to 2.7.2 CE.

My file sizes agree 100% with those posted by @kiokoman. Your file dates of June 30 agree with the rollout of pfSense 2.7.0 CE. The 2.7.1 CE was released in November of this year.

You might want to remove the Suricata package from your VM, then run this command to be sure the Hyperscan library is also removed:

pkg info hyperscan

If anything other than something like "not installed" comes back, then manually remove Hyperscan with this command:

pkg delete hyperscan

You having an older library might be why you are getting the segfault instead of the same fatal error -1 termination that @kiokoman and most others are seeing.

Now reinstall Suricata and you should get the correct latest Hyperscan library.

masons

@bmeeks, I now have the same file sizes and dates. Would you like me to again load the debug package or do you have all that you need now?

bmeeks

@masons said in Suricata process dying due to hyperscan problem:

@bmeeks, I now have the same file sizes and dates. Would you like me to again load the debug package or do you have all that you need now?

It would be interesting to see if now instead of a segfault you get the fatal error termination that @kiokoman is seeing.

You do not have to install the debug package version for that test unless you just want to. I'm hoping maybe your segfault was an issue with the library compilation (being possibly from an older pfSense kernel build) and not a different manifestation of the same Hyperscan error.

So, just set the Pattern Matcher to Auto and attempt to start Suricata. If it dies immediately (or later), check the suricata.log file to see if you have the "Hyperscan returned fatal error -1" message. If you still get a Signal 11 segfault in the system log, then color me thoroughly flummoxed .

Negan

@bmeeks

Just had Suricata crash with this messag on my Netgate 8200:

Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT amd64 1400094 #1 plus-RELENG_23_09_1-n256200-3de1e293f3a: Wed Dec 6 21:00:32 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_09_1-main/obj/amd64/Obhu6gXB/var/jenkins/workspace/pfSense-Plus-snapshots-23_09_1

Crash report details:

PHP Errors:
[19-Dec-2023 13:50:29 Australia/Sydney] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 896747264 bytes) in /usr/local/www/suricata/suricata_logs_browser.php on line 50

No FreeBSD crash data found.

sgnoc

@Negan I've had similar in other php files and these seem to be related to trying to do too much compared to the memory allocated for PHP. Were you trying to load a very large log file in Suricata Logs View? May be related to the specific interface and log type.

I'm no expert on this, so someone else may correct me, but maybe look at the log size and rotation settings for that interface and lower them so they cycle out more frequently, especially if it is a really busy interface?

Negan

@sgnoc
Your probably right, just never seen this after Suricata crashed before.....

bmeeks

@Negan said in Suricata process dying due to hyperscan problem:

@bmeeks

Just had Suricata crash with this messag on my Netgate 8200:

Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT amd64 1400094 #1 plus-RELENG_23_09_1-n256200-3de1e293f3a: Wed Dec 6 21:00:32 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_09_1-main/obj/amd64/Obhu6gXB/var/jenkins/workspace/pfSense-Plus-snapshots-23_09_1

Crash report details:

PHP Errors:
[19-Dec-2023 13:50:29 Australia/Sydney] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 896747264 bytes) in /usr/local/www/suricata/suricata_logs_browser.php on line 50

No FreeBSD crash data found.

Please post this in a new thread so as not to pollute this one with completely unrelated information. Do you see anything in the error report you posted that says "Hyperscan"? If not, then posting in this thread that is titled "Hyperscan problem" breaks standard forum etiquette. Replies to a thread should stay on the topic of the thread.

With 248 posts, this thread is rapidly becoming too long, and burdening it with unrelated issues only makes that worse.

Negan

@bmeeks

Well this was in the log,

74327 - W#07] 2023-12-19 14:29:30 Error: spm-hs: Hyperscan returned fatal error -1.

And what I posted was there also, thought it might be related, which as you say it's not.

bmeeks

@Negan said in Suricata process dying due to hyperscan problem:

@bmeeks

Well this was in the log,

74327 - W#07] 2023-12-19 14:29:30 Error: spm-hs: Hyperscan returned fatal error -1.

And what I posted was there also, thought it might be related, which as you say it's not.

That error is definitely related, but the PHP one is not. This thread has gotten so large I've lost track of which user has posted something related previously and who has not.

Sorry if I sounded like the crusty old guy yelling "you kids get off my lawn!" .

Your PHP error is common to any action on pfSense that attempts to open a large text file for viewing in the PHP GUI. PHP just can't cope with large files as it must first load the entire file into memory and then stream it out line-by-line to a web client. You will need to view the file using command-line tools such as vi or transfer the file off to a PC and view with text editor apps there. The PHP process on pfSense has a fixed memory size that is somewhat independent of how much RAM you might actually have installed on the firewall. In other words, you will get that error no matter if you have 2GB of RAM or 64 GB of RAM installed.

Negan

@bmeeks

The thing is I was not looking at any files at the time, anyway, I just Deleted Sucicata as it's been of no benefit since 23.09. so I won't need to post here again, best of luck tracking down the error..

bmeeks

@Negan said in Suricata process dying due to hyperscan problem:

@bmeeks

The thing is I was not looking at any files at the time, anyway, I just Deleted Sucicata as it's been of no benefit since 23.09. so I need to post here again, best of luck tracking down the error..

You had to have been attempting to view one at some point. That's the only way that error can appear. The thing with the PHP errors, though, is that the Dashboard is the only place they will be shown. So if you tried to view a log two or three days ago, but never visited the Dashboard in the interim, then the next time you do open the Dashboard you will see the PHP error that was recorded in the past.

According to the error, someone attempted to view a log file from the LOGS VIEW tab at this date and time: 19-Dec-2023 13:50:29 Australia/Sydney.

Negan

@bmeeks
Well I only installed Sucicata again today and it was crashed and the PHP Error was there, anyway, thanks for helping....

kiokoman

@masons
i suspected you had a different version of hyperscan because i had that same crash when using hyperscan 5.4.2 instead of 5.4.0
now that you have corrected the situation I expect that you no longer have the crash/core dump but only hyperscan that fails like me

bmeeks

@kiokoman and @masons:
The upstream Suricata developer team took a look at your gdb back trace reports and has asked me to prepare a new debug Suricata package with the Address Sanitizer option enabled and let you run that for a while. That option performs a very detailed memory use analysis at runtime and can print more detailed debugging messages.

I'm working on creating that build now and will test it on my system first to be sure it functions. Once I get everything ready, I will post a link to it back here in this thread.

kiokoman

@bmeeks
Nice, ready when you are

bmeeks

@kiokoman said in Suricata process dying due to hyperscan problem:

@bmeeks
Nice, ready when you are

I haven't forgotten. Ran the ASAN-enabled package first on my virtual machine and right away discovered another bug in the custom blocking module I wrote. Color me doubly embarrassed . This one was a missing address-of ('&') operator in two key places. Surprised the compiler never complained about them, but they both tripped ASAN right away.

So, tidying up and retesting after correcting that bug before posting the ASAN-enabled package for you guys to test.

bmeeks

@kiokoman and @masons:

Here is a link to the ASAN-enabled (Adress Sanitizer) debug build of Suricata 7.0.2. Please install and try this version.

Here is the link to the package binary on my Google Drive: https://drive.google.com/file/d/1y3ohuWyhK3IrHb_p5nVVkc4OwJJBhg0U/view?usp=drive_link. The file has the same name (suricata-7.0.2_6.pkg).

Repeat the steps outlined in this post to install the debug package: https://forum.netgate.com/topic/184101/suricata-process-dying-due-to-hyperscan-problem/223.

You have two options for doing the initial test run. Because the ASAN runtime linked into the binary really only outputs information to the console, you must run Suricata from the command-line to get the ASAN output. But I would first just try to run it from the GUI to see if it crashes still. It is possible the last two bugs I found and fixed this morning in the custom blocking module were the cause of the Hyperscan issue. So, just first try starting this new version from the GUI. If it starts and stays running, then the bug is fixed. If it crashes, then proceed to the steps below to run it from a CLI prompt in order to capture the ASAN output.

To Capture ASAN Output
You will need to start Suricata from a shell prompt on the firewall. You will want to use PuTTY with an SSH session so that you can easily scroll back and see past console output.

Establish an SSH session to the firewall using PuTTY.
Exit the menu into a shell prompt by choosing option #8.
Identify the full path to the correct suricata.yaml configuration for the interface you are testing. Look at the subdirectories under /usr/local/etc/suricata/ and find the suricata_xxxx_yyyy directory that matches the interface you want to test.
Note the physical interface name used by pfSense (for example, em0, vmx1, etc.). The interface name will be part of the subdirectory name.
With the above two pieces of info: (1) path to suricata.yaml for the interface; and (2) the physical interface name, start Suricata using this command:

/usr/local/bin/suricata -i <iface> -c /usr/local/etc/suricata/<suricata_xxxx_yyyy>/suricata.yaml

For example, here is the command for my virtual machine where I am using the em0 interface and the interface's conf directory is suricata_13473_em0:

/usr/local/bin/suricata -i em0 -c /usr/local/suricata/suricata_13473_em0/suricata.yaml

Suricata should start and begin spewing data to the console. If it hits a memory issue, the ASAN module will spit out some colorful data and exit the program. If there are no ASAN-detected errors, the program will just seem to freeze as it was not started in daemon mode. A simple CTRL-C from the keyboard will break out of the program.

kiokoman

@bmeeks
still running man .. still running ....
for me it's a record, 10 min without any hyperscan error

bmeeks

@kiokoman said in Suricata process dying due to hyperscan problem:

@bmeeks
still running man .. still running ....
for me it's a record, 10 min without any hyperscan error

This is good! Maybe I did find it this time.

The ASAN-enabled debug package is going to run quite a bit slower. If your results continue to be positive into the next day, then I will prepare a pull request with the fixes and get it submitted to the Netgate developer team so we can get a new package posted for production use.

There were two places in the custom blocking module code where an IPv4 and an IPv6 address were being copied using memcpy() and I had left off the address-of operator on one of the parameters (the destination memory address). That resulted in heap buffer overflows that would have produced random results depending on the runtime memory configuration of a particular installation. One manifestation could be corrupted pointers that were later passed to Hyperscan.

This would explain the connection between disabling Legacy Blocking Mode and successful execution with Hyperscan enabled.