Testing FileStore

michmoor

As the title suggests i am trying to test File Store but i have no idea how to.

The following settings are enabled.
Enable File-Store
Enable Packet Log
EVE JSON Log -> FILE Output
EVE Logged Info > Tracked Files
Tracked-Files Checksum -> SHA256

How do i trigger this to see if i am able to store output from a SMB share or a HTTP download?

bmeeks

You have to creat or enable rules that contain the filestore keyword. Details are in the official Suricata docs here: https://docs.suricata.io/en/suricata-6.0.13/file-extraction/file-extraction.html.

michmoor

@bmeeks Dont punish me for not doing the simplest documentation check.........

Thanks Bill.

EDIT: Doesnt seem to be working. Per the documentation i tried the simplest rule with the protocol being smb

alert smb any any -> any any (msg:"FILE store all"; filestore; sid:100001; rev:1;)

I copied over a pdf and then i checked the Files tab. Nothing there BUT the alert did trigger.

bmeeks

Yes, this does work for most file types. I do seem to recall some issues with SMB in the past, but I thought those were resolved upstream a few versions back.

There are several required related settings that must be enabled. You must enable EVE.JSON logging, you must enable checksums (typically either SHA1 or MD5), and you must enable the filestore option -- all on the INTERFACE SETTINGS tab. Then restart Suricata after saving the changes to the settings.

The extracted files will be placed in one of 255 sub-directories created under the interface's logging directory. The subdirectories are named using the first two characters of possible SHA checksums. To find which subdirectory corresponds to your particular file, you will need to parse the EVE.JSON log, find the event in question, obtain the SHA checksum for the observed file, and then navigate to that subdirectory in the logging directory.

Here is a thread from the Suricata forums with this information: https://forum.suricata.io/t/file-store-of-suricata/2560.

There are also some built-in file store rules. I belivee they are commented out by default, but you can examine the category and enable some or copy and paste them into your own custom rules. Here is a post about that: https://github.com/OISF/suricata/blob/master/rules/files.rules.

michmoor

@bmeeks Thanks Bill.
Tried the rule from the github link and its not working after a rules reload. Maybe there is still an issue with SMB for filestore but the forum link you posted doesnt appear to say there is a problem. In fact someone got it working.

These are my custom rules:
alert smb any any -> any any (msg:"File Found over SMB and stored"; filestore; sid:32; rev:1;)
alert http any any -> any any (msg:"File Found over SMB and stored"; filestore; sid:32; rev:1;)

How i am testing is doing a copy over SMB from the share to my client desktop.

Enable File store is checked.

bmeeks

So, to clarify:

The rules you have are generating the expected alerts, but when you look at the FILES tab in Suricata you are not seeing any captured files data related to those same alerts?

I can't easily test this now as my testing VMs are out of service for a bit. I know it has worked, though, because I've seen the captured files on the virtual machines.

The code for the FILES tab was created and submitted by a Netgate contract developer a few years ago (maybe two or three years ago ???). The FILES tab should show any files contained in the large subdirectory tree of the filestore.

michmoor

@bmeeks said in Testing FileStore:

The rules you have are generating the expected alerts, but when you look at the FILES tab in Suricata you are not seeing any captured files data related to those same alerts?

That is correct. Nothing is in the Files tab.
A long time ago i remembered this working. That was also a few versions of pf ago so hard to say if this is a configuration problem or a software problem.

token

@michmoor
necro post, did this get resolved? In the GUI I do not see any files saved.

In the /var/log/suricata/suricata<interface>/filestore path I have the 90 something folders that look like hex code but I guess are the first two digits of a hash to organize the files collected by hash. In those folders I have tons of 1-4kb some 1xx kb 'junk' files but no actual jpeg files I've been testing with.

I have ensured to have file-store enabled, hashing enabled (tried MD5 then SHA-1) and upped memory in a lot of non-related settings. In the eve.json logs it shows file-storing=true but file-stored=false.

I don't see in the GUI nor the resultant .yaml file the stream-depth setting mentioned in some suricata documentation. In the tutorial this is set to 10MB to catch decent sized files, pictures etc.

All is see in the pfsense .yaml file generated for suricata in regards to file-store is:

file-store:
version: 2
enabled: yes
length: 0
dir: /var/log/suricata/suricata_em036559/filestore

Additional context, I have the files.rules enabled and have successful alerts of "File Found over SMB and stored" but within that log shows file-stored=false.

I'm tempted to increase the length to some arbitrary large number but it wouldn't survive a service restart to make it valid anyhow- kind of a check-mate here.