Suricata MD5 hashes list

fsansfil · Feb 18, 2015, 3:01 AM

Hello all,

Just wanted to know where to place my MD5 black and white lists for Suricata for rules with filemd5: ?

Cheers

F.

BBcan177 · Feb 18, 2015, 3:11 AM

Are you asking where to put Malicious MD5 files hashes? I would like to know that myself (for both Snort/Suricata)… As i was planning on adding a feature to pfBlockerNG to Download "Malicious MD5's" from several sources on a download frequency...

fsansfil · Feb 18, 2015, 4:44 AM

Yea its one or more lists of MD5 hashes used when making rules like these :

ex: alert http any any -> any any (msg:"FILE MD5 Check PDF against a black list"; filemagic:"pdf"; filemd5:BlackMD5s.txt; sid:9966699; rev:1;)

http://blog.inliniac.net/2012/06/09/suricata-md5-blacklisting/

F.

BBcan177 · Feb 18, 2015, 4:53 AM

So it shouldn't be too hard to implement a new Tab in pfBNG called "MD5" and it can be similar to the IPv4/v6 tabs (and the upcoming DNSBL tab). I have been thinking to do this for awhile, but with the launch of pfBNG and my work on v2.0 DNSBL, time is so limited…

Then you would enter the URLs for the File Hashes... Select an "Alias" for whitelist and another "Alias" for "Blacklist", can actually create as many aliases as you wish...

Then I assume, we could auto create a Rule based on some logic and save that to the config.xml file for Suricata. (Would like to see this in Snort also). Then hopefully execute a Hup command to get Suricata to reload the ruleset. Would need to get some more specifics from Bill to see if he has any issues with the method...

bmeeks · Feb 19, 2015, 4:25 AM

@fsansfil:

Yea its one or more lists of MD5 hashes used when making rules like these :

ex: alert http any any -> any any (msg:"FILE MD5 Check PDF against a black list"; filemagic:"pdf"; filemd5:BlackMD5s.txt; sid:9966699; rev:1;)

http://blog.inliniac.net/2012/06/09/suricata-md5-blacklisting/

F.

Currently the package does not have an option for uploading or maintaining a blacklist file. However, you can manually upload a blacklist file outside of the Suricata GUI and then reference it in your custom rule. There is a "rules path" for each enabled Suricata interface.

Look for something like: /usr/pbi/suricata-amd64/etc/suricata/suricata__{uuid}{interface}_/rules

You would put your blacklist file in that directory.

Bill

fsansfil · Feb 20, 2015, 2:38 AM

Thnaks Bill, it's working.

BBcan177, are you talking about the Virushare http://virusshare.com/hashes.4n6 ?

Because, indeed it would be nice if pfblocker could gather those 150 lists in a smart way and we could use it in suricata…

Also, I think its possible to extract the clamav one, still figuring it out.

Any other MD5 blacklist out there that you know of ?

Cheers.

F.

BBcan177 · Feb 20, 2015, 2:45 AM

Yes I can do this in pfBlockerNG. It already has all the Scheduling code/Alias/Lists etc to handle it. Just need to make a new Tab "MD5"… I have a bunch of URLs for Hashes in my Files... Just have to go thru them and put it together... Put together all the lists you can find and I can write a parser to collect them...

fsansfil · Feb 22, 2015, 7:27 PM

Hello,

Im questioning myself about the mem allocation…

suricata: 22/2/2015 – 13:23:49 - <error>-- [ERRCODE: SC_ERR_FLOW_INIT(205)] - allocating flow hash failed: max flow memcap is smaller than projected hash size. Memcap: 635544320, Hash table size 40674836480. Calculate total hash size by multiplying "flow.hash-size" with 64, which is the hash bucket size.</error>

So lets say I have 2M hashes (2 000 000)…I must multiply it by 64 and get 128 000 000 bytes, roughly 128MB.

So Flow in the LAN Flow/Stream tab of Suricata I should put ??

Flow Memory Cap :

128000000

Flow Hash Table Size :

2 000 000

Is that right ?

Thanks.

F.

bmeeks · Feb 23, 2015, 2:07 AM

When you use IP REP lists and other styles of IP lists, they allocate hash table entries for each IP address. Those can add up quickly with large lists. Other parts of Suricata also use the same hash table. I believe GeoIP uses the hash table as well. Suricata has calculated the hash table size in the error message, but the memcap is set too low for the calculated hash table size.

Bill

fsansfil · Feb 23, 2015, 2:39 AM

Just found this on the Suricata documentation (https://redmine.openinfosecfoundation.org/projects/suricata/wiki/File-keywords)

Memory requirements
Each MD5 uses 16 bytes of memory. 20 Million MD5's use about 310 MiB of memory.

So using all of the virusShare hashes (http://virusshare.com/hashes.4n6) … 135k x 150 = 20 250 000.

Tried loading it with with more than 1024MB of RAM allowed in Flow Memory Cap and it didnt work. Which should plenty for my GeoIP and MD5 hashes list, am I correct ?

bmeeks · Feb 23, 2015, 9:39 PM

Don't stay fixated on just your file MD5 hash table needs. A lot of other Suricata stuff uses the hash table as well. The error message was indicating a computed hash table requirement of 40,674,836,480 bytes. I have no idea is that number is accurate or not, but I can tell it is being computed by the binary part of Suricata and not the PHP GUI part.

Bill