Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Suricata hash matching Please Help

    Scheduled Pinned Locked Moved General pfSense Questions
    21 Posts 4 Posters 3.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      steve40
      last edited by

      Hello team,

      Thank you in advance for making this wealth of knowledge available. First, let me apologize as
      a complete noob to the forum that  my first post is a cry for help. Hopefully, in the near future, I'll have the answer to a question someone else has an issue and I can return the favor.

      If someone can please help with an issue I'm having I would greatly appreciate it. I'm trying to get the Suricata file hash checking module working and cannot get it operational. I can get it to store files and alert on certain attributes of a file but I can't get it to match a hash against a blacklist I'm testing with

      I'm using Pfsense 2.4.2
      suricata version 4.0.4

      Output of suricata.log on startup
      –---------------------------------------

      12/4/2018 -- 06:35:22 - <notice>-- This is Suricata version 4.0.4 RELEASE
      12/4/2018 -- 06:35:22 - <info>-- CPUs/cores online: 2
      12/4/2018 -- 06:35:22 - <info>-- HTTP memcap: 67108864
      12/4/2018 -- 06:35:22 - <notice>-- using flow hash instead of active packets
      12/4/2018 -- 06:35:22 - <info>-- 2 rule files processed. 2 rules successfully loaded, 0 rules failed
      12/4/2018 -- 06:35:22 - <info>-- Threshold config parsed: 0 rule(s) found
      12/4/2018 -- 06:35:22 - <info>-- 3 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 3 inspect application layer, 0 are decoder event only
      12/4/2018 -- 06:35:22 - <info>-- alert-pf -> Creating automatic firewall interface IP address Pass List.
      12/4/2018 -- 06:35:22 - <info>-- alert-pf -> adding firewall interface re0 IPv6 address
      fe80:0000:0000:0000:ca3a:35ff:fed2:0562 to automatic interface IP Pass List.
      12/4/2018 -- 06:35:22 - <info>-- alert-pf -> adding firewall interface re0 IPv4 address
      192.168.0.1 to automatic interface IP Pass List.
      ****** redundant entries removed to conserve post space *****
      12/4/2018 -- 06:35:22 - <info>-- fast output device (regular) initialized: alerts.log
      12/4/2018 -- 06:35:22 - <info>-- http-log output device (regular) initialized: http.log
      12/4/2018 -- 06:35:22 - <info>-- storing files in /var/log/suricata/suricata_re058347/files
      12/4/2018 -- 06:35:22 - <info>-- file-log output device (regular) initialized: files-json.log
      12/4/2018 -- 06:35:22 - <info>-- eve-log output device (regular) initialized: eve.json
      12/4/2018 -- 06:35:22 - <info>-- Going to log the md5 sum of email subject
      12/4/2018 -- 06:35:22 - <info>-- Using 1 live device(s).
      12/4/2018 -- 06:35:22 - <info>-- using interface re0
      12/4/2018 -- 06:35:22 - <info>-- Running in 'auto' checksum mode. Detection of interface state will require 1000 packets.
      12/4/2018 -- 06:35:22 - <info>-- Set snaplen to 1518 for 're0'
      12/4/2018 -- 06:35:22 - <info>-- id 416
      12/4/2018 -- 06:35:22 - <info>-- RunModeIdsPcapAutoFp initialised
      12/4/2018 -- 06:35:22 - <notice>-- all 3 packet processing threads, 2 management threads initialized, engine started.
      12/4/2018 -- 06:36:54 - <info>-- No packets with invalid checksum, assuming checksum offloading is NOT used

      The only two rules loaded are the ones related to the file hashing testing I'm dong

      relevant parts of my suricata.yaml file in the base configure directory /usr/local/etc/suricata/suricata.yaml

      • eve-log:
              enabled: yes

      • files:
                    force-magic: yes  # force logging magic on all logged files
                    # force logging of checksums, available hash functions are md5,
                    # sha1 and sha256
                    force-hash: [md5]
                    #force-md5: yes

      • file-store:
              enabled: yes      # set to yes to enable
              log-dir: files    # directory to store the files
              #dir: /files
              force-magic:  yes  # force logging magic on all stored files
              # force logging of checksums, available hash functions are md5,
              # sha1 and sha256
              #force-hash: no
              #force-md5: yes
              force-filestore: yes
              #force-filestore: yes # force storing of all files
              # override global stream-depth for sessions in which we want to
              # perform file extraction. Set to 0 for unlimited.
              stream-depth: 0

      • file-log:
              enabled: yes
              filename: files-json.log
              append: yes
              #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'

      force-magic: yes  # force logging magic on all logged files
            # force logging of checksums, available hash functions are md5,
            # sha1 and sha256
            force-hash: [md5]
            #force-magic: yes
            #force-md5: yes

      relevant parts of my suricata.yaml file located in the /usr/local/etc/suricata/{Interface}/suricata.yaml

      • eve-log:
              enabled: yes

      • files:
                    force-magic: yes
                    force-hash: [md5]

      • file-store:
              enabled: yes
              log-dir: files
              force-magic: no
              force-hash: [md5]
              waldo: file.waldo

      - file-log:
            enabled: yes
            filename: files-json.log
            append: yes
            filetype: regular
            force-magic: no
            force-hash: [md5]

      To test my configuration I placed "sql.exe" on a web server on the north side of my firewall running these rules.

      I used sigtool –md5 to compute the hash of the file which is

      7083d83ea6c2ad7ad297e6d4b9bccfc8

      My rule is formatted as follows
      –---------------------------------------

      alert http any any -> any any (msg:"Black list checksum match and extract MD5"; fileext:"exe";  filemd5:virushashes; filestore; sid:348; rev:1;)

      virushashes file is located in the rules directory and is formatted as follows

      7083d83ea6c2ad7ad297e6d4b9bccfc8
      a43c71c99bd721ea8e89798f46361730
      7083d83ea6c2ad7ad297e6d4b9bccfc8


      Eve.json log for the interface running suricata says "Truncated" where the MD5 hash value is supposed to be

      {"timestamp":"2018-04-

      12T06:12:37.689009+0000","flow_id":760221975951465,"in_iface":"re0","event_type":"fileinfo","sr

      c_ip":"63.96.49.202","src_port":80,"dest_ip":"192.168.0.2","dest_port":54404,"proto":"TCP","htt

      p":{"hostname":"63.96.49.202","url":"/sql.exe","http_user_agent":"Mozilla/5.0 (Windows NT

      6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0","http_content_type":"application/x-

      msdownload","http_refer":"http://63.96.49.202/","http_method":"GET","protocol":"HTTP

      /1.1","status":200,"length":7300},"app_proto":"http","fileinfo":

      {"filename":"/sql.exe","magic":"PE32+ executable (GUI) x86-64, for MS

      Windows","gaps":false,"state":"TRUNCATED","stored":false,"size":4380,"tx_id":0}}

      No matter what I've done I cannot get the download of sql.exe to match and generate an alert

      any thoughts?

      Thanks in advance</info></notice></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></notice></info></info></notice>

      1 Reply Last reply Reply Quote 0
      • GrimsonG
        Grimson Banned
        last edited by

        Post here: https://forum.pfsense.org/index.php?board=61.0

        1 Reply Last reply Reply Quote 0
        • bmeeksB
          bmeeks
          last edited by

          As referenced by @Grimson's reply, there is a forum here dedicated to Snort and Suricata.

          Where are you placing your hash values?  You said the "rules" directory, but which one?  You would need to use the path related to the Suricata interface such as this:

          /usr/local/etc/suricata/{Interface}/rules

          Everything for a running Suricata instance is contained with the interface sub-directory path.  So the only suricata.yaml that matters is the one in the interface path.  Same for any rules.

          The YAML config file is built and written to the interface sub-directory each time you start/stop Suricata from the GUI.

          My guess is Suricata is not finding your hash values and that's why it is not alerting.  Try putting the hash file here:

          /usr/local/etc/suricata/{Interface}/rules

          That is the actual directory where your active rules are loading from.  Specifically they will be in the suricata.rules file or the custom.rules file in that sub-directory.  If you created your rules as custom rules in the GUI, then they will be the latter location.

          Bill

          1 Reply Last reply Reply Quote 0
          • S
            steve40
            last edited by

            thanks for replying

            My apologies for not clarifying that piece

            the file "virushashes" is located in the /usr/local/etc/suricata/{Interface}/rules directory as shown

            [2.4.2-RELEASE][admin@pfSense.localdomain]/usr/local/etc/suricata/suricata_58347_re0/rules: ls -lash
            total 32
            4 drwxr-xr-x  2 root  wheel  512B Apr 12 17:35 .
            4 drwxr-xr-x  3 root  wheel  512B Apr 12 02:44 ..
            4 -rw-r–r--  1 root  wheel  492B Apr 12 06:35 custom.rules
            4 -rw-r--r--  1 root  wheel  299B Apr 11 21:31 emerging-md5.rules
            4 -rw-r--r--  1 root  wheel  3.8K Apr 11 20:44 files.rules
            0 -rw-r--r--  1 root  wheel    0B Apr 12 06:35 flowbit-required.rules
            4 -rw-r--r--  1 root  wheel  495B Apr 12 03:15 hashcheck.rules
            4 -rw-r--r--  1 root  wheel  367B Apr 12 06:35 suricata.rules
            4 -rw-r--r--  1 root  wheel    99B Apr 12 05:25 virushashes
            [2.4.2-RELEASE][admin@pfSense.localdomain]/usr/local/etc/suricata/suricata_58347_re0/rules:

            I've added the relevant rule to the "custom.rules" section and even went as far as to inject it into the suricata.rules file with no luck :(

            1 Reply Last reply Reply Quote 0
            • bmeeksB
              bmeeks
              last edited by

              I have an idea what may be wrong, but testing the fix will require you to be familiar with using vi or a similar editor in a CLI session on the firewall.

              If you are comfortable making an edit to a file for testing, here are the steps:

              1.  Make a copy of the original file for safekeeping

              
              cp /usr/local/pkg/suricata/suricata_generate_yaml.php /usr/local/pkg/suricata/suricata_generate_yaml.php.orig
              
              

              2. Now open /usr/local/pkg/suricata/suricata_generate_yaml.php in vi or a similar editor on the firewall.  You can also use something like WinSCP connected to the firewall and then right-click on the file to edit it.

              3. Locate line #698 in the file.  It will look like this:

              
              	$http_hosts_default_policy = "     personality: IDS\n     request-body-limit: 4096\n     response-body-limit: 4096\n";
              
              

              Change it to look like this:

              
              	$http_hosts_default_policy = "     personality: IDS\n     request-body-limit: 0\n     response-body-limit: 0\n";
              
              

              Notice the 4096 values are changed to zero.  This sets the HTTP session decoding to unlimited.  Formerly it was limited to 4096 bytes.  Be careful and don't alter the indention or position of spaces in the line!  Just carefully replace "4096" with "0" in both spots.

              4. Save the change to the file and then go into the Suricata GUI and open the interface for editing.  You don't have to make an actual change, but just click the SAVE button at the bottom of the page to force creation of an updated suricata.yaml configuration.

              5. Now on the INTERFACES tab in the GUI restart that Suricata interface and test the file extraction again.  Let me know if that works.

              Thanks,
              Bill

              1 Reply Last reply Reply Quote 0
              • S
                steve40
                last edited by

                Hi meeks…thanks for taking the time to respond

                $http_hosts_default_policy = "    personality: IDS\n    request-body-limit: 0\n    response-body-limit: 0\n";

                I followed your instructions but still no luck

                It stored the 30K in the /files directory as follows

                [2.4.2-RELEASE][admin@pfSense.localdomain]/var/log/suricata/suricata_re058347/files: cat file.1676.meta
                TIME:              04/12/2018-23:00:08.341864
                SRC IP:            63.96.49.202
                DST IP:            192.168.0.2
                PROTO:            6
                SRC PORT:          80
                DST PORT:          63020
                APP PROTO:        http
                HTTP URI:          /sql.exe
                HTTP HOST:        63.96.49.202
                HTTP REFERER:      http://63.96.49.202/
                HTTP USER AGENT:  Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0
                FILENAME:          /sql.exe
                MAGIC:            PE32+ executable (GUI) x86-64, for MS Windows
                STATE:            TRUNCATED
                SIZE:              4380

                /var/log/suricata/suricata_re058347: tail -100f eve.json | grep sql.exe

                I'm going to try with another executable as this one is a UPX packed exe

                1 Reply Last reply Reply Quote 0
                • bmeeksB
                  bmeeks
                  last edited by

                  Hmm… I was hoping the change in decode limits would help.  I'm afraid that was my best idea ...  :(.

                  You might have to try posting a question over on the Suricata Redmine site here: https://redmine.openinfosecfoundation.org/projects/suricata

                  Bill

                  1 Reply Last reply Reply Quote 0
                  • S
                    steve40
                    last edited by

                    Thanks, Bill  :)

                    1 Reply Last reply Reply Quote 0
                    • S
                      steve40
                      last edited by

                      Now I'm stuck trying to figure out how to post this question on their site?

                      lol :(

                      1 Reply Last reply Reply Quote 0
                      • ivorI
                        ivor
                        last edited by

                        https://redmine.openinfosecfoundation.org/account/register

                        Need help fast? Our support is available 24/7 https://www.netgate.com/support/

                        1 Reply Last reply Reply Quote 0
                        • S
                          steve40
                          last edited by

                          Hello Ivor.

                          I did register. Just not seeing a button or anything that would allow for posting

                          Thanks

                          1 Reply Last reply Reply Quote 0
                          • S
                            steve40
                            last edited by

                            Meeks,

                            One thing I noticed that's odd is that even after making the change to the php file as you indicated the suricata.yaml file in the interface subfolder keeps changing back to 4096.

                            If I set it manually to 0 in the .yaml file and leave suricata turned off at the pfsense level and run suricata via the **suricata -c **.yaml  -s single.rulefileIconfigured.rules -i re0 from the shell it works as expected. When I restart the interface it reverts back to 4096 in the yaml file

                            I've tried uninstalling, reinstalling, modifying the php BEFORE creating the interface and all… no luck

                            I even changed the php file to 10x the original size and it simply reverted back to 4096 upon the next restart of the interface

                            1 Reply Last reply Reply Quote 0
                            • bmeeksB
                              bmeeks
                              last edited by

                              Did you also check that parameter within the GUI to be sure it is set to "0"?  You will find it on the APP PARSERS tab.  Check that the default HTTP Engine has those two values set to zero instead of 4096.  I forgot to mention that in my earlier post.

                              Bill

                              1 Reply Last reply Reply Quote 0
                              • S
                                steve40
                                last edited by

                                Meeks. You're the best  :D

                                If you're ever in NY rounds on me

                                Thanks bro

                                1 Reply Last reply Reply Quote 0
                                • S
                                  steve40
                                  last edited by

                                  Hi Meeks.

                                  Im guessing this is normal behavior since suricata is merely intercepting a tcp stream and examining it but ive noticed that with the file hash comparison and blocking turned on the ips will always allow the first instance of a blacklisted file theough.  After that the ips engine will block the DST address and subsequent attempts to download the same file will be blocked along with the entire host.

                                  You know of any way to force it not to allow the first instance through?

                                  Also. Do you know if theres a way to have the executable name included in the alerts section without having to go to the eve.json log?

                                  Thanks

                                  1 Reply Last reply Reply Quote 0
                                  • bmeeksB
                                    bmeeks
                                    last edited by

                                    @steve40:

                                    Hi Meeks.

                                    Im guessing this is normal behavior since suricata is merely intercepting a tcp stream and examining it but ive noticed that with the file hash comparison and blocking turned on the ips will always allow the first instance of a blacklisted file theough.  After that the ips engine will block the DST address and subsequent attempts to download the same file will be blocked along with the entire host.

                                    You know of any way to force it not to allow the first instance through?

                                    Also. Do you know if theres a way to have the executable name included in the alerts section without having to go to the eve.json log?

                                    Thanks

                                    What you are seeing is an artifact of the Legacy Mode blocking.  Suricata, in that mode, is using libpcap to get copies of each packet traversing the interface.  It examines the copy (and in many cases may need to see a number of copies of additional packets) to make a block/no-block decision.  Once it decides to block, it makes a system call to insert the offender's IP address into a built-in pfSense firewall table called snort2c.  That will result in subsequent traffic from that IP address getting blocked until the IP is removed from that pf table.  Suricata will also optionally kill all open states associated with the IP (the default is to kill open states).  However, remember I said "copies of packets", so that initial download can get started as the original packets continued on to the firewall and then the host behind it (assuming the traffic was solicited by the host in the first place).  This is what we call "packet leakage" and is inherent in Legacy Mode.

                                    To prevent this you would need to run with Inline IPS Mode, but that mode uses native Netmap functionality in FreeBSD and also requires the NIC hardware driver to fully support Netmap operation.  Some do, but many to do not.  Also, you must remember that Suricata needs to download the entire file in order to compute the checksum to check against the list.  So a lot of memory might be required for buffering.

                                    Bill

                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      steve40
                                      last edited by

                                      @bmeeks:

                                      @steve40:

                                      Hi Meeks.

                                      Im guessing this is normal behavior since suricata is merely intercepting a tcp stream and examining it but ive noticed that with the file hash comparison and blocking turned on the ips will always allow the first instance of a blacklisted file theough.  After that the ips engine will block the DST address and subsequent attempts to download the same file will be blocked along with the entire host.

                                      You know of any way to force it not to allow the first instance through?

                                      Also. Do you know if theres a way to have the executable name included in the alerts section without having to go to the eve.json log?

                                      Thanks

                                      What you are seeing is an artifact of the Legacy Mode blocking.  Suricata, in that mode, is using libpcap to get copies of each packet traversing the interface.  It examines the copy (and in many cases may need to see a number of copies of additional packets) to make a block/no-block decision.  Once it decides to block, it makes a system call to insert the offender's IP address into a built-in pfSense firewall table called snort2c.  That will result in subsequent traffic from that IP address getting blocked until the IP is removed from that pf table.  Suricata will also optionally kill all open states associated with the IP (the default is to kill open states).  However, remember I said "copies of packets", so that initial download can get started as the original packets continued on to the firewall and then the host behind it (assuming the traffic was solicited by the host in the first place).  This is what we call "packet leakage" and is inherent in Legacy Mode.

                                      To prevent this you would need to run with Inline IPS Mode, but that mode uses native Netmap functionality in FreeBSD and also requires the NIC hardware driver to fully support Netmap operation.  Some do, but many to do not.  Also, you must remember that Suricata needs to download the entire file in order to compute the checksum to check against the list.  So a lot of memory might be required for buffering.

                                      Bill

                                      Hi Bill,

                                      thanks for the reply. I had a feeling that that would be the case so I used Inline mode and saw the same behavior. I belive the interface is a realtek card based on the "re0" …

                                      1 Reply Last reply Reply Quote 0
                                      • bmeeksB
                                        bmeeks
                                        last edited by

                                        When using Inline IPS mode, you would have to manually change the associated rule or rules to DROP action, but I'm still not sure what the impact is with these file matching rules.  As I stated earlier, it's obvious Suricata will have to let the entire file download before it can check the checksum to see if the file is on a blacklist or whitelist.  So the part I'm not sure about is what Suricata is doing with the parts of the file as it is downloaded and before the checksum can be calculated.  If Suricata buffered the entire file, then exactly what will it tell the LAN host that started the transfer?  If that host does not get any packets during the buffering, it will assume the download is failing.  If Suricata feeds it the file parts, then it can't block the download.  I've never investigated this feature, though, so I really am not sure how it works.

                                        Bill

                                        1 Reply Last reply Reply Quote 0
                                        • S
                                          steve40
                                          last edited by

                                          I see where you're coming from.

                                          Oh well, so much for trying to use suricata as a content filter  lol

                                          I wonder how clamav gets around the buffering/leaving client in limbo issue. I wrote a md5 signature for the exact same file and injected it into the .ndb files clam uses and it picked it up and blocked it on first attempt. No issues.

                                          Only thing is, for performance reasons id rather have done the checking lower down the stack

                                          1 Reply Last reply Reply Quote 0
                                          • S
                                            steve40
                                            last edited by

                                            Hiya Meeks… I got all the suricata file matching stuff working ...thanks for your help

                                            I identify binary files and block them via an empty hash whitelist. Which basically turns the box into a carbon black operating at the gateway level.  Works like a charm. (as long as you got pass rules for microsloth and places you wanna get exes from)

                                            It all works like a charm UNTIL.....

                                            you go to download an executable from an HTTPS enabled site.

                                            So out of desperation I'm going to ask a stupid question

                                            Is there a way to intercept these files while passing through an HTTPS session? I've got MITM fully working but I'm guessing that suricata operates at the NIC card and Squid decrypts the packet way higher up the stack...

                                            I really really really don't wanna have to do virus checking via ClamAv

                                            By the way, I've got this whole setup running on a KVM hypervisor so I can get very creative If I need to

                                            thanks

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.