Suricata - File-store location - impossible to change

karadoc

Hello every one,

I am using this version of PfSense 2.4.4-RELEASE-p3 and this version of Suricata Package 4.1.4.

I want to activate the file extraction feature (according to https://suricata.readthedocs.io/en/suricata-4.1.4/file-extraction/file-extraction.html# and https://suricata.readthedocs.io/en/suricata-4.1.4/configuration/suricata-yaml.html#suricata-yaml-file-store).

As explained by documentation, one of the most important part to take into account is the storage (Warning: This will consume a significant amount of disk..), so I mount on the pfsense a remote share on a remote nfs server (/mnt/suricata) and add it to /etc/fstab

here after the df -h command result:

[2.4.4-RELEASE][admin@fw.localdomain]/: df -h
Filesystem                                     Size    Used   Avail Capacity  Mounted on
/dev/ufsid/5c1d5ebb11c8d12f                     29G    3.5G     23G    13%    /
devfs                                          1.0K    1.0K      0B   100%    /dev
/dev/md0                                       3.4M    108K    3.0M     3%    /var/run
devfs                                          1.0K    1.0K      0B   100%    /var/dhcpd/dev
192.168.5.245:/home/nfsuser/nfs/www/pfsense    491G    115G    351G    25%    /mnt/suricata

objective is to store all files in this location (/mnt/suricata/filestore), and only files
Then if I go to the Suricata GUI, I can only set the "Enable File-Store" to enable, but I cannot specify the location.
So I tried to modify the following yml file /usr/local/etc/suricata/suricata.yaml :

   # To prune the filestore directory see the "suricatactl filestore
  # prune" command which can delete files over a certain age.
  - file-store:
      version: 2
      enabled: yes

      # Set the directory for the filestore. If the path is not
      # absolute will be be relative to the default-log-dir.
      dir: /mnt/suricata/filestore

      # Write out a fileinfo record for each occurrence of a
      # file. Disabled by default as each occurrence is already logged
      # as a fileinfo record to the main eve-log.
      #write-fileinfo: yes

      # Force storing of all files. Default: no.
      force-filestore: yes

      # Override the global stream-depth for sessions in which we want
      # to perform file extraction. Set to 0 for unlimited.
      #stream-depth: 0

and

 # File extraction depends on a lot of things to be fully done:
  # - file-store stream-depth. For optimal results, set this to 0 (unlimited)
  # - http request / response body sizes. Again set to 0 for optimal results.
  # - rules that contain the "filestore" keyword.
  - file-store:
      enabled: yes       # set to yes to enable
      log-dir: /mnt/suricata/filestore    # directory to store the files
      force-magic: yes   # force logging magic on all stored files
      # force logging of checksums, available hash functions are md5,
      # sha1 and sha256
      #force-hash: [md5]
      force-filestore: yes # force storing of all files
      # override global stream-depth for sessions in which we want to
      # perform file extraction. Set to 0 for unlimited.
      #stream-depth: 0
      #waldo: file.waldo # waldo file to store the file_id across runs
      # uncomment to disable meta file writing
      #write-meta: no
      # uncomment the following variable to define how many files can
      # remain open for filestore by Suricata. Default value is 0 which
      # means files get closed after each write
      #max-open-files: 1000
      include-pid: no # set to yes to include pid in file names

When I relaunch suricata, the yaml file attached to the interface seems to but updated but without my location settings:

  - file-store:
      version: 2
      enabled: yes
      log-dir: files
      force-magic: yes
      force-hash: [md5]
      waldo: file.waldo

As you can see log_dir is set to file and a new folder is created in /var/log/suricata/[suricata_interface_name]/filestore

Did I do it in the right way ?
Best regards

bmeeks

With all the pfSense packages, Suricata and Snort included, it it normal for the package code to completely rewrite its configuration file (in this case, suricata.yaml) each time you save a change in the GUI or the service is restarted. The package code stores its configuration info in the config.xml file used by the firewall to store all configuration information. This information is read from config.xml by the package code and then used to create (as in rewrite) the package configuration file on disk with each GUI save operation or when the service is restarted. Thus any manual changes you make are lost when the file is recreated.

There is currently no option for changing the logging root for Suricata. There is a feature request in the pfSense Redmine site for that option, but I have not coded it yet. It will require some extra work because of the way logging was originally implemented.

The only way to make the change you tried permanent is to edit the actual PHP code files that generate the suricata.yaml file for each interface. That file is /usr/local/pkg/suricata/suricata_generate_yaml.php. If you are proficient in PHP coding, you can have a look at the file to find the section that creates the file-store configuration and edit it. However, if you edit the code in that file, it will of course be overwritten the next time you update the Suricata package.

I am currently working on the next Suricata update, so I will look into adding the option of specifying a custom file-store logging path to the GUI code.

karadoc

@bmeeks said in Suricata - File-store location - impossible to change:

/usr/local/pkg/suricata/

Ok I understood, so I took a look on the file /usr/local/pkg/suricata/suricata_generate_yaml.php and I did not put any modification but I did an update in the /usr/local/pkg/suricata/suricata_yaml_template.inc:

I replaced the "log-dir:" by "dir:" in file-store section (https://suricata.readthedocs.io/en/suricata-4.1.4/configuration/suricata-yaml.html#suricata-yaml-file-store) and I set my path (yes...harcoded..)

It works ! :)
It will be great to get an option in the GUI in the future :)

thank you for your help

bmeeks

@karadoc said in Suricata - File-store location - impossible to change:

@bmeeks said in Suricata - File-store location - impossible to change:

/usr/local/pkg/suricata/

Ok I understood, so I took a look on the file /usr/local/pkg/suricata/suricata_generate_yaml.php and I did not put any modification but I did an update in the /usr/local/pkg/suricata/suricata_yaml_template.inc:

I replaced the "log-dir:" by "dir:" in file-store section (https://suricata.readthedocs.io/en/suricata-4.1.4/configuration/suricata-yaml.html#suricata-yaml-file-store) and I set my path (yes...harcoded..)

It works ! :)
It will be great to get an option in the GUI in the future :)

thank you for your help

What you did is correct. I did not look before I replied and I did not remember which portions of the suricata.yaml are created from the template and which are populated from runtime string variables. The template is used by the PHP code to create the final file. Some parts are literal strings while others are populated by expanding string variables that are set by PHP code in the suricata_generate_yaml.php file.

bmeeks

The option to specify a custom file-store logging directory is now available in the latest 4.1.6_1 version of the Suricata package. See the Release Notes in this post: https://forum.netgate.com/topic/149490/suricata-v4-1-6_1-package-update-release-notes.