Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    Suricata manual output configuration

    IDS/IPS
    2
    12
    110
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      mkcharlie last edited by

      Hello,

      My use case: I want:

      • to ship alerts with limited information via Telegraf to get alerts (via a Grafana dashboard etc.)
      • to be able to see the details of the alerts in the PFSense GUI

      I was using Suricata with eve.json output successfully to see the alerts in the PFsense GUI, but after adding a unix socket output I do not see the events in the PFsense GUI anymore.

      I enabled socket output via an advanced configuration pass-through because the GUI did not allow me to limit the output to the alert only for the unix socket and at the same time keep those details in the eve.json logging. Therefore I added an 'advanced configuration pass-through' to add the socket output.

      My hope was that multiple output blocks in the Yaml file would be concatenated, however it seems the second output block (which I added via the advanced pass-through configuration) seems to overwrite the first output.

      My yaml:

      %YAML 1.1
      ---
      
      max-pending-packets: 1024
      
      # Runmode the engine should use.
      runmode: autofp
      
      # If set to auto, the variable is internally switched to 'router' in IPS
      # mode and 'sniffer-only' in IDS mode.
      host-mode: auto
      
      # Specifies the kind of flow load balancer used by the flow pinned autofp mode.
      autofp-scheduler: hash
      
      # Daemon working directory
      daemon-directory: /usr/local/etc/suricata/suricata_5210_ix0
      
      default-packet-size: 1514
      
      # The default logging directory.
      default-log-dir: /var/log/suricata/suricata_ix05210
      
      # global stats configuration
      stats:
        enabled: no
        interval: 25
        decoder-events: true
        decoder-events-prefix: "decoder.event"
        stream-events: false
      
      # Configure the type of alert (and other) logging.
      outputs:
      
        # alert-pf blocking plugin
        - alert-pf:
            enabled: no
            kill-state: yes
            block-drops-only: no
            pass-list: /usr/local/etc/suricata/suricata_5210_ix0/passlist
            block-ip: BOTH
            pf-table: snort2c
      
        # a line based alerts log similar to Snort's fast.log
        - fast:
            enabled: yes
            filename: alerts.log
            append: yes
            filetype: regular
      
        - http-log:
            enabled: yes
            filename: http.log
            append: yes
            extended: yes
            filetype: regular
      
        - pcap-log:
            enabled: no
            filename: log.pcap
            limit: 32mb
            max-files: 1000
            mode: normal
      
        - tls-log:
            enabled: no
            filename: tls.log
            extended: yes
      
        - tls-store:
            enabled: no
            certs-log-dir: certs
      
        - stats:
            enabled: no
            filename: stats.log
            append: no
            totals: yes
            threads: no
            null-values: yes
      
        - syslog:
            enabled: no
            identity: suricata
            facility: local1
            level: notice
      
        - drop:
            enabled: no
            filename: drop.log
            append: yes
            filetype: regular
      
        - file-store:
            version: 2
            enabled: no
            length: 0
            dir: filestore
      
        - eve-log:
            enabled: yes
            filetype: regular
            filename: eve.json
            redis:
              server: 127.0.0.1
              port: 6379
              mode: list
              key: "suricata"
            identity: "suricata"
            facility: local1
            level: notice
            xff:
              enabled: no
              mode: extra-data
              deployment: reverse
              header: X-Forwarded-For
            types:
              - alert:
                  payload: yes              # enable dumping payload in Base64
                  payload-buffer-size: 4kb  # max size of payload buffer to output in eve-log
                  payload-printable: yes    # enable dumping payload in printable (lossy) format
                  packet: yes               # enable dumping of packet (without stream segments)
                  http-body: yes            # enable dumping of http body in Base64
                  http-body-printable: yes  # enable dumping of http body in printable format
                  metadata: yes             # enable inclusion of app layer metadata with alert
                  tagged-packets: yes       # enable logging of tagged packets for rules using the 'tag' keyword
      
        - eve-log:
            enabled: no
            filetype: unix_stream
            filename:
            types:
              - stats:
                  threads: yes
      
      # Magic file. The extension .mgc is added to the value here.
      magic-file: /usr/share/misc/magic
      
      # GeoLite2 IP geo-location database file path and filename.
      geoip-database: /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
      
      # Specify a threshold config file
      threshold-file: /usr/local/etc/suricata/suricata_5210_ix0/threshold.config
      
      detect-engine:
        - profile: medium
        - sgh-mpm-context: auto
        - inspection-recursion-limit: 3000
        - delayed-detect: no
      
      # Suricata is multi-threaded. Here the threading can be influenced.
      threading:
        set-cpu-affinity: no
        detect-thread-ratio: 1.0
      
      # Luajit has a strange memory requirement, it's 'states' need to be in the
      # first 2G of the process' memory.
      #
      # 'luajit.states' is used to control how many states are preallocated.
      # State use: per detect script: 1 per detect thread. Per output script: 1 per
      # script.
      luajit:
        states: 128
      
      # Multi pattern algorithm
      # The default mpm-algo value of "auto" will use "hs" if Hyperscan is
      # available, "ac" otherwise.
      mpm-algo: auto
      
      # Single pattern algorithm
      # The default of "auto" will use "hs" if available, otherwise "bm".
      spm-algo: auto
      
      # Defrag settings:
      defrag:
        memcap: 33554432
        hash-size: 65536
        trackers: 65535
        max-frags: 65535
        prealloc: yes
        timeout: 60
      
      # Flow settings:
      flow:
        memcap: 33554432
        hash-size: 65536
        prealloc: 10000
        emergency-recovery: 30
        prune-flows: 5
      
      # This option controls the use of vlan ids in the flow (and defrag)
      # hashing.
      vlan:
        use-for-tracking: true
      
      # Specific timeouts for flows.
      flow-timeouts:
        default:
          new: 30
          established: 300
          closed: 0
          emergency-new: 10
          emergency-established: 100
          emergency-closed: 0
        tcp:
          new: 60
          established: 3600
          closed: 120
          emergency-new: 10
          emergency-established: 300
          emergency-closed: 20
        udp:
          new: 30
          established: 300
          emergency-new: 10
          emergency-established: 100
        icmp:
          new: 30
          established: 300
          emergency-new: 10
          emergency-established: 100
      
      stream:
        memcap: 131217728
        checksum-validation: no
        inline: auto
        prealloc-sessions: 32768
        midstream: false
        async-oneside: false
        max-synack-queued: 5
        bypass: no
        drop-invalid: no
        reassembly:
          memcap: 131217728
          depth: 1048576
          toserver-chunk-size: 2560
          toclient-chunk-size: 2560
      
      # Host table is used by tagging and per host thresholding subsystems.
      host:
        hash-size: 4096
        prealloc: 1000
        memcap: 33554432
      
      # Host specific policies for defragmentation and TCP stream reassembly.
      host-os-policy:
        bsd: [0.0.0.0/0]
      
      # Logging configuration.  This is not about logging IDS alerts, but
      # IDS output about what its doing, errors, etc.
      logging:
      
        # This value is overriden by the SC_LOG_LEVEL env var.
        default-log-level: info
        default-log-format: "%t - <%d> -- "
      
        # Define your logging outputs.
        outputs:
        - console:
            enabled: yes
        - file:
            enabled: yes
            filename: /var/log/suricata/suricata_ix05210/suricata.log
        - syslog:
            enabled: no
            facility: local1
            level: notice
            format: "[%i] <%d> -- "
      
      # IPS Mode Configuration
      # PCAP
      pcap:
        - interface: ix0
          checksum-checks: auto
          promisc: yes
          snaplen: 1518
      
      legacy:
        uricontent: enabled
      
      default-rule-path: /usr/local/etc/suricata/suricata_5210_ix0/rules
      rule-files:
       - suricata.rules
       - flowbit-required.rules
       - custom.rules
      
      classification-file: /usr/local/etc/suricata/suricata_5210_ix0/classification.config
      reference-config-file: /usr/local/etc/suricata/suricata_5210_ix0/reference.config
      
      # Holds variables that would be used by the engine.
      vars:
      
        # Holds the address group vars that would be passed in a Signature.
        address-groups:
          HOME_NET: "[10.10.10.1/32, 10.70.1.0/24, 10.70.2.0/24, 10.70.5.0/24, 10.70.7.0/24, 10.70.8.0/24, 10.70.9.0/24, 10.70.10.0/24, 10.70.11.0/24, 10.70.50.0/24, 10.70.60.0/24, 10.70.80.0/24, 10.70.90.0/24, 10.70.100.0/24, 81.82.192.1/32, 81.82.247.159/32, 127.0.0.1/32, 195.130.130.4/32, 195.130.131.4/32, ::1/128, fe80::3eec:efff:fe21:74e0/128, fe80::208:a2ff:fe11:4d16/128, fe80::208:a2ff:fe11:4d18/128]"
          EXTERNAL_NET: "[!10.10.10.1/32, !10.70.1.0/24, !10.70.2.0/24, !10.70.5.0/24, !10.70.7.0/24, !10.70.8.0/24, !10.70.9.0/24, !10.70.10.0/24, !10.70.11.0/24, !10.70.50.0/24, !10.70.60.0/24, !10.70.80.0/24, !10.70.90.0/24, !10.70.100.0/24, !81.82.192.1/32, !81.82.247.159/32, !127.0.0.1/32, !195.130.130.4/32, !195.130.131.4/32, !::1/128, !fe80::3eec:efff:fe21:74e0/128, !fe80::208:a2ff:fe11:4d16/128, !fe80::208:a2ff:fe11:4d18/128]"
          DNS_SERVERS: "$HOME_NET"
          SMTP_SERVERS: "$HOME_NET"
          HTTP_SERVERS: "$HOME_NET"
          SQL_SERVERS: "$HOME_NET"
          TELNET_SERVERS: "$HOME_NET"
          DNP3_SERVER: "$HOME_NET"
          DNP3_CLIENT: "$HOME_NET"
          MODBUS_SERVER: "$HOME_NET"
          MODBUS_CLIENT: "$HOME_NET"
          ENIP_SERVER: "$HOME_NET"
          ENIP_CLIENT: "$HOME_NET"
          FTP_SERVERS: "$HOME_NET"
          SSH_SERVERS: "$HOME_NET"
          AIM_SERVERS: "64.12.24.0/23, 64.12.28.0/23, 64.12.161.0/24, 64.12.163.0/24, 64.12.200.0/24, 205.188.3.0/24, 205.188.5.0/24, 205.188.7.0/24, 205.188.9.0/24, 205.188.153.0/24, 205.188.179.0/24, 205.188.248.0/24"
          SIP_SERVERS: "$HOME_NET"
      
        # Holds the port group vars that would be passed in a Signature.
        port-groups:
          FTP_PORTS: "21"
          HTTP_PORTS: "80"
          ORACLE_PORTS: "1521"
          SSH_PORTS: "22"
          SHELLCODE_PORTS: "!80"
          DNP3_PORTS: "20000"
          FILE_DATA_PORTS: "$HTTP_PORTS, 110, 143"
          SIP_PORTS: "5060, 5061, 5600"
      
      # Set the order of alerts based on actions
      action-order:
        - pass
        - drop
        - reject
        - alert
      
      # IP Reputation
      
      
      # Limit for the maximum number of asn1 frames to decode (default 256)
      asn1-max-frames: 256
      
      engine-analysis:
        rules-fast-pattern: yes
        rules: yes
      
      #recursion and match limits for PCRE where supported
      pcre:
        match-limit: 3500
        match-limit-recursion: 1500
      
      # Holds details on the app-layer. The protocols section details each protocol.
      app-layer:
        protocols:
          dcerpc:
            enabled: yes
          dhcp:
            enabled: yes
          dnp3:
            enabled: yes
            detection-ports:
              dp: 20000
          dns:
            global-memcap: 16777216
            state-memcap: 524288
            request-flood: 500
            tcp:
              enabled: yes
              detection-ports:
                dp: 53
            udp:
              enabled: yes
              detection-ports:
                dp: 53
          ftp:
            enabled: yes
          ftp-data:
            enabled: no
          http:
            enabled: yes
            memcap: 67108864
          ikev2:
            enabled: yes
          imap:
            enabled: detection-only
          krb5:
            enabled: yes
          modbus:
            enabled: yes
            request-flood: 500
            detection-ports:
              dp: 502
            stream-depth: 0
          msn:
            enabled: detection-only
          nfs:
            enabled: yes
          ntp:
            enabled: yes
          tls:
            enabled: yes
            detection-ports:
              dp: 443
            ja3-fingerprints: off
            encrypt-handling: default
          smb:
            enabled: yes
            detection-ports:
              dp: 139, 445
          smtp:
            enabled: yes
            mime:
              decode-mime: no
              decode-base64: yes
              decode-quoted-printable: yes
              header-value-depth: 2000
              extract-urls: yes
              body-md5: no
            inspected-tracker:
              content-limit: 100000
              content-inspect-min-size: 32768
              content-inspect-window: 4096
          ssh:
            enabled: yes
          tftp:
            enabled: yes
          rdp:
            enabled: yes
          sip:
            enabled: yes
          snmp:
            enabled: yes
          http2:
            enabled: no
          rfb:
            enabled: yes
            detection-ports:
              dp: 5900, 5901, 5902, 5903, 5904, 5905, 5906, 5907, 5908, 5909
      
      ###########################################################################
      # Configure libhtp.
      libhtp:
         default-config:
           personality: IDS
           request-body-limit: 4096
           response-body-limit: 4096
           meta-field-limit: 18432
           double-decode-path: no
           double-decode-query: no
           uri-include-all: no
      
      
      
      coredump:
        max-dump: unlimited
      
      # Suricata user pass through configuration
      outputs:
        - eve-log:
            enabled: yes
            filetype: unix_stream
            filename: /var/run/suricata-stats.sock
            types:
              - alert:
                  payload: no
                  payload-printable: no
                  packet: no
                  http-body: no
                  metadata: no
      

      enabled services:

      • snort
      • pfblockerNG
      1 Reply Last reply Reply Quote 0
      • bmeeks
        bmeeks last edited by bmeeks

        The configuration process of the GUI store all parameters in the config.xml file of the firewall. Each time you click Save in a GUI window, or start or restart Suricata using the GUI icons on the INTERFACES tab, the configuration information is pulled from the config.xml file on the firewall and written into a fresh copy suricata.yaml in the sub-directory for the Suricata interface.

        I believe what you have hit is a current limitation of the Suricata binary. I think you might be hitting an issue similar (but not exactly the same) as the user posting here: https://forum.suricata.io/t/overide-suricata-configuation-file/3216.

        M 1 Reply Last reply Reply Quote 1
        • M
          mkcharlie @bmeeks last edited by

          @bmeeks it seems to be the same problem indeed: the original output is overwritten by the custom config.

          Is there currently a way to achieve the following config using the GUI?

          outputs:
            - eve-log:
                enabled: yes
                filetype: regular
                filename: eve.json
                redis:
                  server: 127.0.0.1
                  port: 6379
                  mode: list
                  key: "suricata"
                identity: "suricata"
                facility: local1
                level: notice
                xff:
                  enabled: no
                  mode: extra-data
                  deployment: reverse
                  header: X-Forwarded-For
                types:
                  - alert:
                      payload: yes              # enable dumping payload in Base64
                      payload-buffer-size: 4kb  # max size of payload buffer to output in eve-log
                      payload-printable: yes    # enable dumping payload in printable (lossy) format
                      packet: yes               # enable dumping of packet (without stream segments)
                      http-body: yes            # enable dumping of http body in Base64
                      http-body-printable: yes  # enable dumping of http body in printable format
                      metadata: yes             # enable inclusion of app layer metadata with alert
                      tagged-packets: yes       # enable logging of tagged packets for rules using the 'tag' keyword
          
            - eve-log:
                enabled: yes
                filetype: unix_stream
                filename: /var/run/suricata-stats.sock
                types:
                  - alert:
                      payload: no
                      payload-printable: no
                      packet: no
                      http-body: no
                      metadata: no
          

          enabled services:

          • snort
          • pfblockerNG
          bmeeks 1 Reply Last reply Reply Quote 0
          • bmeeks
            bmeeks @mkcharlie last edited by bmeeks

            @mkcharlie:
            Sort of, is the best answer. You can't do it directly within the GUI using just tools available there. It will require editing a PHP source code file. Your edit would stay until the next time you either upgraded the Suricata package or removed and reinstalled it.

            One major caveat with this method: this change will be reflected on ALL of your Suricata interfaces as the file you modify for this change is used to build the suricata.yaml file for every configured Suricata interface.

            Here is what you can do. You will need direct access to the firewall file system. My favorite tool for doing this is WinSCP installed on a Windows box, then connect to the firewall from WinSCP using scp:

            1. Locate the file /usr/local/pkg/suricata/suricata_yaml_template.inc.
            2. Make a copy of it for backup in the event you want to restore default behavior or something gets severely messed up.
            3. Double-click the file in WinSCP to open it in the built-in editor.
            4. This file is a template the PHP code uses to create the actual suricata.yaml file for the interface. Be careful not to disturb or change any of the phrases with "$" in front. Those are PHP variables used when writing the actual YAML file.
            5. Scroll down in the file and find the - eve-log: sections. There are two of them.
            6. You can add your third EVE log section under the second one. Here is an example:
            - eve-log:
                enabled: {$enable_eve_log}
                filetype: {$eve_output_type}
                filename: eve.json
                redis: {$eve_redis_output}
                identity: "suricata"
                facility: {$eve_systemlog_facility}
                level: {$eve_systemlog_priority}
                xff:
                  enabled: {$eve_xff_enabled}
                  mode: {$eve_xff_mode}
                  deployment: {$eve_xff_deployment}
                  header: {$eve_xff_header}
                types: {$eve_out_types}
            
            - eve-log:
                enabled: {$enable_telegraf_eve}
                filetype: unix_stream
                filename: {$telegraf_eve_sockname}
                types:
                  - stats:
                      threads: yes
            
            - eve-log:
                enabled: yes
                filetype: unix_stream
                filename: /var/run/suricata-stats.sock
                types:
                  - alert:
                      payload: no
                      payload-printable: no
                      packet: no
                      http-body: no
                      metadata: no
            
            1. Save your edit and that should do it for you if what you want is the EVE log output to a Unix socket.
            M 1 Reply Last reply Reply Quote 0
            • M
              mkcharlie @bmeeks last edited by

              @bmeeks thanks! I can try that. Just, wouldn’t it be easier then to just copy the whole output section to the advanced pass through configuration? That would survive upgrades right?

              enabled services:

              • snort
              • pfblockerNG
              bmeeks 1 Reply Last reply Reply Quote 0
              • bmeeks
                bmeeks @mkcharlie last edited by bmeeks

                @mkcharlie said in Suricata manual output configuration:

                @bmeeks thanks! I can try that. Just, wouldn’t it be easier then to just copy the whole output section to the advanced pass through configuration? That would survive upgrades right?

                I'm not sure putting everything in the passthrough section would work, but you can certainly try it. You would need to copy it from the working suricata.yaml file, though. That code in that template file I referenced earlier is read by the PHP code and then used to spit out a different text file that becomes the YAML config for the interface. So you paste anything with a $ sign in front of it in the passthrough section, that will blow up as it will not get translated. At runtime, as the PHP code is creating the YAML file, it puts content from the config.xml of the firewall into those $variables, and that content is what gets written to the final YAML file.

                That template file is used in a heredoc call within the PHP code. Here is a link that explains what that is: https://andy-carter.com/blog/what-are-php-heredoc-nowdoc.

                M 1 Reply Last reply Reply Quote 0
                • M
                  mkcharlie @bmeeks last edited by

                  @bmeeks I can copy the full output to the pass-through, but that does not seem to solve the issue. The config looks fine though, so now I'm wondering if there is another cause for this issue.

                  enabled services:

                  • snort
                  • pfblockerNG
                  M 1 Reply Last reply Reply Quote 0
                  • M
                    mkcharlie @mkcharlie last edited by

                    Update: had to restart, seems to work. I'll do some more testing.

                    enabled services:

                    • snort
                    • pfblockerNG
                    bmeeks 1 Reply Last reply Reply Quote 0
                    • bmeeks
                      bmeeks @mkcharlie last edited by

                      @mkcharlie said in Suricata manual output configuration:

                      Update: had to restart, seems to work. I'll do some more testing.

                      Nothing in Suricata is dynamic. Changes in the configuration require a restart of the service so the daemon reads the suricata.yaml file again. The only thing you can do "dynamically" is update the rules, then send a special SIGHUP command to the daemon so it will reload the in-memory rules. But any changes to the suricata.yaml require a full restart as that file is only processed during startup.

                      M 1 Reply Last reply Reply Quote 1
                      • M
                        mkcharlie @bmeeks last edited by

                        @bmeeks clear, all seems to work well. I suppose this solution survives updates. Thanks for the help!

                        enabled services:

                        • snort
                        • pfblockerNG
                        bmeeks 1 Reply Last reply Reply Quote 0
                        • bmeeks
                          bmeeks @mkcharlie last edited by bmeeks

                          @mkcharlie said in Suricata manual output configuration:

                          @bmeeks clear, all seems to work well. I suppose this solution survives updates. Thanks for the help!

                          If everything is being passed in the Advanced Passthrough box, then yes it will survive any package or pfSense updates because that data lives as a field in config.xml, the file that holds all the firewall and package configuration information.

                          What would not survive an update is if you modified the PHP source file as I was describing previously.

                          1 Reply Last reply Reply Quote 0
                          • bmeeks
                            bmeeks last edited by bmeeks

                            I will repeat here for clarity something I've mentioned in some other Suricata posts.

                            The Suricata package consists of two unique and separate components. One is a GUI front-end written in PHP. That GUI is what you interact with. It is used to store and manage configuration information for the Suricata interfaces. When you click Save after making a configuration change, the GUI PHP code consolidates all the config parameters and writes them to the suricata.yaml file for the interface.

                            The other piece of the package is the Suricata executable binary that runs as a service. This piece comes from the upstream Suricata developers. That binary is distributed to run on my different operating systems, but it is purely a command-line interface that uses a combination of the suricata.yaml config file and arguments passed on the command line to control its operation. The GUI part of Suricata on pfSense just generates that YAML file and then starts the binary piece with the appropriate command-line arguments.

                            I mention this as a lot of folks seem to misunderstand the distinction between the GUI part they see and binary part they do not. But the binary is where all the real work happens.

                            1 Reply Last reply Reply Quote 0
                            • First post
                              Last post