Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Suricata Fast.log, but in JSON?

    Scheduled Pinned Locked Moved Off-Topic & Non-Support Discussion
    suricataids
    4 Posts 2 Posters 3.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B
      BurningJenkinsContainer
      last edited by BurningJenkinsContainer

      TL;DR: Does anyone have an approach for getting Suricata Alerts in JSON or Syslog formats in its own file? (akin to Fast.log)

      Hey all, new poster here, this board keeps coming up anytime I look up some stuff on Suricata so I figured it may a good place to start.

      I am using Suricata as a HIDS on some hosts, and have some basic Alert Rules setup, mostly for testing purposes. I have another side project going on to do some data analytics at some point on any of the alerts I generate.

      Now lies the crux of my problem, I need the data in JSON to do some ETL and enrichment later on, and EVE does that but I have been playing with the YAML and cannot get EVE to only output Alert Findings. If someone knows a good way to do that, that would be fantastic.

      Another approach I have looked at is parsing out of Fast.log, but it does not seem like you can output that data in JSON or Syslog format. The standard format for Fast.log isn't in Apache-* formats either so I cannot do some hacky on-host ETL either. Is there a way to turn that into JSON or even Syslog easily?

      I played around with the Syslog output, but from what I saw / read in the docs, it looks like it will be collated with all of my other syslogs which I definitely do not want, and the burden of separating it out is not worth it to me either.

      Thanks!

      Edit

      To add some color, this is a lazy copy and paste from what Eve shows that I took off of my SE cross-post

      ,"event_type":"stats","stats":{"uptime":168,"capture":{"kernel_packets":313,"kernel_drops":0,"errors":0},"decoder":{"pkts":313,"bytes":68519,"invalid":0,"ipv4":305,"ipv6":0,"ethernet":313,"r$
      {"timestamp":"2019-08-13T14:29:09.058698+0000","event_type":"stats","stats":{"uptime":176,"capture":{"kernel_packets":313,"kernel_drops":0,"errors":0},"decoder":{"pkts":313,"bytes":68519,"invalid":0,"ipv4":305,"ipv6":0,"ethernet":313,"r$
      {"timestamp":"2019-08-13T14:29:17.059944+0000","event_type":"stats","stats":{"uptime":184,"capture":{"kernel_packets":313,"kernel_drops":0,"errors":0},"decoder":{"pkts":313,"bytes":68519,"invalid":0,"ipv4":305,"ipv6":0,"ethernet":313,"r$
      

      I would only like to see something like this, from Fast.log, but you know...Jsonified?

      [**] [1:200002:6] ET USER_AGENTS Suspicious User Agent (BlackSun) [**] [Classification: A Network Trojan was detected] [Priority: 1] {TCP}
      
      1 Reply Last reply Reply Quote 0
      • bmeeksB
        bmeeks
        last edited by bmeeks

        You probably keep finding references to the pfSense forums because pfSense has a popular add-on GUI package for managing the Suricata binary's configuration.

        Most folks wind up piping the EVE JSON data collected by Suricata on pfSense out to a third-party logs analysis tool on another external host. Two of the more popular mechanisms are graylog and ELK.

        Here is a link to a how-to using Suricata and ELK on Ubuntu: https://www.howtoforge.com/tutorial/suricata-with-elk-and-web-front-ends-on-ubuntu-bionic-beaver-1804-lts/. This particular setup has Suricata on the Ubuntu machine as well, so pfSense would be out of the picture (which is how I understand your current situation to be).

        Here is a graylog link: https://www.graylog.org/post/visualize-and-correlate-ids-alerts-with-open-source-tools.

        I believe you can create your own custom filters for either of these platforms to further refine how logged alerts are presented.

        B 1 Reply Last reply Reply Quote 1
        • B
          BurningJenkinsContainer @bmeeks
          last edited by

          @bmeeks Thank you for the reply, that does make sense now that I am browsing the other board topics.

          You did make a correct assumption, pfSense is out of the picture, however I had already been playing around with ELK, the problem I have is I can move all of EVE easily and it will index it for me, but probably 80% of the lines are garbage noise (to me), which is why I was wondering if you could pare down what EVE outputs to ONLY Alerts.

          To add some color for other posters, the final 2 destinations are a custom BI solution I built that enriches the data with GEOINT and OSINT feeds for consumption by analysts, I will also be sending the non-enriched logs into a ML model that I have do vector autoregression and a few other time-series evil on it to eventually be consumed by yet another side project for needle-in-the-needlestack insights - this is all very much hobbyist, so while ELK and Graylog and Splunk all work great, it doesn't work for what madness I am over-engineering.

          1 Reply Last reply Reply Quote 0
          • B
            BurningJenkinsContainer
            last edited by

            Found an answer, took me long enough given it was right in front of me the whole time...

            On Line 60 in the YAML, you can disable Stats - that probably cuts down 80% of the garbage data in EVE.

            You can further disable logging (in EVE) under metadata for DNS, TLS, TCP, HTTP, etc. -- YMMV, but I feel keeping that stuff is fine since you can filter it out using something like Kibana or Splunk readily.

            1 Reply Last reply Reply Quote 0
            • First post
              Last post
            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.