Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Suricata - Netflow and Hiredis Support

    Scheduled Pinned Locked Moved IDS/IPS
    11 Posts 5 Posters 3.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      securitym0nkey
      last edited by

      Hi,

      the latest version of the Suricata package does come with hiredis support. There is a GUI option now to configure it as well. Currently the GUI is quite limited on configuring what should be included in the eve-output. I'm already working to on a patch to improve that and will file a pull request soon.

      Currently you might want to configure the eve-output by hand using the pass through config option.

      1 Reply Last reply Reply Quote 0
      • W
        wirehead
        last edited by

        I just tried this tonight with the latest 4.0.0 package FYI - I get a JSON not supported / compiled error.  Ideally I don't want to put anythign on pfsense, having it REDIS to a remote machine I have redis and logstash own to grok and inject into my elastic cluster is superb. Keeps the pfsense box foreign object free ;)

        
        suricata		security 	4.0.0		High Performance Network IDS, IPS and Security Monitoring engine by OISF.
        
        Package Dependencies:
          suricata-4.0.0    barnyard2-1.13_1  
        
        

        But Got this error during restarting the interface I setup for redis logging:

        
        22/8/2017 -- 21:09:29 - <error>-- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - redis JSON output option is not compiled</error> 
        

        Digging a little further I noticed hiredis not enabled in the compiled binary on the pfsense system. Is that required to just do 'flows' or is it needed for ANY of the reporting types?

        
        [2.3.4-RELEASE][root@firewall.lan]/root: suricata --build-info
        This is Suricata version 3.2.1 RELEASE
        Features: IPFW PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 NETMAP HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS MAGIC
        SIMD support: none
        Atomic intrisics: 1 2 4 8 byte(s)
        64-bits, Little-endian architecture
        GCC version 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032), C version 199901
        compiled with -fstack-protector
        compiled with _FORTIFY_SOURCE=2
        L1 cache line size (CLS)=64
        thread local storage method: __thread
        compiled with LibHTP v0.5.23, linked against LibHTP v0.5.23
        
        Suricata Configuration:
          AF_PACKET support:                       no
          PF_RING support:                         no
          NFQueue support:                         no
          NFLOG support:                           no
          IPFW support:                            yes
          Netmap support:                          yes
          DAG enabled:                             no
          Napatech enabled:                        no
        
          Unix socket enabled:                     yes
          Detection enabled:                       yes
        
          Libmagic support:                        yes
          libnss support:                          yes
          libnspr support:                         yes
          libjansson support:                      yes
          hiredis support:                         no
          Prelude support:                         no
          PCRE jit:                                yes
          LUA support:                             yes, through luajit
          libluajit:                               yes
          libgeoip:                                yes
          Non-bundled htp:                         yes
          Old barnyard2 support:                   no
          CUDA enabled:                            no
          Hyperscan support:                       yes
          Libnet support:                          yes
        
          Suricatasc install:                      no
        
          Profiling enabled:                       no
          Profiling locks enabled:                 no
        
        Development settings:
          Coccinelle / spatch:                     no
          Unit tests enabled:                      no
          Debug output enabled:                    no
          Debug validation enabled:                no
        
        Generic build parameters:
          Installation prefix:                     /usr/local
          Configuration directory:                 /usr/local/etc/suricata/
          Log directory:                           /var/log/suricata/
        
          --prefix                                 /usr/local
          --sysconfdir                             /usr/local/etc
          --localstatedir                          /var
        
          Host:                                    amd64-portbld-freebsd10.3
          Compiler:                                cc (exec name) / clang (real)
          GCC Protect enabled:                     yes
          GCC march native enabled:                no
          GCC Profile enabled:                     no
          Position Independent Executable enabled: no
          CFLAGS                                   -O2 -pipe  -fstack-protector -fno-strict-aliasing -DOS_FREEBSD
          PCAP_CFLAGS                               -I/usr/local/include
          SECCFLAGS                                -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security
        
        

        So is this```
        hiredis support

        
        I used the GUI, pointed the system at a redis server (did a telnet port confirmation not firewalled to remote redis IP above, sniffed the wire for any attempts for data to be sent and checkboxed all the options in the pfsense GUI minus the 'Flows' option etc.  Here is the relevant config pulled from pfense itself the GUI generated - which is part of the larger```
        output:
        ```YAML section.
        
        Screenshot of config
        
        ![](http://firewall_lan_-_Services__Suricata__Edit_Interface_Settings_-_WAN.png)
        
        Config generated (in part on pfsense box)
        
        
        • eve-log:
                enabled: yes
                type: redis
                redis:
                  server: 192.168.10.178
                  port: 6379
                  mode: list
                  key: "suricata"
                filename: eve.json
                identity: "suricata"
                facility: local1
                level: info
                types:
                  - alert:
                      payload: yes          # enable dumping payload in Base64
                      payload-printable: yes # enable dumping payload in printable (lossy) format
                      packet: yes            # enable dumping of packet (without stream segments)
                      http: yes              # enable dumping of http fields
                      tls: yes              # enable dumping of tls fields
                      ssh: yes              # enable dumping of ssh fields
                      smtp: yes              # enable dumping of smtp fields
                      dnp3: yes              # enable dumping of DNP3 fields
                      tagged-packets: yes    # enable logging of tagged packets
                  - http:
                      extended: yes
                      custom: [accept, accept-charset, accept-encoding, accept-language,
                              accept-datetime, authorization, cache-control, cookie, from,
                              max-forwards, origin, pragma, proxy-authorization, range, te, via,
                              x-requested-with, dnt, x-forwarded-proto, accept-range, age,
                              allow, connection, content-encoding, content-language,
                              content-length, content-location, content-md5, content-range,
                              content-type, date, etags, last-modified, link, location,
                              proxy-authenticate, referrer, refresh, retry-after, server,
                              set-cookie, trailer, transfer-encoding, upgrade, vary, warning,
                              www-authenticate, x-flash-version, x-authenticated-user]
                - dns:
                      query: yes
                      answer: yes
                  - tls:
                      extended: no
                  - files:
                      force-magic: no
                  - ssh
                  - smtp:
                      extended: yes
                      custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
                      md5: [subject]
        
        Suricata runs, but I see this in the logs…
        
        

        22/8/2017 -- 22:15:51 - <error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - redis JSON output option is not compiled</error>

        
        ![firewall_lan_-_Services__Suricata__Edit_Interface_Settings_-_WAN.png](/public/_imported_attachments_/1/firewall_lan_-_Services__Suricata__Edit_Interface_Settings_-_WAN.png)
        ![firewall_lan_-_Services__Suricata__Edit_Interface_Settings_-_WAN.png_thumb](/public/_imported_attachments_/1/firewall_lan_-_Services__Suricata__Edit_Interface_Settings_-_WAN.png_thumb)
        1 Reply Last reply Reply Quote 0
        • W
          wirehead
          last edited by

          Correction. Suricata eventually stops once it hits the JSON error above, thats the last entry in the logs. My bad.  Please let me know if anyone knows whats blocking me? Need to get hiredis compiled in? Or is something else blocking me config wise?

          1 Reply Last reply Reply Quote 0
          • bmeeksB
            bmeeks
            last edited by

            @wirehead:

            Correction. Suricata eventually stops once it hits the JSON error above, thats the last entry in the logs. My bad.  Please let me know if anyone knows whats blocking me? Need to get hiredis compiled in? Or is something else blocking me config wise?

            Your posted screen output indicates you are using Suricata 3.2.1.  Version 4.0.0 is now posted.  I just verified the compiler options are set to enable REDIS support in version 4.0.0.  Upgrade your Suricata installation and try again.

            Bill

            1 Reply Last reply Reply Quote 0
            • W
              wirehead
              last edited by

              Indeed. That is most odd. If I ssh in and open a shell, checking version shows 3.2.1 indeed (sorry missed that)

              
              [2.3.4-RELEASE][root@firewall.lan]/root: suricata -V
              This is Suricata version 3.2.1 RELEASE
              
              

              Looks like somehow I have the old binary and new GUI elements… though as the package manager shows 4.0.0 installed. I can't find a situation of mixed binaries either..

              
              [2.3.4-RELEASE][root@firewall.lan]/root: find / -name "suricata" -type f
              /usr/local/bin/suricata
              /usr/local/etc/rc.d/suricata
              
              

              And when I run that binary…

              
              [2.3.4-RELEASE][root@firewall.lan]/root: /usr/local/bin/suricata -V
              This is Suricata version 3.2.1 RELEASE
              
              

              Very very odd.

              I'll try and re-install it and see what happens. I did the upgrade a week or so ago maybe something didn't go right…

              1 Reply Last reply Reply Quote 0
              • W
                wirehead
                last edited by

                So I had to fully uninstall the package, upgrade or in place re-install did not push out the old binary. Very odd.

                
                [2.3.4-RELEASE][root@firewall.lan]/root: /usr/local/bin/suricata -V
                This is Suricata version 4.0.0 RELEASE
                
                

                Confirmed support etc thx

                
                [2.3.4-RELEASE][root@firewall.lan]/usr/local/etc/suricata/suricata_53398_pppoe0: suricata --build-info |grep redis
                  hiredis support:                         yes
                  hiredis async with libevent:             no
                
                
                1 Reply Last reply Reply Quote 0
                • bmeeksB
                  bmeeks
                  last edited by

                  @wirehead:

                  So I had to fully uninstall the package, upgrade or in place re-install did not push out the old binary. Very odd.

                  
                  [2.3.4-RELEASE][root@firewall.lan]/root: /usr/local/bin/suricata -V
                  This is Suricata version 4.0.0 RELEASE
                  
                  

                  Confirmed support etc thx

                  
                  [2.3.4-RELEASE][root@firewall.lan]/usr/local/etc/suricata/suricata_53398_pppoe0: suricata --build-info |grep redis
                    hiredis support:                         yes
                    hiredis async with libevent:             no
                  
                  

                  Yeah, that is weird.  I would expect the update to recognize the binary change.  This is something I will need to discuss with the pfSense guys.  Maybe something else is needed in the Package Manager configuration for Suricata.

                  Bill

                  1 Reply Last reply Reply Quote 0
                  • W
                    wirehead
                    last edited by

                    So I got this to work! But I had to remove syslog config the GUI configuration yaml generated

                    I removed these entries… or it would not send the REDIS data...

                    
                          filename: eve.json
                          identity: "suricata"
                          facility: local1
                          level: info
                    
                    

                    Here is the working configuration for my testing…

                    
                      - eve-log:
                          enabled: yes
                          type: redis
                          redis:
                            server: 192.168.10.178
                            port: 6379
                            mode: list
                            key: "suricata"
                          types:
                            - alert:
                                payload: yes           # enable dumping payload in Base64
                                payload-printable: yes # enable dumping payload in printable (lossy) format
                                packet: yes            # enable dumping of packet (without stream segments)
                                http: yes              # enable dumping of http fields
                                tls: yes               # enable dumping of tls fields
                                ssh: yes               # enable dumping of ssh fields
                                smtp: yes              # enable dumping of smtp fields
                                dnp3: yes              # enable dumping of DNP3 fields
                                tagged-packets: yes    # enable logging of tagged packets
                            - http:
                                extended: yes
                                custom: [accept, accept-charset, accept-encoding, accept-language,
                                        accept-datetime, authorization, cache-control, cookie, from,
                                        max-forwards, origin, pragma, proxy-authorization, range, te, via,
                                        x-requested-with, dnt, x-forwarded-proto, accept-range, age,
                                        allow, connection, content-encoding, content-language,
                                        content-length, content-location, content-md5, content-range,
                                        content-type, date, etags, last-modified, link, location,
                                        proxy-authenticate, referrer, refresh, retry-after, server,
                                        set-cookie, trailer, transfer-encoding, upgrade, vary, warning,
                                        www-authenticate, x-flash-version, x-authenticated-user]
                            - dns:
                                query: yes
                                answer: yes
                            - tls:
                                extended: no
                            - files:
                                force-magic: no
                            - ssh
                            - smtp:
                                extended: yes
                                custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
                                md5: [subject]
                    
                    

                    This was what the GUI generated that didn't work..

                    
                      - eve-log:
                          enabled: yes
                          type: redis
                          redis:
                            server: 192.168.10.178
                            port: 6379
                            mode: list
                            key: "suricata"
                          filename: eve.json
                          identity: "suricata"
                          facility: local1
                          level: info
                          types:
                            - alert:
                                payload: yes           # enable dumping payload in Base64
                                payload-printable: yes # enable dumping payload in printable (lossy) format
                                packet: yes            # enable dumping of packet (without stream segments)
                                http: yes              # enable dumping of http fields
                                tls: yes               # enable dumping of tls fields
                                ssh: yes               # enable dumping of ssh fields
                                smtp: yes              # enable dumping of smtp fields
                                dnp3: yes              # enable dumping of DNP3 fields
                                tagged-packets: yes    # enable logging of tagged packets
                            - http:
                                extended: yes
                                custom: [accept, accept-charset, accept-encoding, accept-language,
                                        accept-datetime, authorization, cache-control, cookie, from,
                                        max-forwards, origin, pragma, proxy-authorization, range, te, via,
                                        x-requested-with, dnt, x-forwarded-proto, accept-range, age,
                                        allow, connection, content-encoding, content-language,
                                        content-length, content-location, content-md5, content-range,
                                        content-type, date, etags, last-modified, link, location,
                                        proxy-authenticate, referrer, refresh, retry-after, server,
                                        set-cookie, trailer, transfer-encoding, upgrade, vary, warning,
                                        www-authenticate, x-flash-version, x-authenticated-user]
                            - dns:
                                query: yes
                                answer: yes
                            - tls:
                                extended: no
                            - files:
                                force-magic: no
                            - ssh
                            - smtp:
                                extended: yes
                                custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
                                md5: [subject]
                    
                    

                    Here is a screencap from the GUI settings I used to start with…
                    (see attached)

                    ![2017-08-23 12_05_43-firewall.lan - Services_ Suricata_ Edit Interface Settings - WAN.png](/public/imported_attachments/1/2017-08-23 12_05_43-firewall.lan - Services_ Suricata_ Edit Interface Settings - WAN.png)
                    ![2017-08-23 12_05_43-firewall.lan - Services_ Suricata_ Edit Interface Settings - WAN.png_thumb](/public/imported_attachments/1/2017-08-23 12_05_43-firewall.lan - Services_ Suricata_ Edit Interface Settings - WAN.png_thumb)

                    1 Reply Last reply Reply Quote 0
                    • bmeeksB
                      bmeeks
                      last edited by

                      Yeah, this part –

                      
                            filename: eve.json
                            identity: "suricata"
                            facility: local1
                            level: info
                      
                      

                      does not belong there.  Probably coming from an uncleared string in the GUI code (just a guess without looking).  I did not write that code enhancement.  Another user contributed the code.  I will need to take a look and see what the problem is.

                      Bill

                      1 Reply Last reply Reply Quote 0
                      • S
                        shubham_shah
                        last edited by

                        Hi,

                        I am able to implement Suricata-Redis architecture. Please let me know whether we can use Redis-sentinel feature at Suricata Config block. Because my application will be required redis failover support so if I can also configure Suricata with Redis Sentinel then it would be the best for me.

                        Shubham

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.