Suricata - Netflow and Hiredis Support



  • Hi,

    I have a few questions about the Suricata package in pfSense:

    Is Netflow-Export included as in the standard version? - cant find settings for this in the gui and config file [ /usr/local/pkg/suricata/suricata_generate_yaml.php]

    I would like to export eve-log via hiredis to a redis-server  - before building a suricata package with hiredis support - I would like to know if i have to do additional customizing because I cant find anything regarding eve-log in the config file [ /usr/local/pkg/suricata/suricata_generate_yaml.php]. Or is there another config file?

    eve-log -> http://suricata.readthedocs.io/en/latest/output/eve/eve-json-output.html



  • Hi,

    the latest version of the Suricata package does come with hiredis support. There is a GUI option now to configure it as well. Currently the GUI is quite limited on configuring what should be included in the eve-output. I'm already working to on a patch to improve that and will file a pull request soon.

    Currently you might want to configure the eve-output by hand using the pass through config option.



  • I just tried this tonight with the latest 4.0.0 package FYI - I get a JSON not supported / compiled error.  Ideally I don't want to put anythign on pfsense, having it REDIS to a remote machine I have redis and logstash own to grok and inject into my elastic cluster is superb. Keeps the pfsense box foreign object free ;)

    
    suricata		security 	4.0.0		High Performance Network IDS, IPS and Security Monitoring engine by OISF.
    
    Package Dependencies:
      suricata-4.0.0    barnyard2-1.13_1  
    
    

    But Got this error during restarting the interface I setup for redis logging:

    
    22/8/2017 -- 21:09:29 - <error>-- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - redis JSON output option is not compiled</error> 
    

    Digging a little further I noticed hiredis not enabled in the compiled binary on the pfsense system. Is that required to just do 'flows' or is it needed for ANY of the reporting types?

    
    [2.3.4-RELEASE][root@firewall.lan]/root: suricata --build-info
    This is Suricata version 3.2.1 RELEASE
    Features: IPFW PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 NETMAP HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS MAGIC
    SIMD support: none
    Atomic intrisics: 1 2 4 8 byte(s)
    64-bits, Little-endian architecture
    GCC version 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032), C version 199901
    compiled with -fstack-protector
    compiled with _FORTIFY_SOURCE=2
    L1 cache line size (CLS)=64
    thread local storage method: __thread
    compiled with LibHTP v0.5.23, linked against LibHTP v0.5.23
    
    Suricata Configuration:
      AF_PACKET support:                       no
      PF_RING support:                         no
      NFQueue support:                         no
      NFLOG support:                           no
      IPFW support:                            yes
      Netmap support:                          yes
      DAG enabled:                             no
      Napatech enabled:                        no
    
      Unix socket enabled:                     yes
      Detection enabled:                       yes
    
      Libmagic support:                        yes
      libnss support:                          yes
      libnspr support:                         yes
      libjansson support:                      yes
      hiredis support:                         no
      Prelude support:                         no
      PCRE jit:                                yes
      LUA support:                             yes, through luajit
      libluajit:                               yes
      libgeoip:                                yes
      Non-bundled htp:                         yes
      Old barnyard2 support:                   no
      CUDA enabled:                            no
      Hyperscan support:                       yes
      Libnet support:                          yes
    
      Suricatasc install:                      no
    
      Profiling enabled:                       no
      Profiling locks enabled:                 no
    
    Development settings:
      Coccinelle / spatch:                     no
      Unit tests enabled:                      no
      Debug output enabled:                    no
      Debug validation enabled:                no
    
    Generic build parameters:
      Installation prefix:                     /usr/local
      Configuration directory:                 /usr/local/etc/suricata/
      Log directory:                           /var/log/suricata/
    
      --prefix                                 /usr/local
      --sysconfdir                             /usr/local/etc
      --localstatedir                          /var
    
      Host:                                    amd64-portbld-freebsd10.3
      Compiler:                                cc (exec name) / clang (real)
      GCC Protect enabled:                     yes
      GCC march native enabled:                no
      GCC Profile enabled:                     no
      Position Independent Executable enabled: no
      CFLAGS                                   -O2 -pipe  -fstack-protector -fno-strict-aliasing -DOS_FREEBSD
      PCAP_CFLAGS                               -I/usr/local/include
      SECCFLAGS                                -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security
    
    

    So is this```
    hiredis support

    
    I used the GUI, pointed the system at a redis server (did a telnet port confirmation not firewalled to remote redis IP above, sniffed the wire for any attempts for data to be sent and checkboxed all the options in the pfsense GUI minus the 'Flows' option etc.  Here is the relevant config pulled from pfense itself the GUI generated - which is part of the larger```
    output:
    ```YAML section.
    
    Screenshot of config
    
    ![](http://firewall_lan_-_Services__Suricata__Edit_Interface_Settings_-_WAN.png)
    
    Config generated (in part on pfsense box)
    
    
    • eve-log:
            enabled: yes
            type: redis
            redis:
              server: 192.168.10.178
              port: 6379
              mode: list
              key: "suricata"
            filename: eve.json
            identity: "suricata"
            facility: local1
            level: info
            types:
              - alert:
                  payload: yes          # enable dumping payload in Base64
                  payload-printable: yes # enable dumping payload in printable (lossy) format
                  packet: yes            # enable dumping of packet (without stream segments)
                  http: yes              # enable dumping of http fields
                  tls: yes              # enable dumping of tls fields
                  ssh: yes              # enable dumping of ssh fields
                  smtp: yes              # enable dumping of smtp fields
                  dnp3: yes              # enable dumping of DNP3 fields
                  tagged-packets: yes    # enable logging of tagged packets
              - http:
                  extended: yes
                  custom: [accept, accept-charset, accept-encoding, accept-language,
                          accept-datetime, authorization, cache-control, cookie, from,
                          max-forwards, origin, pragma, proxy-authorization, range, te, via,
                          x-requested-with, dnt, x-forwarded-proto, accept-range, age,
                          allow, connection, content-encoding, content-language,
                          content-length, content-location, content-md5, content-range,
                          content-type, date, etags, last-modified, link, location,
                          proxy-authenticate, referrer, refresh, retry-after, server,
                          set-cookie, trailer, transfer-encoding, upgrade, vary, warning,
                          www-authenticate, x-flash-version, x-authenticated-user]
            - dns:
                  query: yes
                  answer: yes
              - tls:
                  extended: no
              - files:
                  force-magic: no
              - ssh
              - smtp:
                  extended: yes
                  custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
                  md5: [subject]
    
    Suricata runs, but I see this in the logs…
    
    

    22/8/2017 -- 22:15:51 - <error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - redis JSON output option is not compiled</error>

    
    ![firewall_lan_-_Services__Suricata__Edit_Interface_Settings_-_WAN.png](/public/_imported_attachments_/1/firewall_lan_-_Services__Suricata__Edit_Interface_Settings_-_WAN.png)
    ![firewall_lan_-_Services__Suricata__Edit_Interface_Settings_-_WAN.png_thumb](/public/_imported_attachments_/1/firewall_lan_-_Services__Suricata__Edit_Interface_Settings_-_WAN.png_thumb)


  • Correction. Suricata eventually stops once it hits the JSON error above, thats the last entry in the logs. My bad.  Please let me know if anyone knows whats blocking me? Need to get hiredis compiled in? Or is something else blocking me config wise?



  • @wirehead:

    Correction. Suricata eventually stops once it hits the JSON error above, thats the last entry in the logs. My bad.  Please let me know if anyone knows whats blocking me? Need to get hiredis compiled in? Or is something else blocking me config wise?

    Your posted screen output indicates you are using Suricata 3.2.1.  Version 4.0.0 is now posted.  I just verified the compiler options are set to enable REDIS support in version 4.0.0.  Upgrade your Suricata installation and try again.

    Bill



  • Indeed. That is most odd. If I ssh in and open a shell, checking version shows 3.2.1 indeed (sorry missed that)

    
    [2.3.4-RELEASE][root@firewall.lan]/root: suricata -V
    This is Suricata version 3.2.1 RELEASE
    
    

    Looks like somehow I have the old binary and new GUI elements… though as the package manager shows 4.0.0 installed. I can't find a situation of mixed binaries either..

    
    [2.3.4-RELEASE][root@firewall.lan]/root: find / -name "suricata" -type f
    /usr/local/bin/suricata
    /usr/local/etc/rc.d/suricata
    
    

    And when I run that binary…

    
    [2.3.4-RELEASE][root@firewall.lan]/root: /usr/local/bin/suricata -V
    This is Suricata version 3.2.1 RELEASE
    
    

    Very very odd.

    I'll try and re-install it and see what happens. I did the upgrade a week or so ago maybe something didn't go right…



  • So I had to fully uninstall the package, upgrade or in place re-install did not push out the old binary. Very odd.

    
    [2.3.4-RELEASE][root@firewall.lan]/root: /usr/local/bin/suricata -V
    This is Suricata version 4.0.0 RELEASE
    
    

    Confirmed support etc thx

    
    [2.3.4-RELEASE][root@firewall.lan]/usr/local/etc/suricata/suricata_53398_pppoe0: suricata --build-info |grep redis
      hiredis support:                         yes
      hiredis async with libevent:             no
    
    


  • @wirehead:

    So I had to fully uninstall the package, upgrade or in place re-install did not push out the old binary. Very odd.

    
    [2.3.4-RELEASE][root@firewall.lan]/root: /usr/local/bin/suricata -V
    This is Suricata version 4.0.0 RELEASE
    
    

    Confirmed support etc thx

    
    [2.3.4-RELEASE][root@firewall.lan]/usr/local/etc/suricata/suricata_53398_pppoe0: suricata --build-info |grep redis
      hiredis support:                         yes
      hiredis async with libevent:             no
    
    

    Yeah, that is weird.  I would expect the update to recognize the binary change.  This is something I will need to discuss with the pfSense guys.  Maybe something else is needed in the Package Manager configuration for Suricata.

    Bill



  • So I got this to work! But I had to remove syslog config the GUI configuration yaml generated

    I removed these entries… or it would not send the REDIS data...

    
          filename: eve.json
          identity: "suricata"
          facility: local1
          level: info
    
    

    Here is the working configuration for my testing…

    
      - eve-log:
          enabled: yes
          type: redis
          redis:
            server: 192.168.10.178
            port: 6379
            mode: list
            key: "suricata"
          types:
            - alert:
                payload: yes           # enable dumping payload in Base64
                payload-printable: yes # enable dumping payload in printable (lossy) format
                packet: yes            # enable dumping of packet (without stream segments)
                http: yes              # enable dumping of http fields
                tls: yes               # enable dumping of tls fields
                ssh: yes               # enable dumping of ssh fields
                smtp: yes              # enable dumping of smtp fields
                dnp3: yes              # enable dumping of DNP3 fields
                tagged-packets: yes    # enable logging of tagged packets
            - http:
                extended: yes
                custom: [accept, accept-charset, accept-encoding, accept-language,
                        accept-datetime, authorization, cache-control, cookie, from,
                        max-forwards, origin, pragma, proxy-authorization, range, te, via,
                        x-requested-with, dnt, x-forwarded-proto, accept-range, age,
                        allow, connection, content-encoding, content-language,
                        content-length, content-location, content-md5, content-range,
                        content-type, date, etags, last-modified, link, location,
                        proxy-authenticate, referrer, refresh, retry-after, server,
                        set-cookie, trailer, transfer-encoding, upgrade, vary, warning,
                        www-authenticate, x-flash-version, x-authenticated-user]
            - dns:
                query: yes
                answer: yes
            - tls:
                extended: no
            - files:
                force-magic: no
            - ssh
            - smtp:
                extended: yes
                custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
                md5: [subject]
    
    

    This was what the GUI generated that didn't work..

    
      - eve-log:
          enabled: yes
          type: redis
          redis:
            server: 192.168.10.178
            port: 6379
            mode: list
            key: "suricata"
          filename: eve.json
          identity: "suricata"
          facility: local1
          level: info
          types:
            - alert:
                payload: yes           # enable dumping payload in Base64
                payload-printable: yes # enable dumping payload in printable (lossy) format
                packet: yes            # enable dumping of packet (without stream segments)
                http: yes              # enable dumping of http fields
                tls: yes               # enable dumping of tls fields
                ssh: yes               # enable dumping of ssh fields
                smtp: yes              # enable dumping of smtp fields
                dnp3: yes              # enable dumping of DNP3 fields
                tagged-packets: yes    # enable logging of tagged packets
            - http:
                extended: yes
                custom: [accept, accept-charset, accept-encoding, accept-language,
                        accept-datetime, authorization, cache-control, cookie, from,
                        max-forwards, origin, pragma, proxy-authorization, range, te, via,
                        x-requested-with, dnt, x-forwarded-proto, accept-range, age,
                        allow, connection, content-encoding, content-language,
                        content-length, content-location, content-md5, content-range,
                        content-type, date, etags, last-modified, link, location,
                        proxy-authenticate, referrer, refresh, retry-after, server,
                        set-cookie, trailer, transfer-encoding, upgrade, vary, warning,
                        www-authenticate, x-flash-version, x-authenticated-user]
            - dns:
                query: yes
                answer: yes
            - tls:
                extended: no
            - files:
                force-magic: no
            - ssh
            - smtp:
                extended: yes
                custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
                md5: [subject]
    
    

    Here is a screencap from the GUI settings I used to start with…
    (see attached)

    ![2017-08-23 12_05_43-firewall.lan - Services_ Suricata_ Edit Interface Settings - WAN.png](/public/imported_attachments/1/2017-08-23 12_05_43-firewall.lan - Services_ Suricata_ Edit Interface Settings - WAN.png)
    ![2017-08-23 12_05_43-firewall.lan - Services_ Suricata_ Edit Interface Settings - WAN.png_thumb](/public/imported_attachments/1/2017-08-23 12_05_43-firewall.lan - Services_ Suricata_ Edit Interface Settings - WAN.png_thumb)



  • Yeah, this part –

    
          filename: eve.json
          identity: "suricata"
          facility: local1
          level: info
    
    

    does not belong there.  Probably coming from an uncleared string in the GUI code (just a guess without looking).  I did not write that code enhancement.  Another user contributed the code.  I will need to take a look and see what the problem is.

    Bill



  • Hi,

    I am able to implement Suricata-Redis architecture. Please let me know whether we can use Redis-sentinel feature at Suricata Config block. Because my application will be required redis failover support so if I can also configure Suricata with Redis Sentinel then it would be the best for me.

    Shubham