4. Suricata - Netflow and Hiredis Support

Suricata - Netflow and Hiredis Support

  • F

    Hi,

    I have a few questions about the Suricata package in pfSense:

    Is Netflow-Export included as in the standard version? - cant find settings for this in the gui and config file [ /usr/local/pkg/suricata/suricata_generate_yaml.php]

    I would like to export eve-log via hiredis to a redis-server  - before building a suricata package with hiredis support - I would like to know if i have to do additional customizing because I cant find anything regarding eve-log in the config file [ /usr/local/pkg/suricata/suricata_generate_yaml.php]. Or is there another config file?

    eve-log -> http://suricata.readthedocs.io/en/latest/output/eve/eve-json-output.html

    0
  • S

    Hi,

    the latest version of the Suricata package does come with hiredis support. There is a GUI option now to configure it as well. Currently the GUI is quite limited on configuring what should be included in the eve-output. I'm already working to on a patch to improve that and will file a pull request soon.

    Currently you might want to configure the eve-output by hand using the pass through config option.

    0
  • W

    I just tried this tonight with the latest 4.0.0 package FYI - I get a JSON not supported / compiled error.  Ideally I don't want to put anythign on pfsense, having it REDIS to a remote machine I have redis and logstash own to grok and inject into my elastic cluster is superb. Keeps the pfsense box foreign object free ;)

    
suricata		security 	4.0.0		High Performance Network IDS, IPS and Security Monitoring engine by OISF.

Package Dependencies:
  suricata-4.0.0    barnyard2-1.13_1 

    But Got this error during restarting the interface I setup for redis logging:

    
22/8/2017 -- 21:09:29 - <error>-- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - redis JSON output option is not compiled</error>

    Digging a little further I noticed hiredis not enabled in the compiled binary on the pfsense system. Is that required to just do 'flows' or is it needed for ANY of the reporting types?

    
[2.3.4-RELEASE][root@firewall.lan]/root: suricata --build-info
This is Suricata version 3.2.1 RELEASE
Features: IPFW PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 NETMAP HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS MAGIC
SIMD support: none
Atomic intrisics: 1 2 4 8 byte(s)
64-bits, Little-endian architecture
GCC version 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032), C version 199901
compiled with -fstack-protector
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.23, linked against LibHTP v0.5.23

Suricata Configuration:
  AF_PACKET support:                       no
  PF_RING support:                         no
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            yes
  Netmap support:                          yes
  DAG enabled:                             no
  Napatech enabled:                        no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  Libmagic support:                        yes
  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  hiredis support:                         no
  Prelude support:                         no
  PCRE jit:                                yes
  LUA support:                             yes, through luajit
  libluajit:                               yes
  libgeoip:                                yes
  Non-bundled htp:                         yes
  Old barnyard2 support:                   no
  CUDA enabled:                            no
  Hyperscan support:                       yes
  Libnet support:                          yes

  Suricatasc install:                      no

  Profiling enabled:                       no
  Profiling locks enabled:                 no

Development settings:
  Coccinelle / spatch:                     no
  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no

Generic build parameters:
  Installation prefix:                     /usr/local
  Configuration directory:                 /usr/local/etc/suricata/
  Log directory:                           /var/log/suricata/

  --prefix                                 /usr/local
  --sysconfdir                             /usr/local/etc
  --localstatedir                          /var

  Host:                                    amd64-portbld-freebsd10.3
  Compiler:                                cc (exec name) / clang (real)
  GCC Protect enabled:                     yes
  GCC march native enabled:                no
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -O2 -pipe  -fstack-protector -fno-strict-aliasing -DOS_FREEBSD
  PCAP_CFLAGS                               -I/usr/local/include
  SECCFLAGS                                -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security

    So is this```
    hiredis support

    
I used the GUI, pointed the system at a redis server (did a telnet port confirmation not firewalled to remote redis IP above, sniffed the wire for any attempts for data to be sent and checkboxed all the options in the pfsense GUI minus the 'Flows' option etc.  Here is the relevant config pulled from pfense itself the GUI generated - which is part of the larger```
output:
```YAML section.

Screenshot of config

![](http://firewall_lan_-_Services__Suricata__Edit_Interface_Settings_-_WAN.png)

Config generated (in part on pfsense box)
    • eve-log:
            enabled: yes
            type: redis
            redis:
              server: 192.168.10.178
              port: 6379
              mode: list
              key: "suricata"
            filename: eve.json
            identity: "suricata"
            facility: local1
            level: info
            types:
              - alert:
                  payload: yes          # enable dumping payload in Base64
                  payload-printable: yes # enable dumping payload in printable (lossy) format
                  packet: yes            # enable dumping of packet (without stream segments)
                  http: yes              # enable dumping of http fields
                  tls: yes              # enable dumping of tls fields
                  ssh: yes              # enable dumping of ssh fields
                  smtp: yes              # enable dumping of smtp fields
                  dnp3: yes              # enable dumping of DNP3 fields
                  tagged-packets: yes    # enable logging of tagged packets
              - http:
                  extended: yes
                  custom: [accept, accept-charset, accept-encoding, accept-language,
                          accept-datetime, authorization, cache-control, cookie, from,
                          max-forwards, origin, pragma, proxy-authorization, range, te, via,
                          x-requested-with, dnt, x-forwarded-proto, accept-range, age,
                          allow, connection, content-encoding, content-language,
                          content-length, content-location, content-md5, content-range,
                          content-type, date, etags, last-modified, link, location,
                          proxy-authenticate, referrer, refresh, retry-after, server,
                          set-cookie, trailer, transfer-encoding, upgrade, vary, warning,
                          www-authenticate, x-flash-version, x-authenticated-user]
            - dns:
                  query: yes
                  answer: yes
              - tls:
                  extended: no
              - files:
                  force-magic: no
              - ssh
              - smtp:
                  extended: yes
                  custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
                  md5: [subject]
    
Suricata runs, but I see this in the logs…

    22/8/2017 -- 22:15:51 - <error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - redis JSON output option is not compiled</error>

    
![firewall_lan_-_Services__Suricata__Edit_Interface_Settings_-_WAN.png](/public/_imported_attachments_/1/firewall_lan_-_Services__Suricata__Edit_Interface_Settings_-_WAN.png)
![firewall_lan_-_Services__Suricata__Edit_Interface_Settings_-_WAN.png_thumb](/public/_imported_attachments_/1/firewall_lan_-_Services__Suricata__Edit_Interface_Settings_-_WAN.png_thumb)
    0
  • W

    Correction. Suricata eventually stops once it hits the JSON error above, thats the last entry in the logs. My bad.  Please let me know if anyone knows whats blocking me? Need to get hiredis compiled in? Or is something else blocking me config wise?

    0

  • @wirehead:

    Correction. Suricata eventually stops once it hits the JSON error above, thats the last entry in the logs. My bad.  Please let me know if anyone knows whats blocking me? Need to get hiredis compiled in? Or is something else blocking me config wise?

    Your posted screen output indicates you are using Suricata 3.2.1.  Version 4.0.0 is now posted.  I just verified the compiler options are set to enable REDIS support in version 4.0.0.  Upgrade your Suricata installation and try again.

    Bill

    0
  • W

    Indeed. That is most odd. If I ssh in and open a shell, checking version shows 3.2.1 indeed (sorry missed that)

    
[2.3.4-RELEASE][root@firewall.lan]/root: suricata -V
This is Suricata version 3.2.1 RELEASE

    Looks like somehow I have the old binary and new GUI elements… though as the package manager shows 4.0.0 installed. I can't find a situation of mixed binaries either..

    
[2.3.4-RELEASE][root@firewall.lan]/root: find / -name "suricata" -type f
/usr/local/bin/suricata
/usr/local/etc/rc.d/suricata

    And when I run that binary…

    
[2.3.4-RELEASE][root@firewall.lan]/root: /usr/local/bin/suricata -V
This is Suricata version 3.2.1 RELEASE

    Very very odd.

    I'll try and re-install it and see what happens. I did the upgrade a week or so ago maybe something didn't go right…

    0
  • W

    So I had to fully uninstall the package, upgrade or in place re-install did not push out the old binary. Very odd.

    
[2.3.4-RELEASE][root@firewall.lan]/root: /usr/local/bin/suricata -V
This is Suricata version 4.0.0 RELEASE

    Confirmed support etc thx

    
[2.3.4-RELEASE][root@firewall.lan]/usr/local/etc/suricata/suricata_53398_pppoe0: suricata --build-info |grep redis
  hiredis support:                         yes
  hiredis async with libevent:             no
    0

  • @wirehead:

    So I had to fully uninstall the package, upgrade or in place re-install did not push out the old binary. Very odd.

    
[2.3.4-RELEASE][root@firewall.lan]/root: /usr/local/bin/suricata -V
This is Suricata version 4.0.0 RELEASE

    Confirmed support etc thx

    
[2.3.4-RELEASE][root@firewall.lan]/usr/local/etc/suricata/suricata_53398_pppoe0: suricata --build-info |grep redis
  hiredis support:                         yes
  hiredis async with libevent:             no

    Yeah, that is weird.  I would expect the update to recognize the binary change.  This is something I will need to discuss with the pfSense guys.  Maybe something else is needed in the Package Manager configuration for Suricata.

    Bill

    0
  • W

    So I got this to work! But I had to remove syslog config the GUI configuration yaml generated

    I removed these entries… or it would not send the REDIS data...

    
      filename: eve.json
      identity: "suricata"
      facility: local1
      level: info

    Here is the working configuration for my testing…

    
  - eve-log:
      enabled: yes
      type: redis
      redis:
        server: 192.168.10.178
        port: 6379
        mode: list
        key: "suricata"
      types:
        - alert:
            payload: yes           # enable dumping payload in Base64
            payload-printable: yes # enable dumping payload in printable (lossy) format
            packet: yes            # enable dumping of packet (without stream segments)
            http: yes              # enable dumping of http fields
            tls: yes               # enable dumping of tls fields
            ssh: yes               # enable dumping of ssh fields
            smtp: yes              # enable dumping of smtp fields
            dnp3: yes              # enable dumping of DNP3 fields
            tagged-packets: yes    # enable logging of tagged packets
        - http:
            extended: yes
            custom: [accept, accept-charset, accept-encoding, accept-language,
                    accept-datetime, authorization, cache-control, cookie, from,
                    max-forwards, origin, pragma, proxy-authorization, range, te, via,
                    x-requested-with, dnt, x-forwarded-proto, accept-range, age,
                    allow, connection, content-encoding, content-language,
                    content-length, content-location, content-md5, content-range,
                    content-type, date, etags, last-modified, link, location,
                    proxy-authenticate, referrer, refresh, retry-after, server,
                    set-cookie, trailer, transfer-encoding, upgrade, vary, warning,
                    www-authenticate, x-flash-version, x-authenticated-user]
        - dns:
            query: yes
            answer: yes
        - tls:
            extended: no
        - files:
            force-magic: no
        - ssh
        - smtp:
            extended: yes
            custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
            md5: [subject]

    This was what the GUI generated that didn't work..

    
  - eve-log:
      enabled: yes
      type: redis
      redis:
        server: 192.168.10.178
        port: 6379
        mode: list
        key: "suricata"
      filename: eve.json
      identity: "suricata"
      facility: local1
      level: info
      types:
        - alert:
            payload: yes           # enable dumping payload in Base64
            payload-printable: yes # enable dumping payload in printable (lossy) format
            packet: yes            # enable dumping of packet (without stream segments)
            http: yes              # enable dumping of http fields
            tls: yes               # enable dumping of tls fields
            ssh: yes               # enable dumping of ssh fields
            smtp: yes              # enable dumping of smtp fields
            dnp3: yes              # enable dumping of DNP3 fields
            tagged-packets: yes    # enable logging of tagged packets
        - http:
            extended: yes
            custom: [accept, accept-charset, accept-encoding, accept-language,
                    accept-datetime, authorization, cache-control, cookie, from,
                    max-forwards, origin, pragma, proxy-authorization, range, te, via,
                    x-requested-with, dnt, x-forwarded-proto, accept-range, age,
                    allow, connection, content-encoding, content-language,
                    content-length, content-location, content-md5, content-range,
                    content-type, date, etags, last-modified, link, location,
                    proxy-authenticate, referrer, refresh, retry-after, server,
                    set-cookie, trailer, transfer-encoding, upgrade, vary, warning,
                    www-authenticate, x-flash-version, x-authenticated-user]
        - dns:
            query: yes
            answer: yes
        - tls:
            extended: no
        - files:
            force-magic: no
        - ssh
        - smtp:
            extended: yes
            custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
            md5: [subject]

    Here is a screencap from the GUI settings I used to start with…
    (see attached)

    ![2017-08-23 12_05_43-firewall.lan - Services_ Suricata_ Edit Interface Settings - WAN.png](/public/imported_attachments/1/2017-08-23 12_05_43-firewall.lan - Services_ Suricata_ Edit Interface Settings - WAN.png)
    ![2017-08-23 12_05_43-firewall.lan - Services_ Suricata_ Edit Interface Settings - WAN.png_thumb](/public/imported_attachments/1/2017-08-23 12_05_43-firewall.lan - Services_ Suricata_ Edit Interface Settings - WAN.png_thumb)

    0

  • Yeah, this part –

    
      filename: eve.json
      identity: "suricata"
      facility: local1
      level: info

    does not belong there.  Probably coming from an uncleared string in the GUI code (just a guess without looking).  I did not write that code enhancement.  Another user contributed the code.  I will need to take a look and see what the problem is.

    Bill

    0
  • S

    Hi,

    I am able to implement Suricata-Redis architecture. Please let me know whether we can use Redis-sentinel feature at Suricata Config block. Because my application will be required redis failover support so if I can also configure Suricata with Redis Sentinel then it would be the best for me.

    Shubham

    0
Log in to reply
 

Products

Services

Support

News

Resources

Company

Our Mission

We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.

Subscribe to our Newsletter

Product information, software announcements, and special offers. See our newsletter archive for past announcements.

© 2020 Rubicon Communications, LLC | Privacy Policy