Suricata - Netflow and Hiredis Support
-
Hi,
I have a few questions about the Suricata package in pfSense:
Is Netflow-Export included as in the standard version? - cant find settings for this in the gui and config file [ /usr/local/pkg/suricata/suricata_generate_yaml.php]
I would like to export eve-log via hiredis to a redis-server - before building a suricata package with hiredis support - I would like to know if i have to do additional customizing because I cant find anything regarding eve-log in the config file [ /usr/local/pkg/suricata/suricata_generate_yaml.php]. Or is there another config file?
eve-log -> http://suricata.readthedocs.io/en/latest/output/eve/eve-json-output.html
-
Hi,
the latest version of the Suricata package does come with hiredis support. There is a GUI option now to configure it as well. Currently the GUI is quite limited on configuring what should be included in the eve-output. I'm already working to on a patch to improve that and will file a pull request soon.
Currently you might want to configure the eve-output by hand using the pass through config option.
-
I just tried this tonight with the latest 4.0.0 package FYI - I get a JSON not supported / compiled error. Ideally I don't want to put anythign on pfsense, having it REDIS to a remote machine I have redis and logstash own to grok and inject into my elastic cluster is superb. Keeps the pfsense box foreign object free ;)
suricata security 4.0.0 High Performance Network IDS, IPS and Security Monitoring engine by OISF. Package Dependencies: suricata-4.0.0 barnyard2-1.13_1But Got this error during restarting the interface I setup for redis logging:
22/8/2017 -- 21:09:29 - <error>-- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - redis JSON output option is not compiled</error>Digging a little further I noticed hiredis not enabled in the compiled binary on the pfsense system. Is that required to just do 'flows' or is it needed for ANY of the reporting types?
[2.3.4-RELEASE][root@firewall.lan]/root: suricata --build-info This is Suricata version 3.2.1 RELEASE Features: IPFW PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 NETMAP HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS MAGIC SIMD support: none Atomic intrisics: 1 2 4 8 byte(s) 64-bits, Little-endian architecture GCC version 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032), C version 199901 compiled with -fstack-protector compiled with _FORTIFY_SOURCE=2 L1 cache line size (CLS)=64 thread local storage method: __thread compiled with LibHTP v0.5.23, linked against LibHTP v0.5.23 Suricata Configuration: AF_PACKET support: no PF_RING support: no NFQueue support: no NFLOG support: no IPFW support: yes Netmap support: yes DAG enabled: no Napatech enabled: no Unix socket enabled: yes Detection enabled: yes Libmagic support: yes libnss support: yes libnspr support: yes libjansson support: yes hiredis support: no Prelude support: no PCRE jit: yes LUA support: yes, through luajit libluajit: yes libgeoip: yes Non-bundled htp: yes Old barnyard2 support: no CUDA enabled: no Hyperscan support: yes Libnet support: yes Suricatasc install: no Profiling enabled: no Profiling locks enabled: no Development settings: Coccinelle / spatch: no Unit tests enabled: no Debug output enabled: no Debug validation enabled: no Generic build parameters: Installation prefix: /usr/local Configuration directory: /usr/local/etc/suricata/ Log directory: /var/log/suricata/ --prefix /usr/local --sysconfdir /usr/local/etc --localstatedir /var Host: amd64-portbld-freebsd10.3 Compiler: cc (exec name) / clang (real) GCC Protect enabled: yes GCC march native enabled: no GCC Profile enabled: no Position Independent Executable enabled: no CFLAGS -O2 -pipe -fstack-protector -fno-strict-aliasing -DOS_FREEBSD PCAP_CFLAGS -I/usr/local/include SECCFLAGS -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-securitySo is this```
hiredis supportI used the GUI, pointed the system at a redis server (did a telnet port confirmation not firewalled to remote redis IP above, sniffed the wire for any attempts for data to be sent and checkboxed all the options in the pfsense GUI minus the 'Flows' option etc. Here is the relevant config pulled from pfense itself the GUI generated - which is part of the larger``` output: ```YAML section. Screenshot of config  Config generated (in part on pfsense box)- eve-log:
enabled: yes
type: redis
redis:
server: 192.168.10.178
port: 6379
mode: list
key: "suricata"
filename: eve.json
identity: "suricata"
facility: local1
level: info
types:
- alert:
payload: yes # enable dumping payload in Base64
payload-printable: yes # enable dumping payload in printable (lossy) format
packet: yes # enable dumping of packet (without stream segments)
http: yes # enable dumping of http fields
tls: yes # enable dumping of tls fields
ssh: yes # enable dumping of ssh fields
smtp: yes # enable dumping of smtp fields
dnp3: yes # enable dumping of DNP3 fields
tagged-packets: yes # enable logging of tagged packets
- http:
extended: yes
custom: [accept, accept-charset, accept-encoding, accept-language,
accept-datetime, authorization, cache-control, cookie, from,
max-forwards, origin, pragma, proxy-authorization, range, te, via,
x-requested-with, dnt, x-forwarded-proto, accept-range, age,
allow, connection, content-encoding, content-language,
content-length, content-location, content-md5, content-range,
content-type, date, etags, last-modified, link, location,
proxy-authenticate, referrer, refresh, retry-after, server,
set-cookie, trailer, transfer-encoding, upgrade, vary, warning,
www-authenticate, x-flash-version, x-authenticated-user]
- dns:
query: yes
answer: yes
- tls:
extended: no
- files:
force-magic: no
- ssh
- smtp:
extended: yes
custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
md5: [subject]
Suricata runs, but I see this in the logs…22/8/2017 -- 22:15:51 - <error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - redis JSON output option is not compiled</error>
  - eve-log:
-
Correction. Suricata eventually stops once it hits the JSON error above, thats the last entry in the logs. My bad. Please let me know if anyone knows whats blocking me? Need to get hiredis compiled in? Or is something else blocking me config wise?
-
Correction. Suricata eventually stops once it hits the JSON error above, thats the last entry in the logs. My bad. Please let me know if anyone knows whats blocking me? Need to get hiredis compiled in? Or is something else blocking me config wise?
Your posted screen output indicates you are using Suricata 3.2.1. Version 4.0.0 is now posted. I just verified the compiler options are set to enable REDIS support in version 4.0.0. Upgrade your Suricata installation and try again.
Bill
-
Indeed. That is most odd. If I ssh in and open a shell, checking version shows 3.2.1 indeed (sorry missed that)
[2.3.4-RELEASE][root@firewall.lan]/root: suricata -V This is Suricata version 3.2.1 RELEASELooks like somehow I have the old binary and new GUI elements… though as the package manager shows 4.0.0 installed. I can't find a situation of mixed binaries either..
[2.3.4-RELEASE][root@firewall.lan]/root: find / -name "suricata" -type f /usr/local/bin/suricata /usr/local/etc/rc.d/suricataAnd when I run that binary…
[2.3.4-RELEASE][root@firewall.lan]/root: /usr/local/bin/suricata -V This is Suricata version 3.2.1 RELEASEVery very odd.
I'll try and re-install it and see what happens. I did the upgrade a week or so ago maybe something didn't go right…
-
So I had to fully uninstall the package, upgrade or in place re-install did not push out the old binary. Very odd.
[2.3.4-RELEASE][root@firewall.lan]/root: /usr/local/bin/suricata -V This is Suricata version 4.0.0 RELEASEConfirmed support etc thx
[2.3.4-RELEASE][root@firewall.lan]/usr/local/etc/suricata/suricata_53398_pppoe0: suricata --build-info |grep redis hiredis support: yes hiredis async with libevent: no -
So I had to fully uninstall the package, upgrade or in place re-install did not push out the old binary. Very odd.
[2.3.4-RELEASE][root@firewall.lan]/root: /usr/local/bin/suricata -V This is Suricata version 4.0.0 RELEASEConfirmed support etc thx
[2.3.4-RELEASE][root@firewall.lan]/usr/local/etc/suricata/suricata_53398_pppoe0: suricata --build-info |grep redis hiredis support: yes hiredis async with libevent: noYeah, that is weird. I would expect the update to recognize the binary change. This is something I will need to discuss with the pfSense guys. Maybe something else is needed in the Package Manager configuration for Suricata.
Bill
-
So I got this to work! But I had to remove syslog config the GUI configuration yaml generated
I removed these entries… or it would not send the REDIS data...
filename: eve.json identity: "suricata" facility: local1 level: infoHere is the working configuration for my testing…
- eve-log: enabled: yes type: redis redis: server: 192.168.10.178 port: 6379 mode: list key: "suricata" types: - alert: payload: yes # enable dumping payload in Base64 payload-printable: yes # enable dumping payload in printable (lossy) format packet: yes # enable dumping of packet (without stream segments) http: yes # enable dumping of http fields tls: yes # enable dumping of tls fields ssh: yes # enable dumping of ssh fields smtp: yes # enable dumping of smtp fields dnp3: yes # enable dumping of DNP3 fields tagged-packets: yes # enable logging of tagged packets - http: extended: yes custom: [accept, accept-charset, accept-encoding, accept-language, accept-datetime, authorization, cache-control, cookie, from, max-forwards, origin, pragma, proxy-authorization, range, te, via, x-requested-with, dnt, x-forwarded-proto, accept-range, age, allow, connection, content-encoding, content-language, content-length, content-location, content-md5, content-range, content-type, date, etags, last-modified, link, location, proxy-authenticate, referrer, refresh, retry-after, server, set-cookie, trailer, transfer-encoding, upgrade, vary, warning, www-authenticate, x-flash-version, x-authenticated-user] - dns: query: yes answer: yes - tls: extended: no - files: force-magic: no - ssh - smtp: extended: yes custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc] md5: [subject]This was what the GUI generated that didn't work..
- eve-log: enabled: yes type: redis redis: server: 192.168.10.178 port: 6379 mode: list key: "suricata" filename: eve.json identity: "suricata" facility: local1 level: info types: - alert: payload: yes # enable dumping payload in Base64 payload-printable: yes # enable dumping payload in printable (lossy) format packet: yes # enable dumping of packet (without stream segments) http: yes # enable dumping of http fields tls: yes # enable dumping of tls fields ssh: yes # enable dumping of ssh fields smtp: yes # enable dumping of smtp fields dnp3: yes # enable dumping of DNP3 fields tagged-packets: yes # enable logging of tagged packets - http: extended: yes custom: [accept, accept-charset, accept-encoding, accept-language, accept-datetime, authorization, cache-control, cookie, from, max-forwards, origin, pragma, proxy-authorization, range, te, via, x-requested-with, dnt, x-forwarded-proto, accept-range, age, allow, connection, content-encoding, content-language, content-length, content-location, content-md5, content-range, content-type, date, etags, last-modified, link, location, proxy-authenticate, referrer, refresh, retry-after, server, set-cookie, trailer, transfer-encoding, upgrade, vary, warning, www-authenticate, x-flash-version, x-authenticated-user] - dns: query: yes answer: yes - tls: extended: no - files: force-magic: no - ssh - smtp: extended: yes custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc] md5: [subject]Here is a screencap from the GUI settings I used to start with…
(see attached)
 -
Yeah, this part –
filename: eve.json identity: "suricata" facility: local1 level: infodoes not belong there. Probably coming from an uncleared string in the GUI code (just a guess without looking). I did not write that code enhancement. Another user contributed the code. I will need to take a look and see what the problem is.
Bill
-
Hi,
I am able to implement Suricata-Redis architecture. Please let me know whether we can use Redis-sentinel feature at Suricata Config block. Because my application will be required redis failover support so if I can also configure Suricata with Redis Sentinel then it would be the best for me.
Shubham
