Kibana+Elasticsearch+Logstash [ELK] v6.3.0 pfSense v2.4.3p1 and Suricata using docker-compose | docker for windows

evaluationcopy

I wanted to share my approach for setting up the ELK stack. I set this up using docker for windows. Docker for windows works by creating a linux VM in Hyper-V for you to run the containers and overall was pretty easy to install and use. In theory this should work on linux systems as well.

https://github.com/evaluationcopy/pfsense-suricata-elk-docker

This setup uses docker-compose to pull the docker images from elastic. The docker images do not require any modifications, and the elastic version to use is specified on the command line. I believe this will allow for easy future upgrades.

The logstash configs and elastic data directories are mapped to local folders outside of the containers, on the host computer.

I've combined logstash filters for pfSense and Suricata so they can both be parsed and viewed in kibana. The filters have been updated to reduce the pfSense logging to just firewall activity (no dhcp, dns requests, etc).

Finally, this builds upon a number of existing works out there. Instructions and original sources are located in the readme on github.

rickygm

do you have plans to remove for linux vm, and with snort?

evaluationcopy

At this point I moved it over to a permanent linux VM. This was better for running long-term. The docker-compose work above is still valid.

As for Snort, I'm now using Snort instead of Suricata. Unfortunately, this ELK setup doesn't parse Snort logs. I believe Snort 3.0 can output json logs which would make integrating Snort much easier. Once Snort 3.0 is released and available in pfSense I'll revisit adding Snort into the stack.

rickygm

@evaluationcopy said in Kibana+Elasticsearch+Logstash [ELK] v6.3.0 pfSense v2.4.3p1 and Suricata using docker-compose | docker for windows:

At this point I moved it over to a permanent linux VM. This was better for running long-term. The docker-compose work above is still valid.

As for Snort, I'm now using Snort instead of Suricata. Unfortunately, this ELK setup doesn't parse Snort logs. I believe Snort 3.0 can output json logs which would make integrating Snort much easier. Once Snort 3.0 is released and available in pfSense I'll revisit adding Snort into the stack.

Ok , let me know, I am interested in that, I write PM.

kozokeith

@evaluationcopy
Hi -
I have been trying for probably 10 or 12 hours to research and parse the pfsense sylog with snort data. I cannot get it to parse. Based on your post, it sounds like you have already concluded that snort syslog data, for example this

snort[12345]

is not parsable in logstash? If you know of a way, id really like to know!
I posted the following on the elastic forum yesterday - no response yet.
Thanks!

I'm attempting to extract pfsense logs, specifically the snort plugin alert logs.

I've pulled the following filter from this site:
https://gist.githubusercontent.com/mrlesmithjr/0799cdc3710f2ae1e182/raw/cfad5b0740928060b9322fcc5a4406073cc446ce/pfsense_snort_logstash

My filter is this:
if [prog] == /snort$ / {
mutate {
add_tag => [ "snort" ]
}
}

Logstash shows this as the [prog] field - so it is parsing this out (which is snort[pid]:
snort[59585]

Question is - how can I tag this text snort[#####]
so the rest of the filter works?

I'm guessing my filter match is incorrect?

Thanks!

Full unfiltered sample log below.

kozokeith

@evaluationcopy
Hi - I have been trying for probably 10 or 12 hours to research and parse the pfsense sylog with snort data. I cannot get it to parse. Based on your sense, it sounds like you have already concluded that snort in particular this - snort[12345] is not parsable in logstash? If you know of a way, id really like to know!
Thanks