PfSense + Docker + ELK stack

[jamesmr89]

just posted this to reddit, we made a docker container to get the ELK stack working with pfSense pf logs:

http://www.reddit.com/r/PFSENSE/comments/2rlm8h/pfsense_docker_elk/

With the syslog-ng package it now supports tcp logging from pfSense as well, enjoy.

NsNetworks

[PantsManUK]

Hate digging up old posts, but I can't get this working with 2.2.5.

The syslog-ng log viewer sees stuff arriving, and docker logs northshore-elk has corresponding entries, but given the tags I'm guessing that the logstash config isn't right somehow (certainly, I have a blank dashboard in Kibana)…

Any help appreciated, this ELK stuff is all brand new to me.

Syslog-ng log entries

Nov 19 13:47:02 localhost filterlog: 9,16777216,,1000000103,sk3,match,block,in,4,0x0,,247,48822,0,DF,17,udp,92,65.55.117.17,46.33.155.243,53,10533,72
Nov 19 13:47:02 localhost filterlog: 9,16777216,,1000000103,sk3,match,block,in,4,0x0,,248,15879,0,DF,17,udp,147,204.79.195.17,46.33.155.243,53,34834,127

Corresponding Docker log entries

{
       "message" => "%{msg}",
      "@version" => "1",
    "@timestamp" => "2015-11-19T13:47:02.000Z",
          "host" => "10.0.0.1:12677",
          "type" => "syslog",
          "prog" => "localhost filterlog",
          "tags" => [
        [0] "nsfilter",
        [1] "_grokparsefailure",
        [2] "PFSense"
    ]
}
{
       "message" => "%{msg}",
      "@version" => "1",
    "@timestamp" => "2015-11-19T13:47:02.000Z",
          "host" => "10.0.0.1:12677",
          "type" => "syslog",
          "prog" => "localhost filterlog",
          "tags" => [
        [0] "nsfilter",
        [1] "_grokparsefailure",
        [2] "PFSense"
    ]
}

[PantsManUK]

Feel like such a dope. Was looking into the config to get a head start, and figured that it must have been me setting the nsfilter IP to the firewall IP that was causing the nsfilter tag. Changed the nsfilter IP back to "192.168.1.1" and restarted the Docker image and it's processing normally now. Should really have paid attention to the README, I guess.

[PantsManUK]

Looks like I spoke too soon - I'm getting events, but the detail is being missed.


[PantsManUK]

And I've managed to get it working again using a blog post I found - https://hml.io/2015/03/28/elk-for-pfsense/

Seems that between the original release on Github and that blog post, the filterlog format got changed and that has thrown out the configs that are on Github. I'll attach my changed configs shortly.

logstash.conf.txt
default.json.txt

[koma]

I'll leave this here:

https://github.com/koma85/pfsense.grok

K.

[usual]

Does this work with 2.3 logs?

[usual]

I got it working with information from various articles. Pretty sweet.

[skyguide]

Hello Usual, could you please elaborate a bit more on what exactly do you mean by "i got this working"?

The current ELK stack plus 2.3.1-RELEASE-p5 (FreeBSD 10.3-RELEASE-p3) dose not seem to work well.

Logstash keeps closing the UDP stream from PFsense, and there may be an issue with appropriate formating of the logs that been feed from PFsense. The inconsistency in logs format may be the cause why Logstash is unable to parse logs and quits.

Could you please let us know if you been able to parse the logs from PFsense 2.3.1 -p5 using ELK stack?

Thank you.

Update @ Mon, June 20, 2016

I can confirm that PFSense v2.3 work very well with ELK stack.
The key is to get a grok filtering set up correctly.

There are many guides out in the inter-pipes. Researching the set-up that actually works I tried many of them but the only one that is straight forward, and actually worked for me is this on:

http://www.stealthshark.com/system-logging-with-elk-stack/

I hope this might help to some one else.

[ando1]

I realize this thread is a bit old but I am having a real hard time trying to understand how to configure ELK for pfsense 2.3 -p3. I have tried to follow may guides and the ELK server running. When I log into Kibana at http://<ip address="">:5601 to configure an index pattern I do not have the "create button". See the screen shot below. I think this is doe to my logstash config file and possibly the pattern file. If someone could tell me what I am doing wrong or has an guide for configuring ELK with pfsense 2.3, I would really appreciate it.

</ip>