ELK + pfSense 2.3 Working

[ando1]

@johnpoz:

While this great, sure many people will be happy.  Why are you using old versions of stuff?

The current is 5 is it not?  And why such an old version of java?  I just looked on my ubuntu 14.04 vm and 111 is current

user@uc:~$ java -version
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
user@uc:~$

I used these versions because these were the ones that worked for me. I asked several times on this forum and received no help so I decided to share a working config with others here as I have read many posts where people said they had tried and could not get it going. If you got a newer version to work, then that's great. Post the instructions so everyone can also enjoy.

[AR15USR]

Thanks a bunch for this post ando1. Been looking forward to getting ELK going, will try it out when I get some free time…

[AR15USR]

I see no Create Index button. The output from your trouble shooting section is:

yellow open .kibana 1 1 1 0 3.1kb 3.1kb

Also, when importing the 3 .json files, the "Firewall External" imports fine but I get this error on the other two:

Error: Could not locate that index-pattern (id: logstash-*)
KbnError@http://0.0.0.0:5601/bundles/commons.bundle.js?v=10000:57463:21
SavedObjectNotFound@http://0.0.0.0:5601/bundles/commons.bundle.js?v=10000:57592:6
applyESResp@http://0.0.0.0:5601/bundles/kibana.bundle.js?v=10000:79296:37
processQueue@http://0.0.0.0:5601/bundles/commons.bundle.js?v=10000:42404:29
scheduleProcessQueue/<@http://0.0.0.0:5601/bundles/commons.bundle.js?v=10000:42420:28
$RootScopeProvider/this.$get$RootScopeProvider/this.$get$RootScopeProvider/this.$getdone@http://0.0.0.0:5601/bundles/commons.bundle.js?v=10000:38205:37
completeRequest@http://0.0.0.0:5601/bundles/commons.bundle.js?v=10000:38403:8
requestLoaded@http://0.0.0.0:5601/bundles/commons.bundle.js?v=10000:38344:10

Also, in steps 4 & 10, the file version numbers don't match fyi…

[AR15USR]

ando1, any idea what is going on?

PS I ran everyone of your troubleshooting commands and they all error out fyi…

[ando1]

@AR15USR:

ando1, any idea what is going on?

PS I ran everyone of your troubleshooting commands and they all error out fyi…

Can you post the output of the logstash debug? You may need to stop the service before you run the command:

/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ –debug

Also what error do you get when you run this?

/opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/

Andy

[ando1]

For anyone interested in getting the newest version of ELK (v5) working with pfSense, I was able to get do it using the instructions on this siye: http://pfelk.3ilson.com/

You need at least Ubuntu server vv16.04.01

[AR15USR]

@ando1:

Can you post the output of the logstash debug? You may need to stop the service before you run the command:

/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ –debug

Also what error do you get when you run this?

/opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/

Andy

/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ –debug

Error: Expected one of #, input, filter, output at line 1, column 1 (byte 1) after  {:level=>:error, :file=>"logstash/agent.rb", :line=>"214", :method=>"execute"}
You may be interested in the '--configtest' flag which you can
use to validate logstash's configuration before you choose
to restart a running system. {:level=>:info, :file=>"logstash/agent.rb", :line=>"216", :method=>"execute"}

/opt/logstash/bin/logstash –configtest -f /etc/logstash/conf.d/

Error: Expected one of #, input, filter, output at line 1, column 1 (byte 1) after  {:level=>:error}

[ando1]

/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ –debug

Error: Expected one of #, input, filter, output at line 1, column 1 (byte 1) after  {:level=>:error, :file=>"logstash/agent.rb", :line=>"214", :method=>"execute"}
You may be interested in the '--configtest' flag which you can
use to validate logstash's configuration before you choose
to restart a running system. {:level=>:info, :file=>"logstash/agent.rb", :line=>"216", :method=>"execute"}

/opt/logstash/bin/logstash –configtest -f /etc/logstash/conf.d/

Error: Expected one of #, input, filter, output at line 1, column 1 (byte 1) after  {:level=>:error}

You definitely have a config file issue. Logstash combines all the configuration files into one and then processes them. Since the error is at Line 1 column 1 it sounds like the problem may be in the 02-inputs file. Have a look at all config files and double check they are OK.

[hamed_forum]

tanks
if can creat ova or ovf from vm machine and upload it its very good :)

[doktornotor]

http://pfelk.3ilson.com/ basically works, but some pointers:

1/ There's a PPA for MaxMind:

sudo add-apt-repository ppa:maxmind/ppa

• see http://dev.maxmind.com/geoip/geoipupdate/ for /etc/GeoIP.conf and run geoipupdate after that. The DB is located in /usr/share/GeoIP/GeoLite2-City.mmdb

2/ You really should set up some authentication:

https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html#xpack-package-installation
https://www.elastic.co/guide/en/x-pack/current/setting-up-authentication.html
https://www.elastic.co/guide/en/x-pack/current/logstash.html

[johnpoz]

Yeah I had issues with the date stuff in logstash config as well.. had to remove the +0400 and timezone..

I have it running, but elasticstack doesn't seem to want to stay running.  Haven't had time to look into why.  And have not had any time to do any visualizations - which is what everyone wants ;)

[doktornotor]

@johnpoz:

I have it running, but elasticstack doesn't seem to want to stay running.  Haven't had time to look into why.

Make sure you've allocated at least 4GiB of RAM to this thing. (Java  >:( ::))

[hamed_forum]

Elasticsearch after 10 sec  start its stop

[bubbawatson]

@doktornotor:

@johnpoz:

I have it running, but elasticstack doesn't seem to want to stay running.  Haven't had time to look into why.

Make sure you've allocated at least 4GiB of RAM to this thing. (Java  >:( ::))

I run elk stack on 1.5  ;D

Small office though. Thx for the info on auth.. I've been wondering how to do that.

[BrunoCAVILLE]

I'm currently going through the process of installing ELK but I have an important question. If I redirect the logs from pfSense to the ELK server will I be able to access the raw logs somewhere? I need to have them somewhere and I'm wondering where they would be if they are sent to ELK.

[BrunoCAVILLE]

Eveything works well except the maps visualization, someone can help?


[BrunoCAVILLE]

Up

Logstash stops after a few seconds (rising heap size didn't help).

[AMizil]

@BrunoCAVILLE:

I'm currently going through the process of installing ELK but I have an important question. If I redirect the logs from pfSense to the ELK server will I be able to access the raw logs somewhere? I need to have them somewhere and I'm wondering where they would be if they are sent to ELK.

Status Menu - System Logs - Settings  - and jump to :  Remote log servers - and you can add another 2 Syslog Servers you have ; ex syslog-ng, Splunk etc

[ronv]

Hi all,

trying to get this going with PFsense 2.3.4 and ELK 5.4 - all components are talking ok, and I can get the JSON Dashboard, Search and Visualization up and running - almost…:

• when I import the visualizations, Kibana complains that the tags geoip.country_name and geoip.city_name are not available.
• I checked 11-pfsense.conf (which I used from this site) against the spec at https://www.elastic.co/guide/en/logstash/current/plugins-filters-geoip.html, and there does not appear to be any issue with this - that is, it looks like those tags should be returned.

Anything else I could check, or logs I could provide?

kind regards

Ron

[hamed_forum]

the log send from pfsense where is save on elk?
i change the elk server and how to export import log on prvise server?