ELK + pfSense 2.3 Working

AR15USR · Nov 16, 2016, 1:56 PM

ando1, any idea what is going on?

PS I ran everyone of your troubleshooting commands and they all error out fyi…

ando1 · Nov 16, 2016, 6:10 PM

@AR15USR:

ando1, any idea what is going on?

PS I ran everyone of your troubleshooting commands and they all error out fyi…

Can you post the output of the logstash debug? You may need to stop the service before you run the command:

/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ –debug

Also what error do you get when you run this?

/opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/

Andy

ando1 · Nov 17, 2016, 11:07 AM

For anyone interested in getting the newest version of ELK (v5) working with pfSense, I was able to get do it using the instructions on this siye: http://pfelk.3ilson.com/

You need at least Ubuntu server vv16.04.01

AR15USR · Nov 17, 2016, 1:44 PM

@ando1:

Can you post the output of the logstash debug? You may need to stop the service before you run the command:

/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ –debug

Also what error do you get when you run this?

/opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/

Andy

/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ –debug

Error: Expected one of #, input, filter, output at line 1, column 1 (byte 1) after  {:level=>:error, :file=>"logstash/agent.rb", :line=>"214", :method=>"execute"}
You may be interested in the '--configtest' flag which you can
use to validate logstash's configuration before you choose
to restart a running system. {:level=>:info, :file=>"logstash/agent.rb", :line=>"216", :method=>"execute"}

/opt/logstash/bin/logstash –configtest -f /etc/logstash/conf.d/

Error: Expected one of #, input, filter, output at line 1, column 1 (byte 1) after  {:level=>:error}

ando1 · Nov 18, 2016, 8:53 PM

/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/ –debug

Error: Expected one of #, input, filter, output at line 1, column 1 (byte 1) after  {:level=>:error, :file=>"logstash/agent.rb", :line=>"214", :method=>"execute"}
You may be interested in the '--configtest' flag which you can
use to validate logstash's configuration before you choose
to restart a running system. {:level=>:info, :file=>"logstash/agent.rb", :line=>"216", :method=>"execute"}

/opt/logstash/bin/logstash –configtest -f /etc/logstash/conf.d/

Error: Expected one of #, input, filter, output at line 1, column 1 (byte 1) after  {:level=>:error}

You definitely have a config file issue. Logstash combines all the configuration files into one and then processes them. Since the error is at Line 1 column 1 it sounds like the problem may be in the 02-inputs file. Have a look at all config files and double check they are OK.

hamed_forum · Nov 22, 2016, 4:56 AM

tanks
if can creat ova or ovf from vm machine and upload it its very good :)

doktornotor · Nov 22, 2016, 6:43 PM

http://pfelk.3ilson.com/ basically works, but some pointers:

1/ There's a PPA for MaxMind:

sudo add-apt-repository ppa:maxmind/ppa

see http://dev.maxmind.com/geoip/geoipupdate/ for /etc/GeoIP.conf and run geoipupdate after that. The DB is located in /usr/share/GeoIP/GeoLite2-City.mmdb

2/ You really should set up some authentication:

https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html#xpack-package-installation
https://www.elastic.co/guide/en/x-pack/current/setting-up-authentication.html
https://www.elastic.co/guide/en/x-pack/current/logstash.html

johnpoz · Nov 22, 2016, 9:18 PM

Yeah I had issues with the date stuff in logstash config as well.. had to remove the +0400 and timezone..

I have it running, but elasticstack doesn't seem to want to stay running. Haven't had time to look into why. And have not had any time to do any visualizations - which is what everyone wants ;)

doktornotor · Nov 22, 2016, 9:42 PM

@johnpoz:

I have it running, but elasticstack doesn't seem to want to stay running. Haven't had time to look into why.

Make sure you've allocated at least 4GiB of RAM to this thing. (Java >:( ::))

hamed_forum · Feb 3, 2017, 9:04 PM

Elasticsearch after 10 sec start its stop

bubbawatson · Mar 7, 2017, 6:34 PM

@doktornotor:

@johnpoz:

I have it running, but elasticstack doesn't seem to want to stay running. Haven't had time to look into why.

Make sure you've allocated at least 4GiB of RAM to this thing. (Java >:( ::))

I run elk stack on 1.5 ;D

Small office though. Thx for the info on auth.. I've been wondering how to do that.

BrunoCAVILLE · May 2, 2017, 12:57 PM

I'm currently going through the process of installing ELK but I have an important question. If I redirect the logs from pfSense to the ELK server will I be able to access the raw logs somewhere? I need to have them somewhere and I'm wondering where they would be if they are sent to ELK.

BrunoCAVILLE · May 5, 2017, 1:46 PM

Eveything works well except the maps visualization, someone can help?

![Capture d’écran 2017-05-05 à 15.18.39.png](/public/imported_attachments/1/Capture d’écran 2017-05-05 à 15.18.39.png)
![Capture d’écran 2017-05-05 à 15.18.39.png_thumb](/public/imported_attachments/1/Capture d’écran 2017-05-05 à 15.18.39.png_thumb)

BrunoCAVILLE · May 9, 2017, 11:38 AM

Up

Logstash stops after a few seconds (rising heap size didn't help).

AMizil · May 13, 2017, 8:26 PM

@BrunoCAVILLE:

I'm currently going through the process of installing ELK but I have an important question. If I redirect the logs from pfSense to the ELK server will I be able to access the raw logs somewhere? I need to have them somewhere and I'm wondering where they would be if they are sent to ELK.

Status Menu - System Logs - Settings - and jump to : Remote log servers - and you can add another 2 Syslog Servers you have ; ex syslog-ng, Splunk etc

ronv · Jun 21, 2017, 2:19 PM

Hi all,

trying to get this going with PFsense 2.3.4 and ELK 5.4 - all components are talking ok, and I can get the JSON Dashboard, Search and Visualization up and running - almost…:

when I import the visualizations, Kibana complains that the tags geoip.country_name and geoip.city_name are not available.
I checked 11-pfsense.conf (which I used from this site) against the spec at https://www.elastic.co/guide/en/logstash/current/plugins-filters-geoip.html, and there does not appear to be any issue with this - that is, it looks like those tags should be returned.

Anything else I could check, or logs I could provide?

kind regards

Ron

hamed_forum · Jun 25, 2017, 4:57 AM

the log send from pfsense where is save on elk?
i change the elk server and how to export import log on prvise server?

pfBasic · Jul 8, 2017, 8:35 AM

Any differences to get this running on 2.4.0 BETA?

pfBasic · Jul 12, 2017, 5:16 PM

I finally got this up & running on pfSense 2.4.0 BETA with the help of AR15USR and some people on IRC.

Initially I was having trouble getting the Index Patterns to populate in the first step of Kibana. I had followed doktornotor's advice for setting up MaxMind. For whatever reason that didn't work for me so I just did it according to http://pfelk.3ilson.com/ and it worked.

Next, I had everything stable and logs being imported, but all logs were being tagged "_grokparsefailure" & "_geoip_lookup_failure" and since the pattern wasn't matching, it wasn't putting out any useful fields/information. This was also preventing me from importing the Visualizations.json due to not having the applicable fields available.

After way too much time troubleshooting and trying to figure out what was happening and why I was given some direction and pointed to the grok debugger by a kind IRC user. https://grokdebug.herokuapp.com/
For anyone looking to troubleshoot or modify their own grok pattern files, here's what I could make of the fields in 2.4.0 BETA's Rsyslog format. https://forum.pfsense.org/index.php?topic=133354.msg733494#msg733494
Run a pcap to see exactly what your pfSense box is sending to your ELK server.

It turned out that all I needed to do was change one character in /etc/logstash/conf.d/patterns/pfsense2-3.grok and reboot.

I changed line 16 (PFSENSE_LOG_DATA)
From:

PFSENSE_LOG_DATA (%{INT:rule}),(%{INT:sub_rule}),,(%{INT:tracker}),(%{WORD:iface}),(%{WORD:reason}),(%{WORD:action}),(%{WORD:direction}),(%{INT:ip_ver}),

To:

PFSENSE_LOG_DATA (%{INT:rule}),(%{INT:sub_rule})?,,(%{INT:tracker}),(%{WORD:iface}),(%{WORD:reason}),(%{WORD:action}),(%{WORD:direction}),(%{INT:ip_ver}),

That's it, one "?".

After that, log files were parsing successfully, I refreshed my Index Pattern Field List to pull in all of the new fields, imported the Visualizations.json and opened up the Dashboard. All is working now on my single core atom with 2GB DDR2!

@doktornotor:

@johnpoz:

I have it running, but elasticstack doesn't seem to want to stay running. Haven't had time to look into why.

Make sure you've allocated at least 4GiB of RAM to this thing. (Java >:( ::))

I have this up and running (for home use) on an old netbook with an atom N450 (Pineview ~2010, single core 1.66GHz) with 2GB DDR2. I had to significantly lower RAM usage in the following two files to get it working. Currently using <1.5GB RAM, the OS is lubuntu with GUI service disabled. It's also running a Unifi controller. Dashboard is slow to load even for a small home network but it works! I couldn't justify buying anything to get an ELK stack for my home network.

/etc/elasticsearch/jvm.options

/etc/logstash/jvm.options

idealanthony · Jul 16, 2017, 8:33 AM

@BrunoCAVILLE:

Eveything works well except the maps visualization, someone can help?

@BrunoCAVILLE - I'm having the same problem as you did. I used the revised visualization file due to the .keyword issue. I've attempted to merge back in the country sections from the http://pfelk.3ilson.com/ visualization file, but still no luck. Just wanted to know if you were able to identify/ resolve the issue?

https://forum.pfsense.org/index.php?topic=125376.0