Nagios Log Server (logstash) filter for pfSense 2.2

[ljorgensen]

Hi,

pfSense 2.2 has changed the log format significantly. If you're sending your logs to Nagios Log Server or another implementation of a logstash stack, you need a new filter. I've spent a couple of days getting this to work the way I want it and it pretty much handles anything in the filterlog from pfSense. It doesn't handle any other log at all because I'm not interested in them at the moment.

This is my first effort at writing a logstash filter so some things might be stupid. But I'll be glad if I can save another poor soul a lot of hours getting this to work.

if [host] =~ /192\.168\.1\.(1|2)/ {
    grok {
      match => [ 'message', '.* %{WORD:program}:%{GREEDYDATA:rest}' ]
    }

	if [program] == "filterlog" {
	  # Grab fields up to IP version. The rest will vary depending on IP version.
	  grok {  
		match => [ 'rest', '%{INT:rule_number},%{INT:sub_rule_number},,%{INT:tracker_id},%{WORD:interface},%{WORD:reason},%{WORD:action},%{WORD:direction},%{WORD:ip_version},%{GREEDYDATA:rest2}' ] 
	  }

	  mutate {
		replace => [ 'message', '%{rest2}' ]
	  }

	  if [ip_version] == "4" {
	    # IPv4\. Grab field up to dest_ip. Rest can vary.
		grok {
		  match => [ 'message', '%{WORD:tos},(\d+)?,%{INT:ttl},%{INT:id},%{INT:offset},%{WORD:flags},%{INT:protocol_id},%{WORD:protocol},%{INT:length},%{IP:src_ip},%{IP:dest_ip},%{GREEDYDATA:rest3}' ]
		}

		if [protocol_id] != 2 {
		  # Non-IGMP has more fields.
		  grok {
		    match => [ 'rest3', '%{WORD:src_port},%{WORD:dest_port}' ]
		  }

		}
	  } else {
	    # IPv6\. Grab field up to dest_ip. Rest can vary.
		grok {
		  match => [ 'message', '%{WORD:class},%{WORD:flow_label},%{INT:hop_limit},%{WORD:protocol},%{INT:protocol_id},%{INT:length},%{IPV6:src_ip},%{IPV6:dest_ip},%{GREEDYDATA:rest3}' ]		
		}

		mutate {
		  replace => [ 'message', '%{rest3}' ]
		  lowercase => [ 'protocol' ]
		}

		if [message] {
		  # Non-ICMP has more fields
  		  grok {
		    match => [ 'message', '%{INT:src_port},%{INT:dest_port},%{INT:data_length}' ]
		  }
		}
	  }

	  mutate {
        	remove_field => [ 'message' ]
		remove_field => [ 'rest' ]
		remove_field => [ 'rest2' ]
		remove_field => [ 'rest3' ]
		remove_tag => [ '_grokparsefailure' ]
		add_tag => [ 'packetfilter' ]
	  }
	}
}

[killmasta93]

awesome work im just curious how did you  Copy SSL Certificate and Logstash Forwarder Package to pfSense(192.168.3.254) to get it work?

i tried

scp /etc/pki/tls/certs/logstash-forwarder.crt root@192.168.3.254:/tmp

but it says he authenticity of host '192.168.3.254 (192.168.3.254)' can't be established.

any ideas

Thanks again for the filter log will implement it after i get it working

[ljorgensen]

@killmasta93:

awesome work im just curious how did you  Copy SSL Certificate and Logstash Forwarder Package to pfSense(192.168.3.254) to get it work?

I didn't do that at all, I just forward syslogs from pfSense to the logstash server. Go to Status -> System Logs -> Settings and fill out the "Remote Logging Options". Under "Remote Syslog Servers" on your Nagios Log Server address and port like this "nagios-log.example.com:5544". The rest happens automatically.

There's a small caveat: Every time you reboot pfSense, it stops logging until you go to Status -> System Logs -> Settings and hit "Save".

[killmasta93]

Hi,
Thanks for your response. When you mean port 5544 would it rather be  (192.168.3.199:5140) (logstash server 192.168.3.199) I tried following this tutorial
https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-centos-7
and
https://elijahpaul.co.uk/monitoring-pfsense-2-1-logs-using-elk-logstash-kibana-elasticsearch/

but im stuck on accessing Kibana web interface :(

also there's no package for logstash forwarder for pfSense?

Thanks

[ljorgensen]

I tried various logstash implementations and didn't get far on any of them. Then I tried Nagios Log Server and it worked out of the box. It's a commercial product built on top og Logstash and Elasticsearch and it's very easy to use.

[mikesamo]

just use this vm.

http://www.sexilog.fr/quickstart/

and do this.

https://elijahpaul.co.uk/monitoring-pfsense-2-1-logs-using-elk-logstash-kibana-elasticsearch/

you can see demo here..

http://demo.sexilog.fr/#/dashboard/elasticsearch/PFSense

very easy.

[killmasta93]

Hi mikesamo thank you soo much for this I really Appreciate it. I am downloading as we speak and I will let you know how it goes. I am going to follow this https://www.rootusers.com/how-to-convert-an-ova-virtual-machine-to-vhd/
because i use Hyper-V  and hopefully it goes well

Thank you again for everything.

also side note I was wondering if its possible to change the password and the name that appears sexilog kinda awkward to put in a work environment with that name haha  :D

[mikesamo]

check this link they explain how to build that vm.

http://www.sexilog.fr/cookbook/

to change the name and setting please read about logstash and kibana you should find it easily.

Thanks,

[killmasta93]

Mikesamo thank you so much Im going to try to build it on Hyper-V and keep you posted if anything

Thank you again  ;D ;D

[mikesamo]

HI,

I just tried and I got the dashboard working in less than one hours.

[killmasta93]

Hi Mikesamo,
What im also trying to do is add install  NGINX for security. But thats were i screw everything up i get the 503 error not sure if to add it before Kibana or after.

Thanks

[killmasta93]

if you guys are interested i made a guide finished it up only thing that needs fixing is the syslogs (pfSense system logs) not filtering  correctly with the sysdashboard
https://forum.pfsense.org/index.php?topic=98740.msg549992#msg549992

and i need to add curator to delete logs after time or gigs and then backup before deleting.

[lolo666666]

Hello,

it would be possible to add the log OPENVPN ?

Laurent

[killmasta93]

yeah thats what im working on :( been harder then I thought

[lolo666666]

it's been several days I'm working on but I do not find solution !!!!

I found that : https://www.reddit.com/r/PFSENSE/comments/3hk4f1/openvpn_logging_format_grok_is_killing_me/

Laurent