• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Nagios Log Server (logstash) filter for pfSense 2.2

Scheduled Pinned Locked Moved General pfSense Questions
15 Posts 4 Posters 11.3k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • L
    ljorgensen
    last edited by Jan 30, 2015, 12:25 PM Jan 30, 2015, 12:19 PM

    Hi,

    pfSense 2.2 has changed the log format significantly. If you're sending your logs to Nagios Log Server or another implementation of a logstash stack, you need a new filter. I've spent a couple of days getting this to work the way I want it and it pretty much handles anything in the filterlog from pfSense. It doesn't handle any other log at all because I'm not interested in them at the moment.

    This is my first effort at writing a logstash filter so some things might be stupid. But I'll be glad if I can save another poor soul a lot of hours getting this to work.

    if [host] =~ /192\.168\.1\.(1|2)/ {
        grok {
          match => [ 'message', '.* %{WORD:program}:%{GREEDYDATA:rest}' ]
        }
    
    	if [program] == "filterlog" {
    	  # Grab fields up to IP version. The rest will vary depending on IP version.
    	  grok {  
    		match => [ 'rest', '%{INT:rule_number},%{INT:sub_rule_number},,%{INT:tracker_id},%{WORD:interface},%{WORD:reason},%{WORD:action},%{WORD:direction},%{WORD:ip_version},%{GREEDYDATA:rest2}' ] 
    	  }
    
    	  mutate {
    		replace => [ 'message', '%{rest2}' ]
    	  }
    
    	  if [ip_version] == "4" {
    	    # IPv4\. Grab field up to dest_ip. Rest can vary.
    		grok {
    		  match => [ 'message', '%{WORD:tos},(\d+)?,%{INT:ttl},%{INT:id},%{INT:offset},%{WORD:flags},%{INT:protocol_id},%{WORD:protocol},%{INT:length},%{IP:src_ip},%{IP:dest_ip},%{GREEDYDATA:rest3}' ]
    		}
    
    		if [protocol_id] != 2 {
    		  # Non-IGMP has more fields.
    		  grok {
    		    match => [ 'rest3', '%{WORD:src_port},%{WORD:dest_port}' ]
    		  }
    
    		}
    	  } else {
    	    # IPv6\. Grab field up to dest_ip. Rest can vary.
    		grok {
    		  match => [ 'message', '%{WORD:class},%{WORD:flow_label},%{INT:hop_limit},%{WORD:protocol},%{INT:protocol_id},%{INT:length},%{IPV6:src_ip},%{IPV6:dest_ip},%{GREEDYDATA:rest3}' ]		
    		}
    
    		mutate {
    		  replace => [ 'message', '%{rest3}' ]
    		  lowercase => [ 'protocol' ]
    		}
    
    		if [message] {
    		  # Non-ICMP has more fields
      		  grok {
    		    match => [ 'message', '%{INT:src_port},%{INT:dest_port},%{INT:data_length}' ]
    		  }
    		}
    	  }
    
    	  mutate {
            	remove_field => [ 'message' ]
    		remove_field => [ 'rest' ]
    		remove_field => [ 'rest2' ]
    		remove_field => [ 'rest3' ]
    		remove_tag => [ '_grokparsefailure' ]
    		add_tag => [ 'packetfilter' ]
    	  }
    	}
    }
    
    1 Reply Last reply Reply Quote 0
    • K
      killmasta93
      last edited by Aug 17, 2015, 11:20 PM

      awesome work im just curious how did you  Copy SSL Certificate and Logstash Forwarder Package to pfSense(192.168.3.254) to get it work?

      i tried

      scp /etc/pki/tls/certs/logstash-forwarder.crt root@192.168.3.254:/tmp
      

      but it says he authenticity of host '192.168.3.254 (192.168.3.254)' can't be established.

      any ideas

      Thanks again for the filter log will implement it after i get it working

      Tutorials:

      https://www.mediafire.com/folder/v329emaz1e9ih/Tutorials

      1 Reply Last reply Reply Quote 0
      • L
        ljorgensen
        last edited by Aug 21, 2015, 7:15 AM Aug 18, 2015, 6:48 AM

        @killmasta93:

        awesome work im just curious how did you  Copy SSL Certificate and Logstash Forwarder Package to pfSense(192.168.3.254) to get it work?

        I didn't do that at all, I just forward syslogs from pfSense to the logstash server. Go to Status -> System Logs -> Settings and fill out the "Remote Logging Options". Under "Remote Syslog Servers" on your Nagios Log Server address and port like this "nagios-log.example.com:5544". The rest happens automatically.

        There's a small caveat: Every time you reboot pfSense, it stops logging until you go to Status -> System Logs -> Settings and hit "Save".

        1 Reply Last reply Reply Quote 0
        • K
          killmasta93
          last edited by Aug 19, 2015, 2:44 AM

          Hi,
          Thanks for your response. When you mean port 5544 would it rather be  (192.168.3.199:5140) (logstash server 192.168.3.199) I tried following this tutorial
          https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-centos-7
          and
          https://elijahpaul.co.uk/monitoring-pfsense-2-1-logs-using-elk-logstash-kibana-elasticsearch/

          but im stuck on accessing Kibana web interface :(

          also there's no package for logstash forwarder for pfSense?

          Thanks

          Tutorials:

          https://www.mediafire.com/folder/v329emaz1e9ih/Tutorials

          1 Reply Last reply Reply Quote 0
          • L
            ljorgensen
            last edited by Aug 20, 2015, 8:09 AM

            I tried various logstash implementations and didn't get far on any of them. Then I tried Nagios Log Server and it worked out of the box. It's a commercial product built on top og Logstash and Elasticsearch and it's very easy to use.

            1 Reply Last reply Reply Quote 0
            • M
              mikesamo
              last edited by Aug 20, 2015, 11:07 PM

              just use this vm.

              http://www.sexilog.fr/quickstart/

              and do this.

              https://elijahpaul.co.uk/monitoring-pfsense-2-1-logs-using-elk-logstash-kibana-elasticsearch/

              you can see demo here..

              http://demo.sexilog.fr/#/dashboard/elasticsearch/PFSense

              very easy.

              1 Reply Last reply Reply Quote 0
              • K
                killmasta93
                last edited by Aug 21, 2015, 1:38 AM Aug 21, 2015, 1:17 AM

                Hi mikesamo thank you soo much for this I really Appreciate it. I am downloading as we speak and I will let you know how it goes. I am going to follow this https://www.rootusers.com/how-to-convert-an-ova-virtual-machine-to-vhd/
                because i use Hyper-V  and hopefully it goes well

                Thank you again for everything.

                also side note I was wondering if its possible to change the password and the name that appears sexilog kinda awkward to put in a work environment with that name haha  :D

                Tutorials:

                https://www.mediafire.com/folder/v329emaz1e9ih/Tutorials

                1 Reply Last reply Reply Quote 0
                • M
                  mikesamo
                  last edited by Aug 21, 2015, 6:28 AM

                  check this link they explain how to build that vm.

                  http://www.sexilog.fr/cookbook/

                  to change the name and setting please read about logstash and kibana you should find it easily.

                  Thanks,

                  1 Reply Last reply Reply Quote 0
                  • K
                    killmasta93
                    last edited by Aug 21, 2015, 8:18 PM

                    Mikesamo thank you so much Im going to try to build it on Hyper-V and keep you posted if anything

                    Thank you again  ;D ;D

                    Tutorials:

                    https://www.mediafire.com/folder/v329emaz1e9ih/Tutorials

                    1 Reply Last reply Reply Quote 0
                    • M
                      mikesamo
                      last edited by Aug 22, 2015, 1:19 PM

                      HI,

                      I just tried and I got the dashboard working in less than one hours.

                      1 Reply Last reply Reply Quote 0
                      • K
                        killmasta93
                        last edited by Aug 22, 2015, 3:08 PM

                        Hi Mikesamo,
                        What im also trying to do is add install  NGINX for security. But thats were i screw everything up i get the 503 error not sure if to add it before Kibana or after.

                        Thanks

                        Tutorials:

                        https://www.mediafire.com/folder/v329emaz1e9ih/Tutorials

                        1 Reply Last reply Reply Quote 0
                        • K
                          killmasta93
                          last edited by Aug 30, 2015, 9:01 PM

                          if you guys are interested i made a guide finished it up only thing that needs fixing is the syslogs (pfSense system logs) not filtering  correctly with the sysdashboard
                          https://forum.pfsense.org/index.php?topic=98740.msg549992#msg549992

                          and i need to add curator to delete logs after time or gigs and then backup before deleting.

                          Clipboarder.2015.08.30-013.png
                          Clipboarder.2015.08.30-013.png_thumb
                          Clipboarder.2015.08.30-014.png
                          Clipboarder.2015.08.30-014.png_thumb

                          Tutorials:

                          https://www.mediafire.com/folder/v329emaz1e9ih/Tutorials

                          1 Reply Last reply Reply Quote 0
                          • L
                            lolo666666
                            last edited by Feb 1, 2016, 2:36 PM

                            Hello,

                            it would be possible to add the log OPENVPN ?

                            Laurent

                            1 Reply Last reply Reply Quote 0
                            • K
                              killmasta93
                              last edited by Feb 1, 2016, 6:24 PM

                              yeah thats what im working on :( been harder then I thought

                              Tutorials:

                              https://www.mediafire.com/folder/v329emaz1e9ih/Tutorials

                              1 Reply Last reply Reply Quote 0
                              • L
                                lolo666666
                                last edited by Feb 1, 2016, 6:30 PM

                                it's been several days I'm working on but I do not find solution !!!!

                                I found that : https://www.reddit.com/r/PFSENSE/comments/3hk4f1/openvpn_logging_format_grok_is_killing_me/

                                Laurent

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post
                                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                  This community forum collects and processes your personal information.
                                  consent.not_received