Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    SOLVED - How to get raw Rsyslog output? 2.4.0 BETA - SOLVED

    General pfSense Questions
    2
    7
    2.9k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      pfBasic Banned
      last edited by

      I'm trying to setup an ELK stack. Everything is up and running, but the filter I used just keeps tagging all of my logs with "_grokparsefailure" and "_geoipfailure", I'm not getting anything usable out of my logs.

      How can I get the raw remote-syslog output from pfSense.

      I'm trying to see exactly what the ELK stack is receiving before anything is processed.

      Also - I found this: https://doc.pfsense.org/index.php/Filter_Log_Format_for_pfSense_2.2

      It shows the pfSense 2.2 filterlog format as```
      <timestamp><hostname>filterlog:</hostname></timestamp>

      
Is that still the same in 2.4.0? Has the Rsyslog format changed since 2.2 (or 2.3)? - If so, what is it in 2.4.0 BETA?
      1 Reply Last reply Reply Quote 0
      • P
        pfBasic Banned
        last edited by

        Bump, anyone?

        1 Reply Last reply Reply Quote 0
        • P
          pfBasic Banned
          last edited by

          I was able to figure most of it out, for anyone who needs it in the future here is what I could figure out for syslog format in 2.4.0 BETA.

          46,,,1000000117,igb0,match,block,in,4,0x0,,245,6029,0,none,6,tcp,40,80.82.65.231,192.168.1.1,56805,3983,0,S,2650484143,,1024,,

          46 = Rule #
          , = nothing?
          , = nothing?
          1000000117 = Tracker
          igb0 = iface
          match = Reason
          block =  action
          in = direction
          4 = ip_ver
          0x0 = tos (type of service)
          , = nothing?
          245 = ttl (time to live)
          6029 = id
          0 = offset OR data_length?
          none = flags
          6 = proto_id
          tcp = proto
          40 = length
          80.82.65.231 = src_ip
          192.168.1.1 - dest_ip
          56805 = src_port
          3983 = dest_port
          0 = offset OR data_length?
          S = unused?
          2650484143 = unused?
          , = nothing?
          1024 = unused?
          , = nothing?

          1 Reply Last reply Reply Quote 0
          • I
            idealanthony
            last edited by

            @pfBasic I'm in the same boat with trying to get ELK working with PFSense.

            Were you able to get everything up and running?  If so, I'd be really interested in seeing you grok file for logstash

            I'm using the article found here http://pfelk.3ilson.com/

            It may be a poor choice, but this is my first attempt at an ELK stack.  I've got everything (Kibana, Elasticsearch, Logstash) installed, and I can see data getting to Kibana, so from a networking standpoint things appear to be working.

            I'm falling flat on the parsing.

            The message tag seems to contain all the data from PFSense in csv format

            message: 5,,,1000000103,igb0,match,block,in,4,0x0,,57,33381,0,DF,17,udp,40,184.105.139.124,172.92.3.122,11775,123,20
``` 

But the tags aren't getting associated

            tags: PFSense, firewall, _grokparsefailure

            
The tutorial I used was for PFsense 2.3 and I'm assuming my pattern/grok file is to blame, but I'm not actually sure how to modify the syntax to fix it.

I've searched for a 2.4 grok pfsense pattern file, but can't seem to find one online.

I'd appreciate any help

Contents of Grok Below

            GROK match pattern for logstash.conf filter: %{PFSENSE_LOG_DATA}%{PFSENSE_IP_SPECIFIC_DATA}%{PFSENSE_IP_DATA}%{PFSENSE_PROTOCOL_DATA}

            GROK Custom Patterns (add to patterns directory and reference in GROK filter for pfSense events):

            GROK Patterns for pfSense 2.3 Logging Format

            Created 27 Jan 2015 by J. Pisano (Handles TCP, UDP, and ICMP log entries)

            Edited 14 Feb 2015 by Elijah Paul elijah.paul@gmail.com

            Edited 10 Mar 2015 by Bernd Zeimetz bernd@bzed.de# taken from https://gist.github.com/elijahpaul/f5f32d4e914dcb7fedd2

            - adding PFSENSE_ prefix

            - adding carp patterns

            Usage: Use with following GROK match pattern

            %{PFSENSE_LOG_DATA}%{PFSENSE_IP_SPECIFIC_DATA}%{PFSENSE_IP_DATA}%{PFSENSE_PROTOCOL_DATA}

            PFSENSE_LOG_DATA (%{INT:rule}),(%{INT:sub_rule}),,(%{INT:tracker}),(%{WORD:iface}),(%{WORD:reason}),(%{WORD:action}),(%{WORD:direction}),(%{INT:ip_ver}),
            PFSENSE_IP_SPECIFIC_DATA (%{PFSENSE_IPv4_SPECIFIC_DATA}|%{PFSENSE_IPv6_SPECIFIC_DATA})
            PFSENSE_IPv4_SPECIFIC_DATA (%{BASE16NUM:tos}),,(%{INT:ttl}),(%{INT:id}),(%{INT:offset}),(%{WORD:flags}),(%{INT:proto_id}),(%{WORD:proto}),
            PFSENSE_IPv4_SPECIFIC_DATA_ECN (%{BASE16NUM:tos}),(%{INT:ecn}),(%{INT:ttl}),(%{INT:id}),(%{INT:offset}),(%{WORD:flags}),(%{INT:proto_id}),(%{WORD:proto}),
            PFSENSE_IPv6_SPECIFIC_DATA (%{BASE16NUM:class}),(%{DATA:flow_label}),(%{INT:hop_limit}),(%{WORD:proto}),(%{INT:proto_id}),
            PFSENSE_IP_DATA (%{INT:length}),(%{IP:src_ip}),(%{IP:dest_ip}),
            PFSENSE_PROTOCOL_DATA (%{PFSENSE_TCP_DATA}|%{PFSENSE_UDP_DATA}|%{PFSENSE_ICMP_DATA}|%{PFSENSE_CARP_DATA})
            PFSENSE_TCP_DATA (%{INT:src_port}),(%{INT:dest_port}),(%{INT:data_length}),(%{WORD:tcp_flags}),(%{INT:sequence_number}),(%{INT:ack_number}),(%{INT:tcp_window}),(%{DATA:urg_data}),(%{DATA:tcp_options})
            PFSENSE_UDP_DATA (%{INT:src_port}),(%{INT:dest_port}),(%{INT:data_length})
            PFSENSE_ICMP_DATA (%{PFSENSE_ICMP_TYPE}%{PFSENSE_ICMP_RESPONSE})
            PFSENSE_ICMP_TYPE (?<icmp_type>(request|reply|unreachproto|unreachport|unreach|timeexceed|paramprob|redirect|maskreply|needfrag|tstamp|tstampreply)),
            PFSENSE_ICMP_RESPONSE (%{PFSENSE_ICMP_ECHO_REQ_REPLY}|%{PFSENSE_ICMP_UNREACHPORT}| %{PFSENSE_ICMP_UNREACHPROTO}|%{PFSENSE_ICMP_UNREACHABLE}|%{PFSENSE_ICMP_NEED_FLAG}|%{PFSENSE_ICMP_TSTAMP}|%{PFSENSE_ICMP_TSTAMP_REPLY})
            PFSENSE_ICMP_ECHO_REQ_REPLY (%{INT:icmp_echo_id}),(%{INT:icmp_echo_sequence})
            PFSENSE_ICMP_UNREACHPORT (%{IP:icmp_unreachport_dest_ip}),(%{WORD:icmp_unreachport_protocol}),(%{INT:icmp_unreachport_port})
            PFSENSE_ICMP_UNREACHPROTO (%{IP:icmp_unreach_dest_ip}),(%{WORD:icmp_unreachproto_protocol})
            PFSENSE_ICMP_UNREACHABLE (%{GREEDYDATA:icmp_unreachable})
            PFSENSE_ICMP_NEED_FLAG (%{IP:icmp_need_flag_ip}),(%{INT:icmp_need_flag_mtu})
            PFSENSE_ICMP_TSTAMP (%{INT:icmp_tstamp_id}),(%{INT:icmp_tstamp_sequence})
            PFSENSE_ICMP_TSTAMP_REPLY (%{INT:icmp_tstamp_reply_id}),(%{INT:icmp_tstamp_reply_sequence}),(%{INT:icmp_tstamp_reply_otime}),(%{INT:icmp_tstamp_reply_rtime}),(%{INT:icmp_tstamp_reply_ttime})

            PFSENSE_CARP_DATA (%{WORD:carp_type}),(%{INT:carp_ttl}),(%{INT:carp_vhid}),(%{INT:carp_version}),(%{INT:carp_advbase}),(%{INT:carp_advskew})

            DHCPD (%{DHCPDISCOVER}|%{DHCPOFFER}|%{DHCPREQUEST}|%{DHCPACK}|%{DHCPINFORM}|%{DHCPRELEASE})
            DHCPDISCOVER %{WORD:dhcp_action} from %{COMMONMAC:dhcp_client_mac}%{SPACE}((%{GREEDYDATA:dhcp_client_hostname}))? via (?<dhcp_client_vlan>[0-9a-z_])(: %{GREEDYDATA:dhcp_load_balance})?
            DHCPOFFER %{WORD:dhcp_action} on %{IPV4:dhcp_client_ip} to %{COMMONMAC:dhcp_client_mac}%{SPACE}((%{GREEDYDATA:dhcp_client_hostname}))? via (?<dhcp_client_vlan>[0-9a-z_]            )
            DHCPREQUEST %{WORD:dhcp_action} for %{IPV4:dhcp_client_ip}%{SPACE}((%{IPV4:dhcp_ip_unknown}))? from %{COMMONMAC:dhcp_client_mac}%{SPACE}((%{GREEDYDATA:dhcp_client_hostname}))? via (?<dhcp_client_vlan>[0-9a-z_])(: %{GREEDYDATA:dhcp_request_message})?
            DHCPACK %{WORD:dhcp_action} on %{IPV4:dhcp_client_ip} to %{COMMONMAC:dhcp_client_mac}%{SPACE}((%{GREEDYDATA:dhcp_client_hostname}))? via (?<dhcp_client_vlan>[0-9a-z_]            )
            DHCPINFORM %{WORD:dhcp_action} from %{IPV4:dhcp_client_ip} via %(?<dhcp_client_vlan>[0-9a-z_]*)
            DHCPRELEASE %{WORD:dhcp_action} of %{IPV4:dhcp_client_ip} from %{COMMONMAC:dhcp_client_mac}%{SPACE}((%{GREEDYDATA:dhcp_client_hostname}))? via</dhcp_client_vlan></dhcp_client_vlan></dhcp_client_vlan></dhcp_client_vlan></dhcp_client_vlan></icmp_type>/bernd@bzed.de

            1 Reply Last reply Reply Quote 0
            • P
              pfBasic Banned
              last edited by

              Give this grok pattern a shot:

              
# GROK match pattern for logstash.conf filter: %{PFSENSE_LOG_DATA}%{PFSENSE_IP_SPECIFIC_DATA}%{PFSENSE_IP_DATA}%{PFSENSE_PROTOCOL_DATA}
# GROK Custom Patterns (add to patterns directory and reference in GROK filter for pfSense events):
# GROK Patterns for pfSense 2.3 Logging Format
#
# Created 27 Jan 2015 by J. Pisano (Handles TCP, UDP, and ICMP log entries)
# Edited 14 Feb 2015 by Elijah Paul elijah.paul@gmail.com
# Edited 10 Mar 2015 by Bernd Zeimetz <bernd@bzed.de># taken from https://gist.github.com/elijahpaul/f5f32d4e914dcb7fedd2
# - adding PFSENSE_ prefix
# - adding carp patterns
#
# Usage: Use with following GROK match pattern
#
# %{PFSENSE_LOG_DATA}%{PFSENSE_IP_SPECIFIC_DATA}%{PFSENSE_IP_DATA}%{PFSENSE_PROTOCOL_DATA}

PFSENSE_LOG_DATA (%{INT:rule}),(%{INT:sub_rule})?,,(%{INT:tracker}),(%{WORD:iface}),(%{WORD:reason}),(%{WORD:action}),(%{WORD:direction}),(%{INT:ip_ver}),
PFSENSE_IP_SPECIFIC_DATA (%{PFSENSE_IPv4_SPECIFIC_DATA}|%{PFSENSE_IPv6_SPECIFIC_DATA})
PFSENSE_IPv4_SPECIFIC_DATA (%{BASE16NUM:tos}),,(%{INT:ttl}),(%{INT:id}),(%{INT:offset}),(%{WORD:flags}),(%{INT:proto_id}),(%{WORD:proto}),
PFSENSE_IPv4_SPECIFIC_DATA_ECN (%{BASE16NUM:tos}),(%{INT:ecn}),(%{INT:ttl}),(%{INT:id}),(%{INT:offset}),(%{WORD:flags}),(%{INT:proto_id}),(%{WORD:proto}),
PFSENSE_IPv6_SPECIFIC_DATA (%{BASE16NUM:class}),(%{DATA:flow_label}),(%{INT:hop_limit}),(%{WORD:proto}),(%{INT:proto_id}),
PFSENSE_IP_DATA (%{INT:length}),(%{IP:src_ip}),(%{IP:dest_ip}),
PFSENSE_PROTOCOL_DATA (%{PFSENSE_TCP_DATA}|%{PFSENSE_UDP_DATA}|%{PFSENSE_ICMP_DATA}|%{PFSENSE_CARP_DATA})
PFSENSE_TCP_DATA (%{INT:src_port}),(%{INT:dest_port}),(%{INT:data_length}),(%{WORD:tcp_flags}),(%{INT:sequence_number}),(%{INT:ack_number}),(%{INT:tcp_window}),(%{DATA:urg_data}),(%{DATA:tcp_options})
PFSENSE_UDP_DATA (%{INT:src_port}),(%{INT:dest_port}),(%{INT:data_length})
PFSENSE_ICMP_DATA (%{PFSENSE_ICMP_TYPE}%{PFSENSE_ICMP_RESPONSE})
PFSENSE_ICMP_TYPE (?<icmp_type>(request|reply|unreachproto|unreachport|unreach|timeexceed|paramprob|redirect|maskreply|needfrag|tstamp|tstampreply)),
PFSENSE_ICMP_RESPONSE (%{PFSENSE_ICMP_ECHO_REQ_REPLY}|%{PFSENSE_ICMP_UNREACHPORT}| %{PFSENSE_ICMP_UNREACHPROTO}|%{PFSENSE_ICMP_UNREACHABLE}|%{PFSENSE_ICMP_NEED_FLAG}|%{PFSENSE_ICMP_TSTAMP}|%{PFSENSE_ICMP_TSTAMP_REPLY})
PFSENSE_ICMP_ECHO_REQ_REPLY (%{INT:icmp_echo_id}),(%{INT:icmp_echo_sequence})
PFSENSE_ICMP_UNREACHPORT (%{IP:icmp_unreachport_dest_ip}),(%{WORD:icmp_unreachport_protocol}),(%{INT:icmp_unreachport_port})
PFSENSE_ICMP_UNREACHPROTO (%{IP:icmp_unreach_dest_ip}),(%{WORD:icmp_unreachproto_protocol})
PFSENSE_ICMP_UNREACHABLE (%{GREEDYDATA:icmp_unreachable})
PFSENSE_ICMP_NEED_FLAG (%{IP:icmp_need_flag_ip}),(%{INT:icmp_need_flag_mtu})
PFSENSE_ICMP_TSTAMP (%{INT:icmp_tstamp_id}),(%{INT:icmp_tstamp_sequence})
PFSENSE_ICMP_TSTAMP_REPLY (%{INT:icmp_tstamp_reply_id}),(%{INT:icmp_tstamp_reply_sequence}),(%{INT:icmp_tstamp_reply_otime}),(%{INT:icmp_tstamp_reply_rtime}),(%{INT:icmp_tstamp_reply_ttime})

PFSENSE_CARP_DATA (%{WORD:carp_type}),(%{INT:carp_ttl}),(%{INT:carp_vhid}),(%{INT:carp_version}),(%{INT:carp_advbase}),(%{INT:carp_advskew})

DHCPD (%{DHCPDISCOVER}|%{DHCPOFFER}|%{DHCPREQUEST}|%{DHCPACK}|%{DHCPINFORM}|%{DHCPRELEASE})
DHCPDISCOVER %{WORD:dhcp_action} from %{COMMONMAC:dhcp_client_mac}%{SPACE}(\(%{GREEDYDATA:dhcp_client_hostname}\))? via (?<dhcp_client_vlan>[0-9a-z_]*)(: %{GREEDYDATA:dhcp_load_balance})?
DHCPOFFER %{WORD:dhcp_action} on %{IPV4:dhcp_client_ip} to %{COMMONMAC:dhcp_client_mac}%{SPACE}(\(%{GREEDYDATA:dhcp_client_hostname}\))? via (?<dhcp_client_vlan>[0-9a-z_]*)
DHCPREQUEST %{WORD:dhcp_action} for %{IPV4:dhcp_client_ip}%{SPACE}(\(%{IPV4:dhcp_ip_unknown}\))? from %{COMMONMAC:dhcp_client_mac}%{SPACE}(\(%{GREEDYDATA:dhcp_client_hostname}\))? via (?<dhcp_client_vlan>[0-9a-z_]*)(: %{GREEDYDATA:dhcp_request_message})?
DHCPACK %{WORD:dhcp_action} on %{IPV4:dhcp_client_ip} to %{COMMONMAC:dhcp_client_mac}%{SPACE}(\(%{GREEDYDATA:dhcp_client_hostname}\))? via (?<dhcp_client_vlan>[0-9a-z_]*)
DHCPINFORM %{WORD:dhcp_action} from %{IPV4:dhcp_client_ip} via %(?<dhcp_client_vlan>[0-9a-z_]*)
DHCPRELEASE %{WORD:dhcp_action} of %{IPV4:dhcp_client_ip} from %{COMMONMAC:dhcp_client_mac}%{SPACE}(\(%{GREEDYDATA:dhcp_client_hostname}\))? via</dhcp_client_vlan></dhcp_client_vlan></dhcp_client_vlan></dhcp_client_vlan></dhcp_client_vlan></icmp_type></bernd@bzed.de>

              Check out my post here: https://forum.pfsense.org/index.php?topic=120937.msg733487#msg733487
              all I did was add a "?" to get it working on 2.4.0.

              Let me know if that works for you!

              If not, there are some troubleshooting tips in the linked post. Those combined with the syslog format should let you sort out your grok file if the easy change I made doesn't help you.

              1 Reply Last reply Reply Quote 0
              • I
                idealanthony
                last edited by

                @pfBasic

                It's alive! I really appreciate the assist! Yup, that one '?' fixed the parsing.

                Also, I'm curious as you used the same setup article.  Were you able to get the geoip capability working?

                I'm thinking it has something to do with the visualization.json.  I compared both the one provided by http://pfelk.3ilson.com/ and the one mentioned in the below post.  Using the revised one from @Starfleet I was able to get the default dashboard setup /wo heatmap or geoip.  Starfleet's version is missing a country and top country section in the json, but adding them in doesn't help.

                @Starfleet:

                Ok, so it looks like ELK changed the way some mappings worked in their latest upgrade. This visualization file will get everything working but the geoip related items. Rename as json and import and it should work.

                If you are interested in the changes, use a diff program to compare the two files.

                (in short, the names of items needed to be changed to name.raw instead of name)

                https://github.com/elastic/elasticsearch/issues/15267

                (Note, you need to be logged in to see the attached file. Sorry, didn't realize that until I looked at this while logged out.)

                1 Reply Last reply Reply Quote 0
                • P
                  pfBasic Banned
                  last edited by

                  Yes I got the dashboard working as advertised. I just had to refresh a few things and reimport the visualizations Jason are some files were successfully parsed and it worked! Check out my linked post, it says more specifically the steps I took.

                  1 Reply Last reply Reply Quote 0
                  • First post
                    Last post
                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.