SOLVED - How to get raw Rsyslog output? 2.4.0 BETA - SOLVED

pfBasic · Jul 9, 2017, 5:14 PM

I'm trying to setup an ELK stack. Everything is up and running, but the filter I used just keeps tagging all of my logs with "_grokparsefailure" and "_geoipfailure", I'm not getting anything usable out of my logs.

How can I get the raw remote-syslog output from pfSense.

I'm trying to see exactly what the ELK stack is receiving before anything is processed.

Also - I found this: https://doc.pfsense.org/index.php/Filter_Log_Format_for_pfSense_2.2

It shows the pfSense 2.2 filterlog format as```
<timestamp><hostname>filterlog:</hostname></timestamp>


Is that still the same in 2.4.0? Has the Rsyslog format changed since 2.2 (or 2.3)? - If so, what is it in 2.4.0 BETA?

pfBasic · Jul 11, 2017, 6:06 PM

Bump, anyone?

pfBasic · Jul 12, 2017, 5:12 PM

I was able to figure most of it out, for anyone who needs it in the future here is what I could figure out for syslog format in 2.4.0 BETA.

46,,,1000000117,igb0,match,block,in,4,0x0,,245,6029,0,none,6,tcp,40,80.82.65.231,192.168.1.1,56805,3983,0,S,2650484143,,1024,,

46 = Rule #
, = nothing?
, = nothing?
1000000117 = Tracker
igb0 = iface
match = Reason
block = action
in = direction
4 = ip_ver
0x0 = tos (type of service)
, = nothing?
245 = ttl (time to live)
6029 = id
0 = offset OR data_length?
none = flags
6 = proto_id
tcp = proto
40 = length
80.82.65.231 = src_ip
192.168.1.1 - dest_ip
56805 = src_port
3983 = dest_port
0 = offset OR data_length?
S = unused?
2650484143 = unused?
, = nothing?
1024 = unused?
, = nothing?

idealanthony · Jul 16, 2017, 2:02 AM

@pfBasic I'm in the same boat with trying to get ELK working with PFSense.

Were you able to get everything up and running? If so, I'd be really interested in seeing you grok file for logstash

I'm using the article found here http://pfelk.3ilson.com/

It may be a poor choice, but this is my first attempt at an ELK stack. I've got everything (Kibana, Elasticsearch, Logstash) installed, and I can see data getting to Kibana, so from a networking standpoint things appear to be working.

I'm falling flat on the parsing.

The message tag seems to contain all the data from PFSense in csv format

message: 5,,,1000000103,igb0,match,block,in,4,0x0,,57,33381,0,DF,17,udp,40,184.105.139.124,172.92.3.122,11775,123,20
``` 

But the tags aren't getting associated

tags: PFSense, firewall, _grokparsefailure


The tutorial I used was for PFsense 2.3 and I'm assuming my pattern/grok file is to blame, but I'm not actually sure how to modify the syntax to fix it.

I've searched for a 2.4 grok pfsense pattern file, but can't seem to find one online.

I'd appreciate any help

Contents of Grok Below

GROK match pattern for logstash.conf filter: %{PFSENSE_LOG_DATA}%{PFSENSE_IP_SPECIFIC_DATA}%{PFSENSE_IP_DATA}%{PFSENSE_PROTOCOL_DATA}

GROK Custom Patterns (add to patterns directory and reference in GROK filter for pfSense events):

GROK Patterns for pfSense 2.3 Logging Format

Created 27 Jan 2015 by J. Pisano (Handles TCP, UDP, and ICMP log entries)

Edited 14 Feb 2015 by Elijah Paul elijah.paul@gmail.com

Edited 10 Mar 2015 by Bernd Zeimetz bernd@bzed.de# taken from https://gist.github.com/elijahpaul/f5f32d4e914dcb7fedd2

- adding PFSENSE_ prefix

- adding carp patterns

Usage: Use with following GROK match pattern

%{PFSENSE_LOG_DATA}%{PFSENSE_IP_SPECIFIC_DATA}%{PFSENSE_IP_DATA}%{PFSENSE_PROTOCOL_DATA}

PFSENSE_LOG_DATA (%{INT:rule}),(%{INT:sub_rule}),,(%{INT:tracker}),(%{WORD:iface}),(%{WORD:reason}),(%{WORD:action}),(%{WORD:direction}),(%{INT:ip_ver}),
PFSENSE_IP_SPECIFIC_DATA (%{PFSENSE_IPv4_SPECIFIC_DATA}|%{PFSENSE_IPv6_SPECIFIC_DATA})
PFSENSE_IPv4_SPECIFIC_DATA (%{BASE16NUM:tos}),,(%{INT:ttl}),(%{INT:id}),(%{INT:offset}),(%{WORD:flags}),(%{INT:proto_id}),(%{WORD:proto}),
PFSENSE_IPv4_SPECIFIC_DATA_ECN (%{BASE16NUM:tos}),(%{INT:ecn}),(%{INT:ttl}),(%{INT:id}),(%{INT:offset}),(%{WORD:flags}),(%{INT:proto_id}),(%{WORD:proto}),
PFSENSE_IPv6_SPECIFIC_DATA (%{BASE16NUM:class}),(%{DATA:flow_label}),(%{INT:hop_limit}),(%{WORD:proto}),(%{INT:proto_id}),
PFSENSE_IP_DATA (%{INT:length}),(%{IP:src_ip}),(%{IP:dest_ip}),
PFSENSE_PROTOCOL_DATA (%{PFSENSE_TCP_DATA}|%{PFSENSE_UDP_DATA}|%{PFSENSE_ICMP_DATA}|%{PFSENSE_CARP_DATA})
PFSENSE_TCP_DATA (%{INT:src_port}),(%{INT:dest_port}),(%{INT:data_length}),(%{WORD:tcp_flags}),(%{INT:sequence_number}),(%{INT:ack_number}),(%{INT:tcp_window}),(%{DATA:urg_data}),(%{DATA:tcp_options})
PFSENSE_UDP_DATA (%{INT:src_port}),(%{INT:dest_port}),(%{INT:data_length})
PFSENSE_ICMP_DATA (%{PFSENSE_ICMP_TYPE}%{PFSENSE_ICMP_RESPONSE})
PFSENSE_ICMP_TYPE (?<icmp_type>(request|reply|unreachproto|unreachport|unreach|timeexceed|paramprob|redirect|maskreply|needfrag|tstamp|tstampreply)),
PFSENSE_ICMP_RESPONSE (%{PFSENSE_ICMP_ECHO_REQ_REPLY}|%{PFSENSE_ICMP_UNREACHPORT}| %{PFSENSE_ICMP_UNREACHPROTO}|%{PFSENSE_ICMP_UNREACHABLE}|%{PFSENSE_ICMP_NEED_FLAG}|%{PFSENSE_ICMP_TSTAMP}|%{PFSENSE_ICMP_TSTAMP_REPLY})
PFSENSE_ICMP_ECHO_REQ_REPLY (%{INT:icmp_echo_id}),(%{INT:icmp_echo_sequence})
PFSENSE_ICMP_UNREACHPORT (%{IP:icmp_unreachport_dest_ip}),(%{WORD:icmp_unreachport_protocol}),(%{INT:icmp_unreachport_port})
PFSENSE_ICMP_UNREACHPROTO (%{IP:icmp_unreach_dest_ip}),(%{WORD:icmp_unreachproto_protocol})
PFSENSE_ICMP_UNREACHABLE (%{GREEDYDATA:icmp_unreachable})
PFSENSE_ICMP_NEED_FLAG (%{IP:icmp_need_flag_ip}),(%{INT:icmp_need_flag_mtu})
PFSENSE_ICMP_TSTAMP (%{INT:icmp_tstamp_id}),(%{INT:icmp_tstamp_sequence})
PFSENSE_ICMP_TSTAMP_REPLY (%{INT:icmp_tstamp_reply_id}),(%{INT:icmp_tstamp_reply_sequence}),(%{INT:icmp_tstamp_reply_otime}),(%{INT:icmp_tstamp_reply_rtime}),(%{INT:icmp_tstamp_reply_ttime})

PFSENSE_CARP_DATA (%{WORD:carp_type}),(%{INT:carp_ttl}),(%{INT:carp_vhid}),(%{INT:carp_version}),(%{INT:carp_advbase}),(%{INT:carp_advskew})

DHCPD (%{DHCPDISCOVER}|%{DHCPOFFER}|%{DHCPREQUEST}|%{DHCPACK}|%{DHCPINFORM}|%{DHCPRELEASE})
DHCPDISCOVER %{WORD:dhcp_action} from %{COMMONMAC:dhcp_client_mac}%{SPACE}((%{GREEDYDATA:dhcp_client_hostname}))? via (?<dhcp_client_vlan>[0-9a-z_])(: %{GREEDYDATA:dhcp_load_balance})?
DHCPOFFER %{WORD:dhcp_action} on %{IPV4:dhcp_client_ip} to %{COMMONMAC:dhcp_client_mac}%{SPACE}((%{GREEDYDATA:dhcp_client_hostname}))? via (?<dhcp_client_vlan>[0-9a-z_])
DHCPREQUEST %{WORD:dhcp_action} for %{IPV4:dhcp_client_ip}%{SPACE}((%{IPV4:dhcp_ip_unknown}))? from %{COMMONMAC:dhcp_client_mac}%{SPACE}((%{GREEDYDATA:dhcp_client_hostname}))? via (?<dhcp_client_vlan>[0-9a-z_])(: %{GREEDYDATA:dhcp_request_message})?
DHCPACK %{WORD:dhcp_action} on %{IPV4:dhcp_client_ip} to %{COMMONMAC:dhcp_client_mac}%{SPACE}((%{GREEDYDATA:dhcp_client_hostname}))? via (?<dhcp_client_vlan>[0-9a-z_])
DHCPINFORM %{WORD:dhcp_action} from %{IPV4:dhcp_client_ip} via %(?<dhcp_client_vlan>[0-9a-z_]*)
DHCPRELEASE %{WORD:dhcp_action} of %{IPV4:dhcp_client_ip} from %{COMMONMAC:dhcp_client_mac}%{SPACE}((%{GREEDYDATA:dhcp_client_hostname}))? via</dhcp_client_vlan></dhcp_client_vlan></dhcp_client_vlan></dhcp_client_vlan></dhcp_client_vlan></icmp_type>/bernd@bzed.de

pfBasic · Jul 16, 2017, 2:22 AM

Give this grok pattern a shot:


# GROK match pattern for logstash.conf filter: %{PFSENSE_LOG_DATA}%{PFSENSE_IP_SPECIFIC_DATA}%{PFSENSE_IP_DATA}%{PFSENSE_PROTOCOL_DATA}
# GROK Custom Patterns (add to patterns directory and reference in GROK filter for pfSense events):
# GROK Patterns for pfSense 2.3 Logging Format
#
# Created 27 Jan 2015 by J. Pisano (Handles TCP, UDP, and ICMP log entries)
# Edited 14 Feb 2015 by Elijah Paul elijah.paul@gmail.com
# Edited 10 Mar 2015 by Bernd Zeimetz <bernd@bzed.de># taken from https://gist.github.com/elijahpaul/f5f32d4e914dcb7fedd2
# - adding PFSENSE_ prefix
# - adding carp patterns
#
# Usage: Use with following GROK match pattern
#
# %{PFSENSE_LOG_DATA}%{PFSENSE_IP_SPECIFIC_DATA}%{PFSENSE_IP_DATA}%{PFSENSE_PROTOCOL_DATA}

PFSENSE_LOG_DATA (%{INT:rule}),(%{INT:sub_rule})?,,(%{INT:tracker}),(%{WORD:iface}),(%{WORD:reason}),(%{WORD:action}),(%{WORD:direction}),(%{INT:ip_ver}),
PFSENSE_IP_SPECIFIC_DATA (%{PFSENSE_IPv4_SPECIFIC_DATA}|%{PFSENSE_IPv6_SPECIFIC_DATA})
PFSENSE_IPv4_SPECIFIC_DATA (%{BASE16NUM:tos}),,(%{INT:ttl}),(%{INT:id}),(%{INT:offset}),(%{WORD:flags}),(%{INT:proto_id}),(%{WORD:proto}),
PFSENSE_IPv4_SPECIFIC_DATA_ECN (%{BASE16NUM:tos}),(%{INT:ecn}),(%{INT:ttl}),(%{INT:id}),(%{INT:offset}),(%{WORD:flags}),(%{INT:proto_id}),(%{WORD:proto}),
PFSENSE_IPv6_SPECIFIC_DATA (%{BASE16NUM:class}),(%{DATA:flow_label}),(%{INT:hop_limit}),(%{WORD:proto}),(%{INT:proto_id}),
PFSENSE_IP_DATA (%{INT:length}),(%{IP:src_ip}),(%{IP:dest_ip}),
PFSENSE_PROTOCOL_DATA (%{PFSENSE_TCP_DATA}|%{PFSENSE_UDP_DATA}|%{PFSENSE_ICMP_DATA}|%{PFSENSE_CARP_DATA})
PFSENSE_TCP_DATA (%{INT:src_port}),(%{INT:dest_port}),(%{INT:data_length}),(%{WORD:tcp_flags}),(%{INT:sequence_number}),(%{INT:ack_number}),(%{INT:tcp_window}),(%{DATA:urg_data}),(%{DATA:tcp_options})
PFSENSE_UDP_DATA (%{INT:src_port}),(%{INT:dest_port}),(%{INT:data_length})
PFSENSE_ICMP_DATA (%{PFSENSE_ICMP_TYPE}%{PFSENSE_ICMP_RESPONSE})
PFSENSE_ICMP_TYPE (?<icmp_type>(request|reply|unreachproto|unreachport|unreach|timeexceed|paramprob|redirect|maskreply|needfrag|tstamp|tstampreply)),
PFSENSE_ICMP_RESPONSE (%{PFSENSE_ICMP_ECHO_REQ_REPLY}|%{PFSENSE_ICMP_UNREACHPORT}| %{PFSENSE_ICMP_UNREACHPROTO}|%{PFSENSE_ICMP_UNREACHABLE}|%{PFSENSE_ICMP_NEED_FLAG}|%{PFSENSE_ICMP_TSTAMP}|%{PFSENSE_ICMP_TSTAMP_REPLY})
PFSENSE_ICMP_ECHO_REQ_REPLY (%{INT:icmp_echo_id}),(%{INT:icmp_echo_sequence})
PFSENSE_ICMP_UNREACHPORT (%{IP:icmp_unreachport_dest_ip}),(%{WORD:icmp_unreachport_protocol}),(%{INT:icmp_unreachport_port})
PFSENSE_ICMP_UNREACHPROTO (%{IP:icmp_unreach_dest_ip}),(%{WORD:icmp_unreachproto_protocol})
PFSENSE_ICMP_UNREACHABLE (%{GREEDYDATA:icmp_unreachable})
PFSENSE_ICMP_NEED_FLAG (%{IP:icmp_need_flag_ip}),(%{INT:icmp_need_flag_mtu})
PFSENSE_ICMP_TSTAMP (%{INT:icmp_tstamp_id}),(%{INT:icmp_tstamp_sequence})
PFSENSE_ICMP_TSTAMP_REPLY (%{INT:icmp_tstamp_reply_id}),(%{INT:icmp_tstamp_reply_sequence}),(%{INT:icmp_tstamp_reply_otime}),(%{INT:icmp_tstamp_reply_rtime}),(%{INT:icmp_tstamp_reply_ttime})

PFSENSE_CARP_DATA (%{WORD:carp_type}),(%{INT:carp_ttl}),(%{INT:carp_vhid}),(%{INT:carp_version}),(%{INT:carp_advbase}),(%{INT:carp_advskew})

DHCPD (%{DHCPDISCOVER}|%{DHCPOFFER}|%{DHCPREQUEST}|%{DHCPACK}|%{DHCPINFORM}|%{DHCPRELEASE})
DHCPDISCOVER %{WORD:dhcp_action} from %{COMMONMAC:dhcp_client_mac}%{SPACE}(\(%{GREEDYDATA:dhcp_client_hostname}\))? via (?<dhcp_client_vlan>[0-9a-z_]*)(: %{GREEDYDATA:dhcp_load_balance})?
DHCPOFFER %{WORD:dhcp_action} on %{IPV4:dhcp_client_ip} to %{COMMONMAC:dhcp_client_mac}%{SPACE}(\(%{GREEDYDATA:dhcp_client_hostname}\))? via (?<dhcp_client_vlan>[0-9a-z_]*)
DHCPREQUEST %{WORD:dhcp_action} for %{IPV4:dhcp_client_ip}%{SPACE}(\(%{IPV4:dhcp_ip_unknown}\))? from %{COMMONMAC:dhcp_client_mac}%{SPACE}(\(%{GREEDYDATA:dhcp_client_hostname}\))? via (?<dhcp_client_vlan>[0-9a-z_]*)(: %{GREEDYDATA:dhcp_request_message})?
DHCPACK %{WORD:dhcp_action} on %{IPV4:dhcp_client_ip} to %{COMMONMAC:dhcp_client_mac}%{SPACE}(\(%{GREEDYDATA:dhcp_client_hostname}\))? via (?<dhcp_client_vlan>[0-9a-z_]*)
DHCPINFORM %{WORD:dhcp_action} from %{IPV4:dhcp_client_ip} via %(?<dhcp_client_vlan>[0-9a-z_]*)
DHCPRELEASE %{WORD:dhcp_action} of %{IPV4:dhcp_client_ip} from %{COMMONMAC:dhcp_client_mac}%{SPACE}(\(%{GREEDYDATA:dhcp_client_hostname}\))? via</dhcp_client_vlan></dhcp_client_vlan></dhcp_client_vlan></dhcp_client_vlan></dhcp_client_vlan></icmp_type></bernd@bzed.de>

Check out my post here: https://forum.pfsense.org/index.php?topic=120937.msg733487#msg733487
all I did was add a "?" to get it working on 2.4.0.

Let me know if that works for you!

If not, there are some troubleshooting tips in the linked post. Those combined with the syslog format should let you sort out your grok file if the easy change I made doesn't help you.

idealanthony · Jul 16, 2017, 4:22 AM

@pfBasic

It's alive! I really appreciate the assist! Yup, that one '?' fixed the parsing.

Also, I'm curious as you used the same setup article. Were you able to get the geoip capability working?

I'm thinking it has something to do with the visualization.json. I compared both the one provided by http://pfelk.3ilson.com/ and the one mentioned in the below post. Using the revised one from @Starfleet I was able to get the default dashboard setup /wo heatmap or geoip. Starfleet's version is missing a country and top country section in the json, but adding them in doesn't help.

@Starfleet:

Ok, so it looks like ELK changed the way some mappings worked in their latest upgrade. This visualization file will get everything working but the geoip related items. Rename as json and import and it should work.

If you are interested in the changes, use a diff program to compare the two files.

(in short, the names of items needed to be changed to name.raw instead of name)

https://github.com/elastic/elasticsearch/issues/15267

(Note, you need to be logged in to see the attached file. Sorry, didn't realize that until I looked at this while logged out.)

pfBasic · Jul 16, 2017, 4:52 AM

Yes I got the dashboard working as advertised. I just had to refresh a few things and reimport the visualizations Jason are some files were successfully parsed and it worked! Check out my linked post, it says more specifically the steps I took.