PfSense syslog and ELSA

thhi

The logs from pfsense for ICMP packets (and ESP, IGMP maybe other protocols as well) have more than one space in front of the ip address part (after applying the "oneline" patch). Therefore you need additional patterns in the patterndb.xml file of elsa, i.e.

for "class 2" - (FIREWALL_ACCESS_DENY)

<pattern>@ESTRING:: block in on @@ESTRING:s0:: (@@ESTRING::proto @@ESTRING:i0: @@ESTRING::   @@IPv4:i1:@@ESTRING:: @@ESTRING:: @@IPv4:i3:@@ANYSTRING@</pattern>

and for "class 3" - (FIREWALL_CONNECTION_END)

<pattern>@ESTRING:: pass in on @@ESTRING:s0:: (@@ESTRING::proto @@ESTRING:i0: @@ESTRING::   @@IPv4:i1:@@ESTRING:: @@ESTRING:: @@IPv4:i3:@@ANYSTRING@</pattern>

There is an additional 'problem' with the pfSense logs in elsa:
The delimiter between ip addresses an the port numbers is a "dot". This is no valid delimiter for the sphinx search engine of elsa. So the search for an ip address isn't working in elsa.

To solve this issue I have added an addition sed command for external logging in pfsense in
/etc/inc/filter.inc to substitute this dots by a colon:

$oneline = isset($config['syslog']['pflog_oneline']) ? " | /usr/bin/sed -l -e 'N;s/\\n //;P;D;' | /usr/bin/sed -l -e 's/\\(.* \
\)\\(\\([0-9]\\{1,3\\}\\.\\)\\{3\\}[0-9]\\{1,3\\}\\)\\.\\([0-9]\\{1,5\\}\\)\\( .* \\)\\(\\([0-9]\\{1,3\\}\\.\\)\\{3\\}[0-9]\\{1,3\\}\\)
\\.\\([0-9]\\{1,5\\}\\)\\(.*\\)/\\1\\2:\\4\\5\\6:\\8\\9/' " : " ";

Maybe there is a better solution.

BBcan177

@thhi:

The logs from pfsense for ICMP packets (and ESP, IGMP maybe other protocols as well) have more than one space in front of the ip address part (after applying the "oneline" patch). Therefore you need additional patterns in the patterndb.xml file of elsa, i.e.

Maybe there is a better solution.

Did you try to post to the ELSA Google Group? Maybe they would have some suggestions?

https://groups.google.com/forum/#!forum/enterprise-log-search-and-archive

A Former User

@jimp:

If you're on 2.1, add this patch:
http://files.pfsense.org/jimp/patches/pf-log-oneline-option.diff

And then check the box on the system log settings to force the firewall logs to one line.

If you're on 2.0.x, use this patch instead:
http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.0.x.diff

Running 2.1.1. Adding that patch always shows that it cannot be applied. Any tips?

jimp

Try this one:

http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.1.1.diff

BBcan177

@jimp:

Try this one:

http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.1.1.diff

Loading this page comes up with a 403 Forbidden error?

jimp

Try again now, I just noticed and fixed that

BBcan177

Works.. Thanks Jim.

A Former User

@jimp:

Try this one:

http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.1.1.diff

Can apply that patch now, but it doesn't work. Logs are still split on 2 lines.

Diagnostics>Command Prompt:


$ /etc/rc.d/syslogd restart
Stopping syslogd.
Starting syslogd.

log sample (sanitized)


2014-04-10T14:38:53+03:00 somehost pf: 00:00:31.932924 rule 3/0(match): block in on em1: (tos 0x0, ttl 54, id 48381, offset 0, flags [DF], proto TCP (6), length 60)
2014-04-10T14:38:53+03:00 somehost pf:     xxx.xxx.xxx.xxx.53883 > yyy.yyy.yyy.yyy.80: Flags [s], cksum 0x158f (correct), seq 1628583023, win 14600, options [mss 1460,sackOK,TS val 2583988370 ecr 0,nop,wscale 7], length 0
[/s]

jimp

Did you enable the option in the system log settings after applying the patch? It doesn't default to on.

A Former User

@jimp:

Did you enable the option in the system log settings after applying the patch? It doesn't default to on.

Knew I would brainfart at some point today. Forgot about that setting, will try when I get back and report back.

Tested after upgrading to 2.1.2 and working as expected after enabling it in the settings. Thank you

miloman

@jimp:

Try this one:

http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.1.1.diff

any love for pfsense 2.1.2 and 2.1.3?

biggsy

That patch works for me on 2.1.3.

BBcan177

I have had some issues with this patch also. Worked on most machines but one of them I had to remove the patch, reboot and then re-enable the patch to get it to work?

ozzbrian

Does this work for 2.5

valshare

don´t work with pfsense 2.2

jimp

The patch is not needed on 2.2.

2.2 changed the native log format to be one line already: https://doc.pfsense.org/index.php/Filter_Log_Format_for_pfSense_2.2

fearnothing

Hi folks, I'm looking into teach myself how to write a syslog-ng parser for pfsense 2.2 output. There are a few fields where I want to query exactly what content might be in them because the documentation page isn't quite specific enough. If someone's already written a parser - great! I'll look at it after I've taught myself to do it first, and maybe learn if I've made any mistakes. But the learning comes first :D

Specifically, I need to know what I might encounter in the following fields:

IP Flags - will it simply be "DNF" or "MF" or could it be more complicated?
ECN - is it going to give the numeric value or a text representation?
URG - will it say "set" or "true" or "urg" or something else?

fearnothing

Further to the above, I'm making good progress with the basic version of pfsense 2.2 firewall syslog events - just about have all IPv4 TCP/UDP working and will start on ICMP and IPv6 after that. One thing I don't have the facility to do at home is CARP, so I would very much appreciate it if people could post me some example CARP event messages for me to make sure my patterns are matching correctly.