PfSense syslog and ELSA

BBcan177

Has anyone had any success getting pfSense Syslogs into ELSA

https://code.google.com/p/enterprise-log-search-and-archive/

I found this article below but I cant get the script to create the file.

http://www.securitygrit.com/2013/03/pfsense-into-elsa.html

jimp

If you're on 2.1, add this patch:
http://files.pfsense.org/jimp/patches/pf-log-oneline-option.diff

And then check the box on the system log settings to force the firewall logs to one line.

If you're on 2.0.x, use this patch instead:
http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.0.x.diff

BBcan177

Thank You,

Are there instruction on installing this patch?

jimp

Use the System Patches package.

BBcan177

Thanks Jimp,

BBcan177

Hi Jimp,

I have updated pfSense with the patch and I am forwarding the logs to an ELSA server (Also tested with Kiwi Syslog) and I am noticing the following:

Here is the output from pfSense for a "Block" in the Firewall log :

01-05-2014 20:08:26 Local0.Info xxx.xxx.xxx.xxx Jan 5 20:08:27 pf: 00:00:00.570566 rule 5/0(match): block in on rl0: (tos 0x0, ttl 114, id 37724, offset 0, flags [none], proto TCP (6), length 48) 185.25.184.184.28796 > xxx.xxx.xxx.xxx.5900: Flags, cksum 0x8635 (correct), seq 979907292, win 65535, options [mss 1460,nop,nop,sackOK], length 0

Is there any way to format this to show the the name of the Rule ie "pfBlocker ET" instead of rule 5/0(match).

Elsa treats this as a single entry and can't pivot on any of the individual contents, ie Src/Dst IP address, or port. I am also investigating if the ELSA parser can decipher this better.

Thanks.

jimp

There isn't a way to get the rule description there.

Parsing might be a bit tricky but ELSA will need to know how to parse the pf or pfSense logs.

Not directly relevant, but in 2.2 we're completely reworking the raw filter logs to be in a nice, easily to parse CSV-like format.

BBcan177

Thanks Jimp,

I will try to see if ELSA can parse the file better.

Looking forward to 2.2.

Thanks

BBcan177

When I modify a Firewall:Rule and hit "apply", I notice that all of the previous log entries in the pfSense Firewall log displays incorrect "Rule Names"?

All new entries in the Log will show the correct Rule name, but the older logs remain indicating an incorrect rule name?

Has anyone seen this issue before? I have tested this on two different boxes with the same issue.

I am using pfBlocker with aliases on pfSense v2.1.

I am also using a patch (http://files.pfsense.org/jimp/patches/pf-log-oneline-option.diff) to allow one line entries for a remote syslog.

Thanks

jimp

@BBcan17:

When I modify a Firewall:Rule and hit "apply", I notice that all of the previous log entries in the pfSense Firewall log displays incorrect "Rule Names"?

All new entries in the Log will show the correct Rule name, but the older logs remain indicating an incorrect rule name?

That is normal. When the rules reload, the rule numbers can change and the descriptions no longer match up. We're working on a way around that for 2.2 as well.

BBcan177

Thanks Jim,

I look forward to the next version.

fsom

Hello,

I'm using pcSense 2.1 with Elsa and your patches and it's working fine for TCP and UDP packages. However, ICMP doesn't get parsed. I was trying to locate the problem the last days but without luck. If pfSense is sending an ICMP log to Elsa, I receive this in my log:

00:00:02.504973 rule 317/0(match): block in on bge1: (tos 0x0, ttl 64, id 315, offset 0, flags [none], proto ICMP (1), length 84) 172.27.0.36 > 172.27.1.3: ICMP echo request, id 44331, seq 0, length 64
host=172.27.1.3 program=pf class=NONE

I was looking at the parsing rules and run some tests by hand, it always get parsed. I then send the same message via netcat:

echo '<14>Jan 22 10:57:48 pf: 00:00:02.504973 rule 317/0(match): block in on bge1: (tos 0x0, ttl 64, id 315, offset 0, flags [none], proto ICMP (1), length 84) 172.27.0.36 > 172.27.1.3: ICMP echo request, id 44331, seq 0, length 64' | nc -v -u -w 0 172.27.50.77 514

and suddenly it gets parsed:

00:00:02.504973 rule 317/0(match): block in on bge1: (tos 0x0, ttl 64, id 315, offset 0, flags [none], proto ICMP (1), length 84) 172.27.0.36 > 172.27.1.3: ICMP echo request, id 44331, seq 0, length 64
host=172.27.0.162 program=pf class=FIREWALL_ACCESS_DENY proto=ICMP srcip=172.27.0.36 srcport=0 dstip=172.27.1.3 dstport=0 o_int=bge1 i_int= access_group=

Does anyone else has this problem?
Thanks in advance,
fsom

BBcan177

What are you using as the search Query in ELSA? Did you look at the Google ELSA group https://groups.google.com/forum/#!forum/enterprise-log-search-and-archive

Try the following example Querys -

class=FIREWALL_CONNECTION_END groupby:host
class=FIREWALL_ACCESS_DENY groupby:host
class=FIREWALL_ACCESS_DENY icmp or tcp or udp dstip="x.x.x.x" groupby:srcip

Are you using a parser similar to this?

http://www.securitygrit.com/2013/03/pfsense-into-elsa.html

I have added the ruleset to the end of patterndb.xml between <patterndb></patterndb>and ran "service syslog-ng restart"

The syslogs are going into the existing database ELSA class/fields "FIREWALL_ACCESS_LOG" and "FIREWALL_CONNECTION_END".

Make sure you Select "Grid Display" to see it parsed properly.

fsom

Hello,
Thank you for your reply. ICMP packages coming from pfSense are going into the CLASS=NONE group, TCP/UDP into the FIREWALL_*. I tried your example queries but the only ICMP package I found was the one I spoofed via netcat. All the other ICMP packages are in the CLASS=none group. The parser I'm using is from http://www.securitygrit.com/2013/03/pfsense-into-elsa.html, before I added it, everything from pfSense was going into CLASS=NONE, after I applied and restarted syslogd they got assigned to the FIREWALL_ACCESS_DENY or FIREWALL_CONNECTION_END class. The thing I don't understand is why ICMP packages aren't going into these classes when coming from pfSense.

The parsing itself is working:
[root@elsa ~]# /usr/local/syslog-ng/bin/pdbtool match -p /usr/local/elsa/node/conf/patterndb.xml -P pf -M "00:03:54.505077 rule 50/0(match): block in on bge0: (tos 0x0, ttl 107, id 25728, offset 0, flags [none], proto ICMP (1), length 28) 172.16.250.251 > xxx.yyy.21.150: ICMP echo request, id 768, seq 51812, length 8"
MESSAGE=00:03:54.505077 rule 50/0(match): block in on bge0: (tos 0x0, ttl 107, id 25728, offset 0, flags [none], proto ICMP (1), length 28) 172.16.250.251 > 194.180.21.150: ICMP echo request, id 768, seq 51812, length 8
PROGRAM=pf
.classifier.class=2
.classifier.rule_id=2
s0=bge0
i0=ICMP
i1=172.16.250.251
i3=xxx.yyy.21.150

Does your ELSA handle ICMP like UDP/TCP ?

Thanks,
fsom

Cino

I guess I've been sleeping, been waiting a long time for FW logs to be parse on a single line. Thanks Jim for the patch!! I'll have to give this a try or just wait for 2.2 ;)

BBcan177

@fsom:

ICMP packages coming from pfSense are going into the CLASS=NONE group

I tested an icmp block rule and can confirm that ELSA is logging this as "class=none" instead of "class=FIREWALL_ACCESS_DENY"

You should post a request to the Google ELSA group.

BBcan177

Hi Jimp,

I have followed the following document -

https://doc.pfsense.org/index.php/Why_can%27t_I_query_SNMP,_use_syslog,_NTP,_or_other_services_initiated_by_the_firewall_itself_over_IPsec_VPN%3F

And it was working for about a month or so, but recently any Ubuntu machine that is in a Site-Site Ipsec tunnel can not get any connectivity when this Manual route is implemented. All other traffic from Windows based machines seems to be unaffected by this, Including the Syslogs from pfSense being directed to a remote syslog server.

I have tried this on two different tunnels with different Ubuntu machines.

When I delete the manual route and restart apinger, it doesn't clear the issue. A full reboot of the pfSense box seems to fix it.

Do you have any suggestions?

Thanks for your help.

jimp

Ditch the route, update to 2.1.1, then Status > System Logs, Settings tab, pick LAN for the source address. :-)

BBcan177

Is 2.1.1 stable enough now? If there are no other choices, i guess I will test it out.

This is the reason why they invented "Alcohol!!"

jimp

It's still got a couple issues yet but it may be good enough for most uses.

Otherwise track down the commit(s) for the syslog source address selection and apply them manually.

BBcan177

@jimp:

Otherwise track down the commit(s) for the syslog source address selection and apply them manually.

Hi Jim,

I found this revision, https://redmine.pfsense.org/projects/pfsense/repository/revisions/53c5407e646028a003b2765a87dd3316b21a9497

Would the steps involved be to replace the two files with the ones on this site.

/etc/inc/system.ini
/usr/local/www/diag_logs_setings.php

jimp

You should be able to use the system patches package to apply that patch. Taking the whole files might get other changes that would have unintended consequences.

BBcan177

Do you have a link that you could share?

jimp

http://doc.pfsense.org/index.php/System_Patches

BBcan177

@jimp:

Ditch the route, update to 2.1.1, then Status > System Logs, Settings tab, pick LAN for the source address. :-)

Hi Jimp,

Thanks for the direction on getting the Syslog to work thru the VPN tunnel. Works well!

I believe that "System:NOTIFICATION / SMTP" has this same issue.

I have "DNS Forwarder" set to forward "mail.domain.com" to a 10.10.10.5, I have the Notification "Email server" set to "mail.domain.com" and the emails never go out.

If I change the "Email Server" in Notification to 10.10.10.5, the emails don't go out.

When i change "mail.domain.com" to the External IP address of the mail server, the email go thru, as this sends the email out thru the internet to get to my mail server.

Would prefer the mail to stay within my VPN tunnel if possible.

jimp

Not relevant to this thread, but that would require a route, the smtp client doesn't have a way to force the source address. Start a new thread if you want to discuss alternatives.

BBcan177

@jimp:

Not relevant to this thread, but that would require a route, the smtp client doesn't have a way to force the source address. Start a new thread if you want to discuss alternatives.

Hi Jimp,

I posted my question to the group without any replies, would you have any suggestions?

https://forum.pfsense.org/index.php/topic,72149.msg394065.html#msg394065

thhi

The logs from pfsense for ICMP packets (and ESP, IGMP maybe other protocols as well) have more than one space in front of the ip address part (after applying the "oneline" patch). Therefore you need additional patterns in the patterndb.xml file of elsa, i.e.

for "class 2" - (FIREWALL_ACCESS_DENY)

<pattern>@ESTRING:: block in on @@ESTRING:s0:: (@@ESTRING::proto @@ESTRING:i0: @@ESTRING::   @@IPv4:i1:@@ESTRING:: @@ESTRING:: @@IPv4:i3:@@ANYSTRING@</pattern>

and for "class 3" - (FIREWALL_CONNECTION_END)

<pattern>@ESTRING:: pass in on @@ESTRING:s0:: (@@ESTRING::proto @@ESTRING:i0: @@ESTRING::   @@IPv4:i1:@@ESTRING:: @@ESTRING:: @@IPv4:i3:@@ANYSTRING@</pattern>

There is an additional 'problem' with the pfSense logs in elsa:
The delimiter between ip addresses an the port numbers is a "dot". This is no valid delimiter for the sphinx search engine of elsa. So the search for an ip address isn't working in elsa.

To solve this issue I have added an addition sed command for external logging in pfsense in
/etc/inc/filter.inc to substitute this dots by a colon:

$oneline = isset($config['syslog']['pflog_oneline']) ? " | /usr/bin/sed -l -e 'N;s/\\n //;P;D;' | /usr/bin/sed -l -e 's/\\(.* \
\)\\(\\([0-9]\\{1,3\\}\\.\\)\\{3\\}[0-9]\\{1,3\\}\\)\\.\\([0-9]\\{1,5\\}\\)\\( .* \\)\\(\\([0-9]\\{1,3\\}\\.\\)\\{3\\}[0-9]\\{1,3\\}\\)
\\.\\([0-9]\\{1,5\\}\\)\\(.*\\)/\\1\\2:\\4\\5\\6:\\8\\9/' " : " ";

Maybe there is a better solution.

BBcan177

@thhi:

The logs from pfsense for ICMP packets (and ESP, IGMP maybe other protocols as well) have more than one space in front of the ip address part (after applying the "oneline" patch). Therefore you need additional patterns in the patterndb.xml file of elsa, i.e.

Maybe there is a better solution.

Did you try to post to the ELSA Google Group? Maybe they would have some suggestions?

https://groups.google.com/forum/#!forum/enterprise-log-search-and-archive

@jimp:

If you're on 2.1, add this patch:
http://files.pfsense.org/jimp/patches/pf-log-oneline-option.diff

And then check the box on the system log settings to force the firewall logs to one line.

If you're on 2.0.x, use this patch instead:
http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.0.x.diff

Running 2.1.1. Adding that patch always shows that it cannot be applied. Any tips?

jimp

Try this one:

http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.1.1.diff

BBcan177

@jimp:

Try this one:

http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.1.1.diff

Loading this page comes up with a 403 Forbidden error?

jimp

Try again now, I just noticed and fixed that

BBcan177

Works.. Thanks Jim.

@jimp:

Try this one:

http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.1.1.diff

Can apply that patch now, but it doesn't work. Logs are still split on 2 lines.

Diagnostics>Command Prompt:


$ /etc/rc.d/syslogd restart
Stopping syslogd.
Starting syslogd.

log sample (sanitized)


2014-04-10T14:38:53+03:00 somehost pf: 00:00:31.932924 rule 3/0(match): block in on em1: (tos 0x0, ttl 54, id 48381, offset 0, flags [DF], proto TCP (6), length 60)
2014-04-10T14:38:53+03:00 somehost pf:     xxx.xxx.xxx.xxx.53883 > yyy.yyy.yyy.yyy.80: Flags [s], cksum 0x158f (correct), seq 1628583023, win 14600, options [mss 1460,sackOK,TS val 2583988370 ecr 0,nop,wscale 7], length 0
[/s]

jimp

Did you enable the option in the system log settings after applying the patch? It doesn't default to on.

@jimp:

Did you enable the option in the system log settings after applying the patch? It doesn't default to on.

Knew I would brainfart at some point today. Forgot about that setting, will try when I get back and report back.

Tested after upgrading to 2.1.2 and working as expected after enabling it in the settings. Thank you

miloman

@jimp:

Try this one:

http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.1.1.diff

any love for pfsense 2.1.2 and 2.1.3?

biggsy

That patch works for me on 2.1.3.

BBcan177

I have had some issues with this patch also. Worked on most machines but one of them I had to remove the patch, reboot and then re-enable the patch to get it to work?