Syslog for Traffic (to Splunk)

[arrmo]

Hi,

I would like to log my traffic (up and down through pfSense) to Splunk - using it to monitor traffic by IP Address / Hostname. I think I can make this work, but I admit, I'm sort of stuck on the first step … :-). I can get pfSense to log blocked items, failures, etc. to Splunk (syslog), but how do I log passed traffic? Sorry if this is a dumb question! Do I add a firewall rule to log passing traffic somehow?

Thanks in advance for any thoughts!

[BBcan177]

Hi arrmo,

I use Security Onion which uses ELSA so I assume the Splunk needs a similar setup:

Logs in pfSense are in a two line format, you need to apply a patch to get the logs into a single line format.

From pfSense add "System Patches" from the "Avalable Packages" repository.
  In the System:Patches menu, select "+" and add a new patch

Here is the latest patch:

http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.1.1.diff

Once you have entered the patch details, you need to "Fetch" and than "Apply"

(HELP LINK) https://doc.pfsense.org/index.php/System_Patches

Finally. check the box on the system log settings to force the firewall logs to one line.

You should now be able to send syslogs to Splunk. You also need to edit the Firewall Rules to enable logging if you want to push that data also.

[arrmo]

Thanks so much for the information - makes sense, and very helpful! I tested before applying the patch, but it "failed" as it's for v2.1.1, and I'm running v2.2. So I didn't apply it for now, a bit afraid to give it a go … :(. Thoughts?

BTW, I think I found an option that enables logging of passing data ... in the System Log settings, there is an option for "Log packets matched from the default pass rules put in the ruleset". Make sense?

Thanks again!

[BBcan177]

I haven't tried it on 2.2..

Did you put the URL in the "URL/Commit" Box? Ignore Whitespace (checked), Base Directory (/)?

Come to think of it, I think the syslogs might be one-line in 2.2 already. If you send the syslogs to Splunk or to any syslog server, you should be able to confirm that.

You don't need to enable these:

**Log packets blocked by the default rule

Log packets blocked by 'Block Bogon Networks' rules

Log packets blocked by 'Block Private Networks' rules**

You need to look at the Firewall:Rules:Edit

Log packets that are handled by this rule
Hint: the firewall has limited local log space. Don't turn on logging for everything. If you want to do a lot of logging, consider using a remote syslog server (see the Diagnostics: System logs: Settings page).

You can also see other options in Status:System Logs:Settings:

And choose what to send to Splunk:

**Everything

System events
Firewall events
DHCP service events
Portal Auth events
VPN (PPTP, IPsec, OpenVPN) events
Gateway Monitor events
Server Load Balancer events
Wireless events**

[arrmo]

Hi,

Yep, I did put the URL in, and fetched it. It just failed when I ran a test on it (rather than just blindly applying it). Make sense? Sorry if I'm not explaining this very well … :(.

I did look in Splunk, but I admit - not sure I know what the expected format is. The end of the line is eol though, so I think it's a single line record. Would eol be the last item expected?

Will look at the firewall rules, what you're saying makes sense. BTW, is there a way to not log this info locally, but rather only send it to Splunk (remote syslog server)?

Thanks!

[BBcan177]

What were the errors for the patch?

Once you configure pfSense to send the logs to Splunk, you probably need to open up the firewall on the Splunk server to accept pfSense's IP address on port 514. If you see the same information in Splunk as you see in the Firewall or System Logs than I assume that its working.

I wouldn't recommend it, but there is an option the the Status:System logs: Settings:
Unless you have space issues, its always nice to have logs in pfSense to be able to review.

Disable writing log files to the local disk

[arrmo]

My fault - sorry! I missed that option. Was looking, just plain missed it … :(.

Yep, Splunk is receiving them, that's where I checked. Just wasn't sure if eol as the last part of the entry signified that they are single line correctly or not, that's all.

Here is the output from the patch Test ...

Output of full patch apply test:
/usr/bin/patch --directory=/ -t -p1 -i /var/patches/53b8c87e8e26a.patch --check --forward --ignore-whitespace

Hmm...  Looks like a unified diff to me...
The text leading up to this was:

|diff --git a/etc/inc/filter.inc b/etc/inc/filter.inc
|index dc8da4f..0089383 100644
|--- a/etc/inc/filter.inc

+++ b/etc/inc/filter.inc
Patching file etc/inc/filter.inc using Plan A...
Hunk #1 failed at 116.
1 out of 1 hunks failed while patching etc/inc/filter.inc
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:

---

|diff --git a/usr/local/www/diag_logs_settings.php b/usr/local/www/diag_logs_settings.php
|index 8e7513c..7cce191 100755
|--- a/usr/local/www/diag_logs_settings.php

+++ b/usr/local/www/diag_logs_settings.php
Patching file usr/local/www/diag_logs_settings.php using Plan A...
Hunk #1 succeeded at 64 with fuzz 2 (offset 2 lines).
Hunk #2 failed at 120.
Hunk #3 failed at 297.
2 out of 3 hunks failed while patching usr/local/www/diag_logs_settings.php
done
Close

Thanks!

[swxdke]

You can follow this great blog post that details how to manage the traffic from the Splunk end by using props.conf and transforms.conf: blog.basementpctech.com/2012/02/splunk-and-pfsense-what-pair.html

I've detailed my pfSense/Splunk configuration on my blog http://secworx.com/2014/09/22/splunking-pfsense/

[arrmo]

This is awesome, thanks! Did you get this running on v2.2? I can't seem to get the patch working there, and also can't seem to log outgoing traffic … :(.

Thanks again.