Splunk, pfSense and Home Monitor

[superweasel]

I am attempting to connect pfSense to Splunk for use with Home Monitor without much success.

Here is what I have setup:

• Splunk 6.2.3 and Home Monitor 4.0 are installed on OS X 10.10.3 Yosemite running OS X Server 4.1

• syslog server is running on OS X Server accepting incoming data on UDP port 514

• pfSense 2.2.2 is sending all logging data to the OS X Server

• OS X Server is accepting all the pfSense logging data as verified in Console

• Splunk is set to receive incoming data on UDP port 514

There is nothing showing up in Splunk or Home Monitor.

I have followed these instructions http://blog.basementpctech.com/2012/02/splunk-and-pfsense-what-pair.html for getting Splunk to work with the current pfSense logs.

Screenshots below.

Any recommendations, corrections or additions would be greatly appreciated.

[tim.mcmanus]

I would take the OS X Server out of the equation entirely.

OS X Server does not follow any known UNIX standards when it deploys itself.  So don't expect syslog to function like it does in another *NIX.  Additionally, pfSense is sending log data to your server and Console has verified this.  The issue isn't with pfSense but more with your Splunk/Home Monitor installation.

Full discloser:  I've been an Apple beta tester for OS X Server since Rhapsody 1.0.  After OS X 10.6, Server became an app and not a collection of standard *NIX services.  Many things "broke" and have remained unresolved since then.

[superweasel]

I reject your reality and substitute my own.

Splunk by default does not run as root in OS X (developers choice), so port UDP 514 is not available to Splunk. You have to redirect the incoming syslog to an unused port and tell Splunk to use the new port as its data port.

Here are the specifics-
    Redirect incoming syslog data to UDP port 1514 by add the following line to /etc/syslog.conf:
   ```
.[tab]@127.0.0.1:1514

    where [tab] is an actual tab.
    Restart syslogd by either restarting OS X or using:
   ```
sudo killall -HUP syslogd

In Splunk Settings>Data Input create a new data input with the following:
        UDP Port = 1514
        Source name override = ""
        set sourcetype = Manual
        Source type = pfsense
        Index = homemonitor

If $SPLUNK_HOME/etc/apps/homemonitor/local/transforms.conf does not exist, copy the file from
        $SPLUNK_HOME/etc/apps/homemonitor/default/transforms.conf
    to
        $SPLUNK_HOME/etc/apps/homemonitor/local/transforms.conf

Just because Apple and in turn OS X fall on the side of security (only root access to UDP port 514), does not mean there is anything wrong with the way OS X works or how it is implemented, it is still UNIX.

[tim.mcmanus]

@superweasel:

I reject your reality and substitute my own.
Just because Apple and in turn OS X fall on the side of security (only root access to UDP port 514), does not mean there is anything wrong with the way OS X works or how it is implemented, it is still UNIX.

I wholeheartedly reject this statement outright.

OS X Server, as I specifically mentions and which I have very deep experience with, is by no means close to standard *NIX implementation.  Very, very far from it.  Their decision to implement all of the services inside of the Server.app and changing the configuration of those services using the CLI command 'serveradmin' has absolutely NOTHING to do with security.  Apple significantly downsized their OS X Server development team and prevent them from implementing standard *NIX builds and installation beginning with OS X Server 10.7.  Most of those developers were laid off or moved to different parts of the OS X development team.  Their offering then became the watered down, non-standard pile of slapped-together code that it is.

Additionally, NOTHING should run as root other than root.  Splunk should have its own service account and should run under that name.  The developer wisely decided not to allow users to run it as root because that's just stupid.  However, giving it a named service account, which is what you'd normally do in *NIX, would have made more sense.

[superweasel]

I guess you missed the point of the original question. I never specifically asked about OS X Server or any of its services, that was just part of the equation. Ultimately, it had nothing to do with the problem or solution. This was strictly about Splunk, syslog port and pfSense. The same issue would have occurred if it was just OS X.

[tim.mcmanus]

The issue was with Splunk on OS X.  That was clear based on your Console output, as I mentioned in my initial post.

@tim.mcmanus:

Additionally, pfSense is sending log data to your server and Console has verified this.  The issue isn't with pfSense but more with your Splunk/Home Monitor installation.

And I stand by my comments that OS X Server is a steaming pile of code.

[biggsy]

If you are using pfSense 2.2.2 the log format is very different from what is was when that blog was posted.

Have you installed version 2.0.2 of the the TA-pfsense Splunk add-on?