Syslog Experience with Pfsense

[KOTRz]

Hi All;
I would like to know the best syslog servers that can be used with pfsense without any problems. there are many syslog servers out there. some of them have limitations, I want to know the best syslog server that is suitable for handling many pfsense devices in one network.
Thanks

[Atom2]

I am using syslog-ng (currently version 3.4.8 ) on linux (gentoo-hardened) and haven't seen any issues so far. The only limitation - of pfSense and not syslog-ng - is that as far as I can tell pfSense's syslog protocol only supports UDP for message transport. syslog-ng supports both TCP and UDP and the majority of other syslog sources in my network deliver their messages through TCP.

Just out of interest: What sort of problems/issues are you actually anticipating?

Regards Atom2

[BBcan177]

I use a software called "Security Onion" that has ELSA installed as its syslog software. It also is a NSM/IDS/HIDS software all in one.

https://code.google.com/p/security-onion/wiki/IntroductionToSecurityOnion

[jimp]

Lots of people seem to love the ELK stack.

But I'm no fan of Java.

ELSA seems OK, though the GUI is not all that great/intuitive for general log viewing.

[BBcan177]

@jimp:

Lots of people seem to love the ELK stack.

But I'm no fan of Java.

ELSA seems OK, though the GUI is not all that great/intuitive for general log viewing.

I think they are all good choices.

I like Security Onion as it is already packaged (as an ISO) with all of the tools integrated and almost ready to go. If you wanted to install ELK, you would have to install it all and maintain the upgrades.

With Security Onion, ELSA is integrated into SGUIL, all of the pcaps can be viewed in SGUIL or ELSA as it is a full-packet capture integration.

Snort/Suricata is already installed, along with BRO and OSSEC.

I also point all of my pfSense Syslogs Boxes to Security Onion. OSSEC is running on all of my Windows/Linux servers and my Windows Server events are also sent to ELSA.

So while ELSA doesn't have all the "beautiful" colored graphs, it is fast and can pivot on any information and output Snort/Suricata, syslogs, Ossec events, BRO, Prads, http etc etc… so that you can review all of this information chronologically and replay exactly what is happening in your network.

So to an Analyst, the Security Onion ELSA integration allows you to find and analyse events really well. Not too mention that its free and can be up and running fairly quickly.

The developer of ELSA Martin Holste has been hired by Mandiant to integrate ELSA into their Software suites. He has also stated that he intends to keep ELSA maintained and open Source.

Not to mention that SO scales well. I have a master/sensor server where ELSA is installed, with two other remote locations that are integrated. So from the Server, I can pivot on any of the above information on any of the nodes very quickly.

Maintaining the nodes is also easy with it "Salt" integration.
https://code.google.com/p/security-onion/wiki/Salt
https://code.google.com/p/security-onion/wiki/FAQ#What's_the_difference_between_a_"server"and_a&q

Sorry for "spamming" but I am trying to spread the good word to my fellow pfSense'rs ...

[KOTRz]

Thanks you all for sharing this.
What Im after is a syslog who can sustain verbose logging in a network environment which generates alot of logs. in three months time I would expect log size to be 3GB Raw.
Can Onionsalt handle this? we need to keep pfsense logs for at least 6 months.

[Atom2]

@KOTRz:

What Im after is a syslog who can sustain verbose logging in a network environment which generates alot of logs. in three months time I would expect log size to be 3GB Raw.
Can Onionsalt handle this? we need to keep pfsense logs for at least 6 months.

Whilst I can't comment om Onionsalt, I guess that your perception of a syslog daemon might somehow be flawed:

In a nutshell, a syslog daemon (or server) is a program that receives messages sent from one or many syslog client(s) (or sources) and processes the incoming messages according to some (user defined) rules. Some of these messages will generally originate from local processes, but in a networked envirronment any such incoming messages could easily (and will usually) come from processes running on remote systems reachable through a network (LAN or WAN). Any such messages are usually delivered from the remote system to the local syslog daemon over port 514 (TCP or UDP) - that's where the local daemon is listening for traffic.

In terms of processing after the messages have been received by the syslog daemon, there are numerous options including for instance filtering (i.e. discarding certain messages according to patterns), processing (i.e. creating a new combined consolidation message out of a number of related raw messages within a so called message context) or rewriting (i.e. changing certain parts of a message). Clearly there are other possible options as well - the limit is your imagination.
After the messages have been processed by all applicable user defined rules, the (left over) raw messages and/or the processed/rewritten messages can then be stored on persistent storage (if desired in separate files according to yet more rules based for instance on originating system, sending process, message context, time, etc.), forwarded to yet another (upstream) syslog daemon or to another process to perform further analysis thereon.

So in essence being able to keep messages after they have been stored on persistent storage is no longer a function of the syslog daemon as such but of the underlying file system - and that I'd guess should easily be able to store files worth 3GB raw.

I hope that helps Atom2

[BBcan177]

In regards to ELSA's capabilities:

https://code.google.com/p/enterprise-log-search-and-archive/wiki/Documentation#Capabilities

ELSA achieves n node scalability by allowing every log receiving node to operate completely independently of the others. Queries from a client through the API against the nodes are sent in parallel so the query will take only the amount of time the of the longest response. Query results are aggregated by the API before being sent to the client as a response. Response times vary depending on the number of query terms and their selectivity, but a given node on modest hardware takes about one half second per billion log entries.

Log reception rates greater than 50,000 events per second per node are achieved through the use of a fast pattern parser in Syslog-NG called PatternDB. The pattern parser allows Syslog-NG to normalize logs without resorting to computationally expensive regular expressions. This allows for sustained high log reception rates in Syslog-NG which are piped directly to a Perl program which further normalizes the logs and prepares large text files for batch inserting into MySQL. MySQL is capable of inserting over 100,000 rows per second when batch loading like this. After each batch is loaded, Sphinx indexes the newly inserted rows in temporary indexes, then again in larger batches every few hours in permanent indexes.

Sphinx can create temporary indexes at a rate of 50,000 logs per second consolidate these temporary indexes at around 35,000 logs per second, which becomes the terminal sustained rate for a given node. The effective bursting rate is around 100,000 logs per second, which is the upper bound of Syslog-NG on most platforms. If indexing cannot keep up, a backlog of raw text files will accumulate. In this way, peaks of several hours or more can be endured without log loss but with an indexing delay.

[onlineph]

Apology for gate crashing!

So would syslog-ng fit the needs if I only intend to log voucher log-in and out? Or do I need to go ELSA? Any one?

[enriluis]

pfsense should by have a lot of agents  to integrate logs with third party solutions like OSSIM , ELSA etc etc, I love OSSIM and I'm waiting for a serious agent to full integrate pfsense with OSSIM, i'm dream with that … lol, i'm not fan of syslog.