Suricata syslog is truncated!
-
I configured suricata to send the alarms by syslog with payload and packet.
But some syslog messages are truncated!
I checked syslog messages and i found PFSense send only first 1024 bytes of syslog messages.
Total packet length is 1066 bytes now, so this issue not related to MTU size.
How to fix it? -
You will have much better luck with something like GrayLog or setting up an ELK stack on a separate host. Suricata is logging more and more of its output in JSON format. This is the direction of the upstream developers. So you need to enable the JSON logging options for each interface and then create a system on your own to suck up the JSON log files and export them off to a separate consolidator host. I believe some users here have tried filebeat and had success with that. It can process JSON logs.
-
Is there any possibility we'll get that JSON option in the PFSense package, so that the beats package stays updated properly? I don't mind messing with installing the beats package myself in my home lab, but I really need that at the work firewall...
-
@asiTechsupport said in Suricata syslog is truncated!:
Is there any possibility we'll get that JSON option in the PFSense package, so that the beats package stays updated properly? I don't mind messing with installing the beats package myself in my home lab, but I really need that at the work firewall...
I honestly do not know. I have not investigated the beats package in FreeBSd very much. About two or three years ago I looked at a logstash-forwarder package for pfSense, but it wanted to pull in several more dependent language packages. There was another option I investigated that used Java! Who in their right mind would think putting Java on your firewall is a good idea? That's one of my frustrations with a lot of these third-party log consolidation packages. They seem to want to use the "hula-hoop language of the month". Stuff like Go and Ruby and so on. These kinds of languages seem to bloom and then fade away, but at the time of their bloom are touted as the best thing to hit computer programming since COBOL ...
. Then, two or three years later, you never hear about them again.OK, rant off --- I do need something to replace the now essentially dead Barnyard2 package. There have been no updates to it on FreeBSD in years. My goal would be to find a very small footprint client that scarfs up the JSON logs and then sends them over the network to another host for indexing, reporting and storage. I don't see a situation where the storage and reporting is done on the firewall itself. You should always try to minimize attack surfaces on your primary security appliance, not add to them.
-
AGREED on all points, LOL!
For now, I'm looking to set up Bro/Zeek somewhere to get a modest NSM going. I was considering using the db capabilities of barnyard2, especially considering that barnyard2 has the option of pointing to Postgres (I'm trying to rid myself of the MySQL habit)... BUUUT, sadly, it seems that Postgres support isn't even compiled into the barnyard2 we have on PFSense.
Ah well, I'll keep my eyes peeled on this front... perhaps a Python logs parsing client is out there? Hmmmm... on to the hunt!
-
@asiTechsupport
The last update to Barnyard2 (which was quite a long time back) messed up its interaction with SQL databases (at least with MySQL). I investigated it a little back then but got sidetracked onto something more pressing at the time. The issue was with the way Barnyard2 calculated a primary key value from the references included in a Snort rule. Rule updates frequently caused Barnyard2 to calculate a primary key that was a duplicate of an existing key, so when BY2 tried to insert the new record the DB would throw a primary key violation and Barnyard2 would croak. The other even bigger issue was BY2 getting hung up processing updates to the Snort rules and running the CPU at 100% for two or three hours. BY2 stores info about the Snort rules along with the alerts data in its database. During this period I was running Snorby on a virtual machine on ESXi. Snorby was based on Ruby and used MySQL as the backend. -
@bmeeks
Thanks for the info!Once I saw it wasn't available, I figured there must be an issue with that side of things and moved on.
-
@bmeeks said in Suricata syslog is truncated!:
You will have much better luck with something like GrayLog or setting up an ELK stack on a separate host. Suricata is logging more and more of its output in JSON format. This is the direction of the upstream developers. So you need to enable the JSON logging options for each interface and then create a system on your own to suck up the JSON log files and export them off to a separate consolidator host. I believe some users here have tried filebeat and had success with that. It can process JSON logs.
Thanks, actually i'm receiving EVE JSON over syslog now (bcoz payload data is needed).
it seems i should try filebeat.
