eve.json log not exported

michmoor

Ive noticed that i am not receiving any eve.json log files in my syslog server but i am getting every other system log dump from the firewall.
When i go check the contents of the eve.json log file in the GUI i receive the following message

Strangely after attempting to view the log file i get the following system alert

22:01:37 PHP ERROR: Type: 1, File: /usr/local/www/suricata/suricata_logs_browser.php, Line: 50, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 738074848 bytes)

edit:

The eve.json log file exists on the system and is being populated

bmeeks

The file will be way too large to view in the pfSense GUI. The text file handling capability of PHP is very low-tech. It must read the entire file into RAM, then stream the content line-by-line to the browser. So large files exceed the amount of free PHP memory (not related to how much other memory pfSense may be showing you as "free"). When initialized, PHP sets a limit on available PHP memory to prevent various types of DoS attacks.

So, the short answer is you can't usually view most of the Suricata log files in the GUI because they get so huge so fast and outrun available PHP memory.

Suricata has one huge shortcoming compared to Snort (and I'm talking at the binary level). Snort lets you set maximum log sizes, and when that size is reached the binary itself (the one doing the logging) will automatically rotate the file. Suricata has no such built-in feature. The best you have available in the binary is an option to rotate some of the log files (but not all) on a timed basis. That's not terribly useful since the files may grow to different sizes at different rates.

The GUI package attempts to emulate Snort's behavior by forcibly closing the log files, rotating them, and then "touching" a new file and telling Suricata to reload.

Most times that kluge works, but not always,. Sometimes the Suricata binary stubbornly refuses to start using the "new" log file and continues to write to the old one. That may be what is wrong in your case. To fix it, stop Suricata on all the interfaces, manually rotate and/or move the logs in question, then restart Suricata.

Suricata upstream could really make things easier and better by offering more log rotation options. The EVE JSON log is a particularly bad offender because SO much data is logged there.

michmoor

@bmeeks thanks bill I’ll give it a whirl.
What’s interesting is that this is on a low throughput interface and yet the log file grows to almost a gig.
Is there a way to tame the eve log file? Should it even be tamed considering it contains important meta data.

bmeeks

@michmoor said in eve.json log not exported:

@bmeeks thanks bill I’ll give it a whirl.
What’s interesting is that this is on a low throughput interface and yet the log file grows to almost a gig.
Is there a way to tame the eve log file? Should it even be tamed considering it contains important meta data.

The only way to tame is to reduce the options enabled on the INTERFACE SETTINGS tab for EVE Logs and perhaps reduce the rules. But I would start with reducing some of the logging depending on circumstances.