Max size logs ?
-
Hi,
I'm having a hard time with setting up my logs file. i tried setting up a special size only for filter.log using the GUI, but it failed and after i reset log to apply it either it doesn't log anymore or it has a strange very inferior size.
So I am trying to roll back to same size for those ~20 logs.
So i go to Status /System Logs /Settings / General Logging Options and try to set up 5GB for each clog ( which is not ideal I would want 1GB for all and 100GB for filter.log but at this point i'm just trying to get it to work).
Here is a screenshot
I save then reset log files on that page ( blue then red button ).
All is logging fine but then i connect with ssh and perform a wc to check and i see i get very little file size ( 705032704 ), here is a screenshot:
I have ample size and dedicated that 250GB SSD to logging, is there no way i can use that space to have big log files ?
Thanks for helping :)
-
Something tells me 100GB is beyond some limitation of syslogd. Maybe the log file size field needs an upper limit entry enforcement.
-
Thing is I'm not even close to 100GB, I ask for 5GB i get 700MB, something must be wrong with my config yet I can't point it out
-
Didn't say the syslogd limit is anywhere near 100GB. Just that it is likely less than that. Probably much less.
Try some smaller sizes. Maybe start at 500MB and double it until it doesn't work. Then split the difference. Then continue by moving up or down by 50% of difference to narrow in on the max.
-
I think the problem is elswhere, on this topic : https://forum.pfsense.org/index.php?topic=82078.0 they create a 1GB with no issue ( an futhermore different sizes before you could do it with the GUI ) of 1GB and the "limit" ( didn't find anything about a limit tho ) if there is one must be even greater, mine is capped at 700MB even tho I did ask for more, and it didnt give me any warning in the GUI.
-
The GUI isn't testing for an upper limit. So there would not be any warning. That's why I stated previously that the field maybe should have that added.
Just like I didn't say the limit was as high as 100GB I also didn't say the limit was as low as 500MB. Just suggested starting there (500MB). I don't know what the limit is. I'm giving you a way to find the limit.
Info on the web indications there may be a hard coded limit within syslogd on Linux systems. I know this is FreeBSD not Linux. But syslogd perhaps shares some source code.
The 705032704 that is resulting is probably the remainder of a multiple of the limit.
Based on the following calculation I'm guessing the maximum is 1GB, 2GB, or 4GB.
5000000000 - 705032704 = 4294967296 = 4GB
4GB / 1GB = 4
4GB / 2GB = 2
4GB / 4GB = 1 -
Install a graylog instance and log to that.
Maintaining logs like that is not a firewall's job.
-
Install a graylog instance and log to that.
Maintaining logs like that is not a firewall's job.
Agree with that!
How do you propose to search or analyze the logs if you keep them on pfSense?
Suck them up from graylog or whatever and feed them to Splunk, ELK or something similar.
-
Ok so I thought that it might have been hardware/bad config on my pfsense so i tried on another pfsense ( spoiler : it was the same ).
So I go on that pfsense, all logs are 1 000 000 000 B, and i test the size of dhcpd.log, all fine.
I then change the specific size of filter.log ( then reset of course ) to 5 000 000 000 B and check : it is working and size is 705032740. Now some maths :
5 000 000 000 - 705032740 = 4294967260 = 2^32I then change the specific size of filter.log ( then reset of course ) to 2 000 000 000 B and check : it is working and size is 2 000 000 000 B
I then change the specific size of filter.log ( then reset of course ) to 5 000 000 000 B and check : it is working and size is 1410065408. Now some maths :
10 000 000 000 - 1410065408= 8589934592= 2^33I then change the specific size of filter.log ( then reset of course ) to 3 000 000 000 B and check : it is NOT working : syslogd : /var/log/filter.log : Operation not supported by device and size is 7239303168 . Don’t see what it adds up to.
I then change the specific size of filter.log ( then reset of course ) to 3 000 000 000 B AGAIN ( after changing it back to a working 2000000000) and check : it is NOT working : syslogd : /var/log/filter.log : Operation not supported by device and size is 7239467008 . Don’t see what it adds up to neither.
But same asked size 3 000 000 00 and different « error size ».I then change the specific size of filter.log ( then reset of course ) to 2147483647 B and check : it is working and size is 2147483647 B
I then change the specific size of filter.log ( then reset of course ) to 2147483648 B and check : it is NOT working : syslogd : /var/log/filter.log : Operation not supported by device and size is 7178878976 . Don’t see what it adds up to.
What i can « conclude » is that max size is 2 147 483 647 B = 2^31 -1
However i don’t understand the Operation not supported by device and the fact that i get different « error sizes » for same 3000000000B.
To answer the remarks saying i should use graylog you are right I already use it on an other pfsense but for that particular pfsense logs will not be used at all and only kept for legal reasons ( if some sick pedophile crack my wifi passwd for example I need to be able to provide 2 years or so of logs to defend myself ).
-
Must be lot of fun with doing maths and waiting where it overflows… but - the circular log is not suitable for archiving purpose, at all. Use a remote syslog server, or at least install the syslog-ng package and log to normal logs, rotating them as needed.