Log files max lines

[jrey]

on the pfBlockerNG options General settings, the "Log Setting (max lines)" section.

Do the values set in here actually work?
Appear to me perhaps they do not, as several of the log files have significantly more lines than the default of 20,000 lines they are all set at. Maybe it is just not clear to me what should happen at these limits.
so what should the logs be doing at the 20,000 lines setting? rolling over x.log.0 - x.log.1 - x.log.2 etc ?

I've been watching the log size and when they get over a couple of MB, I download and delete them in the GUI.

The /var/log/pfblockerng directory has only the logs that are active. (ie never any roll over files)

I just downloaded dns_reply.log when it topped 2MB, for example and at that point in has 23,988 lines. Without checking regularly one file had grown over 8MB.

pfSense 23.01, pfBlockerNG is version 3.2.0_4
system logs (so those in /var/log) are respecting their size limits settings and creating .0 .1 etc) as expected

Thanks

[Gertjan]

@jrey said in Log files max lines:

on the pfBlockerNG options General settings, the "Log Setting (max lines)" section.
Do the values set in here actually work?
Appear to me perhaps they do not, as several of the log files have significantly more lines than the default of 20,000 lines they are all set at. Maybe it is just not clear to me what should happen at these limits.

Not the files sizes.
The number of lines the GUI will show you.
The GUI really is in trouble when it has to read, filter, format, assemble many (like more then 200000) lines.

The files are rotated of course.
Otherwise your pfSense would run out of disk space in no time.

My /var/unbound/var/log/pfblockerng/dns_reply.log is 20 Mbytes right now, and was 'flushed' yesterday just before midnight.

Btw : the file has 212000 lines.
That not the one I want to view with my browser.
The web server (PHP actually) would try it, and go face up and error out while trying.

For these files I use SFTP (FTP over SSH) and then it becomes a one click show and it will be very fast.

[jrey]

@gertjan

Thanks for the response

a) Max Lines = Max Display Lines (got it)
in which case the value could be 1 setting for all instead of 1 setting for each file. Personally am not going to "read" 20,000 lines (the default) in this format anyway.

b) "the files are rotated" or "was flushed yesterday"
do we know which rotated or flushed?
I've never seen a "rotation" as such. So assume you are saying they are just flushed.
is there a control setting to say when this happens? you indicated 20 Mbytes right now, and was flushed yesterday just before midnight.
is the "20 right now" after the flush at midnight or before?
can you tie the "just before midnight" to a source event? (like say a scheduled job) (ie what is the trigger for the flush? file size or time of day).

Thanks again

[jrey]

@gertjan

@jrey said in Log files max lines:

a) Max Lines = Max Display Lines (got it)

Actually, you must mean when viewed somewhere else because the built in viewer only displays 10,000 lines regardless of the settings, which is again set at the default of 20,000 max lines.

Those max lines defaults are these and apply only to pfBlockerNG:

which seem to have nothing to do with display lines, at least under the log browser shown here : Where the max is "Displaying last 10000 lines only"

[Gertjan]

@jrey said in Log files max lines:

b) "the files are rotated" or "was flushed yesterday"

Well, you're right : it's "flushed"

I started to doubt ....
So took the manual of pfBlockerng and looked up what actually happens.
Here : /usr/local/www/pfblockerng/pfblockerng.php, and on line 1134 we have the pfb_log_mgmt() function.
It shows what happens with all the 'log' files, as dns_reply.log is one of them.
The 'limit' number is take, and last 'number' of lines are kept as a history, previous lines are discarded.

The pfb_log_mgmt() function is, among others, called during a 'cron update'.

@jrey said in Log files max lines:

is the "20 right now" after the flush at midnight or before?

Look at your own file, and you'll know ;)

The first line - it has a date / time stamp. Moment before, the file was truncated, or flushed.
The last line will be 'moments away' as it shows the last recorded DNS request or action handled by pfBlockerng.

[jrey]

@gertjan said in Log files max lines:

Well, you're right : it's "flushed"

Actually it was you that suggested flush, but you also said rotated. I was just trying to clarify which one you actually observed.

Thanks for providing the hook showing it processes with the cron job, that will be very helpful.

Just one more quick question. You indicated that you had a file the was 20MB and cleared at midnight.

What schedule are your cron updates for pfB set for ?
someone updating every hour, two hours etc should never see large files as you have at 20MB or mine at 8MB

[Gertjan]

@jrey

pfBlockerng main page :

Related cron setting :

But wait :

So, I guess, I've set to 'every day' at 8h15

The cron settings are now (after a force reload ! - see bottom of the page) :

where is the "8" ? for 8 o'clock 'AM' ? (bug ?)

Anyway.

Note that

 	/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron

will execute

			syslog(LOG_NOTICE, '[pfBlockerNG] Starting cron process.');
			pfblockerng_sync_cron();

and the function pfblockerng_sync_cron(); will do a

	// Call log mgmt function
	// If Update GUI 'Manual view' is selected. Last output will be missed. So sleep for 5 secs.
	sleep(5);
	pfb_log_mgmt();

at the end.

My dns_reply.log was reduced, while testing, to a mere 20000 (my setting) and it grows rapidly, because I see hundreds of lines per minute.