LightSquid Refresh Schedule Data Loss

hvac14400

hi.
i found this topic - https://forum.pfsense.org/index.php?topic=24568.0
it's too old so i can't bump/reply in there, but despite dvserg's post

« Reply #1 on: April 29, 2010, 01:17:37 am »

Thank You

Fixed as

per day fresh will started at 23:45

added new yesterday fresh at 0:15

Thanks!

half of the problem is still there, coz new yesterday fresh doesn't work at all! so if for example you set autorefresh to every 6 hours, you will lose the last 6 hours of data in every day, because when you run this code (from the cron schedule)

/usr/bin/perl /usr/pbi/lightsquid-amd64/local/www/lightsquid3/lightparser.pl yesterday

you will get this output (when debug is enabled):

[2.2.4-RELEASE][root@proxy.*.local]/var/squid/logs3: /usr/bin/perl /usr/pbi/lightsquid-amd64/local/www/lightsquid3/lightparser.pl yesterday

filter today: 20160923
filter yesterday: 20160922
use file :: /var/squid/logs3/access.log
run TIME: 2 sec
LightSquid parser statistic report

1058350 lines processed (average 529175.00 lines per second)
0 lines parsed
0 lines recovered
0 lines notrecovered
0 lines skiped by bad year
1058350 lines skiped by date filter
0 lines skiped by Denied filter
0 lines skiped by skipURL filter

WARNING !!!!, parsed 0 lines from total : 1058350
please check confiuration !!!!
may be wrong log format selected ?

and the reason for this is simple - at midnight squid rotates logfiles and yesterday's access.log becomes access.log.0, BUT as you can see from the debug above (text in bold), lightparser.pl still tries to load access.log, which is now today's already.

so as a temporary fix i opened lightparser.pl, found this section

if ($ARGV[0] eq "yesterday") {
   $filterdatestart=$filterdatestart-(24*60*60);
   $filterdatestop =$filterdatestop -(24*60*60);
   ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($filterdatestart);
   $filterdate=sprintf("%04d%02d%02d",$year+1900,$mon+1,$mday);;
      print ">>> filter yesterday: $filterdate\n" if ($debug);
}

and added the line

$filename="access.log.0";

at the beginning of it, which solved the problem.

but i am in no way a programmer, so maybe there is more correct/appropriate/elegant way to fix this?

thanks.

ccdmas

I just can not believe this bug even exists, let alone after so many many years after it has been created (8 years).