LightSquid Refresh Schedule Data Loss



  • hi.
    i found this topic - https://forum.pfsense.org/index.php?topic=24568.0
    it's too old so i can't bump/reply in there, but despite dvserg's post

    « Reply #1 on: April 29, 2010, 01:17:37 am »

    Thank You

    Fixed as

    • per day fresh will started at 23:45
    • added new yesterday fresh at 0:15

    Thanks!

    half of the problem is still there, coz new yesterday fresh doesn't work at all! so if for example you set autorefresh to every 6 hours, you will lose the last 6 hours of data in every day, because when you run this code (from the cron schedule)

    /usr/bin/perl /usr/pbi/lightsquid-amd64/local/www/lightsquid3/lightparser.pl yesterday

    you will get this output (when debug is enabled):

    [2.2.4-RELEASE][root@proxy.*.local]/var/squid/logs3: /usr/bin/perl /usr/pbi/lightsquid-amd64/local/www/lightsquid3/lightparser.pl yesterday

    filter today: 20160923
    filter yesterday: 20160922
    use file :: /var/squid/logs3/access.log
    run TIME: 2 sec
    LightSquid parser statistic report

    1058350 lines processed (average 529175.00 lines per second)
                        0 lines parsed
                        0 lines recovered
                        0 lines notrecovered
                        0 lines skiped by bad year
                  1058350 lines skiped by date filter
                        0 lines skiped by Denied filter
                        0 lines skiped by skipURL filter

    WARNING !!!!, parsed 0 lines from total : 1058350
    please check confiuration !!!!
    may be wrong log format selected ?

    and the reason for this is simple - at midnight squid rotates logfiles and yesterday's access.log becomes access.log.0, BUT as you can see from the debug above (text in bold), lightparser.pl still tries to load access.log, which is now today's already.

    so as a temporary fix i opened lightparser.pl, found this section

    if ($ARGV[0] eq "yesterday") {
       $filterdatestart=$filterdatestart-(24*60*60);
       $filterdatestop =$filterdatestop -(24*60*60);
       ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($filterdatestart);
       $filterdate=sprintf("%04d%02d%02d",$year+1900,$mon+1,$mday);;
          print ">>> filter yesterday: $filterdate\n" if ($debug);
    }
    
    

    and added the line

    $filename="access.log.0";
    

    at the beginning of it, which solved the problem.

    but i am in no way a programmer, so maybe there is more correct/appropriate/elegant way to fix this?

    thanks.



  • I just can not believe this bug even exists, let alone after so many many years after it has been created (8 years).☹