LightSquid Refresh Schedule Data Loss (bug w/ details to fix)

Erasmus Darwin

pfSense version: 1.2.3-RELEASE
Lightsquid version: 1.7.1 Beta1

I've discovered a bug with LightSquid's Refresh Scheduler option. LightSquid winds up missing the last X minutes of data before midnight where X is the selected refresh rate. So if you set it to refresh every 60 minutes, you'll wind up missing data in the reports on any traffic between 11 pm and midnight each night. I had my refresh rate set to 24h, which made the bug a lot easier to detect since no data at all was being added to the reports.

After some research, it seems that the problem is in /usr/local/pkg/lightsquid.inc, specifically in the code that schedules the cron job to run /usr/local/www/lightsquid/lightparser.pl.

Here's the code:


        // set shedule - refresh data job
		if ($tm) {
            $on = false;
            $opt = array("*", "*", "*", "*", "*", "root", CRONTAB_LS_TEMPLATE . " today");
            switch($tm) {
                  case 'lhp_none': $on = false; break;
                  case 'lhp_10m':  $on = true; $opt[0]= "*/10"; break;
                  case 'lhp_20m':  $on = true; $opt[0]= "*/20"; break;
                  case 'lhp_30m':  $on = true; $opt[0]= "*/30"; break;
                  case 'lhp_40m':  $on = true; $opt[0]= "*/40"; break;
                  case 'lhp_50m':  $on = true; $opt[0]= "*/50"; break;
                  case 'lhp_60m':  $on = true; $opt[0]= "*/60"; break;
                  case 'lhp_2h':   $on = true; $opt[0]= "0"; $opt[1]= "*/2";  break;
                  case 'lhp_3h':   $on = true; $opt[0]= "0"; $opt[1]= "*/3";  break;
                  case 'lhp_4h':   $on = true; $opt[0]= "0"; $opt[1]= "*/4";  break;
                  case 'lhp_6h':   $on = true; $opt[0]= "0"; $opt[1]= "*/6";  break;
                  case 'lhp_8h':   $on = true; $opt[0]= "0"; $opt[1]= "*/8";  break;
                  case 'lhp_12h':  $on = true; $opt[0]= "0"; $opt[1]= "*/12"; break;
                  case 'lhp_24h':  $on = true; $opt[0]= "0"; $opt[1]= "0"; $opt[2]= "*/24"; break;
            }
            ls_setup_cron("lightsquid_parser", $opt, CRONTAB_LS_JOBKEY, $on);
		}  else
            ls_setup_cron("lightsquid_parser", "", "", false);

The line for running the parser every 24 hours (case 'lhp_24h') is completely broken. $opt[2] shouldn't be there at all, since that corresponds to the cron field for the day of the month. This tries to run the schedule every 24 days at midnight. If the $opt[2] part is removed, it'll run with the correct frequency (every day at midnight).

However, that still leaves another bug. The schedule is invoking lightparser with the 'today' option. When invoked that way, lightparser parses the data for everything that's happened during the same day as today (according to the system time). So if you run it at midnight with the today option, it will pick up virtually no records out of the log since a new day has just started. Ideally, the first invocation of lightparser after midnight should, instead, use the 'yesterday' option.

Since I've only got lightsquid processing data daily, I only bothered to fix the 24 hour case. I changed it to:


                  case 'lhp_24h':  $on = true; $opt[0]= "0"; $opt[1]= "0"; $opt[6] = CRONTAB_LS_TEMPLATE . " yesterday"; break;

and everything's fine for my setup. That generates a crontab entry that runs 'lightparser yesterday' at the start of every new day at midnight.

Now fixing the other cases is a bit trickier. For those, you'd want the first invocation after midnight to be 'lightparser yesterday' and all other invocations to be 'lightparser today'. For any case that's a multiple of an hour, that can be handled with just two cron lines:


0 0 * * * root  /usr/bin/perl /usr/local/www/lightsquid/lightparser.pl yesterday
0 X-23/X * * * root  /usr/bin/perl /usr/local/www/lightsquid/lightparser.pl today

where X is the number of hours between refreshes (e.g. 6-23/6 for refreshing every 6 hours).

If you want to refresh every X minutes, then you have to get a bit more creative:


0 0 * * * root  /usr/bin/perl /usr/local/www/lightsquid/lightparser.pl yesterday
X-59/X 0 * * root  /usr/bin/perl /usr/local/www/lightsquid/lightparser.pl today
*/X 1-23 * * root  /usr/bin/perl /usr/local/www/lightsquid/lightparser.pl today

A word of caution: I haven't tested any of these solutions other than the 24h case. That works fine for me, since all historical data is stuffed into lightsquid, and I can use the "refresh now" button if I need data from the current day.

Also, there's still a risk of lost data with this new setup. If the refresh interval is set too short (such as 10 minutes), and there's too much data to process (or the machine is too slow), it won't run 'lightparser.pl yesterday' at midnight if it's still running 'lightparser.pl today' from 11:50 PM. Data loss would be no more than 10 minutes worth, but it's still something to keep in mind as a potential problem.

Anyway, I'll leave the actual PHP hacking to someone more capable. I don't actually know PHP. I just sort of infer based on my familiarity with other languages. Also, I haven't really dug through ls_setup_cron to see how it would handle multiple cron lines – I suspect you'd need a different identifier for each line.

dvserg

Fixed as

per day fresh will started at 23:45
added new yesterday fresh at 0:15

Thanks!