SARG Schedule Starting Multiple force refreshs at scheduled time

bryan.paradis · Jan 22, 2014, 3:53 AM

What about:

Sarg log
/tmp/php_errors.txt
crontab

thestealth · Jan 22, 2014, 4:27 AM

sarg.log - I can't seem to find this, what path should I be looking in?

/tmp/php_errors.txt = empty file

crontab:


#
# pfSense specific crontab entries
# Created: January 21, 2014, 9:51 am
#

0	*	*	*	*	root	/usr/bin/nice -n20 newsyslog
1,31	0-5	*	*	*	root	/usr/bin/nice -n20 adjkerntz -a
1	3	1	*	*	root	/usr/bin/nice -n20 /etc/rc.update_bogons.sh
*/60	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 sshlockout
1	1	*	*	*	root	/usr/bin/nice -n20 /etc/rc.dyndns.update
*/60	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot
30	12	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_urltables
0	0	*	*	*	root	/bin/rm /var/squid/cache/swap.state; /usr/local/sbin/squid -k rotate -f /usr/local/etc/squid/squid.conf
*/15	*	*	*	*	root	/usr/local/pkg/swapstate_check.php
0	5	*	*	*	root	/usr/local/bin/mail_reports_generate.php 0 &
0	*/2	*	*	*	root	/usr/bin/perl /usr/local/www/lightsquid/lightparser.pl today
15	0	*	*	*	root	/usr/bin/perl /usr/local/www/lightsquid/lightparser.pl yesterday
0	*/1	*	*	*	root	/usr/local/bin/php /usr/local/www/sarg.php 0
#
# If possible do not add items to this file manually.
# If you do so, this file must be terminated with a blank line (e.g. new line)
#

bryan.paradis · Jan 22, 2014, 4:43 AM

Does the /tmp/sarg/ directory exist? What files are in there?

Can you post contents of sarg.php that is called in your crontab?

bryan.paradis · Jan 22, 2014, 5:00 AM

@thestealth:

Sure no problem. Just a note, when I manually do a force refresh everything works just fine.

How are you forcing manually

thestealth · Jan 22, 2014, 2:03 PM

@bryan.paradis:

Does the /tmp/sarg/ directory exist? What files are in there?

Can you post contents of sarg.php that is called in your crontab?

There is no /tmp/sarg/

I do a force refresh by doing Status -> Sarg Reports -> Schedule -> Edit this item -> Click the "Force update now" button.

Instead of using the built in scheduling tool, could I just create a custom cron job to pass the additional args to Sarg? If so what is the form of the command?

sarg.php:

/* $Id$ */
/* ========================================================================== */
/*
    sarg.php
    part of pfSense (http://www.pfSense.com)
    Copyright (C) 2012 Marcello Coutinho

    All rights reserved.                                                      
*/
/* ========================================================================== */
/*
    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions are met:

     1\. Redistributions of source code must retain the above copyright notice,
        this list of conditions and the following disclaimer.

     2\. Redistributions in binary form must reproduce the above copyright
        notice, this list of conditions and the following disclaimer in the
        documentation and/or other materials provided with the distribution.

    THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
    INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
    AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
    AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
    OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
    SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
    INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
    CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
    ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
    POSSIBILITY OF SUCH DAMAGE.
                                                                              */
/* ========================================================================== */

require_once("/etc/inc/util.inc");
require_once("/etc/inc/functions.inc");
require_once("/etc/inc/pkg-utils.inc");
require_once("/etc/inc/globals.inc");
require_once("/usr/local/pkg/sarg.inc");

$uname=posix_uname();
if ($uname['machine']=='amd64')
        ini_set('memory_limit', '250M');

if (preg_match ("/(\d+)/",$argv[1],$matches))
	run_sarg($matches[1]);

?>

bryan.paradis · Jan 22, 2014, 9:50 PM

Crontab


0	*/1	*	*	*	root	/usr/local/bin/php /usr/local/www/sarg.php 0

Cron starts the job


//sarg.php
if (preg_match ("/(\d+)/",$argv[1],$matches))
        run_sarg($matches[1]);

sarg.php takes out the "0" and passes it to run_sarg("0")


//sarg.inc
function run_sarg($id=-1) {
        global $config, $g,$sarg_proxy;
        #mount filesystem writeable
        conf_mount_rw();
        $cmd =  SARG_DIR . "/bin/sarg";
        if ($id >= 0 && is_array($config['installedpackages']['sargschedule']['config'])){
                $args=$config['installedpackages']['sargschedule']['config'][$id]['args'];
                $action=$config['installedpackages']['sargschedule']['config'][$id]['action'];
                $gzip=$config['installedpackages']['sargschedule']['config'][$id]['gzip'];
                $find=$config['installedpackages']['sargschedule']['config'][$id]['find'];
                $gziplevel=$config['installedpackages']['sargschedule']['config'][$id]['gziplevel'];
                $daylimit=$config['installedpackages']['sargschedule']['config'][$id]['daylimit'];
                }
        else{
                $args=$_POST['args'];
                $action=$_POST['action'];
                $gzip=$_POST['gzip'];
                $find=$_POST['find'];
                $gziplevel=$_POST['gziplevel'];
                $daylimit="";
                }

The if ($id >= 0 && is_array($config['installedpackages']['sargschedule']['config']) should get triggered as $id should = 0

Force Update Now


//sarg.inc
if ($_POST['Submit'] == 'Force update now')
                run_sarg();

Pressing the button launches off sarg but without the php variable being passed to the function


//sarg.inc
function run_sarg($id=-1) {
        global $config, $g,$sarg_proxy;
        #mount filesystem writeable
        conf_mount_rw();
        $cmd =  SARG_DIR . "/bin/sarg";
        if ($id >= 0 && is_array($config['installedpackages']['sargschedule']['config'])){
                $args=$config['installedpackages']['sargschedule']['config'][$id]['args'];
                $action=$config['installedpackages']['sargschedule']['config'][$id]['action'];
                $gzip=$config['installedpackages']['sargschedule']['config'][$id]['gzip'];
                $find=$config['installedpackages']['sargschedule']['config'][$id]['find'];
                $gziplevel=$config['installedpackages']['sargschedule']['config'][$id]['gziplevel'];
                $daylimit=$config['installedpackages']['sargschedule']['config'][$id]['daylimit'];
                }
        else{
                $args=$_POST['args'];
                $action=$_POST['action'];
                $gzip=$_POST['gzip'];
                $find=$_POST['find'];
                $gziplevel=$_POST['gziplevel'];
                $daylimit="";
                }

Default value id=-1 is used and the ELSE is triggered and I think the schedule args are taken correctly from the POST variables. I wonder if there is a problem

Going forward

1. Could you please copy and paste the installed packages section of your /cf/conf/config.xml ? Like below except yours will have sarg etc

	 <installedpackages><tab><menu>
		 <service><package><name>vnstat2</name>
			<website>http://humdi.net/vnstat/</website>

			<pkginfolink>http://forum.pfsense.org/index.php/topic,14179.0.html</pkginfolink>
			<category>Network Management</category>
			<depends_on_package_base_url>http://192.168.55.4/packages/8/All/</depends_on_package_base_url>
			<depends_on_package>vnstat-1.11.tbz</depends_on_package>
			<depends_on_package_pbi>vnstat-1.11_1-i386.pbi</depends_on_package_pbi>
			<build_port_path>/usr/ports/net/vnstat</build_port_path>
			<version>1.10_2</version>
			<status>Stable</status>
			<required_version>2.0</required_version>
			<maintainer>crazypark2@yahoo.dk</maintainer>
			<config_file>http://www.pfsense.com/packages/config/vnstat2/vnstat2.xml</config_file>
			<configurationfile>vnstat2.xml</configurationfile>
			 <after_install_info></after_install_info></package> 

2\. Please check to make sure you do not having multiple cron processes running

ps aux | grep cron

thestealth · Jan 22, 2014, 11:01 PM

1. Too big to fit here, see link: www.spiralex.net/images/packages.xml

ps aux | grep cron
root     241  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root     819  0.0  0.2  3348   760  ??  INs  30Nov13   0:06.38 /usr/sbin/cron -s
root   13181  0.0  0.3  3348  1308  ??  INs  Sat11PM   0:00.48 /usr/sbin/cron -s
root   16048  0.0  0.3  3348  1308  ??  INs  Sat11PM   0:00.46 /usr/sbin/cron -s
root   17501  0.0  0.0  3256     0  ??  IWs  -         0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
root   17640  0.0  0.0  3256   236  ??  I    18Nov13   0:05.04 minicron: helper /usr/local/bin/ping_hosts.sh  (minicron)
root   17958  0.0  0.0  3256     0  ??  IWs  -         0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /etc/rc.expireaccounts
root   18383  0.0  0.0  3256   236  ??  I    18Nov13   0:00.37 minicron: helper /etc/rc.expireaccounts  (minicron)
root   18565  0.0  0.0  3256     0  ??  IWs  -         0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /etc/rc.update_alias_url_data
root   18866  0.0  0.0  3256   236  ??  I    18Nov13   0:00.02 minicron: helper /etc/rc.update_alias_url_data  (minicron)
root   21905  0.0  0.2  3348   760  ??  INs  18Nov13   0:07.80 /usr/sbin/cron -s
root   28740  0.0  0.2  3348   760  ??  INs   8Dec13   0:05.44 /usr/sbin/cron -s
root   29429  0.0  0.2  3348   760  ??  INs  11Jan14   0:01.31 /usr/sbin/cron -s
root   35786  0.0  0.3  3348  1296  ??  IN    1Dec13   0:00.01 cron: running job (cron)
root   36064  0.0  0.3  3348  1304  ??  IN    1Dec13   0:00.00 cron: running job (cron)
root   37549  0.0  0.3  3348  1296  ??  IN    1Dec13   0:00.00 cron: running job (cron)
root   42116  0.0  0.3  3348  1296  ??  IN    1Dec13   0:00.01 cron: running job (cron)
root   44997  0.0  0.2  3348   760  ??  INs  30Nov13   0:06.40 /usr/sbin/cron -s
root   54704  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   57223  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   59479  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   61592  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   24039  0.0  0.2  3460  1188   0  S+    6:01PM   0:00.00 grep cron

bryan.paradis · Jan 22, 2014, 11:19 PM

@thestealth:

1. Too big to fit here, see link: www.spiralex.net/images/packages.xml

Looks fine.

ps aux | grep cron
root     241  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root     819  0.0  0.2  3348   760  ??  INs  30Nov13   0:06.38 /usr/sbin/cron -s
root   13181  0.0  0.3  3348  1308  ??  INs  Sat11PM   0:00.48 /usr/sbin/cron -s
root   16048  0.0  0.3  3348  1308  ??  INs  Sat11PM   0:00.46 /usr/sbin/cron -s
root   17501  0.0  0.0  3256     0  ??  IWs  -         0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
root   17640  0.0  0.0  3256   236  ??  I    18Nov13   0:05.04 minicron: helper /usr/local/bin/ping_hosts.sh  (minicron)
root   17958  0.0  0.0  3256     0  ??  IWs  -         0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /etc/rc.expireaccounts
root   18383  0.0  0.0  3256   236  ??  I    18Nov13   0:00.37 minicron: helper /etc/rc.expireaccounts  (minicron)
root   18565  0.0  0.0  3256     0  ??  IWs  -         0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /etc/rc.update_alias_url_data
root   18866  0.0  0.0  3256   236  ??  I    18Nov13   0:00.02 minicron: helper /etc/rc.update_alias_url_data  (minicron)
root   21905  0.0  0.2  3348   760  ??  INs  18Nov13   0:07.80 /usr/sbin/cron -s
root   28740  0.0  0.2  3348   760  ??  INs   8Dec13   0:05.44 /usr/sbin/cron -s
root   29429  0.0  0.2  3348   760  ??  INs  11Jan14   0:01.31 /usr/sbin/cron -s
root   35786  0.0  0.3  3348  1296  ??  IN    1Dec13   0:00.01 cron: running job (cron)
root   36064  0.0  0.3  3348  1304  ??  IN    1Dec13   0:00.00 cron: running job (cron)
root   37549  0.0  0.3  3348  1296  ??  IN    1Dec13   0:00.00 cron: running job (cron)
root   42116  0.0  0.3  3348  1296  ??  IN    1Dec13   0:00.01 cron: running job (cron)
root   44997  0.0  0.2  3348   760  ??  INs  30Nov13   0:06.40 /usr/sbin/cron -s
root   54704  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   57223  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   59479  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   61592  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   24039  0.0  0.2  3460  1188   0  S+    6:01PM   0:00.00 grep cron

Wish we had checked for multiple cron processes earlier. Learnt some new things though ;)

Your problem is with a cron package from early or before october. Please upgrade your cron package and if you have to stop the cron service or kill them off and start the cron service.

https://forum.pfsense.org/index.php?topic=67445.0

thestealth · Jan 23, 2014, 12:24 AM

@bryan.paradis:

Wish we had checked for multiple cron processes earlier. Learnt some new things though ;)

Your problem is with a cron package from early or before october. Please upgrade your cron package and if you have to stop the cron service or kill them off and start the cron service.

https://forum.pfsense.org/index.php?topic=67445.0

This seems to have worked:

ps aux | grep cron
root     241  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   17501  0.0  0.0  3256     0  ??  IWs  -         0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
root   17640  0.0  0.0  3256   236  ??  I    18Nov13   0:05.05 minicron: helper /usr/local/bin/ping_hosts.sh  (minicron)
root   17958  0.0  0.0  3256     0  ??  IWs  -         0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /etc/rc.expireaccounts
root   18383  0.0  0.0  3256   236  ??  I    18Nov13   0:00.37 minicron: helper /etc/rc.expireaccounts  (minicron)
root   18565  0.0  0.0  3256     0  ??  IWs  -         0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /etc/rc.update_alias_url_data
root   18866  0.0  0.0  3256   236  ??  I    18Nov13   0:00.02 minicron: helper /etc/rc.update_alias_url_data  (minicron)
root   28032  0.0  0.3  3348  1332  ??  Ss    6:39PM   0:00.01 /usr/sbin/cron -s
root   35786  0.0  0.3  3348  1296  ??  IN    1Dec13   0:00.01 cron: running job (cron)
root   36064  0.0  0.3  3348  1304  ??  IN    1Dec13   0:00.00 cron: running job (cron)
root   37549  0.0  0.3  3348  1296  ??  IN    1Dec13   0:00.00 cron: running job (cron)
root   42116  0.0  0.3  3348  1296  ??  IN    1Dec13   0:00.01 cron: running job (cron)
root   54704  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   57223  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   59479  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   61592  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   29000  0.0  0.2  3460  1188   0  S+    7:25PM   0:00.00 grep cron

What do you think?

bryan.paradis · Jan 23, 2014, 1:20 AM

@thestealth:

@bryan.paradis:

Wish we had checked for multiple cron processes earlier. Learnt some new things though ;)

Your problem is with a cron package from early or before october. Please upgrade your cron package and if you have to stop the cron service or kill them off and start the cron service.

https://forum.pfsense.org/index.php?topic=67445.0

This seems to have worked:

ps aux | grep cron
root     241  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   17501  0.0  0.0  3256     0  ??  IWs  -         0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
root   17640  0.0  0.0  3256   236  ??  I    18Nov13   0:05.05 minicron: helper /usr/local/bin/ping_hosts.sh  (minicron)
root   17958  0.0  0.0  3256     0  ??  IWs  -         0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /etc/rc.expireaccounts
root   18383  0.0  0.0  3256   236  ??  I    18Nov13   0:00.37 minicron: helper /etc/rc.expireaccounts  (minicron)
root   18565  0.0  0.0  3256     0  ??  IWs  -         0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /etc/rc.update_alias_url_data
root   18866  0.0  0.0  3256   236  ??  I    18Nov13   0:00.02 minicron: helper /etc/rc.update_alias_url_data  (minicron)
root   28032  0.0  0.3  3348  1332  ??  Ss    6:39PM   0:00.01 /usr/sbin/cron -s
root   35786  0.0  0.3  3348  1296  ??  IN    1Dec13   0:00.01 cron: running job (cron)
root   36064  0.0  0.3  3348  1304  ??  IN    1Dec13   0:00.00 cron: running job (cron)
root   37549  0.0  0.3  3348  1296  ??  IN    1Dec13   0:00.00 cron: running job (cron)
root   42116  0.0  0.3  3348  1296  ??  IN    1Dec13   0:00.01 cron: running job (cron)
root   54704  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   57223  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   59479  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   61592  0.0  0.3  3348  1308  ??  IN    1Jan14   0:00.00 cron: running job (cron)
root   29000  0.0  0.2  3460  1188   0  S+    7:25PM   0:00.00 grep cron

What do you think?

Only 1 now not 6 :) Looks better!

thestealth · Jan 23, 2014, 1:41 AM

I assume I was looking for this line:

root   28032  0.0  0.3  3348  1332  ??  Ss    6:39PM   0:00.01 /usr/sbin/cron -s

No more weirdness in my log either.

Thank a lot it is much appreciated.

bryan.paradis · Jan 23, 2014, 2:14 AM

@thestealth:

I assume I was looking for this line:
root   28032  0.0  0.3  3348  1332  ??  Ss    6:39PM   0:00.01 /usr/sbin/cron -s
No more weirdness in my log either.

Thank a lot it is much appreciated.

Yes as you can see their were 6 running before and you are welcome!