Sarg package for pfsense

ilium007 · Sep 27, 2012, 2:33 PM

Still having issues here !

The sarg-reports folder is in /usr/local (as shown below) but I log out of the pfsense console and then go to the URL:

http://pfsense.harland/sarg_frame.php?prevent=900651242816820700?

and can still see the reports with no authentication. What am doing wrong !!

[2.0.1-RELEASE][admin@pfSense.harland]/usr/local(35): ls -la
total 54
drwxr-xr-x  16 root  wheel    512 Sep 28 00:27 .
drwxr-xr-x  11 root  wheel    512 Mar 11  2012 ..
drwxr-xr-x   2 root  wheel   4608 Sep 28 00:26 bin
drwxr-xr-x   2 root  wheel    512 Mar 11  2012 captiveportal
drwxr-xr-x  18 root  wheel    512 Sep 28 00:24 etc
drwxr-xr-x  27 root  wheel   1536 Sep 28 00:26 include
drwxr-xr-x   2 root  wheel    512 Sep 26 16:43 info
drwxr-xr-x  16 root  wheel  12288 Sep 28 00:26 lib
drwxr-xr-x   5 root  wheel    512 Sep 28 00:24 libdata
drwxr-xr-x   5 root  wheel    512 Sep 26 20:31 libexec
drwxr-xr-x  28 root  wheel    512 Dec 13  2011 man
drwxr-xr-x   6 root  wheel   2560 Sep 28 00:26 pkg
drwxr-xr-x   4 root  wheel    512 Sep 28 00:27 sarg-reports
drwxr-xr-x   2 root  wheel   1536 Sep 28 00:24 sbin
drwxr-xr-x  38 root  wheel   1024 Sep 26 22:18 share
drwxr-xr-x  20 root  wheel   8192 Sep 28 00:27 www
[2.0.1-RELEASE][admin@pfSense.harland]/usr/local(36): ls -la ./www/sarg*
-rwxr-xr-x  1 root  wheel   2152 Sep 28 00:26 ./www/sarg.php
-rwxr-xr-x  1 root  wheel   4308 Sep 28 00:26 ./www/sarg_about.php
-rwxr-xr-x  1 root  wheel   3120 Sep 28 00:26 ./www/sarg_frame.php
-rwxr-xr-x  1 root  wheel   9739 Sep 28 00:26 ./www/sarg_realtime.php
-rwxr-xr-x  1 root  wheel   3314 Sep 28 00:26 ./www/sarg_reports.php
-rwxr-xr-x  1 root  wheel  16917 Sep 28 00:26 ./www/sarg_sorttable.js

./www/sarg-images:
total 26
drwxr-xr-x   3 root  wheel   512 Sep 28 00:27 .
drwxr-xr-x  20 root  wheel  8192 Sep 28 00:27 ..
-rw-r--r--   1 root  wheel   199 Sep 28 00:27 datetime.png
-rw-r--r--   1 root  wheel    95 Sep 28 00:27 graph.png
-rw-r--r--   1 root  wheel   291 Sep 28 00:27 sarg-squidguard-block.png
-rw-r--r--   1 root  wheel  7153 Sep 28 00:27 sarg.png
drwxr-xr-x   2 root  wheel   512 Sep 28 00:27 temp
[2.0.1-RELEASE][admin@pfSense.harland]/usr/local(37):

marcelloc · Sep 27, 2012, 2:53 PM

@ilium007:

http://pfsense.harland/sarg_frame.php?prevent=900651242816820700?

and can still see the reports with no authentication. What am doing wrong !!

Now I got what you were accessing.

I'm checking sarg_frame.php code.

ilium007 · Sep 27, 2012, 2:55 PM

Glad I am not going mad !!

marcelloc · Sep 27, 2012, 3:23 PM

Reinstall the package in 15 minutes and check if it's ok now.

Thanks for the feedback! :)

ilium007 · Sep 27, 2012, 3:28 PM

Will do !! Thanks !

periko · Oct 1, 2012, 4:01 AM

I had follow this package and read this thread, I had try sarg and works, but I have seen that cron run 2 jobs in my system.

I setup squid without any log rotate, but I see that my system rotate my logs twice.

Sep 29 23:00:02 gw php: : Sarg: force refresh now with args, compress() and rotate action after sarg finish.
Sep 29 23:00:17 gw php: : executing squid log rotate after sarg.
Sep 30 00:00:01 gw php: : Sarg: force refresh now with args, compress() and rotate action after sarg finish.
Sep 30 00:00:17 gw php: : executing squid log rotate after sarg.

I setup sarg to rotate at 23h, I have this on cron:

0 * * * * root /usr/bin/nice -n20 newsyslog
1,31 0-5 * * * root /usr/bin/nice -n20 adjkerntz -a
1 3 1 * * root /usr/bin/nice -n20 /etc/rc.update_bogons.sh
*/60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 sshlockout
1 1 * * * root /usr/bin/nice -n20 /etc/rc.dyndns.update
*/60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot
30 12 * * * root /usr/bin/nice -n20 /etc/rc.update_urltables
0 0 * * * root /bin/rm /var/squid/cache/swap.state; /usr/local/sbin/squid -k rotate
*/15 * * * * root /usr/local/pkg/swapstate_check.php
0 */23 * * * root /usr/local/bin/php /usr/local/www/sarg.php 0

Why it run at 00:00?

sarg-2.3.2_2 Squid log analyzer and HTML report generator
squid-2.7.9_1 HTTP Caching Proxy

Any tip will be appreciate, thanks!!!

Tom70 · Oct 3, 2012, 10:33 AM

Hi,

The daily report works great.
But what are the arguments for a weekly and a monthly report.

Tom

marcelloc · Oct 3, 2012, 2:06 PM

@Tom70:

But what are the arguments for a weekly and a monthly report.

google search could help a lot ;).

I've found this on http://lists.freebsd.org/pipermail/freebsd-ports/2005-November/027512.html

TODAY=$(date +%d/%m/%Y)
YESTERDAY=$(date -v-1d +%d/%m/%Y)
WEEKAGO=$(date -v-1w -v-1d +%d/%m/%Y)
MONTHAGO=$(date -v-1m -v-1d +%d/%m/%Y)
YEARAGO=$(date -v-1y -v-1d +%d/%m/%Y)

Tom70 · Oct 4, 2012, 11:41 AM

@marcelloc:

google search could help a lot ;).

I've found this on http://lists.freebsd.org/pipermail/freebsd-ports/2005-November/027512.html

TODAY=$(date +%d/%m/%Y)
YESTERDAY=$(date -v-1d +%d/%m/%Y)
WEEKAGO=$(date -v-1w -v-1d +%d/%m/%Y)
MONTHAGO=$(date -v-1m -v-1d +%d/%m/%Y)
YEARAGO=$(date -v-1y -v-1d +%d/%m/%Y)

Hi Marcelloc

This page I had already found. But somehow it did not work.
But I get no errors.

Tom

marcelloc · Oct 4, 2012, 12:44 PM

Are you rotating the squid log?

Tom70 · Oct 4, 2012, 2:22 PM

Yes. Every 60 days.

marcelloc · Oct 4, 2012, 2:31 PM

@Tom70:

Yes. Every 60 days.

did you tried -d date -v-1w +%d/%m/%Y-date +%d/%m/%Y for a week report?

stramato · Oct 5, 2012, 2:23 AM

hi guys, i've been reading this thread. I just installed Sarg today.

I'm using Squid Transparent with SquidGuard. Just standard config, with log turned on (log rotate also).

I can view Realtime just fine, but I can't seem to generate a report when I try forcing a sched with the following args:

-d date +%d/%m/%Y-date +%d/%m/%Y

I just get this:

Error: Could not find report index file.
Check and save sarg settings and try to force sarg schedule.

Should I do anything special config to make it work?

marcelloc · Oct 5, 2012, 3:05 AM

@stramato:

Should I do anything special config to make it work?

yes, check all sarg config options, reports to generate and create a schedule to run.

Default sarg options has (yes) after it's description. Select all to create a default config.

stramato · Oct 8, 2012, 1:18 AM

@marcelloc:

@stramato:

Should I do anything special config to make it work?

yes, check all sarg config options, reports to generate and create a schedule to run.

Default sarg options has (yes) after it's description. Select all to create a default config.

Thank you. I had to simply select (ctrl+click to highlight) the config options then click save. I got confused because I thought they're already enabled since they already have a (yes) on them.

ckuecker · Oct 8, 2012, 9:15 PM

I have Sarg running on multiple pfsense boxes. One of my boxes has about 100 users behind it and the report will only work for about the first 4 hours after I wipe out the squid logs. After that I am guessing the squid log gets too big and the sarg report will no longer work.

I am using the -d arguments and I have tried limiting the number of users.

Any suggestions on how I can get sarg to accept a larger log file?

marcelloc · Oct 8, 2012, 10:10 PM

@ckuecker:

Any suggestions on how I can get sarg to accept a larger log file?

I have large files working fine.

try to run sarg on console to check what it returns.

ckuecker · Oct 11, 2012, 3:07 PM

@marcelloc:

@ckuecker:

Any suggestions on how I can get sarg to accept a larger log file?

I have large files working fine.

try to run sarg on console to check what it returns.

Seems to be working fine now. I just need to figure out my schedule because, like others my report is pretty empty at 00:00. I need to figure out Cron now.

I have highlighted what I am questioning. Is this rotating my squid logs even after I have set them not to rotate?

![cron sarg.PNG](/public/imported_attachments/1/cron sarg.PNG)
![cron sarg.PNG_thumb](/public/imported_attachments/1/cron sarg.PNG_thumb)

marcelloc · Oct 11, 2012, 3:22 PM

Check on squid config because it's not created by sarg.

ckuecker · Oct 11, 2012, 5:12 PM

@marcelloc:

Check on squid config because it's not created by sarg.

This is my squid config. Rotation should be disabled.

ckuecker · Oct 12, 2012, 8:43 PM

I think it is working now. Thanks for all your help Marcelloc

ckuecker · Oct 15, 2012, 6:06 PM

Marcelloc, I am not sure if this is a bug or if I am doing something / missing something.

I would like to provide access to the Sarg reports to a few users. When I give them permissions via the user manager to the Sarg reports, it does not work fully.
The real time logs work, but when you try and view reports it just flickers non stop. Looks like it is trying to load the sarg reports frame inside the sarg reports frame.

Attached is the permissions I am giving the user. Is there an easier way or is this a bug?

marcelloc · Oct 15, 2012, 6:27 PM

@ckuecker:

Looks like it is trying to load the sarg reports frame inside the sarg reports frame.

Reinstall sarg package, I've fixed it last week.

ckuecker · Oct 15, 2012, 7:06 PM

awesome! Thanks!!

edit: works like a charm!

LoZio · Oct 22, 2012, 4:32 PM

Using nano 2.0.1 and SARG 2.3.2 pkg v.0.6.1.
No matter what I do, tried everithing I found in this forum.
I always get
Error: Could not find report index file.
Check and save sarg settings and try to force sarg schedul

Running sarg -x results in

SARG: sarg version: 2.3.2 Nov-23-2011
SARG: Reading access log file: /var/squid/logs/access.log
SARG: Records in file: 11460, reading: 100.00%
SARG: Records read: 11460, written: 11459, excluded: 0
SARG: Squid log format
SARG: Period: 22 Oct 2012
SARG: pre-sorting files
SARG: File /usr/local/sarg-reports/22Oct2012-22Oct2012 already exists, moved to /usr/local/sarg-reports/22Oct2012-22Oct2012.4
SARG: Cannot delete /usr/local/sarg-reports/22Oct2012-22Oct2012/d192_168_7_11.html - No such file or directory

Saved, re-saved, re-re-re-saved the config with (yes) options.
Deleted and recreated report directories, gave them 777. Created a schedule with every possible combination of parameters, run it manually, scheduled,…
Each time the no index error.

Running a schedule results in
php: /pkg_edit.php: The command '/usr/local/bin/sarg ' returned exit code '1', the output was 'SARG: Records in file: 11647, reading: 0.00%^MSARG: Records in file: 5000, reading: 42.93%^MSARG: Records in file: 10000, reading: 85.86%^MSARG: Cannot delete /usr/local/sarg-reports/22Oct2012-22Oct2012/d192_168_7_11.html - No such file or directory SARG: Records in file: 11647, reading: 100.00%'

If something is written in these forums, I tried it. :(
Realtime works correctly but what I need i history data.
Any other test/debug I can try?

marcelloc · Oct 22, 2012, 4:46 PM

what config and report options did you selected?

this is my current config

wdowney · Nov 26, 2012, 3:54 PM

I had the same problem as LoZio. To get mine to work I did the following -

de-selected all of the options on the general tab and saved it
forced an update on the schedule tab
re-selected the options on the general tab and saved it
forced an update on the schedule tab

This caused the index.html file to be generated in my /usr/local/sarg-reports folder. Up until this point everything else was working except for the index.html file.

hermanleao · Nov 29, 2012, 6:51 PM

@marcelloc:

Hi all,

I've just published sarg package for pfsense with squid,squidguard and dansguardian log Analysis as well real time report tab.

Squidguard functions are under devel yet but squid and dansguardians(as well as I tested) are working.

After almost everything done, I found an old sarg package published on forum by joaohf and merged some function calls from this old thread.

Another good point is that sarg is able to forward logs via email, so I'm planning to include it for nanobsd installs.

have fun and feedback! :)

att,
Marcello Coutinho

Thanks a lot!

Nachtfalke · Nov 29, 2012, 7:48 PM

Hi,

I would like to use sarg package to get a better overview of the blocked sites from squidguard.
I do not have logging enabled on squid - just on squidguard to watch the blocked sites.

In my company it is not allowed to log accessed sites. The log view of squidguard is not the best I think and so I would like to use squidguard.

On the sarg "general" tab I selected "squidguard" and so options on the multiple-choise lists. When saving the settings I got an error on the top right corner that the squid/access.log was not found.

I took a look at the sarg.inc and I think the problem could be somewhere on line 230. But I am not sure. I added a "break;" but without luck.

So my questions are:
Is it possible to use sarg to just "analyse" the blocked.log file of squidguard but no other log files ?

Any help would be appreciated :-)

marcelloc · Nov 30, 2012, 12:22 PM

@Nachtfalke:

So my questions are:
Is it possible to use sarg to just "analyse" the blocked.log file of squidguard but no other log files ?

Hi Nachtfalke,

I've enabled squidguard config options on gui, but I do not use squidguard. take a look on sarg config options and check manually how it should be configured to work with squidguard. I'll push a fix if you find a way to get it working only with squidguard reports.

The missing break was intentional as it requires squid to work.

att,
Marcello Coutinho

Nachtfalke · Nov 30, 2012, 8:56 PM

I changed the following code on sarg.inc starting on line 227:
From:


		case 'squidguard':
			$squidguard_conf='squidguard_conf '.$sarg_proxy['squidguard_config'];
			$redirector_log_format='redirector_log_format #year#-#mon#-#day# #hour# #tmp#/#list#/#tmp#/#tmp#/#url#/#tmp# #ip#/#tmp# #user# #end#';
			#Leve this case without break to include squid log file on squidguard option

To:


		case 'squidguard':
			$access_log= $sarg_proxy['squidguard_block_log'];
			$squidguard_conf='squidguard_conf '.$sarg_proxy['squidguard_config'];
			$redirector_log_format='redirector_log_format #year#-#mon#-#day# #hour# #tmp#/#list#/#tmp#/#tmp#/#url#/#tmp# #ip#/#tmp# #user# #end#';
			#Leve this case without break to include squid log file on squidguard option
		break;

Now I got this error on system log:


Nov 30 21:53:47 	squid[41070]: Squid Parent: child process 41365 started
Nov 30 21:53:46 	squid[30925]: Squid Parent: child process 28838 exited with status 0
Nov 30 21:53:42 	php: /pkg_edit.php: The command '/usr/local/bin/sarg ' returned exit code '1', the output was 'SARG: Records in file: 30911, reading: 0.00%^MSARG: Maybe you have a broken amount of data in your /var/squidGuard/log/block.log file SARG: getword loop detected after 255 bytes. SARG: Line="2012-11-12 17:40:37 [49110] Request(Einge_Internet/none/-) http://tools.google.com/service/update2?w=6:Ihy13C0hp8xIICE3I3l36cwhjObjYjH-7ezo0Kwjmqdp2WQIYaHezKLduIFlOC07QuSuqJStljIF_EJvqlNqH0mGJEvVnkreJQ2qbW71ZWEQEq24CssCY5d9Ij2SpjptLVmxkQea7O1ZlFABARa472hYaKBlD-inQ1Tv_mhFcwGtSnWPlcze4nm8kf-U3F9frIL5ODG5pU6wvGJhMf50_KfRnn_LxvTASxdUPr_pmKRUeElE6XcQz4FfZJtJxQFcuscJFDwxRAKgT4V4rztyV7DbVScLMNy5y_OfKwesqun5J5bg093aLt-twEi8bFZNxjQnPQSUqYuNivTmpnyQFw 172.17.183.27/- - POST REDIRECT" SARG: Record="http://tools.google.com/service/update2?w=6:Ihy13C0hp8xIICE3I3l36cwhjObjYjH-7ezo0Kwjmqdp2WQIYaHezKLduIFlOC07QuSuqJStljIF_EJvqlNqH0mGJEvVnkreJQ2qbW71ZWEQEq24CssCY5d9Ij2SpjptLVmxkQea7O1ZlFABARa472hYaKBlD-inQ1Tv_mhFcwGtSnWPlcze4n
Nov 30 21:53:42 	php: /pkg_edit.php: Sarg: force refresh now with args, compress() and restart action after sarg finish.
Nov 30 21:53:32 	php: /pkg_edit.php: [sarg] sarg_xmlrpc_sync.php is starting.

Not sure what that means ?

PS: Why is xmlrpc sync starting but I did not enable that !?

marcelloc · Nov 30, 2012, 9:16 PM

@Nachtfalke:

Not sure what that means ?

Maybe a too long line

@Nachtfalke:

PS: Why is xmlrpc sync starting but I did not enable that !?

Maybe a print message before the if :)

move

log_error("[sarg] sarg_xmlrpc_sync.php is starting.");

from line 441 to 445 after

if(!$synconchanges)
                return;

LinuxTracker · Dec 1, 2012, 4:31 AM

2.0.1 Release x86 w/ latest Sarg (which is working pretty well)

Was a solution found for the LDAP issue? I've read the thread a few times and didn't see anything definitive.

I've tried every GUI config possible, forcing updates over and over, tweaking the conf file, reinstalled Sarg, restarted pfSense. etc.

I ran the packet sniffer on the LAN adapter for hours and ran another one on the AD LDAP server.
No port 389 traffic from the pfSense box at all.
From what I see, LDAP is dead.

I'll keep trying but I'm not sure where to look next.

Nachtfalke · Dec 1, 2012, 6:38 PM

@marcelloc:

@Nachtfalke:

Not sure what that means ?

Maybe a too long line

Tried again with a blank block.log file from squidguard with a short entry.
SARG does not generate me any reports on that file.

The access.log from squid is working fine - but as I said I do not want that - or better I am not allowed to do that ;-)

So my conclusion is:
The sarg.inc file needs modification to find the block.log file from squidguard. In the sarg.inc the squidguard_block_log variable is created but it will not be used in further code.

BUT it seems that SARG does not know how to interpret the squidguard log files - even if it has some additional options for that. Google couldn't help me until now. Will do further searches.

marcelloc · Dec 5, 2012, 2:30 AM

@LinuxTracker:

Was a solution found for the LDAP issue? I've read the thread a few times and didn't see anything definitive.

Not yet. It looks like a missing LDAP dependence on compile arts. :(

marcelloc · Dec 5, 2012, 2:34 AM

@Nachtfalke:

BUT it seems that SARG does not know how to interpret the squidguard log files - even if it has some additional options for that. Google couldn't help me until now. Will do further searches.

I agree. But It's hard to test without using squidgurd. I did not found a working setup on Google too.

Nachtfalke · Dec 5, 2012, 8:01 AM

@marcelloc:

@Nachtfalke:

BUT it seems that SARG does not know how to interpret the squidguard log files - even if it has some additional options for that. Google couldn't help me until now. Will do further searches.

I agree. But It's hard to test without using squidgurd. I did not found a working setup on Google too.

Just for information - I posted on the squidguard mailing list:
http://www.shalla.de/mailman/private/squidguard/2012-December/002369.html

marcelloc · Dec 5, 2012, 12:25 PM

@Nachtfalke:

Just for information - I posted on the squidguard mailing list:

The list is private :)

Nachtfalke · Dec 5, 2012, 1:23 PM

@marcelloc:

@Nachtfalke:

Just for information - I posted on the squidguard mailing list:

The list is private :)

This is the answer on my question. I will please him to show me his config for sarg and the versions of squidguard and sarg he is using.


Hi Nachtfalke

> 1.) Is it possible to analyze/read squidguard's blocked websites log with
> SARG ?

Yes, it definitly is. They will be shown as blocked sites in SARG. I
am using this exact setup and its working fine. If you like, I can
send you my config file, alongside with the version numbers of the
programms.

Greetings

B. Brandt

LinuxTracker · Dec 5, 2012, 4:46 PM

@marcelloc:

@LinuxTracker:

Was a solution found for the LDAP issue? I've read the thread a few times and didn't see anything definitive.

Not yet. It looks like a missing LDAP dependence on compile arts. :(

OK. Thank you. If time and attention-span allows I'll poke around a bit.

For now I'll try the Users Association option for manual IP/Name mapping.

I have an related Idea:
I'm fantasizing about a pfSense WINS-like feature that would store+associate Username/Machine Name/IP+MAC Addy
In theory it'd pull info from LDAP, pfSense DHCP & DNS or possibly local LAN DHCP & DNS.

The idea is it'd be a single database that packages could use to pull User Info.
Another option -> Pushing data from this db into whatever table a package is using to store it's LDAP/User info.

Is this worth posting as a forum suggestion? I can't tell.