Sarg package for pfsense
-
Hey guys, I just installed pfSense today and have been trying to get everything set up and am running into some issues with Sarg (and Lightsquid for that matter).
I'm trying to get Sarg and Lightsquid to generate reports from Squid proxy, but for some reason they're both having trouble.
Sarg gives me an error:
[ Sarg config error: squid log file (/var/squid/logs/access.log) does not exists]
and Lightsquid gives me the error:
Error : report folder '/var/lightsquid/report' not contain any valid data! Please run lightparser.pl (and check 'report' folder content)
For Lightsquid, I tried clicking both "Refresh now" and "Refresh full" but still got the same error. For Sarg I checked /var/squid/logs/ and there is only one file - cache.log
Is there something I haven't configured correctly with Squid, or anyone experience and solve these issues in the past? Any help would be greatly appreciated!
-
Please start a new thread for your problem and I'll be happy to take a look. This thread is for a particular Sarge issue.
-
@KOM:
Please start a new thread for your problem and I'll be happy to take a look. This thread is for a particular Sarge issue.
Sorry, didn't mean to hijack this thread! I posted a new thread here:
https://forum.pfsense.org/index.php?topic=79140.0 - 11 days later
-
I'm a newb at this. Forgive me.
Is there a pretty way of exporting logs? If a manager asks for an Internet usage report for certain computer with a date range, how would I do this?
I take it I create a custom Schedule. I want the report to show the Date/Time and website.
-
You should probably start a new thread instead of hijacking this one that's already 27 pages long.
- about a month later
-
Hi Gurus
After some investigation I obtaind some type of reportBy command line I changed some value and obtain these parameter in the sarg.conf file:
[2.1.3-RELEASE][admin@pfSense.localdomain]/usr/pbi/sarg-amd64/etc/sarg(28): cat sarg.conf | sed -e '/^#/d' -e '/^$/d'
access_log /var/squid/logs/access.log
graphs yes
output_dir /usr/local/sarg-reports
anonymous_output_files yes
resolve_ip yes
user_ip no
topuser_sort_field BYTES normal
user_sort_field BYTES normal
exclude_users /usr/pbi/sarg-amd64/etc/sarg/exclude_users.conf
exclude_hosts /usr/pbi/sarg-amd64/etc/sarg/exclude_hosts.conf
date_format e
lastlog 0
remove_temp_files yes
index yes
index_tree file
overwrite_report yes
use_comma yes
exclude_codes /usr/pbi/sarg-amd64/etc/sarg/exclude_codes
max_elapsed 0
report_type sites_users users_sites
usertab none
long_url yes
date_time_by bytes
charset Latin3
privacy no
bytes_in_sites_users_report no
topuser_num 0
dansguardian_conf
show_sarg_info no
show_sarg_logo no
displayed_values bytes
authfail_report_limit 0
denied_report_limit 0
siteusers_report_limit 0
user_report_limit 0
www_document_root /usr/local/www
ntlm_user_format domainname+username
realtime_refresh_time 0
realtime_types GET,PUT,CONNECT
realtime_unauthenticated_records show
sorttable /sarg_sorttable.js
hostalias /usr/pbi/sarg-amd64/etc/sarg/hostalias
[2.1.3-RELEASE][admin@pfSense.localdomain]/usr/pbi/sarg-amd64/etc/sarg(29):And after run this executable command:
[2.1.3-RELEASE][admin@pfSense.localdomain]/usr/pbi/sarg-amd64/etc/sarg(30): sarg -x
Appear a succesfull response:
….....
...........
..............
SARG: Sorting log /tmp/sarg/141.user_unsort
SARG: Making index.html
SARG: Successful report generated on /usr/local/sarg-reports/06Aug2014-20Aug2014
SARG: Purging temporary file sarg-general
SARG: End
[2.1.3-RELEASE][admin@pfSense.localdomain]/usr/pbi/sarg-amd64/etc/sarg(31):See the picture below
But this report appear another problem
Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 103903809 bytes) in /usr/local/www/sarg_frame.php on line 77
This line means in this file means:
[2.1.3-RELEASE][admin@pfSense.localdomain]/usr/local/www(43): sed '77q;d' sarg_frame.php
print preg_replace($pattern,$replace,$report);
[2.1.3-RELEASE][admin@pfSense.localdomain]/usr/local/www(44):The question is how to resolv this event???
I hope your suggestion / Recomendation
Regard
- 18 days later
-
My Sarg doesn't seem to be generating reports anymore since I updated to pfSense 2.1.5, is this expected or do I need to change something? I already changed the conf so that it was pointing to what I think is the correct access.log (or something like that, I forget if that was the exact file name now) and that seems to have fixed an error I was seeing in my system logs, but no reports are being generated anymore, I just see this in the system logs when it is meant to create a report
php: sarg.php: Sarg: force refresh now with args, compress() and none action after sarg finish.
Any ideas what might be wrong?
- 8 days later
-
my sarg problem start after update to 2.1.5.
sarg view report showing:
Error: Could not find report index file.
Check and save sarg settings and try to force sarg schedule.sys log showing:
Sep 16 00:00:01 php: sarg.php: Sarg: force refresh now with -d
date +%d/%m/%Y
-date +%d/%m/%Y
args, compress(on) and none action after sarg finish.
Sep 16 00:00:01 php: sarg.php: Sarg: force refresh now with -ddate +%d/%m/%Y
-date +%d/%m/%Y
args, compress(on) and none action after sarg finish.[2.1.5-RELEASE][admin@xxxxxx.localdomain]/root(1): sarg
SARG: Records in file: 50, reading: 100.00%
SARG: No records found
SARG: End[2.1.5-RELEASE][admin@xxxxxx.localdomain]/root(5): pkg_info
bsdinstaller-2.0.2014.0410 BSD Installer mega-package
gettext-0.18.3.1 GNU gettext package
libiconv-1.14_1 A character set conversion library[2.1.5-RELEASE][admin@xxxxx.localdomain]/root(6): sarg -x
SARG: Init
SARG: Loading configuration from /usr/pbi/sarg-i386/etc/sarg/sarg.conf
SARG: Loading exclude host file from: /usr/pbi/sarg-i386/etc/sarg/exclude_hosts.conf
SARG: Loading exclude file from: /usr/pbi/sarg-i386/etc/sarg/exclude_users.conf
SARG: Reading host alias file "/usr/pbi/sarg-i386/etc/sarg/hostalias"
SARG: List of host names to alias:
SARG: Deleting temporary directory "/tmp/sarg"
SARG: Parameters:
SARG: Hostname or IP address (-a) =
SARG: Useragent log (-b) =
SARG: Exclude file (-c) = /usr/pbi/sarg-i386/etc/sarg/exclude_hosts.conf
SARG: Date from-until (-d) =
SARG: Email address to send reports (-e) =
SARG: Config file (-f) = /usr/pbi/sarg-i386/etc/sarg/sarg.conf
SARG: Date format (-g) = USA (mm/dd/yyyy)
SARG: IP report (-i) = No
SARG: Keep temporary files (-k) = No
SARG: Input log (-l) = /var/squid/logs/access.log
SARG: Resolve IP Address (-n) = No
SARG: Output dir (-o) = /usr/local/sarg-reports/
SARG: Use Ip Address instead of userid (-p) = No
SARG: Accessed site (-s) =
SARG: Time (-t) =
SARG: User (-u) =
SARG: Temporary dir (-w) = /tmp/sarg
SARG: Debug messages (-x) = Yes
SARG: Process messages (-z) = No
SARG: Previous reports to keep (–lastlog) = 0
SARG:
SARG: sarg version: 2.3.6 Arp-21-2013
SARG: Reading access log file: /var/squid/logs/access.log
SARG: Records in file: 50, reading: 100.00%
SARG: Records read: 50, written: 0, excluded: 0
SARG: Squid log format
SARG: No records found
SARG: Endcat /usr/pbi/sarg-i386/etc/sarg/sarg.conf | more
sarg.conf
TAG: access_log file
# Where is the access.log
# sarg -l fileaccess_log /var/squid/logs/access.log
TAG: graphs yes|no
# Use graphics where is possible.
# graph_days_bytes_bar_color blue|green|yellow|orange|brown|redgraphs yes
#graph_days_bytes_bar_color orangeTAG: graph_font
# The full path to the TTF font file to use to create the graphs. It is required
# if graphs is set to yes.#graph_font /usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf
TAG: title
--More--(byte 529)# sarg.conf
- 5 months later
-
Looks like Sarg is not working again.
Updated to 2.2 the other day. Sarg is the latest version available in packages (2.3.9 pkg v.0.6.4) and running squid (2.7.9 pkg v.4.3.6)Attempting to run the Reports gives 'Could not find report index file.' message.
Trying to run Sarg from the CLI gives me 'Cannot set the locale LC_ALL to the environment variable'Anyone have Sarg working with pf 2.2?
-
Doing a quick search for Sarg and picking a result a little more current would have led you to this thread here with a solution.
-
Funny that you refer to that thread, KOM. I had actually read through that prior to posting.
I had already tried creating the symlink on my system with no luck. That's why I wanted to know if anyone actually has this working with 2.2…hoping that they would confirm whether or not any special steps needed to be taken to make it work. -
I have. I did only the symlink.
Try to run sarg on console to see what erros do you get.
If you install cron package, you can see how I call configured report schedule.
-
Cannot set the locale LC_ALL to the environment variable
I saw this on Solaris about 10 years ago. I forget what the fix was. Haven't seen it on 2.2 x64, and I've installed it many, many times. As Mercello said, run sarg or sarg -x from the console and see what it says.
-
@KOM:
Cannot set the locale LC_ALL to the environment variable
This is already applied to the package since pfSense 2.1 . Run an export before the sarg cmd.
export LC_ALL=C && sarg...
-
$ export LC_ALL=C && sarg SARG: Records in file: 0, reading: 0.00% SARG: No records found SARG: End SARG: Records in file: 0, reading: 100.00%
$ sarg -x SARG: Cannot set the locale LC_ALL to the environment variable
-
looks like you have no logs on squid file
sarg -x also need the export LC_ALL=C &&
-
looks like you have no logs on squid file
sarg -x also need the export LC_ALL=C &&
Not entirely following you by this suggestion. But I think you are wanting the output of this from?
sarg -x export LC_ALL=C &&If so, the output just shows the same command entered (this is from the GUI). No error messages or anything else.
In looking at the log files in /var/squid/log, I noticed that all of the access.log files are EMPTY (0 byte files). The cache logs look normal though. Also, all the dates on the log files in that dir are current from the last few days.
I'm not sure how to proceed next though to troubleshoot this. Seems to me that this may be the issue (or at least part of it). Why are the access log files not accumulating data?[EDIT] Might be on to something. I forced an update in a schedule and then noticed that the .0 access log is accumulating. The View Report tab also now no longer gives me the error about the index file.
So what I've done for a test is to disable the log rotation in the report settings. Log rotation is already set for 30 days in the Squid setup.Still seeing " Cannot set the locale LC_ALL to the environment variable" when I try to run sarg -x though.
Something else that might be helpful from my system.
# LC_ALL=C sarg -x SARG: Init SARG: Loading configuration from /usr/local/etc/sarg/sarg.conf SARG: Chaining IP resolving module "dns" SARG: Loading exclude host file from: /usr/pbi/sarg-i386/etc/sarg/exclude_hosts.conf SARG: Loading exclude file from: /usr/pbi/sarg-i386/etc/sarg/exclude_users.conf SARG: Reading host alias file "/usr/pbi/sarg-i386/etc/sarg/hostalias" SARG: List of host names to alias: SARG: Parameters: SARG: Hostname or IP address (-a) = SARG: Useragent log (-b) = SARG: Exclude file (-c) = /usr/pbi/sarg-i386/etc/sarg/exclude_hosts.conf SARG: Date from-until (-d) = SARG: Email address to send reports (-e) = SARG: Config file (-f) = /usr/local/etc/sarg/sarg.conf SARG: Date format (-g) = Sites & Users (yyyy/ww) SARG: IP report (-i) = No SARG: Keep temporary files (-k) = No SARG: Input log (-l) = /var/squid/logs/access.log SARG: Resolve IP Address (-n) = Yes SARG: Output dir (-o) = /usr/local/sarg-reports/ SARG: Use Ip Address instead of userid (-p) = Yes SARG: Accessed site (-s) = SARG: Time (-t) = SARG: User (-u) = SARG: Temporary dir (-w) = /tmp/sarg SARG: Debug messages (-x) = Yes SARG: Process messages (-z) = No SARG: Previous reports to keep (--lastlog) = 0 SARG: SARG: sarg version: 2.3.9 Sep-21-2014 SARG: Loading User table: /usr/pbi/sarg-i386/etc/sarg/usertab.conf SARG: Reading access log file: /var/squid/logs/access.log SARG: Records in file: 174, reading: 100.00% SARG: Records read: 174, written: 174, excluded: 0 SARG: Squid log format SARG: Period: 2015.05 SARG: Sorting log /tmp/sarg/0.user_unsort SARG: Sorting log /tmp/sarg/1.user_unsort SARG: Sorting log /tmp/sarg/2.user_unsort SARG: (repday) Cannot open log file /usr/local/sarg-reports/2015.05/0/d0.html
Regarding that very last line of output, here's what in the 2015.05 directory:
# ls -la /usr/local/sarg-reports/2015.05 total 18 drwxr-xr-x 2 root wheel 512 Feb 4 11:54 . drwxr-xr-x 5 root wheel 512 Feb 4 11:54 .. -rw-r--r-- 1 root wheel 4437 Feb 4 11:54 index.html -rw-r--r-- 1 root wheel 22 Feb 4 11:54 sarg-date -rw-r--r-- 1 root wheel 1398 Feb 4 11:54 sarg-general -rw-r--r-- 1 root wheel 2 Feb 4 11:54 sarg-users -rw-r--r-- 1 root wheel 116 Feb 4 11:54 top
So it's correct in that there's no "0" directory in which to find the d0.html file it's looking for.
A system wide search for this file DOES show that a copy exist here though.
/usr/pbi/sarg-i386/local/sarg-reports/2015.01.1/0/d0.htmlAnd lastly, the Realtime logging appears to be working correctly.
-
Not entirely following you by this suggestion. But I think you are wanting the output of this from?
sarg -x export LC_ALL=C &&export LC_ALL=C && sarg -x
-
Not entirely following you by this suggestion. But I think you are wanting the output of this from?
sarg -x export LC_ALL=C &&export LC_ALL=C && sarg -x
Here we go:
$ export LC_ALL=C && sarg -x SARG: Init SARG: Loading configuration from /usr/local/etc/sarg/sarg.conf SARG: Chaining IP resolving module "dns" SARG: Loading exclude host file from: /usr/pbi/sarg-i386/etc/sarg/exclude_hosts.conf SARG: Loading exclude file from: /usr/pbi/sarg-i386/etc/sarg/exclude_users.conf SARG: Reading host alias file "/usr/pbi/sarg-i386/etc/sarg/hostalias" SARG: List of host names to alias: SARG: Deleting temporary directory "/tmp/sarg" SARG: Parameters: SARG: Hostname or IP address (-a) = SARG: Useragent log (-b) = SARG: Exclude file (-c) = /usr/pbi/sarg-i386/etc/sarg/exclude_hosts.conf SARG: Date from-until (-d) = SARG: Email address to send reports (-e) = SARG: Config file (-f) = /usr/local/etc/sarg/sarg.conf SARG: Date format (-g) = USA (mm/dd/yyyy) SARG: IP report (-i) = No SARG: Keep temporary files (-k) = No SARG: Input log (-l) = /var/squid/logs/access.log SARG: Resolve IP Address (-n) = Yes SARG: Output dir (-o) = /usr/local/sarg-reports/ SARG: Use Ip Address instead of userid (-p) = Yes SARG: Accessed site (-s) = SARG: Time (-t) = SARG: User (-u) = SARG: Temporary dir (-w) = /tmp/sarg SARG: Debug messages (-x) = Yes SARG: Process messages (-z) = No SARG: Previous reports to keep (--lastlog) = 0 SARG: SARG: sarg version: 2.3.9 Sep-21-2014 SARG: Loading User table: /usr/pbi/sarg-i386/etc/sarg/usertab.conf SARG: Reading access log file: /var/squid/logs/access.log SARG: Records in file: 1042, reading: 0.00% SARG: Records read: 1042, written: 1042, excluded: 0 SARG: Squid log format SARG: Period: 2015 Feb 04 SARG: File /usr/local/sarg-reports/2015Feb04-2015Feb04 already exists, moved to /usr/local/sarg-reports/2015Feb04-2015Feb04.2 SARG: Sorting log /tmp/sarg/0.user_unsort SARG: Making file: /tmp/sarg/0 SARG: Sorting log /tmp/sarg/1.user_unsort SARG: Making file: /tmp/sarg/1 SARG: Sorting log /tmp/sarg/2.user_unsort SARG: Making file: /tmp/sarg/2 SARG: Sorting log /tmp/sarg/3.user_unsort SARG: Making file: /tmp/sarg/3 SARG: Sorting log /tmp/sarg/4.user_unsort SARG: Making file: /tmp/sarg/4 SARG: Sorting log /tmp/sarg/5.user_unsort SARG: Making file: /tmp/sarg/5 SARG: Sorting log /tmp/sarg/6.user_unsort SARG: Making file: /tmp/sarg/6 SARG: (repday) Cannot open log file /usr/local/sarg-reports/2015Feb04-2015Feb04/5/d5.html SARG: Records in file: 1042, reading: 100.00%
-
-
KOM: That's pretty odd that something like the chosen report selection is causing this…but that was issue! :o
Is this a bug or is this something that is out of Sarg's control?BTW: Thank you both, KOM and marcelloc!!
-
Is this a bug or is this something that is out of Sarg's control?
Probably a bug in the pfSense Sarg package. Sarg is currently at 2.3.9 while the pfSense package is 2.3.6 so it's 1.5 years older, and as far as I know it's always acted funky like that. Pick the wrong report and the whole thing falls over.
Glad to hear you got it working.
-
looks like its using 2.3.9 on 2.2
https://github.com/pfsense/pfsense-packages/blob/master/pkg_config.10.xml
<depends_on_package_pbi>sarg-2.3.9-##ARCH##.pbi</depends_on_package_pbi>
I haven't downloaded the package myself yet on 2.2
-
I must have been looking at my 2.1.5 box.
-
The only manual fix I had to do on my 2.2 labs was the manual symlink to fix pbi mess.
-
@KOM:
OK, making progress. Sarg seems to be one of the more fragile packages. If you happen to select the wrong report options or report to generate, it won't work. Here is what I use and it seems to work OK:
Hi, I can't get Sarg to produce any report, not even with these settings. Any idea ?
Realtime works fine thoughtI got one Report in the list, which is broken, from last year when I tried it once. How can I delete this report and strat over new ?
-
@Satras:
I got one Report in the list, which is broken, from last year when I tried it once. How can I delete this report and strat over new ?
Did you tried to remove old reports via console/ssh ?
-
I've been digging sarg codes these past days. I tried hacking the template which overrides any changes on the sarg.conf file. I hope we can point the directory of squid rather than have it fixed directory
-
@tux:
rather than have it fixed directory
Do you mean /usr/local/sarg-reports ?
Sarg package needs this to "jail" report access permissions on pfsense gui.
-
No, I mean the sarg.template file. Since whenever there is a change in the config(on the webconfig) it is overridden by the template. So I changed the access log path. Then I also tried creating a folder on a separate drive and symlinked to the default sarg-reports folder but it was a fail. Hope we can configure the path of the access log and same for where to store sarg reports.
-
Hi marcelloc,
i figured out that my sarg dont rotate access.log file. I got a 21GB logfile and wonder why my reports takes so long ;D.
Log: php: /pkg_edit.php: Sarg: force refresh now with -d
date +%d/%m/%Y
args, compress(on) and rotate action after sarg finish.But i saw only cache.log seams to be rotated.
If i use the rotation settings on proxy server tab it works…..but then i have no sarg reports over long period ()eg. 30 days).
Versions:
PfSense 2.1.5 (i386)
squid3 3.1.20 pkg 2.1.2
squidguard 1.4_4 pkg v.1.9.6
havp 0.91_1 pkg v1.05
sarg 2.3.6_2 pkg v.0.6.3
Lightsquid 1.8.2 pkg v.2.33thanks
PS: maybe lightsquid prevent sarg from rotating...so i temporary disabled automatic reports in lightsquid.
-
thanks for the feedback, I'll take a look on rotate call done by sarg.
what pfsense version are you using?
-
hi,
what pfsense version are you using?
PfSense 2.1.5 (i386)
-
Hi I am new to pfsense and i was able to figure out IP sec vpn but i can not get the reports to work nor can I get the realtime to show any dans names.
Can any one help me?
I am running the latest version of pfsense. -
Hi I am new to pfsense and i was able to figure out IP sec vpn but i can not get the reports to work nor can I get the realtime to show any dans names.
Can any one help me?
I am running the latest version of pfsense.Maybe you forgot to enable logging on squid settings.
-
Hi,
My goal with Sarg is to create Squid's daily reports. I have logging and logging rotation both enabled on Squid. My question is why Sarg doesn't show any reports for the previous day (with -d
date -v-1d +%d/%m/%Y
extra args) even though I definitely have yesterday's reports in /var/squid/logs. Is it because of Squid's own logging rotation feature? Do I need to turn it off? Or is it because of 60 minutes restriction on "Find Limit" option? I'm asking this because Sarg easily creates daily reports with -ddate +%d/%m/%Y
extra args.I'm using pfSense 2.2-RELEASE (amd64) with Sarg package 2.3.9 pkg v.0.6.4. Enabled Sarg report options are: 1, 5-10, 13-16.
Thank you!
-
I had a look in the sarg.conf file when I was looking for somthing other and I found that the configuration there only points to the actual access.log file.
access_log /var/log/squid/access.log
So I assume when squid rotates your logs then sarg cannot analyze the logs form access.log.1.
I did some research and found out that since some newer version of sarg (2.3?) - which is installed on pfsense - there is the possibility to set a "*" so sarg analyzes more logfiles.access_log /var/log/squid/access.log*
Source:
http://sourceforge.net/p/sarg/discussion/363374/thread/e2e10ffb/ -
Thanks for your reply!
I modified access_log string in sarg.conf and after that I got "file not found" error (Cannot get the modification time of input log file /var/squid/logs/access.log* (No such file or directory)). So the current version doesn't support this feature.
-
@worldfirst You can not directly edit the sarg.conf since it will be changed backed to its original configuration using the template which is the sarg.template file
-
Yes, I noticed that.