Sarg package for pfsense
-
You can have squidguard denied sites by squidguard by changing squidguard report and squid acl.
Squid3-dev package has this feature, take a look and see how to include it on your current config.
Thanks Marcello!
Finally I get Squid3-dev, SquidGuard-squid3 and sarge to work:
1) after many install/uninstall squidguard started to work only after I selected the transparent proxy interface (not present in the previous squid installed version)
2) On sarge I had to change the squidguard.conf path to /usr/pbi/squidguard-squid3-amd64/etc/squidguard/squidguard.conf on /usr/pbi/sarg-amd64/etc/sarg/sarge.confNow I was trying to understand how to get "denied" sites… sorry but what do you mean by "by changing squidguard report and squid acl"? I can't find any "help" on forum..
Thank you in advance. -
Take a look on squid3-dev general tab
Follow instructions on field "Log denied pages by squidguard"
- 2 months later
-
Hi there !
I have a problem with my Sarg 2.3.6_2 pkg v.0.6.3
Because of my network had many VLANS so i NAT them with a Internet IP, and I put the Pfsensen in edge of the Internet gateway router (next to)
In my Sarg's report , it can't be show the UserID mapping with IP , because these ips were NAT
So , How can i modify the output of report , can its show only UserID field ? guide me ?Thanks so much !
-
So , How can i modify the output of report , can its show only UserID field ? guide me ?
Do you have the usernames logged on you proxy log?
-
Do you have the usernames logged on you proxy log?
NO, there are no Usernames in proxy log, they appear with "-" instead of "Username"
Here is my log in /var/squid/logs/access.log
such as :
1385086726.539 483 192.168.10.10 TCP_MISS/200 1936 POST http://ocsp.thawte.com/ - DIRECT/199.7.52.72 application/ocsp-response
1385086730.465 9 192.168.10.10 TCP_MISS/200 2159 GET http://192.168.10.1/filebrowser/browser.php? - DIRECT/192.168.10.1 text/html
1385086731.484 873 192.168.10.10 TCP_MISS/200 5842 CONNECT vn.data.toolbar.yahoo.com:443 - DIRECT/206.190.42.32 -
1385086732.471 9 192.168.10.10 TCP_MISS/200 2864 GET http://192.168.10.1/filebrowser/browser.php? - DIRECT/192.168.10.1 text/html
1385086732.479 0 192.168.10.10 TCP_MISS/200 1088 GET http://192.168.10.1/filebrowser/images/file_system.gif - DIRECT/192.168.10.1 image/gif -
There is nothing sarg can do if squid logs does not have the client ip.
Look for logging X_forwarded_for info on squid.
This topic may help http://forum.pfsense.org/index.php/topic,54227.msg322323.html#msg322323 -
Yes, I have looked for many other way, but not found something good !
Here is my network :MultiVLAN <–-> Layer3Switch <----> FirewallCisco <----> InternetGW_router <-----> Pfsense (squid+sarg+lightsquid) <----> Internet
Should I change my position of Proxy ?? Where do I put ?
Thanks so much !
-
Enable nat ony on pfsense. Configure all other devices as routers
-
Hi
I am running pfsense 2.1 AMD64 with squid2.7 and squidguard with latest version of SARG.
I have logging enabled for squid and I can see the accessed sites in access.logFurther I have enabled logging on squidguard to log blocked sites.
Unfortunately I only get the access sites on SARG and the DENIED sites which I blacklisted on squid GUI. But I do not get the websites blocked by squidguard.
Here is the debug output of SARG:
[2.1-RELEASE][admin@pfsense2.hpa]/var/squidGuard/log(100): sarg -xz SARG: Init SARG: Loading configuration from /usr/pbi/sarg-amd64/etc/sarg/sarg.conf SARG: TAG: access_log /var/squid/logs/access.log SARG: TAG: graphs yes SARG: TAG: output_dir /usr/local/sarg-reports SARG: TAG: anonymous_output_files no SARG: TAG: resolve_ip no SARG: TAG: user_ip no SARG: TAG: topuser_sort_field BYTES normal SARG: TAG: user_sort_field BYTES normal SARG: TAG: exclude_users /usr/pbi/sarg-amd64/etc/sarg/exclude_users.conf SARG: TAG: exclude_hosts /usr/pbi/sarg-amd64/etc/sarg/exclude_hosts.conf SARG: TAG: date_format e SARG: TAG: lastlog 0 SARG: TAG: remove_temp_files yes SARG: TAG: index yes SARG: TAG: index_tree file SARG: TAG: overwrite_report yes SARG: TAG: use_comma no SARG: TAG: exclude_codes /usr/pbi/sarg-amd64/etc/sarg/exclude_codes SARG: TAG: max_elapsed 0 SARG: TAG: report_type topsites users_sites date_time denied site_user_time_date SARG: TAG: usertab none SARG: TAG: long_url no SARG: TAG: date_time_by bytes elap SARG: TAG: charset UTF-8 SARG: TAG: privacy no SARG: TAG: bytes_in_sites_users_report yes SARG: TAG: topuser_num 0 SARG: TAG: dansguardian_conf SARG: TAG: squidguard_conf /usr/pbi/squidguard-amd64/etc/squidGuard/squidGuard.conf SARG: TAG: redirector_log /var/squidGuard/log/block.log.0 SARG: TAG: redirector_log_format #year#-#mon#-#day# #hour# #tmp#/#list#/#tmp#/#tmp#/#url#/#tmp# #ip#/#tmp# #user# #end# SARG: TAG: show_sarg_info no SARG: TAG: show_sarg_logo no SARG: TAG: displayed_values abbreviation SARG: TAG: authfail_report_limit 0 SARG: TAG: denied_report_limit 0 SARG: TAG: siteusers_report_limit 0 SARG: TAG: user_report_limit 0 SARG: TAG: squidguard_report_limit 0 SARG: TAG: www_document_root /usr/local/www SARG: TAG: ntlm_user_format domainname+username SARG: TAG: realtime_refresh_time 0 SARG: TAG: realtime_types GET,PUT,CONNECT SARG: TAG: realtime_unauthenticated_records show SARG: TAG: sorttable /sarg_sorttable.js SARG: TAG: hostalias /usr/pbi/sarg-amd64/etc/sarg/hostalias SARG: Loading exclude host file from: /usr/pbi/sarg-amd64/etc/sarg/exclude_hosts.conf SARG: Loading exclude file from: /usr/pbi/sarg-amd64/etc/sarg/exclude_users.conf SARG: Reading host alias file "/usr/pbi/sarg-amd64/etc/sarg/hostalias" SARG: List of host names to alias: SARG: Parameters: SARG: Hostname or IP address (-a) = SARG: Useragent log (-b) = SARG: Exclude file (-c) = /usr/pbi/sarg-amd64/etc/sarg/exclude_hosts.conf SARG: Date from-until (-d) = SARG: Email address to send reports (-e) = SARG: Config file (-f) = /usr/pbi/sarg-amd64/etc/sarg/sarg.conf SARG: Date format (-g) = Europe (dd/mm/yyyy) SARG: IP report (-i) = No SARG: Keep temporary files (-k) = No SARG: Input log (-l) = /var/squid/logs/access.log SARG: Redirector log (-L) = /var/squidGuard/log/block.log.0 SARG: Resolve IP Address (-n) = No SARG: Output dir (-o) = /usr/local/sarg-reports/ SARG: Use Ip Address instead of userid (-p) = No SARG: Accessed site (-s) = SARG: Time (-t) = SARG: User (-u) = SARG: Temporary dir (-w) = /tmp/sarg SARG: Debug messages (-x) = Yes SARG: Process messages (-z) = Yes SARG: Previous reports to keep (--lastlog) = 0 SARG: SARG: sarg version: 2.3.6 Arp-21-2013 SARG: Reading access log file: /var/squid/logs/access.log SARG: Records in file: 838, reading: 100.00% SARG: Records read: 838, written: 838, excluded: 0 SARG: Squid log format SARG: (info) date=29/11/2013 SARG: (info) period=29 Nov 2013 SARG: Period: 29 Nov 2013 SARG: (info) outdirname=/usr/local/sarg-reports/29Nov2013-29Nov2013 SARG: Sorting log /tmp/sarg/172_17_0_10.user_unsort SARG: Sorting log /tmp/sarg/172_17_2_61.user_unsort SARG: Sorting log /tmp/sarg/172_17_183_30.user_unsort SARG: Sorting log /tmp/sarg/172_17_0_23.user_unsort SARG: Sorting log /tmp/sarg/172_17_3_144.user_unsort SARG: Sorting log /tmp/sarg/172_17_2_54.user_unsort SARG: Sorting log /tmp/sarg/172_17_2_128.user_unsort SARG: Sorting log /tmp/sarg/172_17_63_83.user_unsort SARG: Sorting log /tmp/sarg/172_17_180_93.user_unsort SARG: Sorting log /tmp/sarg/172_17_2_48.user_unsort SARG: Sorting log /tmp/sarg/172_17_180_86.user_unsort SARG: Sorting log /tmp/sarg/172_17_66_106.user_unsort SARG: Sorting log /tmp/sarg/172_17_0_60.user_unsort SARG: Sorting log /tmp/sarg/172_17_183_73.user_unsort SARG: Sorting log /tmp/sarg/172_17_60_60.user_unsort SARG: Sorting log /tmp/sarg/172_17_180_82.user_unsort SARG: Sorting log /tmp/sarg/172_17_180_96.user_unsort SARG: Sorting log /tmp/sarg/172_17_66_219.user_unsort SARG: Sorting log /tmp/sarg/172_17_180_80.user_unsort SARG: Sorting log /tmp/sarg/172_17_180_85.user_unsort SARG: Sorting log /tmp/sarg/172_17_183_1.user_unsort SARG: Sorting log /tmp/sarg/172_17_2_59.user_unsort SARG: Sorting log /tmp/sarg/172_17_3_123.user_unsort SARG: Sorting log /tmp/sarg/172_17_2_146.user_unsort SARG: Sorting log /tmp/sarg/172_17_3_61.user_unsort SARG: Sorting log /tmp/sarg/172_17_183_75.user_unsort SARG: Sorting log /tmp/sarg/172_17_60_66.user_unsort SARG: Sorting log /tmp/sarg/172_17_60_72.user_unsort SARG: Sorting log /tmp/sarg/172_17_180_94.user_unsort SARG: Sorting log /tmp/sarg/172_17_64_100.user_unsort SARG: Sorting log /tmp/sarg/172_17_63_3.user_unsort SARG: Sorting log /tmp/sarg/172_17_60_61.user_unsort SARG: Sorting log /tmp/sarg/172_17_2_44.user_unsort SARG: Sorting log /tmp/sarg/172_17_66_109.user_unsort SARG: Sorting log /tmp/sarg/172_17_66_220.user_unsort SARG: Sorting log /tmp/sarg/172_17_3_73.user_unsort SARG: Sorting log /tmp/sarg/172_17_2_46.user_unsort SARG: Sorting log /tmp/sarg/172_17_66_104.user_unsort SARG: Sorting log /tmp/sarg/172_17_3_140.user_unsort SARG: Sorting log /tmp/sarg/172_17_2_47.user_unsort SARG: Sorting log /tmp/sarg/172_17_180_83.user_unsort SARG: Sorting log /tmp/sarg/172_17_66_218.user_unsort SARG: Sorting log /tmp/sarg/172_17_2_33.user_unsort SARG: Sorting log /tmp/sarg/172_17_60_62.user_unsort SARG: Sorting log /tmp/sarg/172_17_180_90.user_unsort SARG: Sorting log /tmp/sarg/172_17_63_79.user_unsort SARG: Sorting log /tmp/sarg/172_17_180_95.user_unsort SARG: Sorting log /tmp/sarg/172_17_3_119.user_unsort SARG: Sorting log /tmp/sarg/172_17_183_41.user_unsort SARG: Sorting log /tmp/sarg/172_17_180_84.user_unsort SARG: Sorting log /tmp/sarg/172_17_66_221.user_unsort SARG: Sorting log /tmp/sarg/172_17_0_11.user_unsort SARG: Sorting log /tmp/sarg/172_17_183_74.user_unsort SARG: Sorting log /tmp/sarg/172_17_180_92.user_unsort SARG: (info) Dansguardian report not produced because no dansguardian configuration file was provided SARG: Reading redirector log file /var/squidGuard/log/block.log.0 SARG: Sorting file: /tmp/sarg/redirector.int_log SARG: (info) No top users report because it is not configured in report_type SARG: (info) Downloaded files report not requested in report_type SARG: (info) Sites & users report not requested in report_type SARG: (info) Authentication failures report not requested in report_type SARG: (info) Redirector report not generated because it is empty SARG: Making index.html SARG: Successful report generated on /usr/local/sarg-reports/29Nov2013-29Nov2013 SARG: Purging temporary file sarg-general SARG: End
This is the file of squidguard which contains 5 blocked websites with no special chars or long URLs.
SARG: Reading redirector log file /var/squidGuard/log/block.log.0
I configured that path in sarg.conf
So my problem is why do I get this output:
SARG: (info) Redirector report not generated because it is empty
It shouldn't be empty - it contains blocked websites. I checked the redirector_log_format option on sarg.conf but it was correctly configured for SQUIDGUARD. I changed it to the other possibility just for testing but without luck.
I know that I posted in this thread some months ago with a similar problem but I don't know anymore what to do to get this fixed.
I would appreciate any help!
-
I would appreciate any help!
Try this way:
On squid2 custom options include
acl sglog url_regex -i .*sgrd=ACCESSDENIED;http_access deny sglog;
Edit sgerror.php and include this code
$sge_prefix=(preg_match("/\?/",$cl['u'])?"&":"?"); $str[] = '<iframe src="'.$cl['u'].$sge_prefix.'sgrd=ACCESSDENIED" width="1" height="1"></iframe>';
This way, every time squidguard shows access denied error, it forces client to send the blocked url to squid again to be blocked and logged by sglog acl.
-
Works great! 8)
-
Have an additional question:
SARG logs the denied pages when I open a website in my browser like www.my-website.com and this domain is in my blocklist. Then I got the custom squidguard access denied page with your "pixel" which sends this page to squid.
So if there are any other applications which use http traffic which is blocked by squidguard then this page will not be logged.
Is this correct?
Is this because these apps do not display the blocked page with the "pixel"?Just want to make sure I understand what's happening. So I would check SARG denied pages to check which pages the user directly browsed and squidguard log will tell my everything which is blocked - no matter if via browser or other app.
THANK YOU! :-)
-
If the app does not open/execute the error page, then it will not be logged.
Can you simulate it?
-
If the app does not open/execute the error page, then it will not be logged.
Can you simulate it?
Yes, it is as you said. SquidGuard internal Log page shows blocks on Computers where nobody is logged in and browsing the web via webbrwoser. SARG does not log this.
If I open a webbrowser and go to a webpage which is blocked then I got the denied page and SARG is logging this.
Another question on shedules:
Yesterday - 29. November - I did some "Force Update Now" and always got an updated report. The "Creation date" was always updated on the "View reports" pages.After doing some tests I created a shedule which runs once a day "1d" with no extra arguments.
Today - 30. November - I looked at the "View reports" page and could only see the report from yesterday (29. November). I clicked on "Force update now" and sarg created me a new report for today but did NOT update the report from yesterday. Is this normal!?!I deleted all reports from /usr/local/sarg-reports and forced again an update and only got a report from the 30. November but not from yesterday. I then used custom arduments to get reports from yesterday and so on but did not get the report from yesterday.
Please help! ;)
What I would like to have:
A shedule that will create me a report beginning a 0:00am until 11:59pm for every day.–- edit ---
I think I found an solution:
I disabled log rotation on squid.
I created one shedule with frequency "1d" and logrotate on sarg
I created a second shedule with frequency "1h" and no logrotate.This will update my daily report every hour and will rotate then every day - probably at midnight.
Question:
Where is the difference between:
logrotate
proxy daemon restart
logrotate and restart proxy daemon
?--- edit2 ---
Was it your intention to only sync "General" and "Users" tab? I would think it would be usefull to sync "Shedule", too.
I hat a look at this part in sarg.inc:
/* xml will hold the sections to sync */ $xml = array(); $xml['sarg'] = $config['installedpackages']['sarg']; $xml['sarguser'] = $config['installedpackages']['sarguser']; /* assemble xmlrpc payload */ $params = array( XML_RPC_encode($password), XML_RPC_encode($xml) );
-
I think I found an solution:
I disabled log rotation on squid.
I created one shedule with frequency "1d" and logrotate on sarg
I created a second shedule with frequency "1h" and no logrotate.Question:
Where is the difference between:
logrotate
proxy daemon restart
logrotate and restart proxy daemon-
squid -k rotate
-
squid -k reconfigure
-
squid -k rotate && squid -k reconfigure
Was it your intention to only sync "General" and "Users" tab? I would think it would be usefull to sync "Shedule", too.
Maybe I just forgot to include schedule array on sync.
-
-
I had a look at squidguard log rotation in sarg.inc:
case "squidguard": if ($action =="both" || $action=="rotate"){ log_error('executing squidguard log rotate after sarg.'); log_rotate($sarg_proxy['squidguard_block_log']); file_put_contents($sarg_proxy['squidguard_block_log'],"",LOCK_EX); chown($sarg_proxy['squidguard_block_log'],'proxy'); chgrp($sarg_proxy['squidguard_block_log'],'proxy'); mwexec(SQUID_DIR . '/sbin/squid -k reconfigure'); } #leave this case without break to run squid rotate too.
Log rotation for squidguard's block.log should NOT be done by sarg!
Reason:
Log rotation will be done on squidguard GUI if someone likes it.
squidguard's block.log will not be used by SARG - it is neccessary to use the code marcelloc posted above to send blocked pages back to squid access.log.So I would suggest to disable log rotationg for squidguard's block.log or check if it is enabled/disabled on squidguard GUI.
I for myself jus commented this case part in my sarg.inc for further tests.
-
Push these modifications to github.
-
Push these modifications to github.
Need to do some more tests. Can someone confirm, that:
/usr/pbi/squid-i386/sbin/squid -k rotate
and
/usr/pbi/squid-i386/sbin/squid -k reconfigure
will NOT rotate logs, if:
-
Logging on squid GUI is enabled
-
log rotate field is EMPTY on squid GUI
For me it does not rotate access.log. Could be important if changing the sarg.inc code for squid/squidguard.
-
-
squid -k rotate is the cmd to rotate logs.
-
Hi.
I tried to install Squid3 in place of Squid, and could not get Sarg to work. It worked previously, generating daily reports.
I then re-installed Squid (not squid 3), and cannot get it to work again!
I have it working on one box on one site, but just can't locate this error.
I get:
php: /pkg_edit.php: The command 'export LC_ALL=C && /usr/local/bin/sarg -d
date -v-5m +01/09/%Y
' returned exit code '126', the output was '/usr/local/bin/sarg: Permission denied'Do I look at file permissions somewhere?
Thanks.
-
Are you on latest sarg version?
squid version does not affect sarg. Just keep logging enabled on squid gui.
-
Hi.
I updated Sarg too to try to fix the issue, but it didn't make a difference. Sarg 2.3.6_2 pkg v.0.6.3
What does the error above mean?
Thanks.
-
file permission. but I'm not getting it here. I have some sarg boxes and all are working fine.
What pfsense version are you using? nanobsd?
-
2.0.1-RELEASE (i386)
built on Mon Dec 12 17:53:52 EST 2011
FreeBSD 8.1-RELEASE-p6If it's file permissions, this must be something that Squid changed when I installed squid3? And changing back to squid1 has not altered it?
What file permission would be incorrect?
Thanks.
-
I had a strange issue today after installing squid3 and then sarg (on a new pfsense install) and then uninstalling squid3 then installing squid.
I could not access the reports…complained about missing index.html.
After putting some debug text into the sarg php, I noticed the /usr/local/sarg-reports/index.html was missing and not being created.
To force it to be created, I had to manually select (highlight) "Generate the main index.html (yes)" in Sarg Settings: General Tab: Report Settings/Report Options. Even though is should default to be on(yes).
Then running the schedule (force update) created the main index file and it all worked :)
Not sure what happened there!
-
To force it to be created, I had to manually select (highlight) "Generate the main index.html (yes)" in Sarg Settings: General Tab: Report Settings/Report Options. Even though is should default to be on(yes).
You must select options to enable, that's what field description says.
-
Hi. My index is displaying, but just not for dates since I swapped to Squid 3 and back to Squid.
Where would I find the files to check the permissions on?
Thanks.
-
To force it to be created, I had to manually select (highlight) "Generate the main index.html (yes)" in Sarg Settings: General Tab: Report Settings/Report Options. Even though is should default to be on(yes).
You must select options to enable, that's what field description says.
Thanks for clarifying that.
I think it could be better worded…as it also says default values are in (). I interpret that as these are the Default values. ie. if you do not select anything, it will default to these values....as in Option 2 from the Oxford dictionary:
noun
Pronunciation: /dɪˈfɔːlt, ˈdiːfɔːlt/
1 [mass noun] failure to fulfil an obligation, especially to repay a loan or appear in a law court:
the company will have to restructure its debts to avoid default
[count noun]:
the deteriorating economy pushed defaults to almost $20 billion
2 [in singular] a preselected option adopted by a computer program or other mechanism when no alternative is specified by the user or programmer: -
The next line explains it.
"If you select any option, it will be enabled on conf file."
-
I say this with all due respect and appreciation for making this package available….
I am simply providing user feedback that the wording is unclear and confusing - and does not align with the common usage of the term default nor common UI design practices.
It is your package so feel free to ignore this feedback.
-
Thanks for your feedback
I'll include a longer description to be more clear.
something like
If you select a option, it will be enabled on conf file.
If you do not select a option, it will be disabled on config file.
the () only shows default config values but it does not mean that not selecting it will be enable by default. -
Sounds good.
Thanks for doing that…hopefully will make it easier for newbies to pfsense like me.
- 17 days later
-
Hey, i am a newbie i just installed pfsense 2 days ago… i want to install sarg so that i can log all the sites and downloads that are performed by the users. I was unable to find any guide or anything to proceed. Can someone please help me? :) :)
-
Hey, i am a newbie i just installed pfsense 2 days ago… i want to install sarg so that i can log all the sites and downloads that are performed by the users. I was unable to find any guide or anything to proceed. Can someone please help me? :) :)
SARG does not log anything. SARG just analyzes and visualizes the logs of programs like squid, squidguard and dansguardian.
So before using SARG you should be familar with using a proxy like squid and which sites can be logged and which not or just with some additional configuration.So please first make sure you read the other threads here in the forum about squid, squidguard or dansguardian and if this is working then please explain more in detail what you want to do with sarg and what does not work with your configuration so that we can help.
Good luck!
-
Hi,
I seem to have an issue generating reports from dansguardian using Sarg. I have followed various troubleshooting threads which have not helped fix my problem.
Issue:
Status>Sarg Reports>View Report
Error: Could not find report index file.
Check and save sarg settings and try to force sarg schedule.System/package details:
PFsense 2.1-RELEASE (amd64) FreeBSD 8.3-RELEASE-p11
Dansguardian: 2.12.0.3 pkg v.0.1.8
squid3-dev: 3.3.10 pkg 2.2
Sarg: 2.3.6_2 pkg v.0.6.3
I can see files, including index.html.gz under ']/usr/local/sarg-reports/2013/12/30'
Realtime view in Sarg works ok.
What can I try to troubleshoot this further?
Thanks.
-
Did you enable logging on squid and/or dansguardian?
Did you create a shedule on sarg?
Only enable logrotation on sarg and not on squid.Depending on what reports sarg should create and how big the log files are it could take some time to generate a report.
-
Seems as if it was a time thing. After making no changes and just checking the reporting now, it is working.
-
Seems as if it was a time thing. After making no changes and just checking the reporting now, it is working.
I did have this sometimes when creating a shedule for the first time. Doing a force update now does not work or it seems as it does not but later everything is running.
-
I have installed squid(logging enabled) and i have set transparent mode off(if i switch it on i cannot access most of the websites)
after that if i try sarg, in report: it doesnt show anything(it says you have not configured….) and if i go in realtime and check then it shows only 1 entry of txt/...i have tried to install this soo many times earlier once it showed websites in realtime but only once upon a time that too of only 15 minutes(approx.)
Please help...
I am a newbie
- 5 months later
-
Getting the same message:
rror: Could not find report index file.
Check and save sarg settings and try to force sarg schedule.But I can see the index.html file and other related files in /usr/local/sarg-reports/2014Jun04-2014Jun05
So…I simply copied the index.html up one level and into /usr/local/sarg-reports
the .conf file has the output_dir set to /usr/local/sarg-reports
But...how does this work if files are being placed in these dated sub directories??This seems to have fixed things.
Bug? A configuration setting I need to change somewhere?[EDIT]
Fiddled around with some settings and now it's apparently working?
I'm wondering if disabling the logging function and then re-enabling did something?