Sarg package for pfsense

marcelloc

Not sure what that means ?

Maybe a too long line

PS: Why is xmlrpc sync starting but I did not enable that !?

Maybe a print message before the if :)

move

log_error("[sarg] sarg_xmlrpc_sync.php is starting.");

from line 441 to 445 after

if(!$synconchanges)
                return;

LinuxTracker

2.0.1 Release x86 w/ latest Sarg (which is working pretty well)

Was a solution found for the LDAP issue? I've read the thread a few times and didn't see anything definitive.

I've tried every GUI config possible, forcing updates over and over, tweaking the conf file, reinstalled Sarg, restarted pfSense. etc.

I ran the packet sniffer on the LAN adapter for hours and ran another one on the AD LDAP server.
No port 389 traffic from the pfSense box at all.
From what I see, LDAP is dead.

I'll keep trying but I'm not sure where to look next.

Nachtfalke

@marcelloc:

@Nachtfalke:

Not sure what that means ?

Maybe a too long line

Tried again with a blank block.log file from squidguard with a short entry.
SARG does not generate me any reports on that file.

The access.log from squid is working fine - but as I said I do not want that - or better I am not allowed to do that ;-)

So my conclusion is:
The sarg.inc file needs modification to find the block.log file from squidguard. In the sarg.inc the squidguard_block_log variable is created but it will not be used in further code.

BUT it seems that SARG does not know how to interpret the squidguard log files - even if it has some additional options for that. Google couldn't help me until now. Will do further searches.

marcelloc

@LinuxTracker:

Was a solution found for the LDAP issue? I've read the thread a few times and didn't see anything definitive.

Not yet. It looks like a missing LDAP dependence on compile arts. :(

marcelloc

@Nachtfalke:

BUT it seems that SARG does not know how to interpret the squidguard log files - even if it has some additional options for that. Google couldn't help me until now. Will do further searches.

I agree. But It's hard to test without using squidgurd. I did not found a working setup on Google too.

Nachtfalke

@marcelloc:

@Nachtfalke:

BUT it seems that SARG does not know how to interpret the squidguard log files - even if it has some additional options for that. Google couldn't help me until now. Will do further searches.

I agree. But It's hard to test without using squidgurd. I did not found a working setup on Google too.

Just for information - I posted on the squidguard mailing list:
http://www.shalla.de/mailman/private/squidguard/2012-December/002369.html

marcelloc

@Nachtfalke:

Just for information - I posted on the squidguard mailing list:

The list is private :)

squidguard_private_list.png_thumb

Nachtfalke

@marcelloc:

@Nachtfalke:

Just for information - I posted on the squidguard mailing list:

The list is private :)

This is the answer on my question. I will please him to show me his config for sarg and the versions of squidguard and sarg he is using.


Hi Nachtfalke

> 1.) Is it possible to analyze/read squidguard's blocked websites log with
> SARG ?

Yes, it definitly is. They will be shown as blocked sites in SARG. I
am using this exact setup and its working fine. If you like, I can
send you my config file, alongside with the version numbers of the
programms.

Greetings

B. Brandt

LinuxTracker

@marcelloc:

@LinuxTracker:

Was a solution found for the LDAP issue? I've read the thread a few times and didn't see anything definitive.

Not yet. It looks like a missing LDAP dependence on compile arts. :(

OK. Thank you. If time and attention-span allows I'll poke around a bit.

For now I'll try the Users Association option for manual IP/Name mapping.

I have an related Idea:
I'm fantasizing about a pfSense WINS-like feature that would store+associate Username/Machine Name/IP+MAC Addy
In theory it'd pull info from LDAP, pfSense DHCP & DNS or possibly local LAN DHCP & DNS.

The idea is it'd be a single database that packages could use to pull User Info.
Another option -> Pushing data from this db into whatever table a package is using to store it's LDAP/User info.

Is this worth posting as a forum suggestion? I can't tell.

Nachtfalke

I got this as answer from the mailing list - not sure if this will help me. Need some time to check what he said and the corresponding .conf files.


Hi

Attached you will find my configs, they are from an ubuntu 10.04
system, running squid 2.7Stable7, squidguard 1.4 and sarg 2.3.

Several pitfalls I remember:
- pay special attention to the HTMLOUT of sarg-reports.conf
- pay special attention to the stopped.log directives in squidguard.conf
- triple check that the squid and squidguard log files are readable
and the HTMLOUT is writable by sarg
- there is a known bug in squidguard concerning some escape chars in
urls that cause the squidguard log file to become malformatted. Sarg
dies when this happens. Therefore I am using a self patched version of
squidguard: http://51762846.de.strato-hosting.eu/bene/public/squidguard/

Try running sarg-reports as root in from the console: It should start
with something like

/usr/sbin/sarg-reports daily
SARG: Init
SARG: Loading configuration from /etc/sarg/sarg.conf
SARG: Loading exclude host file from: /etc/sarg/exclude_hosts
SARG: Loading exclude file from: /etc/sarg/exclude_users
SARG: Parameters:
SARG:           Hostname or IP address (-a) =
SARG:                    Useragent log (-b) =
SARG:                     Exclude file (-c) = /etc/sarg/exclude_hosts
SARG:                  Date from-until (-d) = 04/12/2012-04/12/2012
SARG:    Email address to send reports (-e) =
SARG:                      Config file (-f) = /etc/sarg/sarg.conf
SARG:                      Date format (-g) = Europe (dd/mm/yyyy)
SARG:                        IP report (-i) = No
SARG:                        Input log (-l) = /var/log/squid/access.log
SARG:                   Redirector log (-L) = /var/log/squid/stopped.log
SARG:               Resolve IP Address (-n) = No
SARG:                       Output dir (-o) = /var/www/squid-reports/Daily/
SARG: Use Ip Address instead of userid (-p) = No
SARG:                    Accessed site (-s) =
SARG:                             Time (-t) =
SARG:                             User (-u) =
SARG:                    Temporary dir (-w) = /tmp
SARG:                   Debug messages (-x) = Yes
SARG:                 Process messages (-z) = No

If something goes wrong and you don't know what to make of the error
message, just post it here.

Hope this helps

Greetings

B. Brandt

Nachtfalke

Ok, I did some further tests. the sarg.inc is - as far as I tested it - correct.

But for squidguard it means:
If logging in squid is disabled then SARG cannot display only the blocked URL squidguard reported.
So in my situation I cannot use SARG because I am not allowed to have the squid access.log file. :(

cs80

I am still working on recovering from a disaster caused by this package. I figured I'd drop a note here as a possible warning for anyone that is using this package. It may be possible that this was a user issue, rather then the fault of the package.

I sadly can't provide many details at this point, if I can come across anything I will follow back up. Either case:

I just lost my pfsense box due to massive corruption caused by (indirectly?) Sarg. I had Sarg running for ~2 month, maybe a bit more. A few days ago I noticed issues with networking and started digging into it. I found pfsense to be unresponsive. I rebooted it and started getting a lot of wonderful errors..

Either case, somehow Sarg had created enough files to run me out of inodes. It was somewhere near 60GB's of data, and 9.7M (yes, MILLION) inodes in use.
(this is for a 3 user network)

I believe I was using the stock out of the box configuration on it. Sadly, it was a pain to get setup in the first place, that once it did start working I never did go back and look at it again.

marcelloc

@cs80:

Either case, somehow Sarg had created enough files to run me out of inodes. It was somewhere near 60GB's of data, and 9.7M (yes, MILLION) inodes in use.
(this is for a 3 user network)

I believe I was using the stock out of the box configuration on it. Sadly, it was a pain to get setup in the first place, that once it did start working I never did go back and look at it again.

Current sarg version has compress report files and remove reports older then x days.

Sarg reports use a lot of inodes.
On my setup, I've installed a second disc with zfs just for report files. On zfs disc, I got 30million inodes.

marcelloc

@Nachtfalke:

Ok, I did some further tests. the sarg.inc is - as far as I tested it - correct.

But for squidguard it means:
If logging in squid is disabled then SARG cannot display only the blocked URL squidguard reported.
So in my situation I cannot use SARG because I am not allowed to have the squid access.log file. :(

What changes you did to get squidguard working? can you push it ot github?

Try to point sarg to an access.empty.log file on squid config at sarg.inc. this may solve your problem.

Nachtfalke

@marcelloc:

@Nachtfalke:

Ok, I did some further tests. the sarg.inc is - as far as I tested it - correct.

But for squidguard it means:
If logging in squid is disabled then SARG cannot display only the blocked URL squidguard reported.
So in my situation I cannot use SARG because I am not allowed to have the squid access.log file. :(

What changes you did to get squidguard working? can you push it ot github?

Try to point sarg to an access.empty.log file on squid config at sarg.inc. this may solve your problem.

I tried that with an access.log file which just contains some entries but this didn't help me on the SARG reports. It doesn't show me blocked entries newer than the access.log file entries.

So there isn't anything I could push on github ;-)

In general it is working with your config with squidguard but you need the access.log from squid. If this file isn't present and actual you cannot generate reports.

Is dansguardian doing that without squid access.log file ?

caldwell

I have also run into the error that others are seeing:

Error: Could not find report index file.
Check and save sarg settings and try to force sarg schedule.

Here's what I've done.

Totally uninstalled Sarg pkg.
Used "find" command to locate and remove every directory or file referencing sarg in the name.
Upgraded to absolute latest (2nd release from today) pfsense package.
Rebooted.
Reinstalled Sarg.
Selected all report options and report types on the Sarg page in pfsense.
Hit Save.
Set up a 1h schedule and saved it.
Hit "force update" under the schedule.

ls -al /usr/local/sarg-reports/
total 4
drwxr-xr-x 2 root wheel 512 Dec 10 21:19 .
drwxr-xr-x 19 root wheel 512 Dec 10 21:19 ..

No index file(s) of any kind appear there.

This is a drag. What does it take to get a simple package to just install and work the first time?

Does anyone have a solution on how to fix this manually?

Thanks in advance for any help you can offer.

ps - I did find this in system.log:
Dec 10 21:20:24 gw php: /pkg_edit.php: [sarg] sarg_xmlrpc_sync.php is starting.
Dec 10 21:20:32 gw php: /pkg_edit.php: Sarg: force refresh now with args, compress() and none action after sarg finish.
Dec 10 21:20:32 gw php: /pkg_edit.php: The command '/usr/pbi/sarg-i386/bin/sarg ' returned exit code '1', the output was 'SARG: Cannot set the locale LC_ALL to the environment variable'

marcelloc

Caldwell, there is no bug on sarg package for squid and dansguardian logs.

just take a a look on forum for a working config that I'm using and check your squid access log config.

marcelloc

Nachtfalke,

Maybe a grep on squid log file for denied entries????

This way there will be only denied access to report.

Did you tried to select only denied sites on reports to generate?

Nachtfalke

@marcelloc:

Nachtfalke,

Maybe a grep on squid log file for denied entries????

This way there will be only denied access to report.

You think of a possibility that a script could do the grep on the access.log, just save the denied entries in a new file and delete the original one ?
Didn't try that but could be a possibility.

@marcelloc:

Did you tried to select only denied sites on reports to generate?

Not sure if I did that. But I saw all sites so I suppose that I didn't try that. Perhaps I can try this if I find some spare time. I uninstalled SARG some days ago.

expert_az

2.0.1-RELEASE (amd64)

Hello i'm getting this error on logs:

php: : The command '/usr/local/bin/sarg ' returned exit code '1', the output was 'SARG: Records in file: 16119722, reading: 0.00%^MSARG: Records in file: 5000, reading: 0.03%^MSARG: Records in file: 10000, reading: 0.06%^MSARG: Records in file: 15000, reading: 0.09%^MSARG: Records in file: 20000, reading: 0.12%^MSARG: Records in file: 25000, reading: 0.16%^MSARG: Records in file: 30000, reading: 0.19%^MSARG: Records in file: 35000, reading: 0.22%^MSARG: Records in file: 40000, reading: 0.25%^MSARG: Records in file: 45000, reading: 0.28%^MSARG: Records in file: 50000, reading: 0.31%^MSARG: Records in file: 55000, reading: 0.34%^MSARG: Records in file: 60000, reading: 0.37%^MSARG: Records in file: 65000, reading: 0.40%^MSARG: Records in file: 70000, reading: 0.43%^MSARG: Records in file: 75000, reading: 0.47%^MSARG: Records in file: 80000, reading: 0.50%^MSARG: Records in file: 85000, reading: 0.53%^MSARG: Records in file: 90000, reading: 0.56%^MSARG: Records in file: 95000, reading: 0.59%^MS

and when i try running sarg from console getting this log:

sarg
SARG: Records in file: 16121346, reading: 100.00%
sort: open failed: /tmp/sarg/denied.log.unsort: No such file or directory
SARG: sort command return status 2
SARG: sort command: sort -T "/tmp/sarg" -t " " -k 3,3 -k 5,5 -o "/tmp/sarg/denied.log" "/tmp/sarg/denied.log.unsort"

i did reinstall