Want package for squidanalyzer. INSTRUCTIONS AND FILES TO INSTALL

AR15USR

Thank you to Topper and Cino for your input here.

After a lot of reading this is what worked for me:

1. Thousands of additional FreeBSD packages can be installed on pfSense as noted here: https://doc.pfsense.org/index.php/Installing_FreeBSD_Packages

2. In the Index of those FreeBSD packages at this link http://pkg.freebsd.org/freebsd:10:x86:64/latest/All/ you will find that SquidAnalyzer is available as "squidanalyzer-6.5.txz"

3. In the shell enter "pkg add http://pkg.freebsd.org/freebsd:10:x86:64/latest/All/squidanalyzer-6.5.txz" (without quotes)

4. You will receive a confirmation of the install. The squidanalyzer.conf file is in usr/local/etc/squidreport/

5. Run the command manually once: cd /usr/local/bin/ && perl squid-analyzer

6. Set this as a cron job to your desire frequency

7. You should be able to access SquidAnalyzer at x.x.x.x/squidreport

It is cleaner to use pkg to install a FreeBSD package than to use WinSCP to copy and paste items from dropbox into your firewall installation. ;)

At step 5 I get this error:

[2.3.3-RELEASE][admin@xxxx.lan]/usr/local/bin: cd /usr/local/bin/ && perl squid-analyzer -r -c /var/squid/logs/access.log
ERROR: you must give a valid output directory. See option: Output

Also, the output from the pkg install says:

Message from squidanalyzer-6.5:
-----------------------------------------------------------------------------
1\. Modify your httpd.conf to allow access to HTML output like follow:
        Alias /squidreport /usr/local/www/squidreport
        <directory usr="" local="" www="" squidreport="">Options -Indexes FollowSymLinks MultiViews
            AllowOverride None
            Order deny,allow
            Deny from all
            Allow from 127.0.0.1</directory> 
2\. If necessary, give additional host access to SquidAnalyzer in httpd.conf.
   Restart and ensure that httpd is running.
3\. Browse to http://my.host.dom/squidreport/ to ensure that things are working
   properly.
4\. Setup a cronjob to run squid-analyzer daily:

     # SquidAnalyzer log reporting daily
     0 2 * * * /usr/local/bin/squid-analyzer > /dev/null 2>&1

or run it manually.
For more information, see /usr/local/share/doc/squidanalyzer/README file.

Where is the httpd.conf file located?

doktornotor

There is no httpd.conf anywhere, obviously. Apache is not part of pfSense install.

AR15USR

Well not obvious to a non developer home user like myself, but now I know. Thanks.

I'm a bit out of my league without a walk through on this one, so I will have to hope for a package as well..

If I run "pkg delete squidanalyzer-6.5.txz" will I break anything? (will that even work?)

Presbuteros

@AR15USR:

At step 5 I get this error:

[2.3.3-RELEASE][admin@xxxx.lan]/usr/local/bin: cd /usr/local/bin/ && perl squid-analyzer -r -c /var/squid/logs/access.log
ERROR: you must give a valid output directory. See option: Output

AR15USR no need to uninstall the package, you are almost there. I failed to mention in my install notes that there is no httpd.conf. Instead there is squidanalyzer.conf and it is asking for a valid output directory. This file is located in

/usr/local/etc/squidreport/

You can edit squidanalyzer.conf by navigating to Diagnostics>Edit File>Browse and browse to that location. Or you can paste /usr/local/etc/squidreport/ into the "Path to be edited" bar and choose "Browse." Either way will get to the folder you need to be in to see squidanalyzer.conf

You can find your default Log Store Directory settings under Services>Squid Proxy Server>General. That log file location needs to be entered into the squidanalyzer.conf file. It is likely the default directory /var/squid/logs/access.log See below the section of the .conf file you will need to edit.


# Set the path to the Squid, squidGuard and/or ufdbGuard log files
LogFile	/var/squid/logs/access.log

Once you edit the .conf file, run this command again: cd /usr/local/bin/ && perl squid-analyzer

Then navigate to x.x.x.x/squidreport and enjoy.

Let us know how it goes.

AR15USR

Thanks for the help Presbuteros. I got the squidanalyzer.conf file edited and ran the command. It took a while to finish but it did so with no error.

When I pointed my browser to http://192.168.1.1/squidreport I get "403 Forbidden nginx".

EDIT: It works if I go to http://192.168.1.1/squidreport/2017/
I browsed the /squidreport folder and there is no .html file in there:

/usr/local/www/squidreport
  ..
  2016
  2017
  images
  SquidAnalyzer.current
  flotr2.js
  sorttable.js
  squidanalyzer.css

Presbuteros

AR15USR glad you did not give up. Interesting that that x.x.x.x/squidreport doesn't work but x.x.x.x/squidreport/2017/ does.
Also interesting that an "index.html" file was not created when the command was run.

I have recreated your issue on my install by deleting my index.html in /usr/local/www/squidreport/ However, when I run the command

cd /usr/local/bin/ && perl squid-analyzer

the index.html file is created again and I can access Squidanalyzer at my x.x.x.x/squidreport

Try this:

1. Compare your .conf file to mine


####
# This file is the default configuration file for SquidAnalyzer
# Edit it to match your needs and copy it under /usr/local/etc/squidreport/squidanalyzer.conf
#####

# Path where SquidAnalyzer should dump all HTML and images files.
# Choose a path that can be read by a Web browser
Output	/usr/local/www/squidreport

# The URL of the SquidAnalyzer javascript, HTML and images files.
WebUrl	/squidreport

# Set the path to the Squid, squidGuard and/or ufdbGuard log files
LogFile	/var/squid/logs/access.log

# If you want to use DNS name instead of client Ip address as username enable
# this directive. When you don't have authentication, the username is set to
# the client ip address, this allow you to use the DNS name instead.
# Note that you must have a working DNS resolution and that it can really slow
# down the generation of reports.
UseClientDNSName	0

# If you have enabled UseClientDNSName and have lot of ip addresses that do
# not resolve you may want to increase the DNS lookup timeout. By default
# SquidAnalyzer will stop to lookup a DNS name after 0.0001 second (100 ms).
DNSLookupTimeout	0.0001

# Set the file containing network alias name. Network are
# show as Ip addresses so if you want to display name instead
# create a file with this format :
# LOCATION_NAME	IP_NETWORK_ADDRESS
# Separator must be a tabulation
NetworkAlias	/usr/local/etc/squidreport/network-aliases

# Set the file containing user alias name. If you don't have auth_proxy
# enable user are seen as Ip addresses, or if you want to replace login
# name by full user name, create a file with this format :
# FULL_USERNAME	IP_ADDRESS || LOGIN_NAME
# Separator must be a tabulation
UserAlias	/usr/local/etc/squidreport/user-aliases

# How do we sort Network, User and Url report screen
# Value can be: bytes, hits or duration. Default is bytes.
OrderNetwork	bytes
OrderUser	bytes
OrderUrl	bytes

# How do we sort Mime types report screen
# Value can be: bytes or hits. Default is bytes.
OrderMime	bytes

# Should we display user URL details. This will show all URL read
# by user. Take care to have enougth space disk for large user.
UrlReport	1

# Enable this directive if you don't want the tree Top URL and Domain HTML tables.
# You will just have the table of Url/Domain ordered per hits then you can still
# sort the URL/Domain order by clicking on each column
UrlHitsOnly	0

# Should we display user details. This will show statistics per user.
UserReport	1

# Run in quiet mode or print debug information
QuietMode	1

# Cost of the bandwith per Mb. If you want to generate invoice per Mb
# for bandwith traffic this can help you. Value 0 mean no cost.
CostPrice	0

# Currency of the bandwith cost
Currency	$

# Top number of url to show
TopNumber	300

# Path to the file containing client ip addresses, network ip address,
# and/or auth login to exclude from report
Exclude	/usr/local/etc/squidreport/excluded

# Path to the file containing client ip addresses, network ip address,
# and/or auth login to include into the report. Other entries will be
# excluded by default.
Include	/usr/local/etc/squidreport/included

# Translation Lang	/usr/local/etc/squidreport/lang/en_US.txt,
# en_US.txt, ru_RU.txt, uk_UA.txt, cs_CZ.txt, pl_PL.txt and de_DE.txt).
# Default to:
#Lang	/usr/local/etc/squidreport/lang/en_US.txt

# Date format used to display date (year = %y, month = %m and day = %d)
# You can also use %M to replace month by its 3 letters abbreviation.
DateFormat	%y-%m-%d

# Set this to 1 if you want to anonymize all user login. The username
# will be replaced by an unique id that change at each squid-analyzer
# run. Default disable.
AnonymizeLogin	0

# Adds peer cache hit (CD_SIBLING_HIT) to be taken has local cache hit.
# Enabled by default, you must disabled it if you don't want to report
# peer cache hit onto your stats.
SiblingHit	1

# Set the default unit for transfert size. Default is BYTES, other possible
# values are KB, MB and GB
TransfertUnit	BYTES

# Minimum percentage of data in pie's graphs to not be placed in the others item.
MinPie		2

# Set this to your locale to display generated date in your language. Default
# is to use strftime. If you want date in German for example, set it to de_DE.
# For french, fr_FR should do the work.
#Locale		en_US

# By default SquidAnalyzer is saving current collected statistics each time
# a new hour is found in log file. Most of the time this is enough but if
# you have huge log file and don't have enough memory this will slow down the
# parser by forcing Perl to use temporaries files. Use lower value following
# your memory and the size of your log file, on very huge log file with lot of
# requests/seconde a value of 30 minutes (1800) or less should help.
WriteDelay	3600

# Use this directive to show the top N users that look at an URL or a domain.
# Set it to 0 to disable this feature.
TopUrlUser	20

# This directive allow you to replace the SquidAnalyze logo by your custom
# logo. The default value is defined as follow:
# [
# ![]($self->{WebUrl}images/logo-squidanalyzer.png "SquidAnalyzer $VERSION")
# ]($self->{WebUrl}) SquidAnalyzer
# Feel free to define your own header but take care to not break current design.
#CustomHeader	[![](http://my.isp.dom/logo.png "My ISP link")](http://my.isp.dom/) My ISP Company

# This directive allow exclusion of some unwanted methods in report statistics
# like HEAD, POST, CONNECT, etc. Can be a comma separated list of methods.
#ExcludedMethods	HEAD

# This directive allow exclusion of some unwanted mimetypes in report statistics
# like text/html, text/plain, or more generally text/*, etc. Can be a comma separated
# list of perl regular expression.
#ExcludedMimes	text/.*,image/.*

# This directive allow exclusion of some unwanted codes in report statistics
# like TCP_DENIED/403 which are generated when a user accesses a page the first
# time without authentication. Can be a comma separated list of methods.
#ExcludedCodes	TCP_DENIED/403

# When SquidAnalyzer find a corrupted line in his data file, it exit immedialtly.
# You can force him to wait for a certain amount of errors before exiting. Of
# course you might want to remove the corrupted line before the next run. This
# can be useful if you have special characters in some fields like mime type.
#MaxFormatError	0

# Set timezone to use when SquidAnalyzer is used in a different server than
# the one running squid and there is a different timezone between these two
# machine. The value must follow format: +/-HH. Default is to use local time.
#TimeZone	+01

2. Reboot (if you haven't already tried this)

Keep us updated…

AR15USR

Presbuteros, I ran the command again, no change.

The only difference between our .conf files I can see is:

CostPrice	0.5
Currency	€
TopUrlUser	20

I changed mine to dollar fyi and interestingly it did not change in the squidanaylizer/2017 page. I will reboot and report back

Presbuteros

Ok. If the reboot was no joy, I suggest an uninstall then reinstall? I just did it on my installation and all went well.

1. I uninstalled Squidanalyzer from my install with

pkg remove squidanalyzer

It notified me that,

Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):
Installed packages to be REMOVED:
        squidanalyzer-6.5
Number of packages to be removed: 1
Proceed with deinstalling packages? [y/N]: y
[1/1] Deinstalling squidanalyzer-6.5...
You may need to manually remove /usr/local/etc/squidreport/squidanalyzer.conf if it is no longer needed.
[1/1] Deleting files for squidanalyzer-6.5: 100%

2. I used WinSCP to log into my PFSense install and navigated to /usr/local/etc/squidreport/
My "squidanalyzer.conf" file was still there. I deleted it.

3. I reinstalled Squidanalyzer with```
pkg add http://pkg.freebsd.org/freebsd:10:x86:64/latest/All/squidanalyzer-6.5.txz

WinSCP was still open to /usr/local/etc/squidreport/ so I clicked the refresh button which revealed the newly installed files.

**4.** I opened the sqidanalyzer.conf file and made my desired changes.

**5.** I ran```
cd /usr/local/bin/ && perl squid-analyzer

6. I navigated to x.x.x.x/squidreport and all was well.

Good luck.

AR15USR

Well, Presbuteros I did everything you suggested:
Rebooted, uninstalled, reinstalled, ran commands etc

After all that I still have to use the "x.x.x.x/squidreport/2017" to see it. It seems to be working like this though, so I'll just stick with it as is.

Last question, do I need to set up cron to run the "cd /usr/local/bin/ && perl squid-analyzer" command once per day?

Presbuteros

Glad it "works" for you.

@AR15USR:

Last question, do I need to set up cron to run the "cd /usr/local/bin/ && perl squid-analyzer" command once per day?

Yes, you can set the cron job to any frequency you want, even multiple times a day depending on your needs. If you check Squidanalyzer more than once a day then build that into your cron job setting.

AR15USR

Great, and thanks for all your help. Much appreciated…

felda

Had a few issues getting this installed, but got it all figured out after a little bit of tinkering on pfSense 2.3.3 and SquidAnalyzer 6.5

My biggest problem was cron not running the task for some reason.
I adjusted the cron command to:

/usr/local/bin/perl /usr/local/bin/squid-analyzer

The second issue I ran into was this error being thrown at me when I visited specific pages in the report.

Mar 16 15:02:07 pfSense pfsense.rrm.local nginx: 2017/03/16 15:02:07 [error] 30129#100219: *234503 open() "/usr/local/www/squidreport/images/up-arrow.png" failed (2: No such file or directory), client: 192.168.16.101, server: , request: "GET /squidreport/images/up-arrow.png HTTP/1.1", host: "192.168.16.1", referrer: "http://192.168.16.1/squidreport/squidanalyzer.css"

My install of SquidAnalyzer didn't come with up-arrow.png for some reason.
To fix this you just need to put a 16x16 .PNG picture inside of /usr/local/www/squidreport/images folder
You can remotely connect using WinSCP and manually put your own file or you can just copy an icon already in that directory.
Run the following command from shell to copy the back arrow image and use it as the up arrow image.

cd /usr/local/www/squidreport/images
cp back-arrow.png up-arrow.png

If you want Squid Analyzer to include SquidGuard blocked / denied pages you can modify the log files it pulls in the config file at /usr/local/etc/squidreports/squidanalyzer.conf
Using a comma to separate the list.

# Set the path to the Squid, squidGuard and/or ufdbGuard log files
LogFile	/var/squid/logs/access.log,/var/squidGuard/log/block.log

The full SquidAnalyzer readme that can help you fully customize the reports to your needs. https://github.com/darold/squidanalyzer/blob/master/README

andyschmid

@felda:

My biggest problem was cron not running the task for some reason.
I adjusted the cron command to:
/usr/local/bin/perl /usr/local/bin/squid-analyzer

I had the same issue and just to add the problem is the shebang in the first line of /usr/local/bin/squid-analyzer file:

#!/usr/bin/env perl

This tries to read environment variable for perl and depending from where you execute it they might exist or not. If executing via cron the environment variables for perl don't exist hence the script fails.

To fix you can change the first line of /usr/local/bin/squid-analyzer to:

#!/usr/local/bin/perl

This makes the script less pflexible but will ensure you can just execute "/usr/local/bin/squid-analyzer" instead of doing the "cd /usr/local/bin/ && perl squid-analyzer" or "/usr/local/bin/perl /usr/local/bin/squid-analyzer"

marcelloc

The package options looks similar to sarg package.

Do you think it's a good idea to create a gui for it on it on Unofficial pfSense package repo?

https://github.com/marcelloc/Unofficial-pfSense-packages

marcelloc

I've included the squidanalyzer package on my Unofficial repo. It includes all steps I found on english and portuguese forum and added a php file to protect report files using pfSense authetication.

more screenshots form the project on github

http://freecode.com/projects/squidanalyzer/screenshots

report_permissions.PNG_thumb

squidanalyzer.png_thumb

squidanalyzer_php.PNG_thumb

Cino

Thank you Marcelloc!!

Works pretty good. I do have a thought, would it be better to use location: '/var/squidreport' instead of '/usr/local/squidreport' ?

marcelloc

@Cino:

Thank you Marcelloc!!

Works pretty good. I do have a thought, would it be better to use location: '/var/squidreport' instead of '/usr/local/squidreport' ?

I'll see. Is there any pfSense version that cleans /var on boot?

Cino

idk the answer to that.. but lightsquid reports are stored at '/var/lightsquid' (define('LS_REPORTPATH', '/var/lightsquid/report');)

jetberrocal

@marcelloc:

I've included the squidanalyzer package on my Unofficial repo. It includes all steps I found on english and portuguese forum and added a php file to protect report files using pfSense authetication.

more screenshots form the project on github

http://freecode.com/projects/squidanalyzer/screenshots

I see the screen shots with some beautiful graphs, but I do not see the list of users nor the denied sites tables, nor the accessed sites tables.

And if used with e2guardian squid format logs the reason for blocking table.

Topper727

In trying to setup this in my new pfsense 2.4 .. will work on 2.4 but not on 2.3.4 please note. Easy setup..

https://forum.pfsense.org/index.php?topic=127299.0