Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Sarg package for pfsense

    Scheduled Pinned Locked Moved pfSense Packages
    467 Posts 99 Posters 504.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • marcellocM
      marcelloc
      last edited by

      Take a look on squid3-dev general tab

      Follow instructions on field "Log denied pages by squidguard"

      Treinamentos de Elite: http://sys-squad.com

      Help a community developer! ;D

      1 Reply Last reply Reply Quote 0
      • S
        steel_dragon
        last edited by

        Hi there !
        I have a problem with my Sarg 2.3.6_2 pkg v.0.6.3
        Because of my network had many VLANS so i NAT them with a Internet IP, and I put the Pfsensen in edge of the Internet gateway router (next to)
        In my Sarg's report , it can't be show the UserID mapping with IP , because these ips were NAT
        So , How can i modify the output of report , can its show only UserID field ? guide me ?

        Thanks so much !

        1 Reply Last reply Reply Quote 0
        • marcellocM
          marcelloc
          last edited by

          @steel_dragon:

          So , How can i modify the output of report , can its show only UserID field ? guide me ?

          Do you have the usernames logged on you proxy log?

          Treinamentos de Elite: http://sys-squad.com

          Help a community developer! ;D

          1 Reply Last reply Reply Quote 0
          • S
            steel_dragon
            last edited by

            Do you have the usernames logged on you proxy log?

            NO, there are no Usernames in proxy log, they appear with "-" instead of "Username"

            Here is my log in /var/squid/logs/access.log

            such as :

            1385086726.539    483 192.168.10.10 TCP_MISS/200 1936 POST http://ocsp.thawte.com/ - DIRECT/199.7.52.72 application/ocsp-response
            1385086730.465      9 192.168.10.10 TCP_MISS/200 2159 GET http://192.168.10.1/filebrowser/browser.php? - DIRECT/192.168.10.1 text/html
            1385086731.484    873 192.168.10.10 TCP_MISS/200 5842 CONNECT vn.data.toolbar.yahoo.com:443 - DIRECT/206.190.42.32 -
            1385086732.471      9 192.168.10.10 TCP_MISS/200 2864 GET http://192.168.10.1/filebrowser/browser.php? - DIRECT/192.168.10.1 text/html
            1385086732.479      0 192.168.10.10 TCP_MISS/200 1088 GET http://192.168.10.1/filebrowser/images/file_system.gif - DIRECT/192.168.10.1 image/gif

            1 Reply Last reply Reply Quote 0
            • marcellocM
              marcelloc
              last edited by

              There is nothing sarg can do if squid logs does not have the client ip.

              Look for logging X_forwarded_for info on squid.
              This topic may help http://forum.pfsense.org/index.php/topic,54227.msg322323.html#msg322323

              Treinamentos de Elite: http://sys-squad.com

              Help a community developer! ;D

              1 Reply Last reply Reply Quote 0
              • S
                steel_dragon
                last edited by

                Yes, I have looked for many other way, but not found something good !
                Here is my network :

                MultiVLAN <–-> Layer3Switch  <----> FirewallCisco <----> InternetGW_router <-----> Pfsense (squid+sarg+lightsquid) <----> Internet

                Should I change my position of Proxy ?? Where do I put ?

                Thanks so much !

                1 Reply Last reply Reply Quote 0
                • marcellocM
                  marcelloc
                  last edited by

                  Enable nat ony on pfsense. Configure all other devices as routers

                  Treinamentos de Elite: http://sys-squad.com

                  Help a community developer! ;D

                  1 Reply Last reply Reply Quote 0
                  • N
                    Nachtfalke
                    last edited by

                    Hi

                    I am running pfsense 2.1 AMD64 with squid2.7 and squidguard with latest version of SARG.
                    I have logging enabled for squid and I can see the accessed sites in access.log

                    Further I have enabled logging on squidguard to log blocked sites.

                    Unfortunately I only get the access sites on SARG and the DENIED sites which I blacklisted on squid GUI. But I do not get the websites blocked by squidguard.

                    Here is the debug output of SARG:

                    
                    [2.1-RELEASE][admin@pfsense2.hpa]/var/squidGuard/log(100): sarg -xz
                    SARG: Init
                    SARG: Loading configuration from /usr/pbi/sarg-amd64/etc/sarg/sarg.conf
                    SARG: TAG: access_log /var/squid/logs/access.log
                    SARG: TAG: graphs yes
                    SARG: TAG: output_dir /usr/local/sarg-reports
                    SARG: TAG: anonymous_output_files no
                    SARG: TAG: resolve_ip no
                    SARG: TAG: user_ip no
                    SARG: TAG: topuser_sort_field BYTES normal
                    SARG: TAG: user_sort_field BYTES normal
                    SARG: TAG: exclude_users /usr/pbi/sarg-amd64/etc/sarg/exclude_users.conf
                    SARG: TAG: exclude_hosts /usr/pbi/sarg-amd64/etc/sarg/exclude_hosts.conf
                    SARG: TAG: date_format e
                    SARG: TAG: lastlog 0
                    SARG: TAG: remove_temp_files yes
                    SARG: TAG: index yes
                    SARG: TAG: index_tree file
                    SARG: TAG: overwrite_report yes
                    SARG: TAG: use_comma no
                    SARG: TAG: exclude_codes /usr/pbi/sarg-amd64/etc/sarg/exclude_codes
                    SARG: TAG: max_elapsed 0
                    SARG: TAG: report_type topsites users_sites date_time denied site_user_time_date
                    SARG: TAG: usertab none
                    SARG: TAG: long_url no
                    SARG: TAG: date_time_by bytes elap
                    SARG: TAG: charset UTF-8
                    SARG: TAG: privacy no
                    SARG: TAG: bytes_in_sites_users_report yes
                    SARG: TAG: topuser_num 0
                    SARG: TAG: dansguardian_conf
                    SARG: TAG: squidguard_conf /usr/pbi/squidguard-amd64/etc/squidGuard/squidGuard.conf
                    SARG: TAG: redirector_log /var/squidGuard/log/block.log.0
                    SARG: TAG: redirector_log_format #year#-#mon#-#day# #hour# #tmp#/#list#/#tmp#/#tmp#/#url#/#tmp# #ip#/#tmp# #user# #end#
                    SARG: TAG: show_sarg_info no
                    SARG: TAG: show_sarg_logo no
                    SARG: TAG: displayed_values abbreviation
                    SARG: TAG: authfail_report_limit 0
                    SARG: TAG: denied_report_limit 0
                    SARG: TAG: siteusers_report_limit 0
                    SARG: TAG: user_report_limit 0
                    SARG: TAG: squidguard_report_limit 0
                    SARG: TAG: www_document_root /usr/local/www
                    SARG: TAG: ntlm_user_format domainname+username
                    SARG: TAG: realtime_refresh_time 0
                    SARG: TAG: realtime_types GET,PUT,CONNECT
                    SARG: TAG: realtime_unauthenticated_records show
                    SARG: TAG: sorttable /sarg_sorttable.js
                    SARG: TAG: hostalias /usr/pbi/sarg-amd64/etc/sarg/hostalias
                    SARG: Loading exclude host file from: /usr/pbi/sarg-amd64/etc/sarg/exclude_hosts.conf
                    SARG: Loading exclude file from: /usr/pbi/sarg-amd64/etc/sarg/exclude_users.conf
                    SARG: Reading host alias file "/usr/pbi/sarg-amd64/etc/sarg/hostalias"
                    SARG: List of host names to alias:
                    SARG: Parameters:
                    SARG:           Hostname or IP address (-a) =
                    SARG:                    Useragent log (-b) =
                    SARG:                     Exclude file (-c) = /usr/pbi/sarg-amd64/etc/sarg/exclude_hosts.conf
                    SARG:                  Date from-until (-d) =
                    SARG:    Email address to send reports (-e) =
                    SARG:                      Config file (-f) = /usr/pbi/sarg-amd64/etc/sarg/sarg.conf
                    SARG:                      Date format (-g) = Europe (dd/mm/yyyy)
                    SARG:                        IP report (-i) = No
                    SARG:             Keep temporary files (-k) = No
                    SARG:                        Input log (-l) = /var/squid/logs/access.log
                    SARG:                   Redirector log (-L) = /var/squidGuard/log/block.log.0
                    SARG:               Resolve IP Address (-n) = No
                    SARG:                       Output dir (-o) = /usr/local/sarg-reports/
                    SARG: Use Ip Address instead of userid (-p) = No
                    SARG:                    Accessed site (-s) =
                    SARG:                             Time (-t) =
                    SARG:                             User (-u) =
                    SARG:                    Temporary dir (-w) = /tmp/sarg
                    SARG:                   Debug messages (-x) = Yes
                    SARG:                 Process messages (-z) = Yes
                    SARG:  Previous reports to keep (--lastlog) = 0
                    SARG:
                    SARG: sarg version: 2.3.6 Arp-21-2013
                    SARG: Reading access log file: /var/squid/logs/access.log
                    SARG: Records in file: 838, reading: 100.00%
                    SARG:    Records read: 838, written: 838, excluded: 0
                    SARG: Squid log format
                    SARG: (info) date=29/11/2013
                    SARG: (info) period=29 Nov 2013
                    SARG: Period: 29 Nov 2013
                    SARG: (info) outdirname=/usr/local/sarg-reports/29Nov2013-29Nov2013
                    SARG: Sorting log /tmp/sarg/172_17_0_10.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_2_61.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_183_30.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_0_23.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_3_144.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_2_54.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_2_128.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_63_83.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_180_93.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_2_48.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_180_86.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_66_106.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_0_60.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_183_73.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_60_60.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_180_82.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_180_96.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_66_219.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_180_80.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_180_85.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_183_1.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_2_59.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_3_123.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_2_146.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_3_61.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_183_75.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_60_66.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_60_72.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_180_94.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_64_100.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_63_3.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_60_61.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_2_44.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_66_109.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_66_220.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_3_73.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_2_46.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_66_104.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_3_140.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_2_47.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_180_83.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_66_218.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_2_33.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_60_62.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_180_90.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_63_79.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_180_95.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_3_119.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_183_41.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_180_84.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_66_221.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_0_11.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_183_74.user_unsort
                    SARG: Sorting log /tmp/sarg/172_17_180_92.user_unsort
                    SARG: (info) Dansguardian report not produced because no dansguardian configuration file was provided
                    SARG: Reading redirector log file /var/squidGuard/log/block.log.0
                    SARG: Sorting file: /tmp/sarg/redirector.int_log
                    SARG: (info) No top users report because it is not configured in report_type
                    SARG: (info) Downloaded files report not requested in report_type
                    SARG: (info) Sites & users report not requested in report_type
                    SARG: (info) Authentication failures report not requested in report_type
                    SARG: (info) Redirector report not generated because it is empty
                    SARG: Making index.html
                    SARG: Successful report generated on /usr/local/sarg-reports/29Nov2013-29Nov2013
                    SARG: Purging temporary file sarg-general
                    SARG: End
                    
                    

                    This is the file of squidguard which contains 5 blocked websites with no special chars or long URLs.

                    
                    SARG: Reading redirector log file /var/squidGuard/log/block.log.0
                    
                    

                    I configured that path in sarg.conf

                    So my problem is why do I get this output:

                    
                    SARG: (info) Redirector report not generated because it is empty
                    
                    

                    It shouldn't be empty - it contains blocked websites. I checked the redirector_log_format option on sarg.conf but it was correctly configured for SQUIDGUARD. I changed it to the other possibility just for testing but without luck.

                    I know that I posted in this thread some months ago with a similar problem but I don't know anymore what to do to get this fixed.

                    I would appreciate any help!

                    1 Reply Last reply Reply Quote 0
                    • marcellocM
                      marcelloc
                      last edited by

                      @Nachtfalke:

                      I would appreciate any help!

                      Try this way:

                      On squid2 custom options include

                      
                      acl sglog url_regex -i .*sgrd=ACCESSDENIED;http_access deny sglog;
                      
                      

                      Edit sgerror.php and include this code

                      $sge_prefix=(preg_match("/\?/",$cl['u'])?"&":"?");
                      $str[] = '<iframe src="'.$cl['u'].$sge_prefix.'sgrd=ACCESSDENIED" width="1" height="1"></iframe>';
                      

                      This way, every time squidguard shows access denied error, it forces client to send the blocked url to squid again to be blocked and logged by sglog acl.

                      Treinamentos de Elite: http://sys-squad.com

                      Help a community developer! ;D

                      1 Reply Last reply Reply Quote 0
                      • N
                        Nachtfalke
                        last edited by

                        Works great!  8)

                        1 Reply Last reply Reply Quote 0
                        • N
                          Nachtfalke
                          last edited by

                          Have an additional question:

                          SARG logs the denied pages when I open a website in my browser like www.my-website.com and this domain is in my blocklist. Then I got the custom squidguard access denied page with your "pixel" which sends this page to squid.

                          So if there are any other applications which use http traffic which is blocked by squidguard then this page will not be logged.

                          Is this correct?
                          Is this because these apps do not display the blocked page with the "pixel"?

                          Just want to make sure I understand what's happening. So I would check SARG denied pages to check which pages the user directly browsed and squidguard log will tell my everything which is blocked - no matter if via browser or other app.

                          THANK YOU! :-)

                          1 Reply Last reply Reply Quote 0
                          • marcellocM
                            marcelloc
                            last edited by

                            If the app does not open/execute the error page, then it will not be logged.

                            Can you simulate it?

                            Treinamentos de Elite: http://sys-squad.com

                            Help a community developer! ;D

                            1 Reply Last reply Reply Quote 0
                            • N
                              Nachtfalke
                              last edited by

                              @marcelloc:

                              If the app does not open/execute the error page, then it will not be logged.

                              Can you simulate it?

                              Yes, it is as you said. SquidGuard internal Log page shows blocks on Computers where nobody is logged in and browsing the web via webbrwoser. SARG does not log this.

                              If I open a webbrowser and go to a webpage which is blocked then I got the denied page and SARG is logging this.

                              Another question on shedules:
                              Yesterday - 29. November - I did some "Force Update Now" and always got an updated report. The "Creation date" was always updated on the "View reports" pages.

                              After doing some tests I created a shedule which runs once a day "1d" with no extra arguments.
                              Today - 30. November - I looked at the "View reports" page and could only see the report from yesterday (29. November). I clicked on "Force update now" and sarg created me a new report for today but did NOT update the report from yesterday. Is this normal!?!

                              I deleted all reports from /usr/local/sarg-reports and forced again an update and only got a report from the 30. November but not from yesterday. I then used custom arduments to get reports from yesterday and so on but did not get the report from yesterday.

                              Please help! ;)

                              What I would like to have:
                              A shedule that will create me a report beginning a 0:00am until 11:59pm for every day.

                              –- edit ---

                              I think I found an solution:
                              I disabled log rotation on squid.
                              I created one shedule with frequency "1d" and logrotate on sarg
                              I created a second shedule with frequency "1h" and no logrotate.

                              This will update my daily report every hour and will rotate then every day - probably at midnight.

                              Question:
                              Where is the difference between:
                              logrotate
                              proxy daemon restart
                              logrotate and restart proxy daemon
                              ?

                              --- edit2 ---

                              Was it your intention to only sync "General" and "Users" tab? I would think it would be usefull to sync "Shedule", too.

                              I hat a look at this part in sarg.inc:

                              
                                      /* xml will hold the sections to sync */
                                      $xml = array();
                                      $xml['sarg'] = $config['installedpackages']['sarg'];
                                      $xml['sarguser'] = $config['installedpackages']['sarguser'];
                                      /* assemble xmlrpc payload */
                                      $params = array(
                                              XML_RPC_encode($password),
                                              XML_RPC_encode($xml)
                                      );
                              
                              
                              1 Reply Last reply Reply Quote 0
                              • marcellocM
                                marcelloc
                                last edited by

                                @Nachtfalke:

                                I think I found an solution:
                                I disabled log rotation on squid.
                                I created one shedule with frequency "1d" and logrotate on sarg
                                I created a second shedule with frequency "1h" and no logrotate.

                                @Nachtfalke:

                                Question:
                                Where is the difference between:
                                logrotate
                                proxy daemon restart
                                logrotate and restart proxy daemon

                                • squid -k rotate

                                • squid -k reconfigure

                                • squid -k rotate && squid -k reconfigure

                                @Nachtfalke:

                                Was it your intention to only sync "General" and "Users" tab? I would think it would be usefull to sync "Shedule", too.

                                Maybe I just forgot to include schedule array on sync.

                                Treinamentos de Elite: http://sys-squad.com

                                Help a community developer! ;D

                                1 Reply Last reply Reply Quote 0
                                • N
                                  Nachtfalke
                                  last edited by

                                  I had a look at squidguard log rotation in sarg.inc:

                                  
                                                          case "squidguard":
                                                                  if ($action =="both" || $action=="rotate"){
                                                                          log_error('executing squidguard log rotate after sarg.');
                                                                          log_rotate($sarg_proxy['squidguard_block_log']);
                                                                          file_put_contents($sarg_proxy['squidguard_block_log'],"",LOCK_EX);
                                                                          chown($sarg_proxy['squidguard_block_log'],'proxy');
                                                                          chgrp($sarg_proxy['squidguard_block_log'],'proxy');
                                                                          mwexec(SQUID_DIR . '/sbin/squid -k reconfigure');
                                                                          }
                                                          #leave this case without break to run squid rotate too.
                                  
                                  

                                  Log rotation for squidguard's block.log should NOT be done by sarg!

                                  Reason:
                                  Log rotation will be done on squidguard GUI if someone likes it.
                                  squidguard's block.log will not be used by SARG - it is neccessary to use the code marcelloc posted above to send blocked pages back to squid access.log.

                                  So I would suggest to disable log rotationg for squidguard's block.log or check if it is enabled/disabled on squidguard GUI.

                                  I for myself jus commented this case part in my sarg.inc for further tests.

                                  1 Reply Last reply Reply Quote 0
                                  • marcellocM
                                    marcelloc
                                    last edited by

                                    Push these modifications to github.

                                    Treinamentos de Elite: http://sys-squad.com

                                    Help a community developer! ;D

                                    1 Reply Last reply Reply Quote 0
                                    • N
                                      Nachtfalke
                                      last edited by

                                      @marcelloc:

                                      Push these modifications to github.

                                      Need to do some more tests. Can someone confirm, that:

                                      /usr/pbi/squid-i386/sbin/squid -k rotate
                                      

                                      and

                                      /usr/pbi/squid-i386/sbin/squid -k reconfigure
                                      

                                      will NOT rotate logs, if:

                                      • Logging on squid GUI is enabled

                                      • log rotate field is EMPTY on squid GUI

                                      For me it does not rotate access.log. Could be important if changing the sarg.inc code for squid/squidguard.

                                      1 Reply Last reply Reply Quote 0
                                      • marcellocM
                                        marcelloc
                                        last edited by

                                        squid -k rotate is the cmd to rotate logs.

                                        Treinamentos de Elite: http://sys-squad.com

                                        Help a community developer! ;D

                                        1 Reply Last reply Reply Quote 0
                                        • L
                                          Lectrician
                                          last edited by

                                          Hi.

                                          I tried to install Squid3 in place of Squid, and could not get Sarg to work.  It worked previously, generating daily reports.

                                          I then re-installed Squid (not squid 3), and cannot get it to work again!

                                          I have it working on one box on one site, but just can't locate this error.

                                          I get:

                                          php: /pkg_edit.php: The command 'export LC_ALL=C && /usr/local/bin/sarg -d date -v-5m +01/09/%Y' returned exit code '126', the output was '/usr/local/bin/sarg: Permission denied'

                                          Do I look at file permissions somewhere?

                                          Thanks.

                                          1 Reply Last reply Reply Quote 0
                                          • marcellocM
                                            marcelloc
                                            last edited by

                                            Are you on latest sarg version?

                                            squid version does not affect sarg. Just keep logging enabled on squid gui.

                                            Treinamentos de Elite: http://sys-squad.com

                                            Help a community developer! ;D

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.