Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Sarg package for pfsense

    Scheduled Pinned Locked Moved pfSense Packages
    467 Posts 99 Posters 563.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      Averenix
      last edited by

      Potential fix for those experiencing the:

      Error: Could not find report index file.
      Check and save sarg settings and try to force sarg schedule.

      Seems there is a bug with Sarg 2.3.6 (which the pfSense package uses) which means you MUST select sites_users and users_sites in the report types, otherwise Sarg fails to run.
      See bug here: http://sourceforge.net/p/sarg/bugs/154/

      This has been resolved in Sarg 2.4.

      1 Reply Last reply Reply Quote 0
      • M
        MarkVLK
        last edited by

        Hey guys, I just installed pfSense today and have been trying to get everything set up and am running into some issues with Sarg (and Lightsquid for that matter).

        I'm trying to get Sarg and Lightsquid to generate reports from Squid proxy, but for some reason they're both having trouble.

        Sarg gives me an error:

        [ Sarg config error: squid log file (/var/squid/logs/access.log) does not exists]

        and Lightsquid gives me the error:

        Error : report folder '/var/lightsquid/report' not contain any valid data! Please run lightparser.pl (and check 'report' folder content)

        For Lightsquid, I tried clicking both "Refresh now" and "Refresh full" but still got the same error. For Sarg I checked /var/squid/logs/ and there is only one file - cache.log

        Is there something I haven't configured correctly with Squid, or anyone experience and solve these issues in the past? Any help would be greatly appreciated!

        1 Reply Last reply Reply Quote 0
        • KOMK
          KOM
          last edited by

          Please start a new thread for your problem and I'll be happy to take a look.  This thread is for a particular Sarge issue.

          1 Reply Last reply Reply Quote 0
          • M
            MarkVLK
            last edited by

            @KOM:

            Please start a new thread for your problem and I'll be happy to take a look.  This thread is for a particular Sarge issue.

            Sorry, didn't mean to hijack this thread! I posted a new thread here:
            https://forum.pfsense.org/index.php?topic=79140.0

            1 Reply Last reply Reply Quote 0
            • P
              PokerMunkee
              last edited by

              I'm a newb at this.  Forgive me.

              Is there a pretty way of exporting logs?  If a manager asks for an Internet usage report for certain computer with a date range, how would I do this?

              I take it I create a custom Schedule.  I want the report to show the Date/Time and website.

              1 Reply Last reply Reply Quote 0
              • KOMK
                KOM
                last edited by

                You should probably start a new thread instead of hijacking this one that's already 27 pages long.

                1 Reply Last reply Reply Quote 0
                • P
                  peruvichito2014
                  last edited by

                  Hi Gurus
                  After some investigation I obtaind some type of report

                  By command line I changed some value and obtain these parameter in the sarg.conf file:

                  [2.1.3-RELEASE][admin@pfSense.localdomain]/usr/pbi/sarg-amd64/etc/sarg(28): cat sarg.conf | sed -e '/^#/d' -e '/^$/d'
                  access_log /var/squid/logs/access.log
                  graphs yes
                  output_dir /usr/local/sarg-reports
                  anonymous_output_files yes
                  resolve_ip yes
                  user_ip no
                  topuser_sort_field BYTES normal
                  user_sort_field BYTES normal
                  exclude_users /usr/pbi/sarg-amd64/etc/sarg/exclude_users.conf
                  exclude_hosts /usr/pbi/sarg-amd64/etc/sarg/exclude_hosts.conf
                  date_format e
                  lastlog 0
                  remove_temp_files yes
                  index yes
                  index_tree file
                  overwrite_report yes
                  use_comma yes
                  exclude_codes /usr/pbi/sarg-amd64/etc/sarg/exclude_codes
                  max_elapsed 0
                  report_type sites_users users_sites
                  usertab none
                  long_url yes
                  date_time_by bytes
                  charset Latin3
                  privacy no
                  bytes_in_sites_users_report no
                  topuser_num 0
                  dansguardian_conf
                  show_sarg_info no
                  show_sarg_logo no
                  displayed_values bytes
                  authfail_report_limit 0
                  denied_report_limit 0
                  siteusers_report_limit 0
                  user_report_limit 0
                  www_document_root /usr/local/www
                  ntlm_user_format domainname+username
                  realtime_refresh_time 0
                  realtime_types GET,PUT,CONNECT
                  realtime_unauthenticated_records show
                  sorttable /sarg_sorttable.js
                  hostalias /usr/pbi/sarg-amd64/etc/sarg/hostalias
                  [2.1.3-RELEASE][admin@pfSense.localdomain]/usr/pbi/sarg-amd64/etc/sarg(29):

                  And after run this executable command:

                  [2.1.3-RELEASE][admin@pfSense.localdomain]/usr/pbi/sarg-amd64/etc/sarg(30): sarg -x

                  Appear a succesfull response:

                  ….....
                  ...........
                  ..............
                  SARG: Sorting log /tmp/sarg/141.user_unsort
                  SARG: Making index.html
                  SARG: Successful report generated on /usr/local/sarg-reports/06Aug2014-20Aug2014
                  SARG: Purging temporary file sarg-general
                  SARG: End
                  [2.1.3-RELEASE][admin@pfSense.localdomain]/usr/pbi/sarg-amd64/etc/sarg(31):

                  See the picture below

                  But this report appear another problem

                  Fatal error: Allowed memory size of 262144000 bytes exhausted (tried to allocate 103903809 bytes) in /usr/local/www/sarg_frame.php on line 77

                  This line means in this file means:

                  [2.1.3-RELEASE][admin@pfSense.localdomain]/usr/local/www(43): sed '77q;d' sarg_frame.php
                          print preg_replace($pattern,$replace,$report);
                  [2.1.3-RELEASE][admin@pfSense.localdomain]/usr/local/www(44):

                  The question is how to resolv this event???

                  I hope your suggestion / Recomendation

                  Regard

                  Reportes.jpg
                  Reportes.jpg_thumb

                  1 Reply Last reply Reply Quote 0
                  • M
                    MarkVLK
                    last edited by

                    My Sarg doesn't seem to be generating reports anymore since I updated to pfSense 2.1.5, is this expected or do I need to change something? I already changed the conf so that it was pointing to what I think is the correct access.log (or something like that, I forget if that was the exact file name now) and that seems to have fixed an error I was seeing in my system logs, but no reports are being generated anymore, I just see this in the system logs when it is meant to create a report

                    php: sarg.php: Sarg: force refresh now with args, compress() and none action after sarg finish.

                    Any ideas what might be wrong?

                    1 Reply Last reply Reply Quote 0
                    • S
                      sujyo1
                      last edited by

                      my sarg problem start after update to 2.1.5.

                      sarg view report showing:

                      Error: Could not find report index file.
                      Check and save sarg settings and try to force sarg schedule.

                      sys log showing:

                      Sep 16 00:00:01 php: sarg.php: Sarg: force refresh now with -d date +%d/%m/%Y-date +%d/%m/%Y args, compress(on) and none action after sarg finish.
                      Sep 16 00:00:01 php: sarg.php: Sarg: force refresh now with -d date +%d/%m/%Y-date +%d/%m/%Y args, compress(on) and none action after sarg finish.

                      [2.1.5-RELEASE][admin@xxxxxx.localdomain]/root(1): sarg
                      SARG: Records in file: 50, reading: 100.00%
                      SARG: No records found
                      SARG: End

                      [2.1.5-RELEASE][admin@xxxxxx.localdomain]/root(5): pkg_info
                      bsdinstaller-2.0.2014.0410 BSD Installer mega-package
                      gettext-0.18.3.1    GNU gettext package
                      libiconv-1.14_1    A character set conversion library

                      [2.1.5-RELEASE][admin@xxxxx.localdomain]/root(6): sarg -x
                      SARG: Init
                      SARG: Loading configuration from /usr/pbi/sarg-i386/etc/sarg/sarg.conf
                      SARG: Loading exclude host file from: /usr/pbi/sarg-i386/etc/sarg/exclude_hosts.conf
                      SARG: Loading exclude file from: /usr/pbi/sarg-i386/etc/sarg/exclude_users.conf
                      SARG: Reading host alias file "/usr/pbi/sarg-i386/etc/sarg/hostalias"
                      SARG: List of host names to alias:
                      SARG: Deleting temporary directory "/tmp/sarg"
                      SARG: Parameters:
                      SARG:          Hostname or IP address (-a) =
                      SARG:                    Useragent log (-b) =
                      SARG:                    Exclude file (-c) = /usr/pbi/sarg-i386/etc/sarg/exclude_hosts.conf
                      SARG:                  Date from-until (-d) =
                      SARG:    Email address to send reports (-e) =
                      SARG:                      Config file (-f) = /usr/pbi/sarg-i386/etc/sarg/sarg.conf
                      SARG:                      Date format (-g) = USA (mm/dd/yyyy)
                      SARG:                        IP report (-i) = No
                      SARG:            Keep temporary files (-k) = No
                      SARG:                        Input log (-l) = /var/squid/logs/access.log
                      SARG:              Resolve IP Address (-n) = No
                      SARG:                      Output dir (-o) = /usr/local/sarg-reports/
                      SARG: Use Ip Address instead of userid (-p) = No
                      SARG:                    Accessed site (-s) =
                      SARG:                            Time (-t) =
                      SARG:                            User (-u) =
                      SARG:                    Temporary dir (-w) = /tmp/sarg
                      SARG:                  Debug messages (-x) = Yes
                      SARG:                Process messages (-z) = No
                      SARG:  Previous reports to keep (–lastlog) = 0
                      SARG:
                      SARG: sarg version: 2.3.6 Arp-21-2013
                      SARG: Reading access log file: /var/squid/logs/access.log
                      SARG: Records in file: 50, reading: 100.00%
                      SARG:    Records read: 50, written: 0, excluded: 0
                      SARG: Squid log format
                      SARG: No records found
                      SARG: End

                      cat /usr/pbi/sarg-i386/etc/sarg/sarg.conf | more

                      sarg.conf

                      TAG:  access_log file

                      #      Where is the access.log
                      #      sarg -l file

                      access_log /var/squid/logs/access.log

                      TAG: graphs yes|no

                      #      Use graphics where is possible.
                      #          graph_days_bytes_bar_color blue|green|yellow|orange|brown|red

                      graphs yes
                      #graph_days_bytes_bar_color orange

                      TAG:  graph_font

                      #      The full path to the TTF font file to use to create the graphs. It is required
                      #      if graphs is set to yes.

                      #graph_font /usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf

                      TAG:  title

                      --More--(byte 529)# sarg.conf

                      1 Reply Last reply Reply Quote 0
                      • C
                        ck42
                        last edited by

                        Looks like Sarg is not working again.
                        Updated to 2.2 the other day.  Sarg is the latest version available in packages (2.3.9 pkg v.0.6.4) and running squid (2.7.9 pkg v.4.3.6)

                        Attempting to run the Reports gives 'Could not find report index file.' message.
                        Trying to run Sarg from the CLI gives me 'Cannot set the locale LC_ALL to the environment variable'

                        Anyone have Sarg working with pf 2.2?

                        1 Reply Last reply Reply Quote 0
                        • KOMK
                          KOM
                          last edited by

                          Doing a quick search for Sarg and picking a result a little more current would have led you to this thread here with a solution.

                          1 Reply Last reply Reply Quote 0
                          • C
                            ck42
                            last edited by

                            Funny that you refer to that thread, KOM.  I had actually read through that prior to posting.
                            I had already tried creating the symlink on my system with no luck.  That's why I wanted to know if anyone actually has this working with 2.2…hoping that they would confirm whether or not any special steps needed to be taken to make it work.

                            1 Reply Last reply Reply Quote 0
                            • marcellocM
                              marcelloc
                              last edited by

                              I have. I did only the symlink.

                              Try to run sarg on console to see what erros do you get.

                              If you install cron package, you can see how I call configured report schedule.

                              Treinamentos de Elite: http://sys-squad.com

                              Help a community developer! ;D

                              1 Reply Last reply Reply Quote 0
                              • KOMK
                                KOM
                                last edited by

                                Cannot set the locale LC_ALL to the environment variable

                                I saw this on Solaris about 10 years ago.  I forget what the fix was.  Haven't seen it on 2.2 x64, and I've installed it many, many times.  As Mercello said, run sarg or sarg -x from the console and see what it says.

                                1 Reply Last reply Reply Quote 0
                                • marcellocM
                                  marcelloc
                                  last edited by

                                  @KOM:

                                  Cannot set the locale LC_ALL to the environment variable

                                  This is already applied to the package since pfSense 2.1 . Run an export before the sarg cmd.

                                   export LC_ALL=C && sarg...
                                  

                                  Treinamentos de Elite: http://sys-squad.com

                                  Help a community developer! ;D

                                  1 Reply Last reply Reply Quote 0
                                  • C
                                    ck42
                                    last edited by

                                    $  export LC_ALL=C && sarg
                                    SARG: Records in file: 0, reading: 0.00%
                                    SARG: No records found
                                    SARG: End
                                    SARG: Records in file: 0, reading: 100.00%
                                    
                                    $ sarg -x
                                    SARG: Cannot set the locale LC_ALL to the environment variable
                                    
                                    1 Reply Last reply Reply Quote 0
                                    • marcellocM
                                      marcelloc
                                      last edited by

                                      looks like you have no logs on squid file

                                      sarg -x also need the export LC_ALL=C &&

                                      Treinamentos de Elite: http://sys-squad.com

                                      Help a community developer! ;D

                                      1 Reply Last reply Reply Quote 0
                                      • C
                                        ck42
                                        last edited by

                                        @marcelloc:

                                        looks like you have no logs on squid file

                                        sarg -x also need the export LC_ALL=C &&

                                        Not entirely following you by this suggestion.  But I think you are wanting the output of this from?
                                        sarg -x export LC_ALL=C &&

                                        If so, the output just shows the same command entered (this is from the GUI).  No error messages or anything else.

                                        In looking at the log files in /var/squid/log, I noticed that all of the access.log files are EMPTY (0 byte files).  The cache logs look normal though.  Also, all the dates on the log files in that dir are current from the last few days.
                                        I'm not sure how to proceed next though to troubleshoot this.  Seems to me that this may be the issue (or at least part of it).  Why are the access log files not accumulating data?

                                        [EDIT] Might be on to something.  I forced an update in a schedule and then noticed that the .0 access log is accumulating. The View Report tab also now no longer gives me the error about the index file. 
                                        So what I've done for a test is to disable the log rotation in the report settings.  Log rotation is already set for 30 days in the Squid setup.

                                        Still seeing " Cannot set the locale LC_ALL to the environment variable" when I try to run sarg -x though.

                                        Something else that might be helpful from my system.

                                        # LC_ALL=C sarg -x
                                        SARG: Init
                                        SARG: Loading configuration from /usr/local/etc/sarg/sarg.conf
                                        SARG: Chaining IP resolving module "dns"
                                        SARG: Loading exclude host file from: /usr/pbi/sarg-i386/etc/sarg/exclude_hosts.conf
                                        SARG: Loading exclude file from: /usr/pbi/sarg-i386/etc/sarg/exclude_users.conf
                                        SARG: Reading host alias file "/usr/pbi/sarg-i386/etc/sarg/hostalias"
                                        SARG: List of host names to alias:
                                        SARG: Parameters:
                                        SARG:           Hostname or IP address (-a) =
                                        SARG:                    Useragent log (-b) =
                                        SARG:                     Exclude file (-c) = /usr/pbi/sarg-i386/etc/sarg/exclude_hosts.conf
                                        SARG:                  Date from-until (-d) =
                                        SARG:    Email address to send reports (-e) =
                                        SARG:                      Config file (-f) = /usr/local/etc/sarg/sarg.conf
                                        SARG:                      Date format (-g) = Sites & Users (yyyy/ww)
                                        SARG:                        IP report (-i) = No
                                        SARG:             Keep temporary files (-k) = No
                                        SARG:                        Input log (-l) = /var/squid/logs/access.log
                                        SARG:               Resolve IP Address (-n) = Yes
                                        SARG:                       Output dir (-o) = /usr/local/sarg-reports/
                                        SARG: Use Ip Address instead of userid (-p) = Yes
                                        SARG:                    Accessed site (-s) =
                                        SARG:                             Time (-t) =
                                        SARG:                             User (-u) =
                                        SARG:                    Temporary dir (-w) = /tmp/sarg
                                        SARG:                   Debug messages (-x) = Yes
                                        SARG:                 Process messages (-z) = No
                                        SARG:  Previous reports to keep (--lastlog) = 0
                                        SARG:
                                        SARG: sarg version: 2.3.9 Sep-21-2014
                                        SARG: Loading User table: /usr/pbi/sarg-i386/etc/sarg/usertab.conf
                                        SARG: Reading access log file: /var/squid/logs/access.log
                                        SARG: Records in file: 174, reading: 100.00%
                                        SARG:    Records read: 174, written: 174, excluded: 0
                                        SARG: Squid log format
                                        SARG: Period: 2015.05
                                        SARG: Sorting log /tmp/sarg/0.user_unsort
                                        SARG: Sorting log /tmp/sarg/1.user_unsort
                                        SARG: Sorting log /tmp/sarg/2.user_unsort
                                        SARG: (repday) Cannot open log file /usr/local/sarg-reports/2015.05/0/d0.html
                                        

                                        Regarding that very last line of output, here's what in the 2015.05 directory:

                                        # ls -la /usr/local/sarg-reports/2015.05
                                        total 18
                                        drwxr-xr-x  2 root  wheel   512 Feb  4 11:54 .
                                        drwxr-xr-x  5 root  wheel   512 Feb  4 11:54 ..
                                        -rw-r--r--  1 root  wheel  4437 Feb  4 11:54 index.html
                                        -rw-r--r--  1 root  wheel    22 Feb  4 11:54 sarg-date
                                        -rw-r--r--  1 root  wheel  1398 Feb  4 11:54 sarg-general
                                        -rw-r--r--  1 root  wheel     2 Feb  4 11:54 sarg-users
                                        -rw-r--r--  1 root  wheel   116 Feb  4 11:54 top
                                        

                                        So it's correct in that there's no "0" directory in which to find the d0.html file it's looking for.
                                        A system wide search for this file DOES show that a copy exist here though.
                                        /usr/pbi/sarg-i386/local/sarg-reports/2015.01.1/0/d0.html

                                        And lastly, the Realtime logging appears to be working correctly.

                                        1 Reply Last reply Reply Quote 0
                                        • marcellocM
                                          marcelloc
                                          last edited by

                                          @ck42:

                                          Not entirely following you by this suggestion.  But I think you are wanting the output of this from?
                                          sarg -x export LC_ALL=C &&

                                          export LC_ALL=C && sarg -x

                                          Treinamentos de Elite: http://sys-squad.com

                                          Help a community developer! ;D

                                          1 Reply Last reply Reply Quote 0
                                          • C
                                            ck42
                                            last edited by

                                            @marcelloc:

                                            @ck42:

                                            Not entirely following you by this suggestion.  But I think you are wanting the output of this from?
                                            sarg -x export LC_ALL=C &&

                                            export LC_ALL=C && sarg -x

                                            Here we go:

                                            $ export LC_ALL=C && sarg -x
                                            SARG: Init
                                            SARG: Loading configuration from /usr/local/etc/sarg/sarg.conf
                                            SARG: Chaining IP resolving module "dns"
                                            SARG: Loading exclude host file from: /usr/pbi/sarg-i386/etc/sarg/exclude_hosts.conf
                                            SARG: Loading exclude file from: /usr/pbi/sarg-i386/etc/sarg/exclude_users.conf
                                            SARG: Reading host alias file "/usr/pbi/sarg-i386/etc/sarg/hostalias"
                                            SARG: List of host names to alias:
                                            SARG: Deleting temporary directory "/tmp/sarg"
                                            SARG: Parameters:
                                            SARG:           Hostname or IP address (-a) = 
                                            SARG:                    Useragent log (-b) = 
                                            SARG:                     Exclude file (-c) = /usr/pbi/sarg-i386/etc/sarg/exclude_hosts.conf
                                            SARG:                  Date from-until (-d) = 
                                            SARG:    Email address to send reports (-e) = 
                                            SARG:                      Config file (-f) = /usr/local/etc/sarg/sarg.conf
                                            SARG:                      Date format (-g) = USA (mm/dd/yyyy)
                                            SARG:                        IP report (-i) = No
                                            SARG:             Keep temporary files (-k) = No
                                            SARG:                        Input log (-l) = /var/squid/logs/access.log
                                            SARG:               Resolve IP Address (-n) = Yes
                                            SARG:                       Output dir (-o) = /usr/local/sarg-reports/
                                            SARG: Use Ip Address instead of userid (-p) = Yes
                                            SARG:                    Accessed site (-s) = 
                                            SARG:                             Time (-t) = 
                                            SARG:                             User (-u) = 
                                            SARG:                    Temporary dir (-w) = /tmp/sarg
                                            SARG:                   Debug messages (-x) = Yes
                                            SARG:                 Process messages (-z) = No
                                            SARG:  Previous reports to keep (--lastlog) = 0
                                            SARG: 
                                            SARG: sarg version: 2.3.9 Sep-21-2014
                                            SARG: Loading User table: /usr/pbi/sarg-i386/etc/sarg/usertab.conf
                                            SARG: Reading access log file: /var/squid/logs/access.log
                                            SARG: Records in file: 1042, reading: 0.00%
                                            SARG:    Records read: 1042, written: 1042, excluded: 0
                                            SARG: Squid log format
                                            SARG: Period: 2015 Feb 04
                                            SARG: File /usr/local/sarg-reports/2015Feb04-2015Feb04 already exists, moved to /usr/local/sarg-reports/2015Feb04-2015Feb04.2
                                            SARG: Sorting log /tmp/sarg/0.user_unsort
                                            SARG: Making file: /tmp/sarg/0
                                            SARG: Sorting log /tmp/sarg/1.user_unsort
                                            SARG: Making file: /tmp/sarg/1
                                            SARG: Sorting log /tmp/sarg/2.user_unsort
                                            SARG: Making file: /tmp/sarg/2
                                            SARG: Sorting log /tmp/sarg/3.user_unsort
                                            SARG: Making file: /tmp/sarg/3
                                            SARG: Sorting log /tmp/sarg/4.user_unsort
                                            SARG: Making file: /tmp/sarg/4
                                            SARG: Sorting log /tmp/sarg/5.user_unsort
                                            SARG: Making file: /tmp/sarg/5
                                            SARG: Sorting log /tmp/sarg/6.user_unsort
                                            SARG: Making file: /tmp/sarg/6
                                            SARG: (repday) Cannot open log file /usr/local/sarg-reports/2015Feb04-2015Feb04/5/d5.html
                                            SARG: Records in file: 1042, reading: 100.00%
                                            
                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.