Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Sarg package for pfsense

    Scheduled Pinned Locked Moved pfSense Packages
    467 Posts 99 Posters 501.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      Nachtfalke
      last edited by

      I got this as answer from the mailing list - not sure if this will help me. Need some time to check what he said and the corresponding .conf files.

      
      Hi
      
      Attached you will find my configs, they are from an ubuntu 10.04
      system, running squid 2.7Stable7, squidguard 1.4 and sarg 2.3.
      
      Several pitfalls I remember:
      - pay special attention to the HTMLOUT of sarg-reports.conf
      - pay special attention to the stopped.log directives in squidguard.conf
      - triple check that the squid and squidguard log files are readable
      and the HTMLOUT is writable by sarg
      - there is a known bug in squidguard concerning some escape chars in
      urls that cause the squidguard log file to become malformatted. Sarg
      dies when this happens. Therefore I am using a self patched version of
      squidguard: http://51762846.de.strato-hosting.eu/bene/public/squidguard/
      
      Try running sarg-reports as root in from the console: It should start
      with something like
      
      /usr/sbin/sarg-reports daily
      SARG: Init
      SARG: Loading configuration from /etc/sarg/sarg.conf
      SARG: Loading exclude host file from: /etc/sarg/exclude_hosts
      SARG: Loading exclude file from: /etc/sarg/exclude_users
      SARG: Parameters:
      SARG:           Hostname or IP address (-a) =
      SARG:                    Useragent log (-b) =
      SARG:                     Exclude file (-c) = /etc/sarg/exclude_hosts
      SARG:                  Date from-until (-d) = 04/12/2012-04/12/2012
      SARG:    Email address to send reports (-e) =
      SARG:                      Config file (-f) = /etc/sarg/sarg.conf
      SARG:                      Date format (-g) = Europe (dd/mm/yyyy)
      SARG:                        IP report (-i) = No
      SARG:                        Input log (-l) = /var/log/squid/access.log
      SARG:                   Redirector log (-L) = /var/log/squid/stopped.log
      SARG:               Resolve IP Address (-n) = No
      SARG:                       Output dir (-o) = /var/www/squid-reports/Daily/
      SARG: Use Ip Address instead of userid (-p) = No
      SARG:                    Accessed site (-s) =
      SARG:                             Time (-t) =
      SARG:                             User (-u) =
      SARG:                    Temporary dir (-w) = /tmp
      SARG:                   Debug messages (-x) = Yes
      SARG:                 Process messages (-z) = No
      
      If something goes wrong and you don't know what to make of the error
      message, just post it here.
      
      Hope this helps
      
      Greetings
      
      B. Brandt
      
      
      1 Reply Last reply Reply Quote 0
      • N
        Nachtfalke
        last edited by

        Ok, I did some further tests. the sarg.inc is - as far as I tested it - correct.

        But for squidguard it means:
        If logging in squid is disabled then SARG cannot display only the blocked URL squidguard reported.
        So in my situation I cannot use SARG because I am not allowed to have the squid access.log file.  :(

        1 Reply Last reply Reply Quote 0
        • C
          cs80
          last edited by

          I am still working on recovering from a disaster caused by this package. I figured I'd drop a note here as a possible warning for anyone that is using this package. It may be possible that this was a user issue, rather then the fault of the package.

          I sadly can't provide many details at this point, if I can come across anything I will follow back up. Either case:

          I just lost my pfsense box due to massive corruption caused by (indirectly?) Sarg. I had Sarg running for ~2 month, maybe a bit more. A few days ago I noticed issues with networking and started digging into it. I found pfsense to be unresponsive. I rebooted it and started getting a lot of wonderful errors..

          Either case, somehow Sarg had created enough files to run me out of inodes. It was somewhere near 60GB's of data, and 9.7M (yes, MILLION) inodes in use.
          (this is for a 3 user network)

          I believe I was using the stock out of the box configuration on it. Sadly, it was a pain to get setup in the first place, that once it did start working I never did go back and look at it again.

          1 Reply Last reply Reply Quote 0
          • marcellocM
            marcelloc
            last edited by

            @cs80:

            Either case, somehow Sarg had created enough files to run me out of inodes. It was somewhere near 60GB's of data, and 9.7M (yes, MILLION) inodes in use.
            (this is for a 3 user network)

            I believe I was using the stock out of the box configuration on it. Sadly, it was a pain to get setup in the first place, that once it did start working I never did go back and look at it again.

            Current sarg version has compress report files and remove reports older then x days.

            Sarg reports use a lot of inodes.
            On my setup, I've installed a second disc with zfs just for report files. On zfs disc, I got 30million inodes.

            Treinamentos de Elite: http://sys-squad.com

            Help a community developer! ;D

            1 Reply Last reply Reply Quote 0
            • marcellocM
              marcelloc
              last edited by

              @Nachtfalke:

              Ok, I did some further tests. the sarg.inc is - as far as I tested it - correct.

              But for squidguard it means:
              If logging in squid is disabled then SARG cannot display only the blocked URL squidguard reported.
              So in my situation I cannot use SARG because I am not allowed to have the squid access.log file.  :(

              What changes you did to get squidguard working? can you push it ot github?

              Try to point sarg to an access.empty.log file on squid config at sarg.inc. this may solve your problem.

              Treinamentos de Elite: http://sys-squad.com

              Help a community developer! ;D

              1 Reply Last reply Reply Quote 0
              • N
                Nachtfalke
                last edited by

                @marcelloc:

                @Nachtfalke:

                Ok, I did some further tests. the sarg.inc is - as far as I tested it - correct.

                But for squidguard it means:
                If logging in squid is disabled then SARG cannot display only the blocked URL squidguard reported.
                So in my situation I cannot use SARG because I am not allowed to have the squid access.log file.  :(

                What changes you did to get squidguard working? can you push it ot github?

                Try to point sarg to an access.empty.log file on squid config at sarg.inc. this may solve your problem.

                I tried that with an access.log file which just contains some entries but this didn't help me on the SARG reports. It doesn't show me blocked entries newer than the access.log file entries.

                So there isn't anything I could push on github ;-)

                In general it is working with your config with squidguard but you need the access.log from squid. If this file isn't present and actual you cannot generate reports.

                Is dansguardian doing that without squid access.log file ?

                1 Reply Last reply Reply Quote 0
                • C
                  caldwell
                  last edited by

                  I have also run into the error that others are seeing:

                  Error: Could not find report index file.
                  Check and save sarg settings and try to force sarg schedule.

                  Here's what I've done.

                  1. Totally uninstalled Sarg pkg.
                  2. Used "find" command to locate and remove every directory or file referencing sarg in the name.
                  3. Upgraded to absolute latest (2nd release from today) pfsense package.
                  4. Rebooted.
                  5. Reinstalled Sarg.
                  6. Selected all report options and report types on the Sarg page in pfsense.
                  7. Hit Save.
                  8. Set up a 1h schedule and saved it.
                  9. Hit "force update" under the schedule.

                  ls -al /usr/local/sarg-reports/
                  total 4
                  drwxr-xr-x   2 root  wheel  512 Dec 10 21:19 .
                  drwxr-xr-x  19 root  wheel  512 Dec 10 21:19 ..

                  No index file(s) of any kind appear there.

                  This is a drag.  What does it take to get a simple package to just install and work the first time?

                  Does anyone have a solution on how to fix this manually?

                  Thanks in advance for any help you can offer.

                  ps - I did find this in system.log:
                  Dec 10 21:20:24 gw php: /pkg_edit.php: [sarg] sarg_xmlrpc_sync.php is starting.
                  Dec 10 21:20:32 gw php: /pkg_edit.php: Sarg: force refresh now with  args, compress() and none action after sarg finish.
                  Dec 10 21:20:32 gw php: /pkg_edit.php: The command '/usr/pbi/sarg-i386/bin/sarg ' returned exit code '1', the output was 'SARG: Cannot set the locale LC_ALL to the environment variable'

                  1 Reply Last reply Reply Quote 0
                  • marcellocM
                    marcelloc
                    last edited by

                    Caldwell, there is no bug on sarg package for squid and dansguardian logs.

                    just take a a look on forum for a working config that I'm using and check your squid access log config.

                    Treinamentos de Elite: http://sys-squad.com

                    Help a community developer! ;D

                    1 Reply Last reply Reply Quote 0
                    • marcellocM
                      marcelloc
                      last edited by

                      Nachtfalke,

                      Maybe a grep on squid log file for denied entries????

                      This way there will be only denied access to report.

                      Did you tried to select only denied sites on reports to generate?

                      Treinamentos de Elite: http://sys-squad.com

                      Help a community developer! ;D

                      1 Reply Last reply Reply Quote 0
                      • N
                        Nachtfalke
                        last edited by

                        @marcelloc:

                        Nachtfalke,

                        Maybe a grep on squid log file for denied entries????

                        This way there will be only denied access to report.

                        You think of a possibility that a script could do the grep on the access.log, just save the denied entries in a new file and delete the original one ?
                        Didn't try that but could be a possibility.

                        @marcelloc:

                        Did you tried to select only denied sites on reports to generate?

                        Not sure if I did that. But I saw all sites so I suppose that I didn't try that. Perhaps I can try this if I find some spare time. I uninstalled SARG some days ago.

                        1 Reply Last reply Reply Quote 0
                        • E
                          expert_az
                          last edited by

                          2.0.1-RELEASE (amd64)

                          Hello i'm getting this error on logs:

                          php: : The command '/usr/local/bin/sarg ' returned exit code '1', the output was 'SARG: Records in file: 16119722, reading: 0.00%^MSARG: Records in file: 5000, reading: 0.03%^MSARG: Records in file: 10000, reading: 0.06%^MSARG: Records in file: 15000, reading: 0.09%^MSARG: Records in file: 20000, reading: 0.12%^MSARG: Records in file: 25000, reading: 0.16%^MSARG: Records in file: 30000, reading: 0.19%^MSARG: Records in file: 35000, reading: 0.22%^MSARG: Records in file: 40000, reading: 0.25%^MSARG: Records in file: 45000, reading: 0.28%^MSARG: Records in file: 50000, reading: 0.31%^MSARG: Records in file: 55000, reading: 0.34%^MSARG: Records in file: 60000, reading: 0.37%^MSARG: Records in file: 65000, reading: 0.40%^MSARG: Records in file: 70000, reading: 0.43%^MSARG: Records in file: 75000, reading: 0.47%^MSARG: Records in file: 80000, reading: 0.50%^MSARG: Records in file: 85000, reading: 0.53%^MSARG: Records in file: 90000, reading: 0.56%^MSARG: Records in file: 95000, reading: 0.59%^MS

                          and when i try running sarg from console getting this log:

                          sarg
                          SARG: Records in file: 16121346, reading: 100.00%
                          sort: open failed: /tmp/sarg/denied.log.unsort: No such file or directory
                          SARG: sort command return status 2
                          SARG: sort command: sort -T "/tmp/sarg" -t "    " -k 3,3 -k 5,5 -o "/tmp/sarg/denied.log" "/tmp/sarg/denied.log.unsort"

                          i did reinstall

                          1 Reply Last reply Reply Quote 0
                          • B
                            bernie156
                            last edited by

                            Hi, I just did a fresh installation of pfSense, then squid 2.7.9 pkg v.4.3.1 and after that Sarg 2.3.2 pkg v.0.6.1.

                            Running a simple report generation with "force update now" gives this output:

                            php: /pkg_edit.php: The command '/usr/local/bin/sarg ' returned exit code '1', the output was 'SARG: Records in file: 13455, reading: 0.00%^MSARG: Records in file: 5000, reading: 37.16%^MSARG: Records in file: 10000, reading: 74.32%^MSARG: cannot open /usr/local/sarg-reports/2013/01/18-19/sarg-date for writing SARG:: No such file or directory SARG: Records in file: 13455, reading: 100.00%'

                            Sarg created a folder at that time: /usr/local/sarg-reports/2013/01/18-19.5  with the content
                            drwxr-xr-x  2 root  wheel  114176 Jan 19 16:00 192_168_24_10
                            drwxr-xr-x  2 root  wheel    512 Jan 19 16:00 192_168_24_201
                            -rw-r–r--  1 root  wheel    1402 Jan 19 16:00 download.html.gz
                            -rw-r--r--  1 root  wheel    1581 Jan 19 16:00 index.html.gz
                            -rw-r--r--  1 root  wheel      22 Jan 19 16:00 sarg-date
                            -rw-r--r--  1 root  wheel  177652 Jan 19 16:00 sarg-general
                            -rw-r--r--  1 root  wheel  65450 Jan 19 16:00 sarg-sites
                            -rw-r--r--  1 root  wheel      2 Jan 19 16:00 sarg-users
                            -rw-r--r--  1 root  wheel  23027 Jan 19 16:00 siteuser.html.gz
                            -rw-r--r--  1 root  wheel    4893 Jan 19 16:00 topsites.html.gz

                            So I do not understand what to with "MSARG: cannot open /usr/local/sarg-reports/2013/01/18-19/sarg-date for writing SARG:: No such file or directory"

                            Can someone help me?

                            1 Reply Last reply Reply Quote 0
                            • marcellocM
                              marcelloc
                              last edited by

                              What sarg options did you selected on GUI? Did you tried to remove this report before running sarg again?

                              Treinamentos de Elite: http://sys-squad.com

                              Help a community developer! ;D

                              1 Reply Last reply Reply Quote 0
                              • B
                                bernie156
                                last edited by

                                I only selected Report Options "Convert to IP address" and "Top Users" and "Top Sites" on the General Tab. The scheduled report has no Sarg args set.

                                No, I didn't try to remove a report. Tab "View Report" says always
                                Error: Could not find report index file.
                                Check and save sarg settings and try to force sarg schedule.

                                1 Reply Last reply Reply Quote 0
                                • marcellocM
                                  marcelloc
                                  last edited by

                                  Check config options. One you will need is create index file

                                  Treinamentos de Elite: http://sys-squad.com

                                  Help a community developer! ;D

                                  1 Reply Last reply Reply Quote 0
                                  • B
                                    bernie156
                                    last edited by

                                    Selected all options wich were default "(yes)".  And - as expected - got: "Error: Could not find report index file. Check and save sarg settings and try to force sarg schedule."

                                    As you can see in my first post, the index.html is there but cannot be found.

                                    Today /usr/local/sarg-reports/18Jan2013-20Jan2013 contains
                                    -rw-r–r--  1 root  wheel    1156 Jan 20 11:38 index.html.gz
                                    -rw-r--r--  1 root  wheel      22 Jan 20 11:38 sarg-date
                                    -rw-r--r--  1 root  wheel  408865 Jan 20 11:38 sarg-general
                                    -rw-r--r--  1 root  wheel    100 Jan 20 11:38 top

                                    Log says today:
                                    php: /pkg_edit.php: The command '/usr/local/bin/sarg ' returned exit code '1', the output was 'SARG: Records in file: 29632, reading: 0.00%^MSARG: Records in file: 5000, reading: 16.87%^MSARG: Records in file: 10000, reading: 33.75%^MSARG: Records in file: 15000, reading: 50.62%^MSARG: Records in file: 20000, reading: 67.49%^MSARG: Records in file: 25000, reading: 84.37%^MSARG: Cannot delete /usr/local/sarg-reports/18Jan2013-20Jan2013/d192_168_24_201.html - No such file or directory SARG: Records in file: 29632, reading: 100.00%'

                                    1 Reply Last reply Reply Quote 0
                                    • B
                                      bernie156
                                      last edited by

                                      I did a fresh install of pfSense, squid 3 and Sarg, selected all Sarg default options and it works. Thanks for your effort anyway.

                                      1 Reply Last reply Reply Quote 0
                                      • K
                                        KeltecRFB
                                        last edited by

                                        Raising a Necro-Thread instead of creating a new one.

                                        Is there way to configure Sarg to show denied access reporting and what Proxy\Dans acl triggered it?  Can that be done in the GUI or is it in CLI only?

                                        Thanks!

                                        1 Reply Last reply Reply Quote 0
                                        • marcellocM
                                          marcelloc
                                          last edited by

                                          @KeltecRFB:

                                          Is there way to configure Sarg to show denied access reporting and what Proxy\Dans acl triggered it?  Can that be done in the GUI or is it in CLI only?

                                          Sarg only understands squid log format, so I think it sarg is not able to log what ACL denied a url.

                                          Do you have a sarg config that does it?

                                          Treinamentos de Elite: http://sys-squad.com

                                          Help a community developer! ;D

                                          1 Reply Last reply Reply Quote 0
                                          • N
                                            novicenaja
                                            last edited by

                                            @marcelloc:

                                            Hi all,

                                            I've just published sarg package for pfsense with squid,squidguard and dansguardian log Analysis as well real time report tab.

                                            Squidguard functions are under devel yet but squid and dansguardians(as well as I tested) are working.

                                            After almost everything done, I found an old sarg package published on forum by joaohf and merged some function calls from this old thread.

                                            Another good point is that sarg is able to forward logs via email, so I'm planning to include it for nanobsd installs.

                                            have fun and feedback!  :)

                                            att,
                                            Marcello Coutinho

                                            ขอบคุณครับ (khob kun krub) Thank you verymuch

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.