Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    $100 bounty - List sites visited by ip

    Scheduled Pinned Locked Moved Expired/Withdrawn Bounties
    20 Posts 9 Posters 13.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • I
      imoex2
      last edited by

      yes

      1 Reply Last reply Reply Quote 0
      • M
        MJK
        last edited by

        Just noticed this thread…

        I am also very interested in some logging facility - I tried BandWidthD, and had some issues as documented elsewhere here, and am still experimenting with it!

        If "gbelandger"'s code is still available, and if it's moderately easy to get "on top of it", I would be very happy to try it, and make any improvements I can, and share it back with whomsoever... (I've been writing software for way over 30 years, but am new to the pfSense/FreeBSD environment).

        Thank you,
          - Mike

        1 Reply Last reply Reply Quote 0
        • S
          sullrich
          last edited by

          Any suggestions of a suitable package?  Maybe query freshmeat and find one that you would find suitable.

          1 Reply Last reply Reply Quote 0
          • M
            MJK
            last edited by

            I don't know enough about the main packages to make any sensible suggestions, Scott.

            A few years ago, I implemented Freesco, which, I think, is very similar to pfSense, and which had some suitable logging packages - but, right now, I don't recall which ones! I can check out, though, if needed. Freesco was chosen then because it seemed the best option to support many (up to 10, I think) LAN cards, on separate sub-nets, etc. The needs now are different, and PfSense matches them very well (a relatively old PC, VPN, etc) - apart from the logging!

            Any package that covers the main spec mentioned here by IMOEX2 and GBELANGER would be very nice indeed!!

            Best regards,
              - Mike

            1 Reply Last reply Reply Quote 0
            • S
              sullrich
              last edited by

              Have you tried darkstat?

              1 Reply Last reply Reply Quote 0
              • M
                MJK
                last edited by

                Apologies for the huge delay, Scott.

                I looked at DarkStat some time back, but it shows very very little info, and the little that's there is not intuitive, IMO, (those few graphs are just a few vertical and horizontal lines ;) ), and what I've seen is nowhere close to the "spec" mentioned earlier in this thread.

                Best regards,
                  - Mike

                1 Reply Last reply Reply Quote 0
                • J
                  Justinw
                  last edited by

                  I believe that lightsquid is what you are looking for.  This is already working without modification from the repositories.  There is no gui configuration for it however, so you have to do all the conf files by hand at current.  This would be easy money for someone with more time on their hands than myself.  pkg_add -r lightsquid should do the trick.  Fix the confs, add the cron job and have fun.  This will keep track of sites visited by ip, and associate them with a calendar date.  You can also see other stats like most visited site for the month etc.  Just a suggestion!

                  1 Reply Last reply Reply Quote 0
                  • M
                    MJK
                    last edited by

                    Thank you, JustinW.

                    LightSquid looks VERY appropriate indeed.

                    A few further comments:
                      - "My" pfSense is now "live", so I cannot easily mess with it. I'd need to build a similar box for experiments.

                    - In other long threads here, the installation of LightSquid was discussed "a lot", with much of it over my head. Perhaps, sometimes, installation can be a bit complex ? And there was mention of having an integrated LightSquid installation option available in the next release fo pfSense - which would be real nice.

                    - In other threads here also, I note that other apps are required, and I don't know if some/all of these are in the standard pfSense build (Squid, etc).

                    - At my site, DHCP is running, so the IP addresses assigned to users are transient. Again, I've seen references to LightSquid scripts which require users to log-in/log-out, and which will then update a database to relate Usernames to IP addresses (based on time). This might add another layer of complexity!

                    - Dang - "nothing's easy"!

                    Thank you again,
                      - Mike

                    1 Reply Last reply Reply Quote 0
                    • K
                      kchung
                      last edited by

                      I've just installed lightsquid on my pfsense and wow, it's wonderfully easy and useful! Now I know where the good pr0n are all at!

                      1 Reply Last reply Reply Quote 0
                      • D
                        dvserg
                        last edited by

                        Look this
                        http://forum.pfsense.org/index.php/topic,4314.msg30643.html#msg30643

                        I make two themes for LightSquid Anybody test this :D
                        I use novosea theme
                        All themes tested on IE6 and FFox.

                        SquidGuardDoc EN  RU Tutorial
                        Localization ru_PFSense

                        1 Reply Last reply Reply Quote 0
                        • S
                          Sproggit
                          last edited by

                          SARG is the solution, IF you are forcing users to use squid via ACL's
                          Transparent proxying may be a bit more hairy, but still do-able (the advantage with the ACL solution is that you can get squid to authenticate agains a winderz AD server, so you get reports on actual users and not just IP's).

                          If you wan a SARG solution, say so (and maybe an e-mail to remind me), and I'll take a looky.

                          Regards
                          The Sproggg

                          1 Reply Last reply Reply Quote 0
                          • M
                            MJK
                            last edited by

                            Thank you. SARG looks superb!

                            From the info I read on it, I could not verify that it can cope in a DHCP environment, but I'm hoping it can.

                            At the time that any traffic occurs through pfSense (with DHCP), pfSense knows the IP and a "Name" associated with that IP - I don't recall if it's a "Computer-Name" (which would be very good!), or a "User-Name" (which would be only GREAT!!). So, hopefully someone (SQUID, or SARG, or…) can associate these names at the time the traffic occurs, and log them, without any separate logins, etc...

                            I would be most interested in pursuing this option for pfSense, though, clearly, I can help only at a basic technical level, as well as pay for all assistance. I notice a few other threads here discussing the installation of SARG, so, obviously, there's interest in such tools.

                            Thank you again,
                              - Mike

                            1 Reply Last reply Reply Quote 0
                            • S
                              Sproggit
                              last edited by

                              Okeydokey.
                              I've downloaded the developer iso, and will start looking this evening.
                              In interest of full disclosure :) .. there IS a freebsd 6.2 package available, so all I have to figure out is how to build a pfsense addon package, and make the menu structure pfsense style pretty.

                              Milestone 1:
                              Incorporate the package with an installer (This will probably give you the option to see SARG via an unlinked, custum URL)

                              Milestone 2:
                              Integrate package with pfsense…

                              1 Should be done in a few days (read, about a week).

                              2 ... I dunno, the package templates don't look TOO nasty, I just don't want to break anything else :)

                              If anyone has already developed packages, or is great at PhP, I'm more than happy to share the bounty if this will speed up a solution

                              SARG has no problems with a DHCP environment, just take a look at the squid access.log file (tail -f it to see it in motion). SARG uses the data in this file to compile reports.
                              AGAIN, if we want to do transparent squid, we need to get X-Forwarded for switched on, else SARG thinks all traffic comes from the pfsense box itsef!
                              Also, I will take a look at the NMB authentication (or RADIUS, or any other PAM) available in psense squid for USER as opposed to IP logging
                              This may need a tweak, and I'm loathe to mess with someone elses code.

                              regards
                              The Sproggg

                              1 Reply Last reply Reply Quote 0
                              • M
                                MJK
                                last edited by

                                Thank you again,

                                In case you end up re-inventing "The Wheel", run a search here on SARG, and you'll see that some others put in some serious work on pfSense install procedures for SARG, early this year. In one thread, Scott indicated that the forthcoming pfSense 1.2 might be needed, because of some very recent tweaks to the CRON features.

                                - Mike

                                1 Reply Last reply Reply Quote 0
                                • D
                                  dvserg
                                  last edited by

                                  @MJK:

                                  In one thread, Scott indicated that the forthcoming pfSense 1.2 might be needed, because of some very recent tweaks to the CRON features.

                                  - Mike

                                  If you need - can use my code for define cron task pfSense 1.2

                                  
                                  // setup cron tasks
                                  // original source from '/etc/inc/pfsense-utils.inc' function 'tdr_install_cron'
                                  // this function safe for other tasks
                                  // *****************************************************************************
                                  // - $task_name: cron task name (for config identification) /for searching my cron tasks/
                                  // - $options:   array=[0:minute][1:hour][2:mday][3:month][4:wday][5:who][6:cmd]
                                  // - $task_key:  cron command key for searching
                                  // - $on_off:    true-'on task', false-'off' task
                                  // required: $task_nameand $on_off
                                  // *****************************************************************************
                                  define('FIELD_TASKNAME', 'task_name');
                                  
                                  function ls_setup_cron($task_name, $options, $task_key, $on_off) {
                                          global $config;
                                          update_log("ls_setup_cron: start task_name=$task_name, task_key=$task_key, on_off=$on_off");
                                  
                                          // check input params
                                          if(!$task_name) {
                                              update_log("ls_setup_cron: exit - uncomplete input params.");
                                              return;
                                          }
                                          // search cron config settings
                                          if(!$config['cron']['item']) {
                                              update_log("ls_setup_cron: exit - 'config.xml'->[cron]->[items] not found.");
                                              return;
                                          }
                                  
                                          // searching task
                                          $x_name='';
                                          $x=0;
                                          foreach($config['cron']['item'] as $item) {
                                              if($item[FIELD_TASKNAME] and $task_name and ($item[FIELD_TASKNAME]==$task_name)) {
                                                 update_log("ls_setup_cron: found cron task with name=$task_name on [$x_name].");
                                                 $x_name = $x;
                                              }
                                              $x++;
                                          }
                                          unset($x);
                                  
                                          // install cron:
                                          //  - if not found with such name and not found 'task_key', when install task
                                          //  - if found task with such name, when renew this item (delete and add new with all check's)
                                          // deinstall cron:
                                          //  - deinstall only, if found such name
                                          switch($on_off) {
                                                  case true:
                                                       if($task_key) {
                                                            // searching task
                                                            $x=0;
                                                            $x_task='';
                                                            foreach($config['cron']['item'] as $item) {
                                                               if(strstr($item['command'], $task_key)) {
                                                                  $x_task = $x;
                                                                  update_log("ls_setup_cron: found cron task with key=$task_key on [$x].");
                                                               }
                                                               $x++;
                                                            }
                                                            unset($x);
                                  
                                                            if($x_task and (!$x_name or ($x_task != $x_name))) { // other task with $task_key alredy installed
                                                                     update_log("ls_setup_cron: can't add cron task, while such task exists $task_key");
                                                                     break;
                                                            } else {
                                                                if(is_array($options)) {
                                  
                                                                     // delete this item (by name)
                                                                     if($x_name > 0)
                                                                        unset($config['cron']['item'][$x_name]);
                                                                     // and add new
                                                                     $cron_item = array();
                                                                     $cron_item[FIELD_TASKNAME] = $task_name;
                                                                     $cron_item['minute']    = $options[0];
                                                                     $cron_item['hour']      = $options[1];
                                                                     $cron_item['mday']      = $options[2];
                                                                     $cron_item['month']     = $options[3];
                                                                     $cron_item['wday']      = $options[4];
                                                                     $cron_item['who']       = $options[5];
                                                                     $cron_item['command']   = $options[6];
                                                                     // check options
                                                                     if(!$cron_item['who']) $cron_item['who'] = "nobody";
                                                                     $config['cron']['item'][] = $cron_item;
                                                                     write_config("Installed cron task '$task_name' for 'lightsquid' package");
                                                                     configure_cron();
                                                                     // log
                                                                     update_log("ls_setup_cron: add cron task '$task_name'='" . $cron_item['command'] . "'");
                                                                }
                                                            }
                                                       } else
                                                            // log
                                                            update_log("ls_setup_cron: input prm 'task_key' not defined");
                                                  break;
                                                  case false:
                                                            // delete cron task only with name $task_name
                                                            if($x_name > 0) {
                                                               unset($config['cron']['item'][$x_name]);
                                                               write_config();
                                                               // log
                                                               update_log("ls_setup_cron: delete cron task '$task_name'");
                                                            }
                                                  break;
                                          }
                                          configure_cron();
                                          update_log("ls_setup_cron: end");
                                  }
                                  

                                  SquidGuardDoc EN  RU Tutorial
                                  Localization ru_PFSense

                                  1 Reply Last reply Reply Quote 0
                                  • First post
                                    Last post
                                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.