$100 bounty - List sites visited by ip
-
If it is possible to list sites visited by ip, I am willing to put a $100 bounty on it
-
If it is possible to list sites visited by ip, I am willing to put a $100 bounty on it
Care to elaborate a bit? I can think of a couple ways to do this with already built in functionality, but I'm skeptical that this is what you want. Please give us a little more detail on what it is you expect. Thanks
–Bill
-
I would think he means Squid log analysis?
I've performed some research to find something similar when I was building my own Linux-based proxy/firewalls, and turned out nothing that quite cut it. Most available non-commercial proxy analysis tools were either web log analysis tools that recognized the proxy format, or very basic tools with limited reporting capability. I found nothing solid that concentrated on 'user monitoring' or resource usage stats.
I did write my own code, albeit not completely polished, I can find it and send it over if you want.
Basically it's a regex that does squid log analysis and groups requests by IP addresses. Its definately something that an office manager would ask, as it allows preventive monitoring of web resources usage by employees.
I'm not a 'big brotherism' fan myself, but it is an efficient deterrent when employees know it will be noticed when they spend 3 hours a day on YouTube. It also becomes a good ROI justification.
-
I want to be able to(in no particular order):
- see a list of what sites were visited
- click on a site and see what ips visited it
-
of times the site was visited by an ip or group of ip's
- time stamps
- host resolution would be nice for the lan side so its not just listed as ip paired with a site
- top 5 top 10 sites visited etc…
- maybe an option to block that site
might add more to this...of course i'll add more to the bounty if necessary
-
I want to be able to(in no particular order):
- see a list of what sites were visited
- click on a site and see what ips visited it
-
of times the site was visited by an ip or group of ip's
- time stamps
- host resolution would be nice for the lan side so its not just listed as ip paired with a site
- top 5 top 10 sites visited etc…
- maybe an option to block that site
might add more to this...of course i'll add more to the bounty if necessary
Minus number 7 (you repeated 2 twice), it sounds like the ntop package may already do what you want.
BTW, nowhere here am I getting that this is limited to HTTP. A number of people (only one in the thread) have suggested that you are talking about HTTP and getting the squid logs. I'm assuming you want this for all IP traffic, all ports, etc. Am I right?–Bill
-
yes
-
Just noticed this thread…
I am also very interested in some logging facility - I tried BandWidthD, and had some issues as documented elsewhere here, and am still experimenting with it!
If "gbelandger"'s code is still available, and if it's moderately easy to get "on top of it", I would be very happy to try it, and make any improvements I can, and share it back with whomsoever... (I've been writing software for way over 30 years, but am new to the pfSense/FreeBSD environment).
Thank you,
- Mike -
Any suggestions of a suitable package? Maybe query freshmeat and find one that you would find suitable.
-
I don't know enough about the main packages to make any sensible suggestions, Scott.
A few years ago, I implemented Freesco, which, I think, is very similar to pfSense, and which had some suitable logging packages - but, right now, I don't recall which ones! I can check out, though, if needed. Freesco was chosen then because it seemed the best option to support many (up to 10, I think) LAN cards, on separate sub-nets, etc. The needs now are different, and PfSense matches them very well (a relatively old PC, VPN, etc) - apart from the logging!
Any package that covers the main spec mentioned here by IMOEX2 and GBELANGER would be very nice indeed!!
Best regards,
- Mike -
Have you tried darkstat?
-
Apologies for the huge delay, Scott.
I looked at DarkStat some time back, but it shows very very little info, and the little that's there is not intuitive, IMO, (those few graphs are just a few vertical and horizontal lines ;) ), and what I've seen is nowhere close to the "spec" mentioned earlier in this thread.
Best regards,
- Mike -
I believe that lightsquid is what you are looking for. This is already working without modification from the repositories. There is no gui configuration for it however, so you have to do all the conf files by hand at current. This would be easy money for someone with more time on their hands than myself. pkg_add -r lightsquid should do the trick. Fix the confs, add the cron job and have fun. This will keep track of sites visited by ip, and associate them with a calendar date. You can also see other stats like most visited site for the month etc. Just a suggestion!
-
Thank you, JustinW.
LightSquid looks VERY appropriate indeed.
A few further comments:
- "My" pfSense is now "live", so I cannot easily mess with it. I'd need to build a similar box for experiments.- In other long threads here, the installation of LightSquid was discussed "a lot", with much of it over my head. Perhaps, sometimes, installation can be a bit complex ? And there was mention of having an integrated LightSquid installation option available in the next release fo pfSense - which would be real nice.
- In other threads here also, I note that other apps are required, and I don't know if some/all of these are in the standard pfSense build (Squid, etc).
- At my site, DHCP is running, so the IP addresses assigned to users are transient. Again, I've seen references to LightSquid scripts which require users to log-in/log-out, and which will then update a database to relate Usernames to IP addresses (based on time). This might add another layer of complexity!
- Dang - "nothing's easy"!
Thank you again,
- Mike -
I've just installed lightsquid on my pfsense and wow, it's wonderfully easy and useful! Now I know where the good pr0n are all at!
-
Look this
http://forum.pfsense.org/index.php/topic,4314.msg30643.html#msg30643I make two themes for LightSquid Anybody test this :D
I use novosea theme
All themes tested on IE6 and FFox. -
SARG is the solution, IF you are forcing users to use squid via ACL's
Transparent proxying may be a bit more hairy, but still do-able (the advantage with the ACL solution is that you can get squid to authenticate agains a winderz AD server, so you get reports on actual users and not just IP's).If you wan a SARG solution, say so (and maybe an e-mail to remind me), and I'll take a looky.
Regards
The Sproggg -
Thank you. SARG looks superb!
From the info I read on it, I could not verify that it can cope in a DHCP environment, but I'm hoping it can.
At the time that any traffic occurs through pfSense (with DHCP), pfSense knows the IP and a "Name" associated with that IP - I don't recall if it's a "Computer-Name" (which would be very good!), or a "User-Name" (which would be only GREAT!!). So, hopefully someone (SQUID, or SARG, or…) can associate these names at the time the traffic occurs, and log them, without any separate logins, etc...
I would be most interested in pursuing this option for pfSense, though, clearly, I can help only at a basic technical level, as well as pay for all assistance. I notice a few other threads here discussing the installation of SARG, so, obviously, there's interest in such tools.
Thank you again,
- Mike -
Okeydokey.
I've downloaded the developer iso, and will start looking this evening.
In interest of full disclosure :) .. there IS a freebsd 6.2 package available, so all I have to figure out is how to build a pfsense addon package, and make the menu structure pfsense style pretty.Milestone 1:
Incorporate the package with an installer (This will probably give you the option to see SARG via an unlinked, custum URL)Milestone 2:
Integrate package with pfsense…1 Should be done in a few days (read, about a week).
2 ... I dunno, the package templates don't look TOO nasty, I just don't want to break anything else :)
If anyone has already developed packages, or is great at PhP, I'm more than happy to share the bounty if this will speed up a solution
SARG has no problems with a DHCP environment, just take a look at the squid access.log file (tail -f it to see it in motion). SARG uses the data in this file to compile reports.
AGAIN, if we want to do transparent squid, we need to get X-Forwarded for switched on, else SARG thinks all traffic comes from the pfsense box itsef!
Also, I will take a look at the NMB authentication (or RADIUS, or any other PAM) available in psense squid for USER as opposed to IP logging
This may need a tweak, and I'm loathe to mess with someone elses code.regards
The Sproggg -
Thank you again,
In case you end up re-inventing "The Wheel", run a search here on SARG, and you'll see that some others put in some serious work on pfSense install procedures for SARG, early this year. In one thread, Scott indicated that the forthcoming pfSense 1.2 might be needed, because of some very recent tweaks to the CRON features.
- Mike
-
@MJK:
In one thread, Scott indicated that the forthcoming pfSense 1.2 might be needed, because of some very recent tweaks to the CRON features.
- Mike
If you need - can use my code for define cron task pfSense 1.2
// setup cron tasks // original source from '/etc/inc/pfsense-utils.inc' function 'tdr_install_cron' // this function safe for other tasks // ***************************************************************************** // - $task_name: cron task name (for config identification) /for searching my cron tasks/ // - $options: array=[0:minute][1:hour][2:mday][3:month][4:wday][5:who][6:cmd] // - $task_key: cron command key for searching // - $on_off: true-'on task', false-'off' task // required: $task_nameand $on_off // ***************************************************************************** define('FIELD_TASKNAME', 'task_name'); function ls_setup_cron($task_name, $options, $task_key, $on_off) { global $config; update_log("ls_setup_cron: start task_name=$task_name, task_key=$task_key, on_off=$on_off"); // check input params if(!$task_name) { update_log("ls_setup_cron: exit - uncomplete input params."); return; } // search cron config settings if(!$config['cron']['item']) { update_log("ls_setup_cron: exit - 'config.xml'->[cron]->[items] not found."); return; } // searching task $x_name=''; $x=0; foreach($config['cron']['item'] as $item) { if($item[FIELD_TASKNAME] and $task_name and ($item[FIELD_TASKNAME]==$task_name)) { update_log("ls_setup_cron: found cron task with name=$task_name on [$x_name]."); $x_name = $x; } $x++; } unset($x); // install cron: // - if not found with such name and not found 'task_key', when install task // - if found task with such name, when renew this item (delete and add new with all check's) // deinstall cron: // - deinstall only, if found such name switch($on_off) { case true: if($task_key) { // searching task $x=0; $x_task=''; foreach($config['cron']['item'] as $item) { if(strstr($item['command'], $task_key)) { $x_task = $x; update_log("ls_setup_cron: found cron task with key=$task_key on [$x]."); } $x++; } unset($x); if($x_task and (!$x_name or ($x_task != $x_name))) { // other task with $task_key alredy installed update_log("ls_setup_cron: can't add cron task, while such task exists $task_key"); break; } else { if(is_array($options)) { // delete this item (by name) if($x_name > 0) unset($config['cron']['item'][$x_name]); // and add new $cron_item = array(); $cron_item[FIELD_TASKNAME] = $task_name; $cron_item['minute'] = $options[0]; $cron_item['hour'] = $options[1]; $cron_item['mday'] = $options[2]; $cron_item['month'] = $options[3]; $cron_item['wday'] = $options[4]; $cron_item['who'] = $options[5]; $cron_item['command'] = $options[6]; // check options if(!$cron_item['who']) $cron_item['who'] = "nobody"; $config['cron']['item'][] = $cron_item; write_config("Installed cron task '$task_name' for 'lightsquid' package"); configure_cron(); // log update_log("ls_setup_cron: add cron task '$task_name'='" . $cron_item['command'] . "'"); } } } else // log update_log("ls_setup_cron: input prm 'task_key' not defined"); break; case false: // delete cron task only with name $task_name if($x_name > 0) { unset($config['cron']['item'][$x_name]); write_config(); // log update_log("ls_setup_cron: delete cron task '$task_name'"); } break; } configure_cron(); update_log("ls_setup_cron: end"); }