Proxy Report (LightSquid)
-
This may have been addressed earlier, but I haven't seen (or found) a solution in the forums:
SETUP
Dual Xeon Processor, 8 Gig Ram, 250gig HD, Six Interfaces ( 1 LAN, 5 WAN, 1 spare )
Latest pfsense version (upgraded today in the hope that the issue will be resolved)Packages:
Squid3-dev ( Captive Portal Authentication, ClamAV & I-Cap Disabled )
LightsquidLightsquid produces the required report ( username, sites accessed ) as seen by the attached lightsquid.png file.
BUT in the proxy state tab of lightsquid I get the following
Warning: gethostbyaddr(): Address is not a valid IPv4 or IPv6 address in /usr/local/www/sqstat/sqstat.class.php on line 426
as captured by squid.pngSquid is working properly and logging is enabled with the log store directory correctly configured as /var/squid/logs
Tried changing the /var/squid/logs directory to /var/squid/log with no desired results.Any suggestions?
-
I have the same error.
Config of PFsense packages:
squid3-dev Network 3.3.10 pkg 2.2.6
Lightsquid Network Report 1.8.2 pkg v.2.33
squidGuard-squid3 Network Management 1.4_4 pkg v.1.9.5
Squid RealTime stat 1.20 for the proxy server.This recipt at http://vicryhc.wordpress.com/2014/06/10/pfsense-proxy-squid-sqstat-error-1cannot-get-data-server-answered-http1-0-403-forbidden/
can't help me. So this error and problem has actual status.BUT THE SAME FUNCTIONALITY CAN BE REACHED BY INSTALLING & USING Sarg (Network Report) (2.3.6_2 pkg v.0.6.3).
Status: Sarg Realtime - MORE FLEXIBLE. -
And at all - this problem NOT present in configuration:
squid3 Network 3.1.20 pkg 2.1.1 (NOT dev version)
+
squidGuard-squid3 Network Management 1.4_4 pkg v.1.9.5
+
Lightsquid Network Report 1.8.2 pkg v.2.33 -
I have this problem too
Lightsquid 1.8.2 pkg v.2.33
squid3-dev 3.3.10 pkg 2.2.6
squidGuard-squid3 1.4_4 pkg v.1.9.5 -
You must be using IPv6 or have it enabled. I've noticed this only pops with up when using IPv6. Its a problem with sqstat code. Reach out to the original developer and see if he can fix it, Thats probably the only way it will get fix in the pfSense package.
http://samm.kiev.ua/sqstat/
-
I found the problem!
In squid 3.4.10_2 was change in the name of two fields as follows
peer = remote
me = localSquid 3.1.20 (pfsense 2.1.5)
Connection: 0x287c6190 FD 16, read 10546, wrote 24779 FD desc: mail.google.com:443 in: buf 0x2a3fc000, offset 0, size 4096 peer: 192.168.0.196:3592 me: 192.168.0.2:3128 nrequests: 1 uri mail.google.com:443 logType TCP_MISS out.offset 0, out.size 24740 req_sz 208 entry 0x0/N/A start 1426612080.376224 (612.153023 seconds ago) username delay_pool 1
Squid 3.4.10_2 (pfsense 2.2)
Connection: 0x8265326d8 FD 466, read 39330, wrote 256466 FD desc: 0.client-channel.google.com:443 in: buf 0x82ff41000, offset 0, size 4096 remote: 192.168.0.51:49829 local: 192.168.0.1:3128 nrequests: 1 uri 0.client-channel.google.com:443 logType TCP_MISS out.offset 0, out.size 256427 req_sz 232 entry 0x0/N/A start 1426601559.846703 (11113.823216 seconds ago) username delay_pool 0
I made the alterations in /usr/local/www/sqstat/sqstat.class.php file and is now operating normally.
Changed file:/* $Id$ */ /* sqstat.class.php Squid Proxy Server realtime stat (c) Alex Samorukov, samm@os2.kiev.ua modification by 2011 Serg Dvoriancev, dv_serg@mail.ru Squid Proxy Server realtime stat part of pfSense (www.pfSense.com) Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1\. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2\. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ // sqstat class DEFINE('SQSTAT_VERSION', '1.20'); DEFINE('SQSTAT_SHOWLEN', 60); class squidstat{ var $fp; # conection var $squidhost; var $squidport; # hosts var $hosts_file; var $hosts; # versions var $server_version; var $sqstat_version; # other var $group_by; var $resolveip; var $autorefresh; var $use_sessions = false; # cache manager var $cachemgr_passwd; # errors var $errno; var $errstr; function squidstat(){ $this->sqstat_version = SQSTAT_VERSION; $this->squidhost = '127.0.0.1'; $this->squidport = '3128'; $this->group_by = 'host'; $this->resolveip = true; $this->hosts_file = ''; $this->autorefresh = 0; $this->cachemgr_passwd = ''; $errno = 0; $errstr = ''; if (!function_exists("preg_match")) { $this->errorMsg(5, 'You need to install [PHP pcre extension](http://www.php.net/pcre/) to run this script'); $this->showError(); exit(5); } // we need session support to gather avg. speed if (function_exists("session_start")){ $this->use_sessions=true; } } function formatXHTML($body, $refresh, $use_js = false){ $text=''."\n". ''."\n" .'' .'' .''; if($refresh) $text.=''; $text.='<title>SqStat '.SQSTAT_VERSION.'</title>' .($use_js?'':'').'' .($use_js?'':'') .$body.''; return $text; } function showError(){ $text=' # SqStat error '. ' ## Error ('.$this->errno.'): '.$this->errstr.''; echo $this->formatXHTML($text,0); } function connect($squidhost, $squidport){ $this->fp = false; # connecting to the squidhost $this->fp = @fsockopen($squidhost, $squidport, $this->errno, $this->errstr, 10); if (!$this->fp) { # failed to connect return false; } return true; } # based @ (c) moritz at barafranca dot com function duration ($seconds) { $takes_time = array(604800,86400,3600,60,0); $suffixes = array("w","d","h","m","s"); $output = ""; foreach ($takes_time as $key=>$val) { ${$suffixes[$key]} = ($val == 0) ? $seconds : floor(($seconds/$val)); $seconds -= ${$suffixes[$key]} * $val; if (${$suffixes[$key]} > 0) { $output .= ${$suffixes[$key]}; $output .= $suffixes[$key]." "; } } return trim($output); } /** * Format a number of bytes into a human readable format. * Optionally choose the output format and/or force a particular unit * * @param int $bytes The number of bytes to format. Must be positive * @param string $format Optional. The output format for the string * @param string $force Optional. Force a certain unit. B|KB|MB|GB|TB * @return string The formatted file size */ function filesize_format($bytes, $format = '', $force = '') { $force = strtoupper($force); $defaultFormat = '%01d %s'; if (strlen($format) == 0) $format = $defaultFormat; $bytes = max(0, (int) $bytes); $units = array('b', 'Kb', 'Mb', 'Gb', 'Tb', 'Pb'); $power = array_search($force, $units); if ($power === false) $power = $bytes > 0 ? floor(log($bytes)/log(1024)) : 0; return sprintf($format, $bytes / pow(1024, $power), $units[$power]); } function makeQuery($pass = ""){ $raw = array(); # sending request if(!$this->fp) die("Please connect to server"); $out = "GET cache_object://localhost/active_requests HTTP/1.0\r\n"; if ($pass != "") $out .= "Authorization: Basic ".base64_encode("cachemgr:$pass")."\r\n"; $out .= "\r\n"; fwrite($this->fp, $out); while (!feof($this->fp)) { $raw[] = trim(fgets($this->fp, 2048)); } fclose($this->fp); if (!preg_match("/^HTTP.* 200 OK$/", $raw[0])) { $this->errorMsg(1, "Cannot get data. Server answered: $raw[0]"); return false; } # parsing output; $header = 1; $connection = 0; $parsed["server_version"] = "Unknown"; foreach($raw as $key=>$v){ # cutoff http header if ($header==1 && $v=="") $header=0; if ($header) { if(substr(strtolower($v),0,7) == "server:") { # parsing server version $parsed["server_version"] = substr($v,8); } } else { if(substr($v,0,11) == "Connection:") { # parsing connection $connection = substr($v,12); } if ($connection) { # username field is avaible in Squid 2.6 stable if(substr($v,0,9) == "username ") $parsed["con"][$connection]["username"] = substr($v, 9); if(substr($v,0,5) == "peer:") $parsed["con"][$connection]["peer"] = substr($v, 6); if(substr($v,0,7) == "remote:") $parsed["con"][$connection]["peer"] = substr($v, 8); if(substr($v,0,3) == "me:") $parsed["con"][$connection]["me"] = substr($v, 4); if(substr($v,0,6) == "local:") $parsed["con"][$connection]["me"] = substr($v, 7); if(substr($v,0,4) == "uri ") $parsed["con"][$connection]["uri"] = substr($v, 4); if(substr($v,0,10) == "delay_pool") $parsed["con"][$connection]["delay_pool"] = substr($v, 11); if (preg_match('/out.offset \d+, out.size (\d+)/', $v, $matches)) { $parsed["con"][$connection]["bytes"] = $matches[1]; } if (preg_match('/start \d+\.\d+ \((\d+).\d+ seconds ago\)/', $v, $matches)){ $parsed["con"][$connection]["seconds"] = $matches[1]; } } } } return $parsed; } function implode_with_keys($array, $glue) { foreach ($array as $key => $v){ $ret[] = $key . '=' . htmlspecialchars($v); } return implode($glue, $ret); } function makeHtmlReport($data, $resolveip = false, $hosts_array = array(), $use_js = true) { global $group_by; if($this->use_sessions){ session_name('SQDATA'); session_start(); } $total_avg = $total_curr = 0; // resort data array $users=array(); switch($group_by){ case "host": $group_by_name="Host"; $group_by_key='return $ip;'; break; case "username": $group_by_name="User"; $group_by_key='return $v["username"];'; break; default: die("wrong group_by!"); } foreach($data["con"] as $key => $v){ if(substr($v["uri"],0,13)=="cache_object:") continue; // skip myself $ip=substr($v["peer"],0,strpos($v["peer"],":")); if(isset($hosts_array[$ip])){ $ip=$hosts_array[$ip]; } // i use ip2long() to make ip sorting work correctly elseif($resolveip){ $hostname=gethostbyaddr($ip); if($hostname==$ip) $ip=ip2long($ip);// resolve failed else $ip=$hostname; } else{ $ip=ip2long(substr($v["peer"],0,strpos($v["peer"],":"))); } $v['connection'] = $key; if(!isset($v["username"])) $v["username"]="N/A"; $users[eval($group_by_key)][]=$v; } ksort($users); $refresh=0; if(isset($_GET["refresh"]) && !isset($_GET["stop"])) $refresh=(int)$_GET["refresh"]; $text=''; if(count($GLOBALS["configs"])==1) $servers=$GLOBALS["squidhost"].':'.$GLOBALS["squidport"]; else{ $servers='<select onchange="this.form.submit();" name="config">'; foreach ($GLOBALS["configs"] as $key=>$v){ $servers.='<option '.($globals["config"]="=$key?'" selected="selected" ':'').'="" value="'.$key.'">'.htmlspecialchars($v).'</option>'; } $servers.='</select>'; } $text.=' <form method="get" action="'.$_SERVER[" php_self"].'"="">'. 'Squid RealTime stat for the '.$servers.' proxy server ('.$data["server_version"].'). '. 'Auto refresh: sec. Created at: <tt>'.date("h:i:s d/m/Y").'</tt> '. '</form> '. ''. ''. ''. ($this->use_sessions?'':''). ''. ''; $ausers=$acon=0; unset($session_data); if (isset($_SESSION['time']) && ((time() - $_SESSION['time']) < 3*60) && isset($_SESSION['sqdata']) && is_array($_SESSION['sqdata'])) { //only if the latest data was less than 3 minutes ago $session_data = $_SESSION['sqdata']; } $table=''; foreach($users as $key=>$v){ $ausers++; $table.=''. ''; $user_avg = $user_curr = $con_color = 0; foreach ($v as $con){ if(substr($con["uri"],0,7)=="http://" || substr($con["uri"],0,6)=="ftp://"){ if(strlen($con["uri"])>SQSTAT_SHOWLEN) $uritext=htmlspecialchars(substr($con["uri"],0,SQSTAT_SHOWLEN)).' ....'; else $uritext=htmlspecialchars($con["uri"]).''; $uri='['.$uritext; } else $uri=htmlspecialchars($con["uri"]); $acon++; //speed stuff $con_id = $con['connection']; $is_time = time(); $curr_speed=0; $avg_speed=0; if (isset($session_data[$con_id]) && $con_data = $session_data[$con_id] ) { // if we have info about current connection, we do analyze its data // current speed $was_time = $con_data['time']; $was_size = $con_data['size']; if ($was_time && $was_size) { $delta = $is_time - $was_time; if ($delta == 0) { $delta = 1; } if ($con['bytes'] >= $was_size) { $curr_speed = ($con['bytes'] - $was_size) / 1024 / $delta; } } else { $curr_speed = $con['bytes'] / 1024; } //avg speed $avg_speed = $con['bytes'] / 1024; if ($con['seconds'] > 0) { $avg_speed /= $con['seconds']; } } $new_data[$con_id]['time'] = $is_time; $new_data[$con_id]['size'] = $con['bytes']; //sum speeds $total_avg += $avg_speed; $user_avg += $avg_speed; $total_curr += $curr_speed; $user_curr += $curr_speed; if($use_js) $js='onMouseout="hideddrivetip()" onMouseover="ddrivetip(\''.$this->implode_with_keys($con,' ').'\')"'; else $js=''; $table.=']('.htmlspecialchars($con[)['. ']('.htmlspecialchars($con[)['; if($this->use_sessions){ $table .= ']('.htmlspecialchars($con[)['. ']('.htmlspecialchars($con[)['; } $table .= ']('.htmlspecialchars($con[)['. ']('.htmlspecialchars($con[)['. ']('.htmlspecialchars($con[)['; } if($this->use_sessions){ $table.=sprintf("]('.htmlspecialchars($con[)[", $user_curr, $user_avg); } } $_SESSION['time'] = time(); if(isset($new_data)) $_SESSION['sqdata'] = $new_data; $stat_row=''; if($this->use_sessions){ $stat_row.=sprintf("]('.htmlspecialchars($con[)[", $ausers, $acon, $total_curr, $total_avg); } else { $stat_row.=sprintf("]('.htmlspecialchars($con[)[", $ausers, $acon); } if($ausers==0){ $text.=']('.htmlspecialchars($con[)['; } else { $text.=$stat_row.$table.$stat_row; } $text .= ']('.htmlspecialchars($con[) | '.$group_by_name.' | URI | Curr. Speed | Avg. Speed | Size | Time | | **'.(is_int($key)?long2ip($key):$key).'** | | | | '.$uri.' | '.( (round($curr_speed, 2) > 0) ? sprintf("%01.2f KB/s", $curr_speed) : '' ).' | '.( (round($avg_speed, 2) > 0) ? sprintf("%01.2f KB/s", $avg_speed) : '' ). ' | '.$this->filesize_format($con["bytes"]).' | '.$this->duration($con["seconds"],"short").' | | | %01.2f KB/s | %01.2f KB/s | | | **Total:** | **%d** users and **%d** connections @ **%01.2f/%01.2f** KB/s (CURR/AVG) | | **Total:** | **%d** users and **%d** connections | | **No active connections** | ['. ']('.htmlspecialchars($con[) [©]('.htmlspecialchars($con[) [Alex Samorukov](mailto:samm@os2.kiev.ua?subject=SqStat '.SQSTAT_VERSION.'), 2006 '; return $this->formatXHTML($text,$refresh,$use_js); } function parseRequest($data, $group_by = 'host', $resolveip = false) { $parsed = array(); if ($this->use_sessions) { session_name('SQDATA'); session_start(); } # resort data array $users = array(); switch ($group_by) { case "username": $group_by_name = "User"; $group_by_key = "username"; break; case "host": default: $group_by_name = "Host"; $group_by_key = "peer"; break; } # resolve IP & group foreach ($data["con"] as $key => $v) { # skip myself if (substr($v["uri"], 0, 13) == "cache_object:") continue; $ip = substr($v["peer"], 0, strpos($v["peer"], ":")); $v["peer"] = $ip; # name from hosts if (isset($this->hosts[$ip])) { $ip = $this->hosts[$ip]; } else # i use ip2long() to make ip sorting work correctly if ($resolveip) { $hostname = gethostbyaddr($ip); if ($hostname == $ip) $ip = ip2long($ip); # resolve failed. use (ip2long) key else $ip = $hostname; } else { $ip = ip2long(substr($v["peer"], 0, strpos($v["peer"], ":"))); } $v['con_id'] = $key; $v["username"] = isset($v["username"]) ? $v["username"] : "N/A"; # users [key => conn_array] $users[$v[$group_by_key]][] = $v; } ksort($users); unset($session_data); if (isset($_SESSION['time']) && ((time() - $_SESSION['time']) < 3*60) && isset($_SESSION['sqdata']) && is_array($_SESSION['sqdata'])) { # only if the latest data was less than 3 minutes ago $session_data = $_SESSION['sqdata']; } # users count & con cont $ausers = $acon = 0; $total_avg = $total_curr = 0; foreach ($users as $key => $v) { $ausers++; $user_avg = $user_curr = $con_color = 0; foreach ($v as $con_key => $con){ $cres = array(); $acon++; $uritext = $con["uri"]; if (substr($con["uri"], 0, 7) == "http://" || substr($con["uri"], 0, 6) == "ftp://") { if (strlen($uritext) > SQSTAT_SHOWLEN) $uritext = htmlspecialchars(substr($uritext, 0, SQSTAT_SHOWLEN)) . ' ....'; } else $uritext = htmlspecialchars($uritext); $cres['uritext'] = $uritext; $cres['uri'] = $con["uri"]; # speed stuff $con_id = $con['connection']; $is_time = time(); $curr_speed = $avg_speed = 0; if (isset($session_data[$con_id]) && $con_data = $session_data[$con_id] ) { # if we have info about current connection, we do analyze its data # current speed $was_time = $con_data['time']; $was_size = $con_data['size']; if ($was_time && $was_size) { $delta = $is_time - $was_time; if ($delta == 0) { $delta = 1; } if ($con['bytes'] >= $was_size) { $curr_speed = ($con['bytes'] - $was_size) / 1024 / $delta; } } else { $curr_speed = $con['bytes'] / 1024; } # avg speed $avg_speed = $con['bytes'] / 1024; if ($con['seconds'] > 0) { $avg_speed /= $con['seconds']; } } $cres['cur_speed'] = $curr_speed; $cres['avg_speed'] = $avg_speed; $cres['seconds'] = $con["seconds"]; $cres['bytes'] = $con["bytes"]; # groupped parsed[key => conn_key] $parsed['users'][$key]['con'][$con_key] = $cres; # for sessions $new_data[$con_id]['time'] = $is_time; $new_data[$con_id]['size'] = $con['bytes']; # sum speeds $total_avg += $avg_speed; $user_avg += $avg_speed; $total_curr += $curr_speed; $user_curr += $curr_speed; } # total per user $parsed['users'][$key]['user_curr'] = $user_curr; $parsed['users'][$key]['user_avg'] = $user_avg; } # total info $parsed['ausers'] = $ausers; $parsed['acon'] = $acon; $parsed['total_avg'] = $total_avg; $parsed['total_curr'] = $total_curr; # update session info $_SESSION['time'] = time(); if (isset($new_data)) $_SESSION['sqdata'] = $new_data; return $parsed; } function errorMsg($errno, $errstr) { $this->errno = $errno; $this->errstr = $errstr; } function load_hosts() { # loading hosts file $hosts_array = array(); if (!empty($this->hosts_file)) { if (is_file($this->hosts_file)) { $handle = @fopen($this->hosts_file, "r"); if ($handle) { while (!feof($handle)) { $buffer = fgets($handle, 4096); unset($matches); if (preg_match('/^([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})[ \t]+(.+)$/i', $buffer, $matches)) { $hosts_array[$matches[1]]=$matches[2]; } } fclose($handle); } $this->hosts = $hosts_array; } else { #error $this->errorMsg(4, "Hosts file not found. Cant read <tt>'{$this->hosts_file}'</tt>."); return $this->errno; } } return 0; } function query_exec() { $data = ""; $this->server_version = '(unknown)'; if ($this->connect($this->squidhost, $this->squidport)) { $data = $this->makeQuery($this->cachemgr_passwd); if ($this->errno == 0) { $this->server_version = $data['server_version']; $data = $this->parseRequest($data, 'host', true); } } return $data; } } ?>
-
I found the problem!
Yes, you did. Thank you very much for posting the fix. I applied it by patching the 2 new lines of code into my local version; it was a perfect diff save for those so I could also just have dropped your new file in.
It worked perfectly on the following configuration:
pfsense 2.2.1-RELEASE (amd64)
squid3 3.4.10_2 pkg 0.2.6
Lightsquid 1.8.2 pkg v 2.35Again, thank you very much.