Monitor Squid Status

Bai Shen · Oct 4, 2010, 2:06 AM

Is there a way to monitor the status of Squid? I'd like to know how much memory/disk space it's using, and what the hit rate is, etc.

I didn't see an option anywhere to do this.

Visseroth · Oct 4, 2010, 10:53 AM

I second that. Lightsquid doesn't seem to work for me.

Bai Shen · Oct 4, 2010, 5:29 PM

I installed Lightsquid, but it doesn't really do what I want. It's just a list of access by IP. I'm assuming that the items in the list are what got cached, but it doesn't really say.

Also, it doesn't give any information about squid's status AFAIK.

wagonza · Oct 5, 2010, 8:16 AM

@Bai:

I installed Lightsquid, but it doesn't really do what I want. It's just a list of access by IP. I'm assuming that the items in the list are what got cached, but it doesn't really say.

Also, it doesn't give any information about squid's status AFAIK.

Correct - lightsquid parses the squid access log and generates a report on what sites have been visited. It lists access times per IP, how much bandwidth each IP has used and how much traffic has been served out of squid's cache.

The squid package is compiled with SNMP, so you could utilize another external tool, on another machine, to query squid and have it output the relevant info (cpu usage, mem info, cache hits, etc.) and have that tool graph the results.
To do this you will you need allow access, so add the following ACL snippet under Services->Proxy Server->General in the Custom Options field.


acl aclname snmp_community public 
snmp_access allow aclname

This will enable SNMP and squid will accept snmp queries (read-only) on port 3401.

Otherwise you can use the cachemgr.cgi which also provides this info. Steps to set it up as follows:


cd /usr/local/www && ln -s /usr/local/libexec/squid/cachemgr.cgi
cd /usr/local/etc/squid && cp cachemgr.conf.default cachemgr.conf

edit cachemgr.conf and add the IP address of the interface your squid is listening on, for example:


# This file controls which servers may be managed by
# the cachemgr.cgi script
#
# The file consists of one server per line on the format
#   hostname:port  description
#
# Specifying :port is optional. If not specified then
# the default proxy port is assumed. :* or :any matches
# any port on the target server.
#
# hostname is matched using shell filename matching, allowing
# * and other shell wildcards.
192.168.1.1

Then in the pfSense GUI go to Services->Proxy Server->Access Control and add the IP you inserted above, into the 'External Cache-Managers' field. Click save and then surf to http://gui_ip/cachemgr.cgi

Hope that helps!

Bai Shen · Oct 5, 2010, 2:16 PM

@wagonza:

Correct - lightsquid parses the squid access log and generates a report on what sites have been visited. It lists access times per IP, how much bandwidth each IP has used and how much traffic has been served out of squid's cache.

Where does it show how much was served from squid's cache? I didn't see that.

Also, Lightsquid keeps giving me a 500 error whenever I try and look at the graphs. Any ideas?

The squid package is compiled with SNMP, so you could utilize another external tool, on another machine, to query squid and have it output the relevant info (cpu usage, mem info, cache hits, etc.) and have that tool graph the results.
To do this you will you need allow access, so add the following ACL snippet under Services->Proxy Server->General in the Custom Options field.

Not sure I want to make that many changes just yet since 2.0 should hopefully be out soon. But I'll definitely take a look. Thanks. :)

wagonza · Oct 5, 2010, 5:23 PM

@Bai:

Where does it show how much was served from squid's cache? I didn't see that.
Also, Lightsquid keeps giving me a 500 error whenever I try and look at the graphs. Any ideas?

Where it says 'Hit %' on the landing page - that is a percentage of what has been served out of cache.

@Bai:

Not sure I want to make that many changes just yet since 2.0 should hopefully be out soon. But I'll definitely take a look. Thanks. :)

You have nothing to lose, as the changes should be safe across 2.0 updates.

scooterdouglas · Oct 5, 2010, 8:17 PM

@wagonza,

Thanks for that little snippet on the cachemgr.cgi, works a treat (albeit the tables are a bit basic compared to Lightsquid :) )

@Bai Shen,

The problem with Error 500 in Lightsquid is that Squid uses Perl v5.10.xx but Lightsquid uses Perl v5.8.xx, so you will probably have a symlink pointing to Perl v5.10.xx.

I tried helping some one over in the Spanish forum (Click here) but Spanish is not my first language! Take a look at the post, as it does tell you which directories you need to check.

;)

johnnybe · Oct 5, 2010, 8:51 PM

@wagonza:

Otherwise you can use the cachemgr.cgi which also provides this info. Steps to set it up as follows:
cd /usr/local/www && ln -s /usr/local/libexec/squid/cachemgr.cgi
cd /usr/local/etc/squid && cp cachemgr.conf.default cachemgr.conf
edit cachemgr.conf and add the IP address of the interface your squid is listening on, for example:
# This file controls which servers may be managed by
# the cachemgr.cgi script
#
# The file consists of one server per line on the format
#   hostname:port  description
#
# Specifying :port is optional. If not specified then
# the default proxy port is assumed. :* or :any matches
# any port on the target server.
#
# hostname is matched using shell filename matching, allowing
# * and other shell wildcards.
192.168.1.1
Then in the pfSense GUI go to Services->Proxy Server->Access Control and add the IP you inserted above, into the 'External Cache-Managers' field. Click save and then surf to http://gui_ip/cachemgr.cgi

Hope that helps!

Now, that was really really nice and easy! Thank you wagonza!

It was so easy that it must be added to pfsense squid package gui, IMHO.

wagonza · Oct 6, 2010, 8:56 AM

@johnnybe:

Now, that was really really nice and easy! Thank you wagonza!

It was so easy that it must be added to pfsense squid package gui, IMHO.

no problem - let me see what I can do about getting that added.

mhab12 · Oct 6, 2010, 2:38 PM

Just ran through this myself and agree that it is a great tool. Thank you for finally shedding some light on how to use cachemgr!

Bai Shen · Oct 6, 2010, 2:38 PM

@wagonza:

Where it says 'Hit %' on the landing page - that is a percentage of what has been served out of cache.

Ah, gotcha. So far I have yet to break 1%. :)

Any suggestions for squid settings? I currently have the following, and it seems that some things have actually gotten slower. I'm trying to figure out what will give me the most benefit.

5G disk cache size
256M memory cache size
256k max file size

@wagonza:

You have nothing to lose, as the changes should be safe across 2.0 updates.

Okay. I'll give it a shot then when I get some free time.

@scooterdouglas:

@Bai Shen,

The problem with Error 500 in Lightsquid is that Squid uses Perl v5.10.xx but Lightsquid uses Perl v5.8.xx, so you will probably have a symlink pointing to Perl v5.10.xx.

I tried helping some one over in the Spanish forum (Click here) but Spanish is not my first language! Take a look at the post, as it does tell you which directories you need to check.

;)

Interesting. I wonder why they're not synced. I'll take a look over there when I get a chance, but I haven't done any Spanish since HS. :)

wagonza · Oct 6, 2010, 7:54 PM

@Bai:

Ah, gotcha. So far I have yet to break 1%. :)

Any suggestions for squid settings? I currently have the following, and it seems that some things have actually gotten slower. I'm trying to figure out what will give me the most benefit.

5G disk cache size
256M memory cache size
256k max file size

Depends on the number of users that are using the cache.
256M is great, specially for caching hot replies/objects (saves it having to serve it off the disk). Just bare in mind that if your request rate requires more than 256M of memory, squid will increase the memory usage so that it can serve these new requests. Consider that if you dont have a lot of memory.

Also consider raising the max file size if you plan on downloading larger stuff and want it cached :)

Otherwise for more tuning have a look at http://doc.pfsense.org/index.php/Squid_Package_Tuning - that has some performance tuning and how to cache windows updates etc.

Also have a look at http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube to give you an idea of what is possible with caching of dynamic content.

Bai Shen · Oct 7, 2010, 2:24 PM

@wagonza:

Depends on the number of users that are using the cache.

It's just for my house, so only a couple people. However, we tend to be power users. :)

256M is great, specially for caching hot replies/objects (saves it having to serve it off the disk). Just bare in mind that if your request rate requires more than 256M of memory, squid will increase the memory usage so that it can serve these new requests. Consider that if you dont have a lot of memory.

So it won't hard limit itself to what I set? Hmm. I might have to drop it, then. I only have 1GB in this machine(can prolly bump it to 1.5), and since I set up squid, I've gone from 15% memory used to 75%.

Also consider raising the max file size if you plan on downloading larger stuff and want it cached :)

I was going to do that initially, as I'm barely using any hard drive space with my current install. Although I only have a 20GB drive. The 160 I was initially going to use wouldn't work correctly. I think the mobo didn't support it.

The reason I went with the default 4 and then 256 was because it mentioned that a lower setting would help more with speed while a higher one would save bandwidth.

Otherwise for more tuning have a look at http://doc.pfsense.org/index.php/Squid_Package_Tuning - that has some performance tuning and how to cache windows updates etc.

Also have a look at http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube to give you an idea of what is possible with caching of dynamic content.

I'll take a look. Squid is just checking urls, correct? So if I had the same file served from multiple urls(updates, etc), it wouldn't recognize that, right?

Also, yesterday I got up to 9.99% hit rate. :) However, looking at it now, it's down to 4.36% :(

So I guess I'll tweek the settings and also install the monitoring setup posted earlier.

Bai Shen · Oct 7, 2010, 4:07 PM

Just did the cachemgr.cgi setup. Man, that's a lot of info. Is there anyplace that explains what each of those links mean?

Also, any suggestions for a good SNMP setup?

wagonza · Oct 7, 2010, 5:20 PM

Have a look at http://wiki.squid-cache.org/SquidFaq/CacheManager - which provides a lot of info on the statistics that are generated.

For a good SNMP graph tool check out www.cacti.net and if/once you have that setup download squid template from http://forums.cacti.net/about4142.html

it will create some nice pretty graphs.

Bai Shen · Oct 7, 2010, 7:59 PM

Cool. I'll take a look. Thanks.

dlawley · Oct 14, 2010, 12:29 AM

Sorry been a long week, must be missing something..

Configured as shown but get a Cache Manager Error.

connect: (60) Operation timed out

Not sure which user/password combo its looking for.. but I might not be getting that far that user/pw make a difference

ran the two CMDs as shown, set the following

External Cache-Managers set to 192.168.5.1
cachemgr.conf also set to 192.168.5.1

You made it sound easy…

wagonza · Oct 14, 2010, 8:32 AM

@dlawley:

Sorry been a long week, must be missing something..

Configured as shown but get a Cache Manager Error.

connect: (60) Operation timed out

Not sure which user/password combo its looking for.. but I might not be getting that far that user/pw make a difference

ran the two CMDs as shown, set the following

External Cache-Managers set to 192.168.5.1
cachemgr.conf also set to 192.168.5.1

You made it sound easy…

You should be getting a web page thats says "Cache Manager Interface" and below it a drop down list with the IP address 192.168.5.1 in it.
The username and password fields you can ignore and just click continue. It should still allow you in. The only difference is that some of the links will not be active for example 'Shut Down the Squid Process'. If you want this links set to active, then in General Settings->Custom options add:

'cachemgr_passwd none all'

and click save. You wont have to enter a password. To make it secure have a look at the cachemgr_passwd config option @ http://www.squid-cache.org/Doc/config/cachemgr_passwd/

dlawley · Oct 14, 2010, 9:59 AM

Thanks for the quick reply.

I was all around that option reading the squidfaq…

retraced my steps, and a reboot

Still getting Cache Manager timeout...

Running transparent maybe an issue?? I know it causes some limitations in other areas when you use it.

Maybe proxy filter service is also having an effect?

maybe I need to be hitting 127.0.0.1 instead. ?? guess its easy enough to try..

Do not edit manually !

http_port 192.168.5.1:800
http_port 127.0.0.1:80 transparent
icp_port 0

wagonza · Oct 14, 2010, 9:54 AM

@dlawley:

Thanks for the quick reply.

I was all around that option reading the squidfaq…

retraced my steps, and a reboot

Still getting Cache Manager timeout...

Running transparent maybe an issue?? I know it causes some limitations in other areas when you use it.

maybe I need to be hitting 127.0.0.1 instead. ?? guess its easy enough to try..

Do not edit manually !

http_port 192.168.5.1:800
http_port 127.0.0.1:80 transparent
icp_port 0

I see you have squid listening on the non default port of 800 - did you add 192.168.5.1:800 to the cachemgr.conf file? Not just 192.168.5.1 - otherwise cachemgr.cgi will try connect to 3128.

Shouldn't make a difference whether the proxy is in transparent mode or not.