Monitor Squid Status



  • Is there a way to monitor the status of Squid?  I'd like to know how much memory/disk space it's using, and what the hit rate is, etc.

    I didn't see an option anywhere to do this.



  • I second that. Lightsquid doesn't seem to work for me.



  • I installed Lightsquid, but it doesn't really do what I want.  It's just a list of access by IP.  I'm assuming that the items in the list are what got cached, but it doesn't really say.

    Also, it doesn't give any information about squid's status AFAIK.



  • @Bai:

    I installed Lightsquid, but it doesn't really do what I want.  It's just a list of access by IP.  I'm assuming that the items in the list are what got cached, but it doesn't really say.

    Also, it doesn't give any information about squid's status AFAIK.

    Correct - lightsquid parses the squid access log and generates a report on what sites have been visited. It lists access times per IP, how much bandwidth each IP has used and how much traffic has been served out of squid's cache.

    The squid package is compiled with SNMP, so you could utilize another external tool, on another machine, to query squid and have it output the relevant info (cpu usage, mem info, cache hits, etc.) and have that tool graph the results.
    To do this you will you need allow access, so add the following ACL snippet under Services->Proxy Server->General in the Custom Options field.

    
    acl aclname snmp_community public 
    snmp_access allow aclname
    
    

    This will enable SNMP and squid will accept snmp queries (read-only) on port 3401.

    Otherwise you can use the cachemgr.cgi which also provides this info. Steps to set it up as follows:

    
    cd /usr/local/www && ln -s /usr/local/libexec/squid/cachemgr.cgi
    cd /usr/local/etc/squid && cp cachemgr.conf.default cachemgr.conf
    
    

    edit cachemgr.conf and add the IP address of the interface your squid is listening on, for example:

    
    # This file controls which servers may be managed by
    # the cachemgr.cgi script
    #
    # The file consists of one server per line on the format
    #   hostname:port  description
    #
    # Specifying :port is optional. If not specified then
    # the default proxy port is assumed. :* or :any matches
    # any port on the target server.
    #
    # hostname is matched using shell filename matching, allowing
    # * and other shell wildcards.
    192.168.1.1
    
    

    Then in the pfSense GUI go to Services->Proxy Server->Access Control and add the IP you inserted above, into the 'External Cache-Managers' field. Click save and then surf to http://gui_ip/cachemgr.cgi

    Hope that helps!



  • @wagonza:

    Correct - lightsquid parses the squid access log and generates a report on what sites have been visited. It lists access times per IP, how much bandwidth each IP has used and how much traffic has been served out of squid's cache.

    Where does it show how much was served from squid's cache?  I didn't see that.

    Also, Lightsquid keeps giving me a 500 error whenever I try and look at the graphs.  Any ideas?

    The squid package is compiled with SNMP, so you could utilize another external tool, on another machine, to query squid and have it output the relevant info (cpu usage, mem info, cache hits, etc.) and have that tool graph the results.
    To do this you will you need allow access, so add the following ACL snippet under Services->Proxy Server->General in the Custom Options field.

    Not sure I want to make that many changes just yet since 2.0 should hopefully be out soon. But I'll definitely take a look.  Thanks. :)



  • @Bai:

    Where does it show how much was served from squid's cache?  I didn't see that.
    Also, Lightsquid keeps giving me a 500 error whenever I try and look at the graphs.  Any ideas?

    Where it says 'Hit %' on the landing page - that is a percentage of what has been served out of cache.

    @Bai:

    Not sure I want to make that many changes just yet since 2.0 should hopefully be out soon. But I'll definitely take a look.  Thanks. :)

    You have nothing to lose, as the changes should be safe across 2.0 updates.



  • @wagonza,

    Thanks for that little snippet on the cachemgr.cgi, works a treat (albeit the tables are a bit basic compared to Lightsquid :) )

    @Bai Shen,

    The problem with Error 500 in Lightsquid is that Squid uses Perl v5.10.xx but Lightsquid uses Perl v5.8.xx, so you will probably have a symlink pointing to Perl v5.10.xx.

    I tried helping some one over in the Spanish forum (Click here) but Spanish is not my first language!  Take a look at the post, as it does tell you which directories you need to check.

    ;)



  • @wagonza:

    Otherwise you can use the cachemgr.cgi which also provides this info. Steps to set it up as follows:

    
    cd /usr/local/www && ln -s /usr/local/libexec/squid/cachemgr.cgi
    cd /usr/local/etc/squid && cp cachemgr.conf.default cachemgr.conf
    
    

    edit cachemgr.conf and add the IP address of the interface your squid is listening on, for example:

    
    # This file controls which servers may be managed by
    # the cachemgr.cgi script
    #
    # The file consists of one server per line on the format
    #   hostname:port  description
    #
    # Specifying :port is optional. If not specified then
    # the default proxy port is assumed. :* or :any matches
    # any port on the target server.
    #
    # hostname is matched using shell filename matching, allowing
    # * and other shell wildcards.
    192.168.1.1
    
    

    Then in the pfSense GUI go to Services->Proxy Server->Access Control and add the IP you inserted above, into the 'External Cache-Managers' field. Click save and then surf to http://gui_ip/cachemgr.cgi

    Hope that helps!

    Now, that was really really nice and easy! Thank you wagonza!

    It was so easy that it must be added to pfsense squid package gui, IMHO.



  • @johnnybe:

    Now, that was really really nice and easy! Thank you wagonza!

    It was so easy that it must be added to pfsense squid package gui, IMHO.

    no problem - let me see what I can do about getting that added.



  • Just ran through this myself and agree that it is a great tool.  Thank you for finally shedding some light on how to use cachemgr!



  • @wagonza:

    Where it says 'Hit %' on the landing page - that is a percentage of what has been served out of cache.

    Ah, gotcha.  So far I have yet to break 1%. :)

    Any suggestions for squid settings?  I currently have the following, and it seems that some things have actually gotten slower.  I'm trying to figure out what will give me the most benefit.

    5G disk cache size
    256M memory cache size
    256k max file size

    @wagonza:

    You have nothing to lose, as the changes should be safe across 2.0 updates.

    Okay.  I'll give it a shot then when I get some free time.

    @scooterdouglas:

    @Bai Shen,

    The problem with Error 500 in Lightsquid is that Squid uses Perl v5.10.xx but Lightsquid uses Perl v5.8.xx, so you will probably have a symlink pointing to Perl v5.10.xx.

    I tried helping some one over in the Spanish forum (Click here) but Spanish is not my first language!  Take a look at the post, as it does tell you which directories you need to check.

    ;)

    Interesting.  I wonder why they're not synced.  I'll take a look over there when I get a chance, but I haven't done any Spanish since HS. :)



  • @Bai:

    Ah, gotcha.  So far I have yet to break 1%. :)

    Any suggestions for squid settings?  I currently have the following, and it seems that some things have actually gotten slower.  I'm trying to figure out what will give me the most benefit.

    5G disk cache size
    256M memory cache size
    256k max file size

    Depends on the number of users that are using the cache.
    256M is great, specially for caching hot replies/objects (saves it having to serve it off the disk). Just bare in mind that if your request rate requires more than 256M of memory, squid will increase the memory usage so that it can serve these new requests. Consider that if you dont have a lot of memory.

    Also consider raising the max file size if you plan on downloading larger stuff and want it cached :)

    Otherwise for more tuning have a look at http://doc.pfsense.org/index.php/Squid_Package_Tuning - that has some performance tuning and how to cache windows updates etc.

    Also have a look at http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube to give you an idea of what is possible with caching of dynamic content.



  • @wagonza:

    Depends on the number of users that are using the cache.

    It's just for my house, so only a couple people.  However, we tend to be power users. :)

    256M is great, specially for caching hot replies/objects (saves it having to serve it off the disk). Just bare in mind that if your request rate requires more than 256M of memory, squid will increase the memory usage so that it can serve these new requests. Consider that if you dont have a lot of memory.

    So it won't hard limit itself to what I set?  Hmm.  I might have to drop it, then.  I only have 1GB in this machine(can prolly bump it to 1.5), and since I set up squid, I've gone from 15% memory used to 75%.

    Also consider raising the max file size if you plan on downloading larger stuff and want it cached :)

    I was going to do that initially, as I'm barely using any hard drive space with my current install.  Although I only have a 20GB drive.  The 160 I was initially going to use wouldn't work correctly.  I think the mobo didn't support it.

    The reason I went with the default 4 and then 256 was because it mentioned that a lower setting would help more with speed while a higher one would save bandwidth.

    Otherwise for more tuning have a look at http://doc.pfsense.org/index.php/Squid_Package_Tuning - that has some performance tuning and how to cache windows updates etc.

    Also have a look at http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube to give you an idea of what is possible with caching of dynamic content.

    I'll take a look.  Squid is just checking urls, correct?  So if I had the same file served from multiple urls(updates, etc), it wouldn't recognize that, right?

    Also, yesterday I got up to 9.99% hit rate.  :) However, looking at it now, it's down to 4.36% :(

    So I guess I'll tweek the settings and also install the monitoring setup posted earlier.



  • Just did the cachemgr.cgi setup.  Man, that's a lot of info.  Is there anyplace that explains what each of those links mean?

    Also, any suggestions for a good SNMP setup?



  • Have a look at http://wiki.squid-cache.org/SquidFaq/CacheManager - which provides a lot of info on the statistics that are generated.

    For a good SNMP graph tool check out www.cacti.net and if/once you have that setup download squid template from http://forums.cacti.net/about4142.html

    it will create some nice pretty graphs.



  • Cool.  I'll take a look.  Thanks.



  • Sorry been a long week, must be missing something..

    Configured as shown but get a Cache Manager Error.

    connect: (60) Operation timed out

    Not sure which user/password combo its looking for.. but I might not be getting that far that user/pw make a difference

    ran the two CMDs as shown, set the following

    External Cache-Managers set to 192.168.5.1
    cachemgr.conf also set to 192.168.5.1

    You made it sound easy…



  • @dlawley:

    Sorry been a long week, must be missing something..

    Configured as shown but get a Cache Manager Error.

    connect: (60) Operation timed out

    Not sure which user/password combo its looking for.. but I might not be getting that far that user/pw make a difference

    ran the two CMDs as shown, set the following

    External Cache-Managers set to 192.168.5.1
    cachemgr.conf also set to 192.168.5.1

    You made it sound easy…

    You should be getting a web page thats says "Cache Manager Interface" and below it a drop down list with the IP address 192.168.5.1 in it.
    The username and password fields you can ignore and just click continue. It should still allow you in. The only difference is that some of the links will not be active for example 'Shut Down the Squid Process'. If you want this links set to active, then in General Settings->Custom options add:

    'cachemgr_passwd none all'

    and click save. You wont have to enter a password. To make it secure have a look at the cachemgr_passwd config option @ http://www.squid-cache.org/Doc/config/cachemgr_passwd/



  • Thanks for the quick reply.

    I was all around that option reading the squidfaq…

    retraced my steps, and a reboot

    Still getting Cache Manager timeout...

    Running transparent maybe an issue??  I know it causes some limitations in other areas when you use it.

    Maybe proxy filter service is also having an effect?

    maybe I need to be hitting 127.0.0.1 instead.  ?? guess its easy enough to try..

    Do not edit manually !

    http_port 192.168.5.1:800
    http_port 127.0.0.1:80 transparent
    icp_port 0



  • @dlawley:

    Thanks for the quick reply.

    I was all around that option reading the squidfaq…

    retraced my steps, and a reboot

    Still getting Cache Manager timeout...

    Running transparent maybe an issue??  I know it causes some limitations in other areas when you use it.

    maybe I need to be hitting 127.0.0.1 instead.  ?? guess its easy enough to try..

    Do not edit manually !

    http_port 192.168.5.1:800
    http_port 127.0.0.1:80 transparent
    icp_port 0

    I see you have squid listening on the non default port of 800 - did you add 192.168.5.1:800 to the cachemgr.conf file? Not just 192.168.5.1 - otherwise cachemgr.cgi will try connect to 3128.

    Shouldn't make a difference whether the proxy is in transparent mode or not.



  • That did it!, thanks!!



  • The proxy filter might impact your access to the cache mgr, if you have ip addresses blocked.  Which I did on my production server.

    But I'm all good now.



  • i always use cachemgr to monitoring squid …
    ifen not understand much ...  ;D
    i keep this link on my email to help me remember how to it ...

    http://forum.pfsense.org/index.php?topic=14609.0



  • So I have the cachemgr page working, but one thing concerns me.  On the Store Directory page, it says the following: Accepted object sizes: 0 - (unlimited) bytes

    However, on the Cache Mgmt page, I have max object size set to 1024.  So why the difference?



  • No workie for me :(

    I get this:

    Cache Manager menu for x.x.x.x:

    GIF89a

    Generated Mon, 25 Oct 2010 06:32:27 GMT, by cachemgr.cgi/2.7.STABLE9@r

    Any help would be appreciated. Thanks!



  • @Pistolero:

    No workie for me :(

    I get this:

    Cache Manager menu for x.x.x.x:

    GIF89a

    Generated Mon, 25 Oct 2010 06:32:27 GMT, by cachemgr.cgi/2.7.STABLE9@r

    Any help would be appreciated. Thanks!

    I am not sure what that reference to GIF89a is? But you should see a page with:

    Cache server with drop down list of your pfSense's IP address.
    Input for a Manager name
    Input for a Password

    and finally a "Continue" button which will take you to the stats page. What do you see when you click on this button?



  • @wagonza:

    @Pistolero:

    No workie for me :(

    I get this:

    Cache Manager menu for x.x.x.x:

    GIF89a

    Generated Mon, 25 Oct 2010 06:32:27 GMT, by cachemgr.cgi/2.7.STABLE9@r

    Any help would be appreciated. Thanks!

    I am not sure what that reference to GIF89a is? But you should see a page with:

    Cache server with drop down list of your pfSense's IP address.
    Input for a Manager name
    Input for a Password

    and finally a "Continue" button which will take you to the stats page. What do you see when you click on this button?

    I do see the drop-down list with the input fields. I get that weird screen after clicking continue. The weird page with the GIF89a I get only in Firefox. IE only displays



  • Sounds like you dont have your cachemgr.conf file setup correctly and after clicking submit it is timing out.



  • Hi pl tel me
    i Installed  pfsense

    installed proxy server with mod security
    '
    squid  package
    squidguard package  installed
    tel me how to block unwanted or porn sites

    pl reply
    Ganesh



  • Please dont hijack another thread but rather start a new one.

    Have a look at http://doc.pfsense.org/index.php/SquidGuard_package for instructions.



  • Did the cachemgr.cgi trick. Working great with squid in transparent mode. Thank you!



  • Mrtg work well too.

    http://oss.oetiker.ch/mrtg/

    ![MRTG Index Page.png](/public/imported_attachments/1/MRTG Index Page.png)
    ![MRTG Index Page.png_thumb](/public/imported_attachments/1/MRTG Index Page.png_thumb)



  • I found what was breaking my cachemgr.cgi!

    I use SquidGuard to filter ads and other inappropriate sites, redirecting them to an internal web server with a single transparent pixel. I also disallow IP addresses in the address field, to keep users too-smart-for-their-own-good from bypassing the filter by using an IP. If you configure the cachemgr to connect to the proxy via IP, it will be blocked. Solved by connecting to localhost or the host name of the firewall.



  • Hi guys!
    Sorry for bringing this topic out of its grave but I'm testing the new 2.0 RC1 and the cachemgr.cgi doesn't work anymore!
    At least, it does half work : I can reach the cachemgr.cgi page with login info but hitting the connection brings me to "Access denied"…
    Is this a squid.conf acl problem or maybe an Apache acl?
    Also, I'll try the mrtg trick but the graphs don't print anything...
    Any help will be appreciated!

    Thanks!



  • What is your LAN IP address?

    For example my LAN IP address is: 192.168.3.1

    So, in cacehmgr.cgi I have the following:

    
    # This file controls which servers may be managed by
    # the cachemgr.cgi script
    #
    # The file consists of one server per line on the format
    #   hostname:port  description
    #
    # Specifying :port is optional. If not specified then
    # the default proxy port is assumed. :* or :any matches
    # any port on the target server.
    #
    # hostname is matched using shell filename matching, allowing
    # * and other shell wildcards.
    # localhost
    192.168.3.1
    
    

    Now, goto Services => Proxy Server, select the Access Control tab, scroll down to External Cache Managers and you should see some similar to this:

    
    192.168.3.1;127.0.0.1
    
    

    Ignore the 127.0.0.1, that's something else I've been playing with ;)

    Now to access cachemgr.cgi by following this link http://<your_pfsense_server>/cachemgr.cgi</your_pfsense_server>.

    ;)



  • thanks …
    tutorials that you provide goes well
    even now I'm dizzy reading the information provided  ;D

    but I still have one question, how to give a password when accessing cachemgr ?

    thanks a lot


Log in to reply