Squid is eating all my memory FIXED



  • I notice that squid is consuming a lot of memory, even when the size of the memory cache is set to < 1/2 the physical memory. The service watchdog fortunately restarts squid an squidguard when they run out of memory, about every four days. I see the following.

    Another view (taken from Observium) is a little clearer and shows where I upgraded the memory from 1G to 2G on Fri 16th May. The memory usage of squid was even clearer after this.

    Using top -o size its easy to see that squid is the big memory hog, and how it grows.

    last pid: 99281;  load averages:  0.00,  0.00,  0.00   up 15+21:51:16  19:39:58
    73 processes:  2 running, 67 sleeping, 4 zombie
    CPU:  0.4% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.3% idle
    Mem: 242M Active, 234M Inact, 229M Wired, 1684K Cache, 112M Buf, 1275M Free
    Swap: 
    
      PID USERNAME       THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
     6715 root             6  44    0   333M  8896K uwait    0:20  0.00% radiusd
    88143 proxy           17  44    0   120M   107M ucond    0:10  0.00% squid
    35763 root             1  64   20 89424K  7644K nanslp   3:14  0.00% php
    74686 root             1  54    0 83920K 27796K accept   0:27  0.00% php
    92912 root             1  60    0 78160K 19336K accept   0:04  0.00% php
    
    

    I read that squid a rather leak ridden. Anybody else seen this?

    Steve
    ![Screen Shot 2014-06-01 at 19.31.43.png_thumb](/public/imported_attachments/1/Screen Shot 2014-06-01 at 19.31.43.png_thumb)
    ![Screen Shot 2014-06-01 at 19.31.43.png](/public/imported_attachments/1/Screen Shot 2014-06-01 at 19.31.43.png)
    ![Screen Shot 2014-06-01 at 19.30.33.png](/public/imported_attachments/1/Screen Shot 2014-06-01 at 19.30.33.png)
    ![Screen Shot 2014-06-01 at 19.30.33.png_thumb](/public/imported_attachments/1/Screen Shot 2014-06-01 at 19.30.33.png_thumb)


  • Rebel Alliance Developer Netgate

    How large is your disk cache?

    IIRC, the RAM specified for the memory cache is a minimum, not a maximum. It will use more depending on your other settings.



  • Hi,

    I've reduced the cache size to 512M. The webpage describes it thus, so I think it's a max amount to use.

    This is the amount of physical RAM (in megabytes) to be used for negative cache and in-transit objects. This value should not exceed more than 50% of the installed RAM. The minimum value is 1MB.

    After making the above post I noticed the following message in /var/log/squid/cache.log, repeated endlessly.

    2014-06-01 18:41:19 [50045] (squidGuard): can't write to logfile /var/log/squidGuard.log
    

    Having changed the file owner from root to proxy things seem to have improved. The memory usage has more or less stabilised.

    last pid:  2272;  load averages:  0.05,  0.06,  0.02   up 18+00:01:32  21:50:14
    77 processes:  4 running, 69 sleeping, 4 zombie
    CPU:  3.4% user,  1.1% nice,  5.2% system,  0.4% interrupt, 89.9% idle
    Mem: 874M Active, 742M Inact, 229M Wired, 56M Cache, 112M Buf, 81M Free
    Swap: 
    
      PID USERNAME       THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
    88143 proxy           17  44    0   744M   730M ucond    1:36  0.00% squid
     6715 root             6  44    0   333M  9176K uwait    0:26  0.00% radiusd
    35763 root             1  64   20 89424K  7644K nanslp   3:39  0.00% php
    28756 root             1  96    0 86992K 31364K RUN      0:24  0.00% php
    74075 root             1  45    0 78160K 19392K accept   0:09  0.00% php
    40552 root             1  76    0 77136K  7988K wait     0:00  0.00% php
    36424 root             1  47    0 77136K  6804K wait     0:00  0.00% php
    55104 proxy            1  44    0 15564K  4084K sbwait   0:00  0.00% squidGuard
    62286 proxy            1  44    0 15564K  3956K sbwait   0:00  0.00% squidGuard
    60459 proxy            1  44    0 15564K  3892K sbwait   0:00  0.00% squidGuard
    88094 proxy            1  76    0 13756K  7300K wait     0:00  0.00% squid
     5500 root             3  44    0 13020K  4016K ucond    5:30  0.00% bacula-fd
    
    


  • I believe I'm getting somewhere now. Having set the cache to 1024M and used wget to haul down a large number of web pages very fast, I'm seeing that the reported memory usage from squid to be within range, but the memory consumption of it's process to be roughly double this.

    Cache information for squid:
    	Hits as % of all requests:	5min: 0.0%, 60min: 0.5%
    	Hits as % of bytes sent:	5min: 0.7%, 60min: 0.7%
    	Memory hits as % of hit requests:	5min: 0.0%, 60min: 16.9%
    	Disk hits as % of hit requests:	5min: 0.0%, 60min: 71.8%
    	Storage Swap size:	2829518 KB
    	Storage Swap capacity:	27.6% used, 72.4% free
    	Storage Mem size:	444924 KB
    	Storage Mem capacity:	42.4% used, 57.6% free
    	Mean Object Size:	53.03 KB
    
    last pid: 88230;  load averages:  0.05,  0.05,  0.07   up 18+02:49:17  00:37:59
    71 processes:  2 running, 65 sleeping, 4 zombie
    CPU:  1.1% user,  0.0% nice,  1.1% system,  0.4% interrupt, 97.4% idle
    Mem: 1065M Active, 591M Inact, 249M Wired, 268K Cache, 112M Buf, 78M Free
    Swap: 
    
      PID USERNAME       THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
    53129 proxy           17  44    0   941M   928M RUN      0:35  0.98% squid
    52037 root             1  45    0 84944K 25744K accept   0:21  0.00% php
    74075 root             1  76    0 78160K 17900K accept   0:09  0.00% php
    82484 root             1  64   20  8224K  8244K select  34:10  0.00% ntpd
     6715 root             6  44    0   333M  8036K uwait    0:27  0.00% radiusd
    
    

    And then waiting a couple of minutes we get.

    Cache information for squid:
    	Hits as % of all requests:	5min: 0.0%, 60min: 0.5%
    	Hits as % of bytes sent:	5min: 0.7%, 60min: 0.7%
    	Memory hits as % of hit requests:	5min: 0.0%, 60min: 16.9%
    	Disk hits as % of hit requests:	5min: 0.0%, 60min: 71.8%
    	Storage Swap size:	2862396 KB
    	Storage Swap capacity:	28.0% used, 72.0% free
    	Storage Mem size:	478360 KB
    	Storage Mem capacity:	45.6% used, 54.4% free
    	Mean Object Size:	52.86 KB
    
    last pid: 75787;  load averages:  0.18,  0.08,  0.07   up 18+02:50:54  00:39:36
    71 processes:  1 running, 66 sleeping, 4 zombie
    CPU:  0.0% user,  0.0% nice,  2.2% system,  1.9% interrupt, 95.9% idle
    Mem: 1138M Active, 530M Inact, 239M Wired, 31M Cache, 112M Buf, 45M Free
    Swap: 
    
      PID USERNAME       THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
    53129 proxy           17  44    0  1010M   996M ucond    0:38  1.95% squid
    52037 root             1  45    0 84944K 25616K accept   0:21  0.00% php
    74075 root             1  76    0 78160K 17900K accept   0:09  0.00% php
    82484 root             1  64   20  8224K  8244K select  34:10  0.00% ntpd
    40552 root             1  73    0 77136K  7984K wait     0:00  0.00% php
    83319 root             1  44    0  9460K  7768K select   3:11  0.00% bsnmpd
    52679 proxy            1  76    0 13756K  7684K wait     0:00  0.00% squid
    

    So, an extra 34M or so of cached data takes up another 68M or so of resident memory.

    Looks like I need to be halving my cache size down to nearer 512M.

    Steve


  • Rebel Alliance Developer Netgate

    You'll probably find this most helpful: http://wiki.squid-cache.org/SquidFaq/SquidMemory



  • Thanks for that; I'd been looking at that and similar pages before comparing the General Runtime Information from cachemgr.cgi with the output of top -o res, as shown below. With the cache size limit set to 512M it has now stabilised using 1084M of resident memory when the memory cache is full.

    Cache information for squid:
    	Hits as % of all requests:	5min: 1.9%, 60min: 1.2%
    	Hits as % of bytes sent:	5min: 1.1%, 60min: 0.8%
    	Memory hits as % of hit requests:	5min: 10.0%, 60min: 30.3%
    	Disk hits as % of hit requests:	5min: 90.0%, 60min: 62.8%
    	Storage Swap size:	2978610 KB
    	Storage Swap capacity:	29.1% used, 70.9% free
    	Storage Mem size:	521188 KB
    	Storage Mem capacity:	99.4% used,  0.6% free
    	Mean Object Size:	52.09 KB
    	Requests given to unlinkd:	0
    
    last pid:  4916;  load averages:  0.03,  0.08,  0.02   up 18+23:40:34  21:29:16
    72 processes:  1 running, 67 sleeping, 4 zombie
    CPU:  5.6% user,  0.0% nice,  3.4% system,  9.0% interrupt, 82.0% idle
    Mem: 1206M Active, 408M Inact, 227M Wired, 53M Cache, 112M Buf, 88M Free
    Swap: 
    
      PID USERNAME       THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
    53129 proxy           17  44    0  1135M  1084M ucond    2:26  1.95% squid
    
    

    The recommendation to not exceed 50% of available RAM appears bogus and I believe should be changed to 25%.

    Steve



  • I am having similar issues and would like to troubleshoot further.  What command line option did you run to get your squid cache stats?  I also found that there were memory leak fixes in two Squid releases immediately after the build that the pfSense package uses (3.1.20).  Perhaps we can get the package rebuilt to the final 3.1 release (3.1.23).

    Here are the release notes for the memory leak issues, one fix in 3.1.21 and one in 3.1.22:
    http://www.squid-cache.org/Versions/v3/3.1/changesets/SQUID_3_1_22.html
    http://www.squid-cache.org/Versions/v3/3.1/changesets/SQUID_3_1_21.html



  • See HERE for how to get Cache Manager stats. Select General Runtime Information from the menu to get the stats above.

    Steve


Log in to reply