Squid is eating all my memory FIXED

Steve Evans · Jun 4, 2014, 8:31 PM

I notice that squid is consuming a lot of memory, even when the size of the memory cache is set to < 1/2 the physical memory. The service watchdog fortunately restarts squid an squidguard when they run out of memory, about every four days. I see the following.

Another view (taken from Observium) is a little clearer and shows where I upgraded the memory from 1G to 2G on Fri 16th May. The memory usage of squid was even clearer after this.

Using top -o size its easy to see that squid is the big memory hog, and how it grows.

last pid: 99281;  load averages:  0.00,  0.00,  0.00   up 15+21:51:16  19:39:58
73 processes:  2 running, 67 sleeping, 4 zombie
CPU:  0.4% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.3% idle
Mem: 242M Active, 234M Inact, 229M Wired, 1684K Cache, 112M Buf, 1275M Free
Swap: 

  PID USERNAME       THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
 6715 root             6  44    0   333M  8896K uwait    0:20  0.00% radiusd
88143 proxy           17  44    0   120M   107M ucond    0:10  0.00% squid
35763 root             1  64   20 89424K  7644K nanslp   3:14  0.00% php
74686 root             1  54    0 83920K 27796K accept   0:27  0.00% php
92912 root             1  60    0 78160K 19336K accept   0:04  0.00% php

I read that squid a rather leak ridden. Anybody else seen this?

Steve
![Screen Shot 2014-06-01 at 19.31.43.png_thumb](/public/imported_attachments/1/Screen Shot 2014-06-01 at 19.31.43.png_thumb)
![Screen Shot 2014-06-01 at 19.31.43.png](/public/imported_attachments/1/Screen Shot 2014-06-01 at 19.31.43.png)
![Screen Shot 2014-06-01 at 19.30.33.png](/public/imported_attachments/1/Screen Shot 2014-06-01 at 19.30.33.png)
![Screen Shot 2014-06-01 at 19.30.33.png_thumb](/public/imported_attachments/1/Screen Shot 2014-06-01 at 19.30.33.png_thumb)

jimp · Jun 3, 2014, 5:29 PM

How large is your disk cache?

IIRC, the RAM specified for the memory cache is a minimum, not a maximum. It will use more depending on your other settings.

Steve Evans · Jun 3, 2014, 9:07 PM

Hi,

I've reduced the cache size to 512M. The webpage describes it thus, so I think it's a max amount to use.

This is the amount of physical RAM (in megabytes) to be used for negative cache and in-transit objects. This value should not exceed more than 50% of the installed RAM. The minimum value is 1MB.

After making the above post I noticed the following message in /var/log/squid/cache.log, repeated endlessly.

2014-06-01 18:41:19 [50045] (squidGuard): can't write to logfile /var/log/squidGuard.log

Having changed the file owner from root to proxy things seem to have improved. The memory usage has more or less stabilised.

last pid:  2272;  load averages:  0.05,  0.06,  0.02   up 18+00:01:32  21:50:14
77 processes:  4 running, 69 sleeping, 4 zombie
CPU:  3.4% user,  1.1% nice,  5.2% system,  0.4% interrupt, 89.9% idle
Mem: 874M Active, 742M Inact, 229M Wired, 56M Cache, 112M Buf, 81M Free
Swap: 

  PID USERNAME       THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
88143 proxy           17  44    0   744M   730M ucond    1:36  0.00% squid
 6715 root             6  44    0   333M  9176K uwait    0:26  0.00% radiusd
35763 root             1  64   20 89424K  7644K nanslp   3:39  0.00% php
28756 root             1  96    0 86992K 31364K RUN      0:24  0.00% php
74075 root             1  45    0 78160K 19392K accept   0:09  0.00% php
40552 root             1  76    0 77136K  7988K wait     0:00  0.00% php
36424 root             1  47    0 77136K  6804K wait     0:00  0.00% php
55104 proxy            1  44    0 15564K  4084K sbwait   0:00  0.00% squidGuard
62286 proxy            1  44    0 15564K  3956K sbwait   0:00  0.00% squidGuard
60459 proxy            1  44    0 15564K  3892K sbwait   0:00  0.00% squidGuard
88094 proxy            1  76    0 13756K  7300K wait     0:00  0.00% squid
 5500 root             3  44    0 13020K  4016K ucond    5:30  0.00% bacula-fd

Steve Evans · Jun 3, 2014, 11:41 PM

I believe I'm getting somewhere now. Having set the cache to 1024M and used wget to haul down a large number of web pages very fast, I'm seeing that the reported memory usage from squid to be within range, but the memory consumption of it's process to be roughly double this.

Cache information for squid:
	Hits as % of all requests:	5min: 0.0%, 60min: 0.5%
	Hits as % of bytes sent:	5min: 0.7%, 60min: 0.7%
	Memory hits as % of hit requests:	5min: 0.0%, 60min: 16.9%
	Disk hits as % of hit requests:	5min: 0.0%, 60min: 71.8%
	Storage Swap size:	2829518 KB
	Storage Swap capacity:	27.6% used, 72.4% free
	Storage Mem size:	444924 KB
	Storage Mem capacity:	42.4% used, 57.6% free
	Mean Object Size:	53.03 KB

last pid: 88230;  load averages:  0.05,  0.05,  0.07   up 18+02:49:17  00:37:59
71 processes:  2 running, 65 sleeping, 4 zombie
CPU:  1.1% user,  0.0% nice,  1.1% system,  0.4% interrupt, 97.4% idle
Mem: 1065M Active, 591M Inact, 249M Wired, 268K Cache, 112M Buf, 78M Free
Swap: 

  PID USERNAME       THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
53129 proxy           17  44    0   941M   928M RUN      0:35  0.98% squid
52037 root             1  45    0 84944K 25744K accept   0:21  0.00% php
74075 root             1  76    0 78160K 17900K accept   0:09  0.00% php
82484 root             1  64   20  8224K  8244K select  34:10  0.00% ntpd
 6715 root             6  44    0   333M  8036K uwait    0:27  0.00% radiusd

And then waiting a couple of minutes we get.

Cache information for squid:
	Hits as % of all requests:	5min: 0.0%, 60min: 0.5%
	Hits as % of bytes sent:	5min: 0.7%, 60min: 0.7%
	Memory hits as % of hit requests:	5min: 0.0%, 60min: 16.9%
	Disk hits as % of hit requests:	5min: 0.0%, 60min: 71.8%
	Storage Swap size:	2862396 KB
	Storage Swap capacity:	28.0% used, 72.0% free
	Storage Mem size:	478360 KB
	Storage Mem capacity:	45.6% used, 54.4% free
	Mean Object Size:	52.86 KB

last pid: 75787;  load averages:  0.18,  0.08,  0.07   up 18+02:50:54  00:39:36
71 processes:  1 running, 66 sleeping, 4 zombie
CPU:  0.0% user,  0.0% nice,  2.2% system,  1.9% interrupt, 95.9% idle
Mem: 1138M Active, 530M Inact, 239M Wired, 31M Cache, 112M Buf, 45M Free
Swap: 

  PID USERNAME       THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
53129 proxy           17  44    0  1010M   996M ucond    0:38  1.95% squid
52037 root             1  45    0 84944K 25616K accept   0:21  0.00% php
74075 root             1  76    0 78160K 17900K accept   0:09  0.00% php
82484 root             1  64   20  8224K  8244K select  34:10  0.00% ntpd
40552 root             1  73    0 77136K  7984K wait     0:00  0.00% php
83319 root             1  44    0  9460K  7768K select   3:11  0.00% bsnmpd
52679 proxy            1  76    0 13756K  7684K wait     0:00  0.00% squid

So, an extra 34M or so of cached data takes up another 68M or so of resident memory.

Looks like I need to be halving my cache size down to nearer 512M.

Steve

jimp · Jun 4, 2014, 1:48 PM

You'll probably find this most helpful: http://wiki.squid-cache.org/SquidFaq/SquidMemory

Steve Evans · Jun 4, 2014, 8:31 PM

Thanks for that; I'd been looking at that and similar pages before comparing the General Runtime Information from cachemgr.cgi with the output of top -o res, as shown below. With the cache size limit set to 512M it has now stabilised using 1084M of resident memory when the memory cache is full.

Cache information for squid:
	Hits as % of all requests:	5min: 1.9%, 60min: 1.2%
	Hits as % of bytes sent:	5min: 1.1%, 60min: 0.8%
	Memory hits as % of hit requests:	5min: 10.0%, 60min: 30.3%
	Disk hits as % of hit requests:	5min: 90.0%, 60min: 62.8%
	Storage Swap size:	2978610 KB
	Storage Swap capacity:	29.1% used, 70.9% free
	Storage Mem size:	521188 KB
	Storage Mem capacity:	99.4% used,  0.6% free
	Mean Object Size:	52.09 KB
	Requests given to unlinkd:	0

last pid:  4916;  load averages:  0.03,  0.08,  0.02   up 18+23:40:34  21:29:16
72 processes:  1 running, 67 sleeping, 4 zombie
CPU:  5.6% user,  0.0% nice,  3.4% system,  9.0% interrupt, 82.0% idle
Mem: 1206M Active, 408M Inact, 227M Wired, 53M Cache, 112M Buf, 88M Free
Swap: 

  PID USERNAME       THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
53129 proxy           17  44    0  1135M  1084M ucond    2:26  1.95% squid

The recommendation to not exceed 50% of available RAM appears bogus and I believe should be changed to 25%.

Steve

mhab12 · Sep 17, 2014, 7:53 PM

I am having similar issues and would like to troubleshoot further. What command line option did you run to get your squid cache stats? I also found that there were memory leak fixes in two Squid releases immediately after the build that the pfSense package uses (3.1.20). Perhaps we can get the package rebuilt to the final 3.1 release (3.1.23).

Here are the release notes for the memory leak issues, one fix in 3.1.21 and one in 3.1.22:
http://www.squid-cache.org/Versions/v3/3.1/changesets/SQUID_3_1_22.html
http://www.squid-cache.org/Versions/v3/3.1/changesets/SQUID_3_1_21.html

Steve Evans · Sep 17, 2014, 9:22 PM

See HERE for how to get Cache Manager stats. Select General Runtime Information from the menu to get the stats above.

Steve