Squid is eating all my memory FIXED
-
I notice that squid is consuming a lot of memory, even when the size of the memory cache is set to < 1/2 the physical memory. The service watchdog fortunately restarts squid an squidguard when they run out of memory, about every four days. I see the following.
Another view (taken from Observium) is a little clearer and shows where I upgraded the memory from 1G to 2G on Fri 16th May. The memory usage of squid was even clearer after this.
Using top -o size its easy to see that squid is the big memory hog, and how it grows.
last pid: 99281; load averages: 0.00, 0.00, 0.00 up 15+21:51:16 19:39:58 73 processes: 2 running, 67 sleeping, 4 zombie CPU: 0.4% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.3% idle Mem: 242M Active, 234M Inact, 229M Wired, 1684K Cache, 112M Buf, 1275M Free Swap: PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 6715 root 6 44 0 333M 8896K uwait 0:20 0.00% radiusd 88143 proxy 17 44 0 120M 107M ucond 0:10 0.00% squid 35763 root 1 64 20 89424K 7644K nanslp 3:14 0.00% php 74686 root 1 54 0 83920K 27796K accept 0:27 0.00% php 92912 root 1 60 0 78160K 19336K accept 0:04 0.00% php
I read that squid a rather leak ridden. Anybody else seen this?
Steve



 -
How large is your disk cache?
IIRC, the RAM specified for the memory cache is a minimum, not a maximum. It will use more depending on your other settings.
-
Hi,
I've reduced the cache size to 512M. The webpage describes it thus, so I think it's a max amount to use.
This is the amount of physical RAM (in megabytes) to be used for negative cache and in-transit objects. This value should not exceed more than 50% of the installed RAM. The minimum value is 1MB.
After making the above post I noticed the following message in /var/log/squid/cache.log, repeated endlessly.
2014-06-01 18:41:19 [50045] (squidGuard): can't write to logfile /var/log/squidGuard.log
Having changed the file owner from root to proxy things seem to have improved. The memory usage has more or less stabilised.
last pid: 2272; load averages: 0.05, 0.06, 0.02 up 18+00:01:32 21:50:14 77 processes: 4 running, 69 sleeping, 4 zombie CPU: 3.4% user, 1.1% nice, 5.2% system, 0.4% interrupt, 89.9% idle Mem: 874M Active, 742M Inact, 229M Wired, 56M Cache, 112M Buf, 81M Free Swap: PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 88143 proxy 17 44 0 744M 730M ucond 1:36 0.00% squid 6715 root 6 44 0 333M 9176K uwait 0:26 0.00% radiusd 35763 root 1 64 20 89424K 7644K nanslp 3:39 0.00% php 28756 root 1 96 0 86992K 31364K RUN 0:24 0.00% php 74075 root 1 45 0 78160K 19392K accept 0:09 0.00% php 40552 root 1 76 0 77136K 7988K wait 0:00 0.00% php 36424 root 1 47 0 77136K 6804K wait 0:00 0.00% php 55104 proxy 1 44 0 15564K 4084K sbwait 0:00 0.00% squidGuard 62286 proxy 1 44 0 15564K 3956K sbwait 0:00 0.00% squidGuard 60459 proxy 1 44 0 15564K 3892K sbwait 0:00 0.00% squidGuard 88094 proxy 1 76 0 13756K 7300K wait 0:00 0.00% squid 5500 root 3 44 0 13020K 4016K ucond 5:30 0.00% bacula-fd
-
I believe I'm getting somewhere now. Having set the cache to 1024M and used wget to haul down a large number of web pages very fast, I'm seeing that the reported memory usage from squid to be within range, but the memory consumption of it's process to be roughly double this.
Cache information for squid: Hits as % of all requests: 5min: 0.0%, 60min: 0.5% Hits as % of bytes sent: 5min: 0.7%, 60min: 0.7% Memory hits as % of hit requests: 5min: 0.0%, 60min: 16.9% Disk hits as % of hit requests: 5min: 0.0%, 60min: 71.8% Storage Swap size: 2829518 KB Storage Swap capacity: 27.6% used, 72.4% free Storage Mem size: 444924 KB Storage Mem capacity: 42.4% used, 57.6% free Mean Object Size: 53.03 KB
last pid: 88230; load averages: 0.05, 0.05, 0.07 up 18+02:49:17 00:37:59 71 processes: 2 running, 65 sleeping, 4 zombie CPU: 1.1% user, 0.0% nice, 1.1% system, 0.4% interrupt, 97.4% idle Mem: 1065M Active, 591M Inact, 249M Wired, 268K Cache, 112M Buf, 78M Free Swap: PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 53129 proxy 17 44 0 941M 928M RUN 0:35 0.98% squid 52037 root 1 45 0 84944K 25744K accept 0:21 0.00% php 74075 root 1 76 0 78160K 17900K accept 0:09 0.00% php 82484 root 1 64 20 8224K 8244K select 34:10 0.00% ntpd 6715 root 6 44 0 333M 8036K uwait 0:27 0.00% radiusd
And then waiting a couple of minutes we get.
Cache information for squid: Hits as % of all requests: 5min: 0.0%, 60min: 0.5% Hits as % of bytes sent: 5min: 0.7%, 60min: 0.7% Memory hits as % of hit requests: 5min: 0.0%, 60min: 16.9% Disk hits as % of hit requests: 5min: 0.0%, 60min: 71.8% Storage Swap size: 2862396 KB Storage Swap capacity: 28.0% used, 72.0% free Storage Mem size: 478360 KB Storage Mem capacity: 45.6% used, 54.4% free Mean Object Size: 52.86 KB
last pid: 75787; load averages: 0.18, 0.08, 0.07 up 18+02:50:54 00:39:36 71 processes: 1 running, 66 sleeping, 4 zombie CPU: 0.0% user, 0.0% nice, 2.2% system, 1.9% interrupt, 95.9% idle Mem: 1138M Active, 530M Inact, 239M Wired, 31M Cache, 112M Buf, 45M Free Swap: PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 53129 proxy 17 44 0 1010M 996M ucond 0:38 1.95% squid 52037 root 1 45 0 84944K 25616K accept 0:21 0.00% php 74075 root 1 76 0 78160K 17900K accept 0:09 0.00% php 82484 root 1 64 20 8224K 8244K select 34:10 0.00% ntpd 40552 root 1 73 0 77136K 7984K wait 0:00 0.00% php 83319 root 1 44 0 9460K 7768K select 3:11 0.00% bsnmpd 52679 proxy 1 76 0 13756K 7684K wait 0:00 0.00% squid
So, an extra 34M or so of cached data takes up another 68M or so of resident memory.
Looks like I need to be halving my cache size down to nearer 512M.
Steve
-
You'll probably find this most helpful: http://wiki.squid-cache.org/SquidFaq/SquidMemory
-
Thanks for that; I'd been looking at that and similar pages before comparing the General Runtime Information from cachemgr.cgi with the output of top -o res, as shown below. With the cache size limit set to 512M it has now stabilised using 1084M of resident memory when the memory cache is full.
Cache information for squid: Hits as % of all requests: 5min: 1.9%, 60min: 1.2% Hits as % of bytes sent: 5min: 1.1%, 60min: 0.8% Memory hits as % of hit requests: 5min: 10.0%, 60min: 30.3% Disk hits as % of hit requests: 5min: 90.0%, 60min: 62.8% Storage Swap size: 2978610 KB Storage Swap capacity: 29.1% used, 70.9% free Storage Mem size: 521188 KB Storage Mem capacity: 99.4% used, 0.6% free Mean Object Size: 52.09 KB Requests given to unlinkd: 0
last pid: 4916; load averages: 0.03, 0.08, 0.02 up 18+23:40:34 21:29:16 72 processes: 1 running, 67 sleeping, 4 zombie CPU: 5.6% user, 0.0% nice, 3.4% system, 9.0% interrupt, 82.0% idle Mem: 1206M Active, 408M Inact, 227M Wired, 53M Cache, 112M Buf, 88M Free Swap: PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 53129 proxy 17 44 0 1135M 1084M ucond 2:26 1.95% squid
The recommendation to not exceed 50% of available RAM appears bogus and I believe should be changed to 25%.
Steve
-
I am having similar issues and would like to troubleshoot further. What command line option did you run to get your squid cache stats? I also found that there were memory leak fixes in two Squid releases immediately after the build that the pfSense package uses (3.1.20). Perhaps we can get the package rebuilt to the final 3.1 release (3.1.23).
Here are the release notes for the memory leak issues, one fix in 3.1.21 and one in 3.1.22:
http://www.squid-cache.org/Versions/v3/3.1/changesets/SQUID_3_1_22.html
http://www.squid-cache.org/Versions/v3/3.1/changesets/SQUID_3_1_21.html -
See HERE for how to get Cache Manager stats. Select General Runtime Information from the menu to get the stats above.
Steve