• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Squid is eating all my memory FIXED

Scheduled Pinned Locked Moved pfSense Packages
8 Posts 3 Posters 12.2k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S
    Steve Evans
    last edited by Jun 4, 2014, 8:31 PM Jun 1, 2014, 6:36 PM

    I notice that squid is consuming a lot of memory, even when the size of the memory cache is set to < 1/2 the physical memory. The service watchdog fortunately restarts squid an squidguard when they run out of memory, about every four days. I see the following.

    Another view (taken from Observium) is a little clearer and shows where I upgraded the memory from 1G to 2G on Fri 16th May. The memory usage of squid was even clearer after this.

    Using top -o size its easy to see that squid is the big memory hog, and how it grows.

    last pid: 99281;  load averages:  0.00,  0.00,  0.00   up 15+21:51:16  19:39:58
    73 processes:  2 running, 67 sleeping, 4 zombie
    CPU:  0.4% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.3% idle
    Mem: 242M Active, 234M Inact, 229M Wired, 1684K Cache, 112M Buf, 1275M Free
    Swap: 
    
      PID USERNAME       THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
     6715 root             6  44    0   333M  8896K uwait    0:20  0.00% radiusd
    88143 proxy           17  44    0   120M   107M ucond    0:10  0.00% squid
    35763 root             1  64   20 89424K  7644K nanslp   3:14  0.00% php
    74686 root             1  54    0 83920K 27796K accept   0:27  0.00% php
    92912 root             1  60    0 78160K 19336K accept   0:04  0.00% php
    
    

    I read that squid a rather leak ridden. Anybody else seen this?

    Steve
    ![Screen Shot 2014-06-01 at 19.31.43.png_thumb](/public/imported_attachments/1/Screen Shot 2014-06-01 at 19.31.43.png_thumb)
    ![Screen Shot 2014-06-01 at 19.31.43.png](/public/imported_attachments/1/Screen Shot 2014-06-01 at 19.31.43.png)
    ![Screen Shot 2014-06-01 at 19.30.33.png](/public/imported_attachments/1/Screen Shot 2014-06-01 at 19.30.33.png)
    ![Screen Shot 2014-06-01 at 19.30.33.png_thumb](/public/imported_attachments/1/Screen Shot 2014-06-01 at 19.30.33.png_thumb)

    1 Reply Last reply Reply Quote 0
    • J
      jimp Rebel Alliance Developer Netgate
      last edited by Jun 3, 2014, 5:29 PM

      How large is your disk cache?

      IIRC, the RAM specified for the memory cache is a minimum, not a maximum. It will use more depending on your other settings.

      Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

      Need help fast? Netgate Global Support!

      Do not Chat/PM for help!

      1 Reply Last reply Reply Quote 0
      • S
        Steve Evans
        last edited by Jun 3, 2014, 9:07 PM Jun 3, 2014, 9:00 PM

        Hi,

        I've reduced the cache size to 512M. The webpage describes it thus, so I think it's a max amount to use.

        This is the amount of physical RAM (in megabytes) to be used for negative cache and in-transit objects. This value should not exceed more than 50% of the installed RAM. The minimum value is 1MB.

        After making the above post I noticed the following message in /var/log/squid/cache.log, repeated endlessly.

        2014-06-01 18:41:19 [50045] (squidGuard): can't write to logfile /var/log/squidGuard.log
        

        Having changed the file owner from root to proxy things seem to have improved. The memory usage has more or less stabilised.

        last pid:  2272;  load averages:  0.05,  0.06,  0.02   up 18+00:01:32  21:50:14
        77 processes:  4 running, 69 sleeping, 4 zombie
        CPU:  3.4% user,  1.1% nice,  5.2% system,  0.4% interrupt, 89.9% idle
        Mem: 874M Active, 742M Inact, 229M Wired, 56M Cache, 112M Buf, 81M Free
        Swap: 
        
          PID USERNAME       THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
        88143 proxy           17  44    0   744M   730M ucond    1:36  0.00% squid
         6715 root             6  44    0   333M  9176K uwait    0:26  0.00% radiusd
        35763 root             1  64   20 89424K  7644K nanslp   3:39  0.00% php
        28756 root             1  96    0 86992K 31364K RUN      0:24  0.00% php
        74075 root             1  45    0 78160K 19392K accept   0:09  0.00% php
        40552 root             1  76    0 77136K  7988K wait     0:00  0.00% php
        36424 root             1  47    0 77136K  6804K wait     0:00  0.00% php
        55104 proxy            1  44    0 15564K  4084K sbwait   0:00  0.00% squidGuard
        62286 proxy            1  44    0 15564K  3956K sbwait   0:00  0.00% squidGuard
        60459 proxy            1  44    0 15564K  3892K sbwait   0:00  0.00% squidGuard
        88094 proxy            1  76    0 13756K  7300K wait     0:00  0.00% squid
         5500 root             3  44    0 13020K  4016K ucond    5:30  0.00% bacula-fd
        
        
        1 Reply Last reply Reply Quote 0
        • S
          Steve Evans
          last edited by Jun 3, 2014, 11:41 PM

          I believe I'm getting somewhere now. Having set the cache to 1024M and used wget to haul down a large number of web pages very fast, I'm seeing that the reported memory usage from squid to be within range, but the memory consumption of it's process to be roughly double this.

          Cache information for squid:
          	Hits as % of all requests:	5min: 0.0%, 60min: 0.5%
          	Hits as % of bytes sent:	5min: 0.7%, 60min: 0.7%
          	Memory hits as % of hit requests:	5min: 0.0%, 60min: 16.9%
          	Disk hits as % of hit requests:	5min: 0.0%, 60min: 71.8%
          	Storage Swap size:	2829518 KB
          	Storage Swap capacity:	27.6% used, 72.4% free
          	Storage Mem size:	444924 KB
          	Storage Mem capacity:	42.4% used, 57.6% free
          	Mean Object Size:	53.03 KB
          
          last pid: 88230;  load averages:  0.05,  0.05,  0.07   up 18+02:49:17  00:37:59
          71 processes:  2 running, 65 sleeping, 4 zombie
          CPU:  1.1% user,  0.0% nice,  1.1% system,  0.4% interrupt, 97.4% idle
          Mem: 1065M Active, 591M Inact, 249M Wired, 268K Cache, 112M Buf, 78M Free
          Swap: 
          
            PID USERNAME       THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
          53129 proxy           17  44    0   941M   928M RUN      0:35  0.98% squid
          52037 root             1  45    0 84944K 25744K accept   0:21  0.00% php
          74075 root             1  76    0 78160K 17900K accept   0:09  0.00% php
          82484 root             1  64   20  8224K  8244K select  34:10  0.00% ntpd
           6715 root             6  44    0   333M  8036K uwait    0:27  0.00% radiusd
          
          

          And then waiting a couple of minutes we get.

          Cache information for squid:
          	Hits as % of all requests:	5min: 0.0%, 60min: 0.5%
          	Hits as % of bytes sent:	5min: 0.7%, 60min: 0.7%
          	Memory hits as % of hit requests:	5min: 0.0%, 60min: 16.9%
          	Disk hits as % of hit requests:	5min: 0.0%, 60min: 71.8%
          	Storage Swap size:	2862396 KB
          	Storage Swap capacity:	28.0% used, 72.0% free
          	Storage Mem size:	478360 KB
          	Storage Mem capacity:	45.6% used, 54.4% free
          	Mean Object Size:	52.86 KB
          
          last pid: 75787;  load averages:  0.18,  0.08,  0.07   up 18+02:50:54  00:39:36
          71 processes:  1 running, 66 sleeping, 4 zombie
          CPU:  0.0% user,  0.0% nice,  2.2% system,  1.9% interrupt, 95.9% idle
          Mem: 1138M Active, 530M Inact, 239M Wired, 31M Cache, 112M Buf, 45M Free
          Swap: 
          
            PID USERNAME       THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
          53129 proxy           17  44    0  1010M   996M ucond    0:38  1.95% squid
          52037 root             1  45    0 84944K 25616K accept   0:21  0.00% php
          74075 root             1  76    0 78160K 17900K accept   0:09  0.00% php
          82484 root             1  64   20  8224K  8244K select  34:10  0.00% ntpd
          40552 root             1  73    0 77136K  7984K wait     0:00  0.00% php
          83319 root             1  44    0  9460K  7768K select   3:11  0.00% bsnmpd
          52679 proxy            1  76    0 13756K  7684K wait     0:00  0.00% squid
          

          So, an extra 34M or so of cached data takes up another 68M or so of resident memory.

          Looks like I need to be halving my cache size down to nearer 512M.

          Steve

          1 Reply Last reply Reply Quote 0
          • J
            jimp Rebel Alliance Developer Netgate
            last edited by Jun 4, 2014, 1:48 PM

            You'll probably find this most helpful: http://wiki.squid-cache.org/SquidFaq/SquidMemory

            Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

            Need help fast? Netgate Global Support!

            Do not Chat/PM for help!

            1 Reply Last reply Reply Quote 0
            • S
              Steve Evans
              last edited by Jun 4, 2014, 8:31 PM

              Thanks for that; I'd been looking at that and similar pages before comparing the General Runtime Information from cachemgr.cgi with the output of top -o res, as shown below. With the cache size limit set to 512M it has now stabilised using 1084M of resident memory when the memory cache is full.

              Cache information for squid:
              	Hits as % of all requests:	5min: 1.9%, 60min: 1.2%
              	Hits as % of bytes sent:	5min: 1.1%, 60min: 0.8%
              	Memory hits as % of hit requests:	5min: 10.0%, 60min: 30.3%
              	Disk hits as % of hit requests:	5min: 90.0%, 60min: 62.8%
              	Storage Swap size:	2978610 KB
              	Storage Swap capacity:	29.1% used, 70.9% free
              	Storage Mem size:	521188 KB
              	Storage Mem capacity:	99.4% used,  0.6% free
              	Mean Object Size:	52.09 KB
              	Requests given to unlinkd:	0
              
              last pid:  4916;  load averages:  0.03,  0.08,  0.02   up 18+23:40:34  21:29:16
              72 processes:  1 running, 67 sleeping, 4 zombie
              CPU:  5.6% user,  0.0% nice,  3.4% system,  9.0% interrupt, 82.0% idle
              Mem: 1206M Active, 408M Inact, 227M Wired, 53M Cache, 112M Buf, 88M Free
              Swap: 
              
                PID USERNAME       THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
              53129 proxy           17  44    0  1135M  1084M ucond    2:26  1.95% squid
              
              

              The recommendation to not exceed 50% of available RAM appears bogus and I believe should be changed to 25%.

              Steve

              1 Reply Last reply Reply Quote 0
              • M
                mhab12
                last edited by Sep 17, 2014, 7:53 PM Sep 17, 2014, 7:47 PM

                I am having similar issues and would like to troubleshoot further.  What command line option did you run to get your squid cache stats?  I also found that there were memory leak fixes in two Squid releases immediately after the build that the pfSense package uses (3.1.20).  Perhaps we can get the package rebuilt to the final 3.1 release (3.1.23).

                Here are the release notes for the memory leak issues, one fix in 3.1.21 and one in 3.1.22:
                http://www.squid-cache.org/Versions/v3/3.1/changesets/SQUID_3_1_22.html
                http://www.squid-cache.org/Versions/v3/3.1/changesets/SQUID_3_1_21.html

                1 Reply Last reply Reply Quote 0
                • S
                  Steve Evans
                  last edited by Sep 17, 2014, 9:22 PM

                  See HERE for how to get Cache Manager stats. Select General Runtime Information from the menu to get the stats above.

                  Steve

                  1 Reply Last reply Reply Quote 0
                  • First post
                    Last post
                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                    This community forum collects and processes your personal information.
                    consent.not_received