Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Squid not using disk cache?

    Cache/Proxy
    3
    7
    1.2k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      shafylus
      last edited by shafylus

      Hi,
      new pfsense user, well not new, just inexperienced.

      I have the transparent squid proxy set up and I notice its getting hits but the disk cache has zero utilization? Why? Is this normal? It's specifically the Disk Hits line that I don't understand:
      Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0%

      Thank you for any suggestions or replies

      Squid Object Cache: Version 5.8
      Build Info:
      Service Name: squid
      Start Time:	Sun, 29 Oct 2023 01:48:01 GMT
      Current Time:	Sun, 29 Oct 2023 02:06:43 GMT
      Connection information for squid:
      	Number of clients accessing cache:	20
      	Number of HTTP requests received:	193
      	Number of ICP messages received:	0
      	Number of ICP messages sent:	0
      	Number of queued ICP replies:	0
      	Number of HTCP messages received:	0
      	Number of HTCP messages sent:	0
      	Request failure ratio:	 0.00
      	Average HTTP requests per minute since start:	10.3
      	Average ICP messages per minute since start:	0.0
      	Select loop called: 743218 times, 1.509 ms avg
      Cache information for squid:
      	Hits as % of all requests:	5min: 16.7%, 60min: 20.5%
      	Hits as % of bytes sent:	5min: 0.4%, 60min: 0.4%
      	Memory hits as % of hit requests:	5min: 0.0%, 60min: 7.7%
      	Disk hits as % of hit requests:	5min: 0.0%, 60min: 0.0%
      	Storage Swap size:	74531696 KB
      	Storage Swap capacity:	 2.4% used, 97.6% free
      	Storage Mem size:	234568 KB
      	Storage Mem capacity:	 4.6% used, 95.4% free
      	Mean Object Size:	1645.33 KB
      	Requests given to unlinkd:	7
      Median Service Times (seconds)  5 min    60 min:
      	HTTP Requests (All):   0.04519  0.03829
      	Cache Misses:          0.06640  0.05046
      	Cache Hits:            0.01745  0.00307
      	Near Hits:             0.00000  0.00000
      	Not-Modified Replies:  0.00000  0.00307
      	DNS Lookups:           0.00000  0.00000
      	ICP Queries:           0.00000  0.00000
      Resource usage for squid:
      	UP Time:	1121.650 seconds
      	CPU Time:	24.818 seconds
      	CPU Usage:	2.21%
      	CPU Usage, 5 minute avg:	2.78%
      	CPU Usage, 60 minute avg:	2.21%
      	Maximum Resident Size: 2836352 KB
      	Page faults with physical i/o: 21
      Memory accounted for:
      	Total accounted:       243225 KB
      	memPoolAlloc calls:    621165
      	memPoolFree calls:     727224
      File descriptor usage for squid:
      	Maximum number of file descriptors:   466812
      	Largest file desc currently in use:     36
      	Number of file desc currently in use:   23
      	Files queued for open:                   0
      	Available number of file descriptors: 466789
      	Reserved number of file descriptors:   100
      	Store Disk files open:                   0
      Internal Data Structures:
      	 45353 StoreEntries
      	    93 StoreEntries with MemObjects
      	    39 Hot Object Cache Items
      	 45299 on-disk objects
      
      1 Reply Last reply Reply Quote 0
      • JonathanLeeJ
        JonathanLee
        last edited by

        Screenshot 2023-11-01 at 10.01.03 AM.png

        Hi,

        Have you set the "Hard Disk Cache System?"

        Make sure to upvote

        KOMK S 2 Replies Last reply Reply Quote 0
        • KOMK
          KOM @JonathanLee
          last edited by

          @JonathanLee In my experience, squid no longer does a good job of caching. The nature of the dynamic web makes it hard for squid when it's trying to cache static content but all content appears to be dynamic. When I was using squid cache, I could only ever get in the 4-7% hit range which isn't worth the time to configure the cache. These days I use squid as a base for squidguard filtering only. All disk caching is disabled.

          JonathanLeeJ 1 Reply Last reply Reply Quote 0
          • JonathanLeeJ
            JonathanLee @KOM
            last edited by

            @KOM

            Dynamic cache use with updates. I can get lot's of hits if you are using SSL intercept even in transpartment mode the updates come over the network as port 80.

            I have 80 percent hits in the last hour on mine right now.

            It can also be used to accelerate servers sitting behind the proxy I have read. With use of StoreID you can get vastly higher hit rate rates also per Squids website.

            Screenshot 2023-11-01 at 11.44.15 AM.png

            Make sure to upvote

            KOMK 1 Reply Last reply Reply Quote 0
            • KOMK
              KOM @JonathanLee
              last edited by

              @JonathanLee I'm not sure you're interpreting that chart correctly. The key metric is Hits as % of all requests. Yours is less than 10%. That 88% is the % of hits that were serviced from disk versus RAM. So, of all your traffic, only 9.8% was cached, but of that 9.8%, 88.6% came from disk and not RAM. That's how I understand it.

              1 Reply Last reply Reply Quote 0
              • S
                shafylus @JonathanLee
                last edited by

                @JonathanLee

                Hi Jonathan, I have the system cache set to ufs; as shown:

                Screenshot 2023-11-02 121455.png

                My goal was to make downloads push faster, I need speed.
                My favorite games, i.e. maplestory tend to update and force me
                to redownload the client (23GB) thought squid could help speed it up
                Same with pushing windows updates.

                1 Reply Last reply Reply Quote 0
                • JonathanLeeJ
                  JonathanLee
                  last edited by JonathanLee

                  If you want to up the cache time you may need to use Squids StoreID helper program,

                  https://forum.netgate.com/topic/182487/storeid-and-squid-helper-program/

                  I have been researching this. It does increase cache rates. However I use SSL intercept.

                  You may also want to enable Dynamic cache for Windows updates.

                  #WINDOWS
                  refresh_pattern -i windowsupdate.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
                  refresh_pattern -i microsoft.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
                  refresh_pattern -i windows.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
                  refresh_pattern -i microsoft.com.akadns.net/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
                  refresh_pattern -i deploy.akamaitechnologies.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
                  refresh_pattern -i msedge.b.tlu.dl.delivery.mp.microsoft.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
                  
                  #FACEBOOK
                  refresh_pattern ^http.*facebook.com.* 720 100% 4320
                  
                  #FACEBOOK IMAGES  
                  refresh_pattern -i pixel.facebook.com..(jpg|png|gif|ico|css|js|jpg?) 241920 80% 241920    
                  refresh_pattern -i .akamaihd.net..(jpg|png|gif|ico|css|js|jpg?) 241920 80% 241920    
                  refresh_pattern -i ((facebook.com)|(85.131.151.39)).(jpg|png|gif|jpg?) 241920 99% 241920 store-stale   
                  refresh_pattern static.(xx|ak).fbcdn.net.(jpg|gif|png|jpg?) 241920 99% 241920   
                  refresh_pattern ^http.*profile.ak.fbcdn.net.*(jpg|gif|png|jpg?) 241920 99% 241920
                  refresh_pattern ^http.*fbcdn.net.*(jpg|gif|png|jpg?) 241920 99% 241920
                  
                  #FACEBOOK VIDEO
                  refresh_pattern -i .video.ak.fbcdn.net.*.(mp4|flv|mp3|amf) 10080 80% 43200      
                  refresh_pattern (audio|video)/(webm|mp4) 129600 99% 129600 store-stale
                  #refresh_pattern -i ^http.*squid.internal.*  241920 100% 241920 store-stale
                  
                  #YAHOO
                  refresh_pattern ^http.*mail.yahoo.com.* 720 100% 4320
                  refresh_pattern ^http.*yahoo.* 720 100% 4320
                  refresh_pattern ^http.*yimg.* 720 100% 4320
                  
                  #apple update
                  refresh_pattern -i apple.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip|dist)$ 0 80% 43200 refresh-ims
                  #refresh_pattern -i (download|adcdownload).apple.com/.*\.(pkg|dmg) 4320 100% 43200
                  refresh_pattern -i appldnld\.apple\.com 129600 100% 129600
                  refresh_pattern -i phobos\.apple\.com 129600 100% 129600
                  refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600
                  
                  #GOOGLE STUFF
                  refresh_pattern ^http.*gmail.* 720 100% 4320
                  refresh_pattern ^http.*google.* 720 100% 4320
                  
                  #AntiVirus
                  refresh_pattern ^http.*kaspersky.* 43200 100% 43200
                  refresh_pattern ^http.*geo\.kaspersky\.com.* 43200 100% 43200
                  refresh_pattern kaspersky.*\.avc$ 43200 100% 43200     
                  refresh_pattern kaspersky 43200 100% 43200  
                  
                  #Office/Windows
                  refresh_pattern ^http.*officecdn\.microsoft\.com.* 43200 100% 43200
                  refresh_pattern ^http.*officecdn\.microsoft\.com\.edgesuite\.net.* 43200 100% 43200
                  refresh_pattern ^http.*delivery\.mp\.microsoft\.com\/filestreamingservice\/files.* 43200 100% 43200
                  refresh_pattern ^http.*download\.windowsupdate\.com.* 43200 100% 43200
                  refresh_pattern ^http.*download\.windowsupdate\.com\/.* 43200 100% 43200
                  refresh_pattern ^http.*au\.download\.windowsupdate\.com\/.* 43200 100% 43200
                  refresh_pattern ^http.*tlu\.dl\.delivery\.mp\.microsoft\.com\/filestreamingservice\/files.* 43200 100% 43200
                  
                  #Website
                  refresh_pattern -i (\.|-)(xml|js|jsp|txt|css|swf|html|htm|txt|gif|tiff)(\?.*)?$ 360 80% 1440
                  #end new refresh patterns
                  
                  refresh_pattern \.(ico|video-stats)$ 129600 100% 129600 
                  
                  #MISC FILE CACHING HERE
                  refresh_pattern -i \.(3gp|7z|ace|asx|avi|bin|cab|dat|deb|rpm|divx|dvr-ms)(\?|$) 43800 100% 129600
                  refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso)(\?|$) 43800 100% 129600
                  refresh_pattern -i \.(m1v|M2V|M2P|MOD|MOV|FLV)(\?|$) 43800 100% 129600
                  refresh_pattern -i \.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf)(\?|$) 43800 100% 129600
                  refresh_pattern -i \.(mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p))(\?|$) 43800 100% 129600
                  refresh_pattern -i \.(og(x|v|a|g)|rar|rm|r(a|p)m|snd|vob|wav)(\?|$) 43800 100% 129600
                  refresh_pattern -i \.(pp(s|t)|wax|wm(a|v)|wmx|wpl|zip|cb(r|z|t))(\?|$) 43800 100% 129600 
                  refresh_pattern -i \.(woff|txt|exe|dmg|webm)(\?|$) 43800 100% 129600
                  refresh_pattern -i \.(doc|pdf)(\?|$) 10080 90% 43200 # DOC | PDF
                  refresh_pattern -i .(iso|avi|wav|mp3|mp4|mpeg|swf|flv|x-flv)$ 43200 90% 432000
                  refresh_pattern -i .(deb|rpm|exe|zip|tar|tgz|ram|rar|bin|ppt|doc|docx|tiff)$ 10080 90% 43200   
                  refresh_pattern -i .index.(html|htm)$ 0 40% 10080     
                  refresh_pattern -i .(ppt|pptx|doc|docx|docm|docb|dot|pdf|pub|ps)$ 100000 90% 200000 refresh-ims
                  refresh_pattern -i .(xls|xlsx|xlt|xlm|xlsm|xltm|xlw|csv|txt)$ 100000 90% 200000 refresh-ims
                  refresh_pattern -i .(app|bin|deb|rpm|drpm|exe|zip|zipx|tar|tgz|tbz2|tlz|iso|arj|cfs|dar|jar)$ 100000 90% 200000 refresh-ims
                  refresh_pattern -i .(bz|bz2|ipa|ram|rar|uxx|gz|msi|dll|lz|lzma|7z|s7z|Z|z|zz|sz)$ 100000 90% 200000 refresh-ims
                  refresh_pattern -i .(exe|msi)$ 0 90% 200000 refresh-ims
                  refresh_pattern -i .(cab|psf|vidt|apk|wtex|hz|ova|ovf)$ 100000 90% 200000 refresh-ims
                  refresh_pattern -i .(xml|flow|asp|aspx)$ 0 90% 200000 refresh-ims
                  refresh_pattern -i .(json)$ 0 90% 200000 refresh-ims
                  refresh_pattern -i .(asx|mp2|mp3|mp4|mp5|wmv|flv|mts|f4v|f4|pls|midi|mid)$ 100000 90% 200000 refresh-ims
                  refresh_pattern -i .(mpa|m2a|mpe|avi|mov|mpg|mpeg|mpg3|mpg4|mpg5)$ 100000 90% 200000 refresh-ims
                  refresh_pattern -i .(m1s|mp2v|m2v|m2s|m2ts|mp2t|wmx|rm|rmvb|3pg|3gpp|omg|ogm|asf|war)$ 100000 90% 200000 refresh-ims
                  refresh_pattern -i .(wav|class|dat|zsci|ver|advcs)$ 100000 90% 200000 refresh-ims
                  refresh_pattern -i .(gif|png|ico|jpg|jpeg|jp2|webp)$ 100000 90% 200000 refresh-ims
                  refresh_pattern -i .(jpx|j2k|j2c|fpx|bmp|tif|tiff|bif)$ 100000 90% 20000 refresh-ims
                  refresh_pattern -i .(pcd|pict|rif|exif|hdr|bpg|img|jif|jfif)$ 100000 90% 200000 refresh-ims
                  refresh_pattern -i .(woff|woff2|eps|ttf|otf|svg|svgi|svgz|ps|ps1|acsm|eot)$ 100000 90% 200000 refresh-ims
                  refresh_pattern -i (\.|-)(mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t)(\?.*)?$ 43200 100% 432000        
                  refresh_pattern -i (.|-)(mp3|m4a|aa?c3?|wm?av?|og(x|v|a|g)|ape|mka|au|aiff|zip|flac|m4(b|r)|m1v|m2(v|p)|mo(d|v)|arj|appx|lha|lzh|on2) 43200 100% 432000        
                  refresh_pattern -i (.|-)(exe|bin|(n|t)ar|acv|(r|j)ar|t?gz|(g|b)z(ip)?2?|7?z(ip)?|wm[v|a]|patch|diff|mar|vpu|inc|r(a|p)m|kom|iso|sys|[ap]sf|ms[i|u|f]|dat|msi|cab|psf|dvr-ms|ace|asx|qt|xt|esd) 43200 100% 432000               
                  refresh_pattern -i (.|-)(webm|(x-)?swf|mp(eg)?(3|4)|mpe?g(av)?|(x-)?f(l|4)v|divx?|rmvb?|mov|trp|ts|avi|m38u|wmv|wmp|m4v|mkv|asf|dv|vob|3gp?2?) 43200 100% 432000        
                  refresh_pattern -i (.|-)(docx?|xlsx?|pptx?|rtf|xml|pdf|tiff?|txt) 43200 100% 432000
                  
                  #new refresh patterns 2
                  refresh_pattern -i (\.|-)(ini|def|sig|upt|mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|esd|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t)(\?.*)?$ 43200 100% 432000        
                  #end new refresh patterns 2
                  
                  #new refresh patterns
                  refresh_pattern -i (\.|-)(mp3|m4a|aa?c3?|wm?av?|og(x|v|a|g)|ape|mka|au|aiff|zip|flac|m4(b|r)|m1v|m2(v|p)|mo(d|v)|arj|appx|lha|lzh|on2)(\?.*)?$ 43200 100% 432000        
                  refresh_pattern -i (\.|-)(exe|bin|(n|t)ar|acv|(r|j)ar|t?gz|(g|b)z(ip)?2?|7?z(ip)?|wm[v|a]|patch|diff|mar|vpu|inc|r(a|p)m|kom|iso|sys|[ap]sf|ms[i|u|f]|dat|msi|cab|psf|dvr-ms|ace|asx|qt|xt|esd)(\?.*)?$ 43200 100% 432000        
                  refresh_pattern -i (\.|-)(ico(.*)?|pn[pg]|css|(g|t)iff?|jpe?g(2|3|4)?|psd|c(d|b)r|cad|bmp|img)(\?.*)?$ 43200 100% 432000        
                  refresh_pattern -i (\.|-)(webm|(x-)?swf|mp(eg)?(3|4)|mpe?g(av)?|(x-)?f(l|4)v|divx?|rmvb?|mov|trp|ts|avi|m38u|wmv|wmp|m4v|mkv|asf|dv|vob|3gp?2?)(\?.*)?$ 43200 100% 432000        
                  refresh_pattern -i (\.|-)(docx?|xlsx?|pptx?|rtf|xml|pdf|tiff?|txt)(\?.*)?$ 43200 100% 432000        
                  refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|flv) 129600 100% 129600                            
                  refresh_pattern (Release|Packages(.gz)*)$ 0 20% 2880
                  
                  # GENERIC CACHING BELOW
                  refresh_pattern -i \.(cdn) 10800 100% 43800
                  refresh_pattern -i (cdn)   10800 100% 43800
                  refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?) 129600 100% 129600      
                  refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.flv?) 129600 100% 129600      
                  refresh_pattern ^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 129600 20% 129600       
                  refresh_pattern ^.*safebrowsing.*google 129600 100% 129600      
                  refresh_pattern ^http://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.uk) 129600 100% 129600   
                  refresh_pattern ytimg\.com.*\.jpg 129600 100% 129600            
                  
                  #YOUTUBE
                  refresh_pattern \.ytimg\? 10800 90% 10800
                  refresh_pattern -i (yimg|twimg).com.* 1440 100% 129600    
                  refresh_pattern -i (ytimg|ggpht).com.* 1440 80% 129600     
                  refresh_pattern -i (get_video?|videoplayback?|videodownload?|.mp4|.webm|.flv|((audio|video)/(webm|mp4))) 241920 100% 241920 store-stale
                  refresh_pattern -i ^https?://..googlevideo.com/videoplayback. 10080 99% 43200 store-stale
                  refresh_pattern -i ^https?://..googlevideo.com/videoplayback.$ 241920 100% 241920 store-stale
                  
                  #XBOX
                  refresh_pattern -i lancache-microsoft 525600 100% 525600
                  refresh_pattern -i images-eds.xboxlive.com 525600 100% 525600
                  
                  #catch all
                  refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
                  refresh_pattern  . 0 20% 4320
                  range_offset_limit 0
                  

                  Ref:
                  https://github.com/mmd123/squid-cache-dynamic_refresh-list/blob/master/pfsense%20squid%20refresh%20patterns

                  https://wiki.squid-cache.org/Features/StoreID#:~:text=“Store%20ID”%20is%20another%20name,algorithm%20via%20a%20helper%20program.

                  https://github.com/rudiservo/pfsense_storeid

                  http://www.squid-cache.org/Doc/config/store_id_program/

                  https://wiki.squid-cache.org/Features/StoreID/DB

                  https://forum.netgate.com/topic/106614/dynamic-cache-not-work/4

                  I hope that gives you some good info on ways to get better hit rates.

                  Make sure to upvote

                  1 Reply Last reply Reply Quote 0
                  • First post
                    Last post
                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.