Squid not using disk cache?
-
Hi,
new pfsense user, well not new, just inexperienced.I have the transparent squid proxy set up and I notice its getting hits but the disk cache has zero utilization? Why? Is this normal? It's specifically the Disk Hits line that I don't understand:
Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0%Thank you for any suggestions or replies
Squid Object Cache: Version 5.8 Build Info: Service Name: squid Start Time: Sun, 29 Oct 2023 01:48:01 GMT Current Time: Sun, 29 Oct 2023 02:06:43 GMT Connection information for squid: Number of clients accessing cache: 20 Number of HTTP requests received: 193 Number of ICP messages received: 0 Number of ICP messages sent: 0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 10.3 Average ICP messages per minute since start: 0.0 Select loop called: 743218 times, 1.509 ms avg Cache information for squid: Hits as % of all requests: 5min: 16.7%, 60min: 20.5% Hits as % of bytes sent: 5min: 0.4%, 60min: 0.4% Memory hits as % of hit requests: 5min: 0.0%, 60min: 7.7% Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0% Storage Swap size: 74531696 KB Storage Swap capacity: 2.4% used, 97.6% free Storage Mem size: 234568 KB Storage Mem capacity: 4.6% used, 95.4% free Mean Object Size: 1645.33 KB Requests given to unlinkd: 7 Median Service Times (seconds) 5 min 60 min: HTTP Requests (All): 0.04519 0.03829 Cache Misses: 0.06640 0.05046 Cache Hits: 0.01745 0.00307 Near Hits: 0.00000 0.00000 Not-Modified Replies: 0.00000 0.00307 DNS Lookups: 0.00000 0.00000 ICP Queries: 0.00000 0.00000 Resource usage for squid: UP Time: 1121.650 seconds CPU Time: 24.818 seconds CPU Usage: 2.21% CPU Usage, 5 minute avg: 2.78% CPU Usage, 60 minute avg: 2.21% Maximum Resident Size: 2836352 KB Page faults with physical i/o: 21 Memory accounted for: Total accounted: 243225 KB memPoolAlloc calls: 621165 memPoolFree calls: 727224 File descriptor usage for squid: Maximum number of file descriptors: 466812 Largest file desc currently in use: 36 Number of file desc currently in use: 23 Files queued for open: 0 Available number of file descriptors: 466789 Reserved number of file descriptors: 100 Store Disk files open: 0 Internal Data Structures: 45353 StoreEntries 93 StoreEntries with MemObjects 39 Hot Object Cache Items 45299 on-disk objects
-
Hi,
Have you set the "Hard Disk Cache System?"
-
@JonathanLee In my experience, squid no longer does a good job of caching. The nature of the dynamic web makes it hard for squid when it's trying to cache static content but all content appears to be dynamic. When I was using squid cache, I could only ever get in the 4-7% hit range which isn't worth the time to configure the cache. These days I use squid as a base for squidguard filtering only. All disk caching is disabled.
-
Dynamic cache use with updates. I can get lot's of hits if you are using SSL intercept even in transpartment mode the updates come over the network as port 80.
I have 80 percent hits in the last hour on mine right now.
It can also be used to accelerate servers sitting behind the proxy I have read. With use of StoreID you can get vastly higher hit rate rates also per Squids website.
-
@JonathanLee I'm not sure you're interpreting that chart correctly. The key metric is Hits as % of all requests. Yours is less than 10%. That 88% is the % of hits that were serviced from disk versus RAM. So, of all your traffic, only 9.8% was cached, but of that 9.8%, 88.6% came from disk and not RAM. That's how I understand it.
-
Hi Jonathan, I have the system cache set to ufs; as shown:
My goal was to make downloads push faster, I need speed.
My favorite games, i.e. maplestory tend to update and force me
to redownload the client (23GB) thought squid could help speed it up
Same with pushing windows updates. -
If you want to up the cache time you may need to use Squids StoreID helper program,
https://forum.netgate.com/topic/182487/storeid-and-squid-helper-program/
I have been researching this. It does increase cache rates. However I use SSL intercept.
You may also want to enable Dynamic cache for Windows updates.
#WINDOWS refresh_pattern -i windowsupdate.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i microsoft.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i windows.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i microsoft.com.akadns.net/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i deploy.akamaitechnologies.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i msedge.b.tlu.dl.delivery.mp.microsoft.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims #FACEBOOK refresh_pattern ^http.*facebook.com.* 720 100% 4320 #FACEBOOK IMAGES refresh_pattern -i pixel.facebook.com..(jpg|png|gif|ico|css|js|jpg?) 241920 80% 241920 refresh_pattern -i .akamaihd.net..(jpg|png|gif|ico|css|js|jpg?) 241920 80% 241920 refresh_pattern -i ((facebook.com)|(85.131.151.39)).(jpg|png|gif|jpg?) 241920 99% 241920 store-stale refresh_pattern static.(xx|ak).fbcdn.net.(jpg|gif|png|jpg?) 241920 99% 241920 refresh_pattern ^http.*profile.ak.fbcdn.net.*(jpg|gif|png|jpg?) 241920 99% 241920 refresh_pattern ^http.*fbcdn.net.*(jpg|gif|png|jpg?) 241920 99% 241920 #FACEBOOK VIDEO refresh_pattern -i .video.ak.fbcdn.net.*.(mp4|flv|mp3|amf) 10080 80% 43200 refresh_pattern (audio|video)/(webm|mp4) 129600 99% 129600 store-stale #refresh_pattern -i ^http.*squid.internal.* 241920 100% 241920 store-stale #YAHOO refresh_pattern ^http.*mail.yahoo.com.* 720 100% 4320 refresh_pattern ^http.*yahoo.* 720 100% 4320 refresh_pattern ^http.*yimg.* 720 100% 4320 #apple update refresh_pattern -i apple.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip|dist)$ 0 80% 43200 refresh-ims #refresh_pattern -i (download|adcdownload).apple.com/.*\.(pkg|dmg) 4320 100% 43200 refresh_pattern -i appldnld\.apple\.com 129600 100% 129600 refresh_pattern -i phobos\.apple\.com 129600 100% 129600 refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600 #GOOGLE STUFF refresh_pattern ^http.*gmail.* 720 100% 4320 refresh_pattern ^http.*google.* 720 100% 4320 #AntiVirus refresh_pattern ^http.*kaspersky.* 43200 100% 43200 refresh_pattern ^http.*geo\.kaspersky\.com.* 43200 100% 43200 refresh_pattern kaspersky.*\.avc$ 43200 100% 43200 refresh_pattern kaspersky 43200 100% 43200 #Office/Windows refresh_pattern ^http.*officecdn\.microsoft\.com.* 43200 100% 43200 refresh_pattern ^http.*officecdn\.microsoft\.com\.edgesuite\.net.* 43200 100% 43200 refresh_pattern ^http.*delivery\.mp\.microsoft\.com\/filestreamingservice\/files.* 43200 100% 43200 refresh_pattern ^http.*download\.windowsupdate\.com.* 43200 100% 43200 refresh_pattern ^http.*download\.windowsupdate\.com\/.* 43200 100% 43200 refresh_pattern ^http.*au\.download\.windowsupdate\.com\/.* 43200 100% 43200 refresh_pattern ^http.*tlu\.dl\.delivery\.mp\.microsoft\.com\/filestreamingservice\/files.* 43200 100% 43200 #Website refresh_pattern -i (\.|-)(xml|js|jsp|txt|css|swf|html|htm|txt|gif|tiff)(\?.*)?$ 360 80% 1440 #end new refresh patterns refresh_pattern \.(ico|video-stats)$ 129600 100% 129600 #MISC FILE CACHING HERE refresh_pattern -i \.(3gp|7z|ace|asx|avi|bin|cab|dat|deb|rpm|divx|dvr-ms)(\?|$) 43800 100% 129600 refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso)(\?|$) 43800 100% 129600 refresh_pattern -i \.(m1v|M2V|M2P|MOD|MOV|FLV)(\?|$) 43800 100% 129600 refresh_pattern -i \.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf)(\?|$) 43800 100% 129600 refresh_pattern -i \.(mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p))(\?|$) 43800 100% 129600 refresh_pattern -i \.(og(x|v|a|g)|rar|rm|r(a|p)m|snd|vob|wav)(\?|$) 43800 100% 129600 refresh_pattern -i \.(pp(s|t)|wax|wm(a|v)|wmx|wpl|zip|cb(r|z|t))(\?|$) 43800 100% 129600 refresh_pattern -i \.(woff|txt|exe|dmg|webm)(\?|$) 43800 100% 129600 refresh_pattern -i \.(doc|pdf)(\?|$) 10080 90% 43200 # DOC | PDF refresh_pattern -i .(iso|avi|wav|mp3|mp4|mpeg|swf|flv|x-flv)$ 43200 90% 432000 refresh_pattern -i .(deb|rpm|exe|zip|tar|tgz|ram|rar|bin|ppt|doc|docx|tiff)$ 10080 90% 43200 refresh_pattern -i .index.(html|htm)$ 0 40% 10080 refresh_pattern -i .(ppt|pptx|doc|docx|docm|docb|dot|pdf|pub|ps)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(xls|xlsx|xlt|xlm|xlsm|xltm|xlw|csv|txt)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(app|bin|deb|rpm|drpm|exe|zip|zipx|tar|tgz|tbz2|tlz|iso|arj|cfs|dar|jar)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(bz|bz2|ipa|ram|rar|uxx|gz|msi|dll|lz|lzma|7z|s7z|Z|z|zz|sz)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(exe|msi)$ 0 90% 200000 refresh-ims refresh_pattern -i .(cab|psf|vidt|apk|wtex|hz|ova|ovf)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(xml|flow|asp|aspx)$ 0 90% 200000 refresh-ims refresh_pattern -i .(json)$ 0 90% 200000 refresh-ims refresh_pattern -i .(asx|mp2|mp3|mp4|mp5|wmv|flv|mts|f4v|f4|pls|midi|mid)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(mpa|m2a|mpe|avi|mov|mpg|mpeg|mpg3|mpg4|mpg5)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(m1s|mp2v|m2v|m2s|m2ts|mp2t|wmx|rm|rmvb|3pg|3gpp|omg|ogm|asf|war)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(wav|class|dat|zsci|ver|advcs)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(gif|png|ico|jpg|jpeg|jp2|webp)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(jpx|j2k|j2c|fpx|bmp|tif|tiff|bif)$ 100000 90% 20000 refresh-ims refresh_pattern -i .(pcd|pict|rif|exif|hdr|bpg|img|jif|jfif)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(woff|woff2|eps|ttf|otf|svg|svgi|svgz|ps|ps1|acsm|eot)$ 100000 90% 200000 refresh-ims refresh_pattern -i (\.|-)(mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t)(\?.*)?$ 43200 100% 432000 refresh_pattern -i (.|-)(mp3|m4a|aa?c3?|wm?av?|og(x|v|a|g)|ape|mka|au|aiff|zip|flac|m4(b|r)|m1v|m2(v|p)|mo(d|v)|arj|appx|lha|lzh|on2) 43200 100% 432000 refresh_pattern -i (.|-)(exe|bin|(n|t)ar|acv|(r|j)ar|t?gz|(g|b)z(ip)?2?|7?z(ip)?|wm[v|a]|patch|diff|mar|vpu|inc|r(a|p)m|kom|iso|sys|[ap]sf|ms[i|u|f]|dat|msi|cab|psf|dvr-ms|ace|asx|qt|xt|esd) 43200 100% 432000 refresh_pattern -i (.|-)(webm|(x-)?swf|mp(eg)?(3|4)|mpe?g(av)?|(x-)?f(l|4)v|divx?|rmvb?|mov|trp|ts|avi|m38u|wmv|wmp|m4v|mkv|asf|dv|vob|3gp?2?) 43200 100% 432000 refresh_pattern -i (.|-)(docx?|xlsx?|pptx?|rtf|xml|pdf|tiff?|txt) 43200 100% 432000 #new refresh patterns 2 refresh_pattern -i (\.|-)(ini|def|sig|upt|mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|esd|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t)(\?.*)?$ 43200 100% 432000 #end new refresh patterns 2 #new refresh patterns refresh_pattern -i (\.|-)(mp3|m4a|aa?c3?|wm?av?|og(x|v|a|g)|ape|mka|au|aiff|zip|flac|m4(b|r)|m1v|m2(v|p)|mo(d|v)|arj|appx|lha|lzh|on2)(\?.*)?$ 43200 100% 432000 refresh_pattern -i (\.|-)(exe|bin|(n|t)ar|acv|(r|j)ar|t?gz|(g|b)z(ip)?2?|7?z(ip)?|wm[v|a]|patch|diff|mar|vpu|inc|r(a|p)m|kom|iso|sys|[ap]sf|ms[i|u|f]|dat|msi|cab|psf|dvr-ms|ace|asx|qt|xt|esd)(\?.*)?$ 43200 100% 432000 refresh_pattern -i (\.|-)(ico(.*)?|pn[pg]|css|(g|t)iff?|jpe?g(2|3|4)?|psd|c(d|b)r|cad|bmp|img)(\?.*)?$ 43200 100% 432000 refresh_pattern -i (\.|-)(webm|(x-)?swf|mp(eg)?(3|4)|mpe?g(av)?|(x-)?f(l|4)v|divx?|rmvb?|mov|trp|ts|avi|m38u|wmv|wmp|m4v|mkv|asf|dv|vob|3gp?2?)(\?.*)?$ 43200 100% 432000 refresh_pattern -i (\.|-)(docx?|xlsx?|pptx?|rtf|xml|pdf|tiff?|txt)(\?.*)?$ 43200 100% 432000 refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|flv) 129600 100% 129600 refresh_pattern (Release|Packages(.gz)*)$ 0 20% 2880 # GENERIC CACHING BELOW refresh_pattern -i \.(cdn) 10800 100% 43800 refresh_pattern -i (cdn) 10800 100% 43800 refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?) 129600 100% 129600 refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.flv?) 129600 100% 129600 refresh_pattern ^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 129600 20% 129600 refresh_pattern ^.*safebrowsing.*google 129600 100% 129600 refresh_pattern ^http://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.uk) 129600 100% 129600 refresh_pattern ytimg\.com.*\.jpg 129600 100% 129600 #YOUTUBE refresh_pattern \.ytimg\? 10800 90% 10800 refresh_pattern -i (yimg|twimg).com.* 1440 100% 129600 refresh_pattern -i (ytimg|ggpht).com.* 1440 80% 129600 refresh_pattern -i (get_video?|videoplayback?|videodownload?|.mp4|.webm|.flv|((audio|video)/(webm|mp4))) 241920 100% 241920 store-stale refresh_pattern -i ^https?://..googlevideo.com/videoplayback. 10080 99% 43200 store-stale refresh_pattern -i ^https?://..googlevideo.com/videoplayback.$ 241920 100% 241920 store-stale #XBOX refresh_pattern -i lancache-microsoft 525600 100% 525600 refresh_pattern -i images-eds.xboxlive.com 525600 100% 525600 #catch all refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 range_offset_limit 0
Ref:
https://github.com/mmd123/squid-cache-dynamic_refresh-list/blob/master/pfsense%20squid%20refresh%20patternshttps://wiki.squid-cache.org/Features/StoreID#:~:text=“Store%20ID”%20is%20another%20name,algorithm%20via%20a%20helper%20program.
https://github.com/rudiservo/pfsense_storeid
http://www.squid-cache.org/Doc/config/store_id_program/
https://wiki.squid-cache.org/Features/StoreID/DB
https://forum.netgate.com/topic/106614/dynamic-cache-not-work/4
I hope that gives you some good info on ways to get better hit rates.