Improve Custom refresh pattern



  • Hi all, been trying to get the most out of squid cache and its refresh patterns.

    Please post improvements.

    Updated - Added kivimart refresh patterns

    Updated - Added deajan refresh patterns

    Updated - Added Valeriy refresh patterns

    Updated - added kikawala refresh patterns

    Updated on 3 March 2017

    
    #new refresh patterns 3
    acl Windows_Update dstdomain windowsupdate.microsoft.com
    acl Windows_Update dstdomain .update.microsoft.com
    acl Windows_Update dstdomain download.windowsupdate.com
    acl Windows_Update dstdomain www.download.windowsupdate.com
    acl Windows_Update dstdomain au.download.windowsupdate.com
    acl Windows_Update dstdomain bg.v4.pr.dl.ws.microsoft.com
    
    #new refresh patterns 2
    refresh_pattern -i (\.|-)(ini|def|sig|upt|mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|esd|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    #end new refresh patterns 2
    
    #new refresh patterns
    refresh_pattern -i (\.|-)(mp3|m4a|aa?c3?|wm?av?|og(x|v|a|g)|ape|mka|au|aiff|zip|flac|m4(b|r)|m1v|m2(v|p)|mo(d|v)|arj|appx|lha|lzh|on2)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    refresh_pattern -i (\.|-)(exe|bin|(n|t)ar|acv|(r|j)ar|t?gz|(g|b)z(ip)?2?|7?z(ip)?|wm[v|a]|patch|diff|mar|vpu|inc|r(a|p)m|kom|iso|sys|[ap]sf|ms[i|u|f]|dat|msi|cab|psf|dvr-ms|ace|asx|qt|xt|esd)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    refresh_pattern -i (\.|-)(ico(.*)?|pn[pg]|css|(g|t)iff?|jpe?g(2|3|4)?|psd|c(d|b)r|cad|bmp|img)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    refresh_pattern -i (\.|-)(webm|(x-)?swf|mp(eg)?(3|4)|mpe?g(av)?|(x-)?f(l|4)v|divx?|rmvb?|mov|trp|ts|avi|m38u|wmv|wmp|m4v|mkv|asf|dv|vob|3gp?2?)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    refresh_pattern -i (\.|-)(docx?|xlsx?|pptx?|rtf|xml|pdf|tiff?|txt)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    #Website
    refresh_pattern -i (\.|-)(xml|js|jsp|txt|css)(\?.*)?$ 360 40% 1440 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    refresh_pattern -i .index.(html|htm)$ 0 40% 1440
    refresh_pattern . 30 25% 1440
    #end new refresh patterns
    
    refresh_pattern -i \.(3gp|7z|ace|asx|avi|bin|cab|dat|deb|rpm|divx|dvr-ms)      129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|(x-|)flv) 129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    refresh_pattern -i \.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js)         129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    refresh_pattern -i \.(mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p))                   129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    refresh_pattern -i \.(og(x|v|a|g)|rar|rm|r(a|p)m|snd|vob|wav)                  129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    refresh_pattern -i \.(pp(s|t)|wax|wm(a|v)|wmx|wpl|zip|cb(r|z|t))               129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    
    refresh_pattern ^gopher:  1440  0%  1440
    refresh_pattern ^ftp:    10080 95% 43200 override-lastmod reload-into-ims
    
    refresh_pattern -i \.(doc|pdf)$           100080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private reload-into-ims
    refresh_pattern -i \.(html|htm)$          1440   40% 40320 ignore-no-cache ignore-no-store ignore-private override-expire reload-into-ims
    refresh_pattern (Release|Packages(.gz)*)$    0   20%  2880
    refresh_pattern .                          180   95% 43200 override-lastmod reload-into-ims
    
    # 1 year = 525600 mins, 1 month = 43800 mins
    refresh_pattern -i (/cgi-bin/|\?)         0      0%      0
    refresh_pattern \.(ico|video-stats)$ 129600 100% 129600 override-expire ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth override-lastmod ignore-must-revalidate
    
    refresh_pattern imeem.*\.flv$                           0     0%         0 override-lastmod override-expire
    refresh_pattern \.rapidshare.*\/[0-9]*\/.*\/[^\/]* 161280    90%    161280 ignore-reload
    
    refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?) 129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims
    refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.flv?) 129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims
    refresh_pattern ^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 129600 20% 129600 ignore-no-cache ignore-no-store ignore-private override-expire ignore-reload ignore-auth ignore-must-revalidate
    
    refresh_pattern ^.*safebrowsing.*google                                  129600 100% 129600 override-expire ignore-reload ignore-no-cache ignore-private ignore-auth ignore-must-revalidate
    refresh_pattern ^http://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.uk)     129600 100% 129600 override-expire ignore-reload ignore-private
    refresh_pattern ytimg\.com.*\.jpg                                        129600 100% 129600 override-expire ignore-reload
    refresh_pattern images\.friendster\.com.*\.(png|gif)                     129600 100% 129600 override-expire ignore-reload
    refresh_pattern garena\.com                                              129600 100% 129600 override-expire reload-into-ims
    refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png)           129600 100% 129600 override-expire ignore-reload
    refresh_pattern vid\.akm\.dailymotion\.com.*\.on2\?                      129600 100% 129600 ignore-no-cache override-expire override-lastmod
    refresh_pattern mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 129600 100% 129600 reload-into-ims override-expire ignore-private
    refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\.                      129600 100% 129600 reload-into-ims ignore-no-cache ignore-no-store ignore-reload override-expire
    refresh_pattern ^http:\/\/www.onemanga.com.*\/                           129600 100% 129600 reload-into-ims ignore-no-cache ignore-no-store ignore-reload override-expire
    
    # ANTI VIRUS
    refresh_pattern guru.avg.com/.*\.(bin)                              43200 100% 43200  ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    refresh_pattern (avgate|avira).*(idx|gz)$                           43200 100% 43200  ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    refresh_pattern kaspersky.*\.avc$                                   43200 100% 43200  ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    refresh_pattern kaspersky                                           43200 100% 43200  ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    refresh_pattern update.nai.com/.*\.(gem|zip|mcs)                    43200 100% 43200  ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    refresh_pattern ^http:\/\/liveupdate.symantecliveupdate.com.*\(zip) 43200 100% 43200  ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    refresh_pattern -i symantecliveupdate.com/.*\.(zip|exe)             43200 100% 43200 reload-into-ims
    refresh_pattern -i avast.com/.*\.(vpu|vpaa) 4320 100% 43200 reload-into-ims
    refresh_pattern -i avira-update.com/.*\.* 720 100% 10800 reload-into-ims
    
    #windows update NEW UPDATE 0.04
    refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    refresh_pattern update.microsoft.com/.*\.(cab|exe)                  43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf) 4320 100% 43200 reload-into-ims 
    refresh_pattern update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    
    refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims
    refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims
    refresh_pattern -i .*windowsupdate.com/.*\.(cab|exe)                     259200 100% 259200 ignore-no-store ignore-reload reload-into-ims
    refresh_pattern -i .*update.microsoft.com/.*\.(cab|exe|dll|msi|psf)                  259200 100% 259200 ignore-no-store ignore-reload reload-into-ims
    refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims
    refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims
    
    #apple update
    refresh_pattern -i (download|adcdownload).apple.com/.*\.(pkg|dmg) 4320 100% 43200 reload-into-ims
    refresh_pattern -i appldnld\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate
    refresh_pattern -i phobos\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate
    refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate
    
    #images facebook
    refresh_pattern ((facebook.com)|(85.131.151.39)).*\.(jpg|png|gif) 129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
    refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|swf|mp3)           129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
    refresh_pattern static\.ak\.fbcdn\.net*\.(jpg|gif|png)            129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
    refresh_pattern ^http:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png)  129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
    
    #banner IIX
    refresh_pattern ^http:\/\/openx.*\.(jp(e?g|e|2)|gif|pn[pg]|swf|ico|css|tiff?) 129600 100% 129600 reload-into-ims ignore-reload override-expire ignore-no-cache ignore-no-store
    refresh_pattern ^http:\/\/ads(1|2|3).kompas.com.*\/                           43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-cache ignore-no-store
    refresh_pattern ^http:\/\/img.ads.kompas.com.*\/                              43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-cache ignore-no-store
    refresh_pattern .kompasimages.com.*\.(jpg|gif|png|swf)                        43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-cache ignore-no-store
    refresh_pattern ^http:\/\/openx.kompas.com.*\/                                43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-cache ignore-no-store
    refresh_pattern kaskus.\us.*\.(jp(e?g|e|2)|gif|png|swf)                       43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-cache ignore-no-store
    refresh_pattern ^http:\/\/img.kaskus.us.*\.(jpg|gif|png|swf)                  43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-cache ignore-no-store
    
    #IIX DOWNLOAD
    refresh_pattern ^http:\/\/\.www[0-9][0-9]\.indowebster\.com\/(.*)(mp3|rar|zip|flv|wmv|3gp|mp(4|3)|exe|msi|zip) 43200 100% 129600 reload-into-ims  ignore-reload override-expire ignore-no-cache ignore-no-store  ignore-auth
    
    refresh_pattern -i ^http://(khm?)([^/]*?)\.google\.(de|com)     129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    
    refresh_pattern -i ^http://ecn\.t\d\.tiles\.virtualearth\.net/tiles/\w*\.jpeg     129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    
    


  • This is my custom

    All Files

    ######################
    refresh_pattern -i (.|-)(mp3|m4a|aa?c3?|wm?av?|og(x|v|a|g)|ape|mka|au|aiff|zip|flac|m4(b|r)|m1v|m2(v|p)|mo(d|v)|arj|appx|lha|lzh|on2)(?.)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    refresh_pattern -i (.|-)(exe|bin|(n|t)ar|acv|(r|j)ar|t?gz|(g|b)z(ip)?2?|7?z(ip)?|wm[v|a]|patch|diff|mar|vpu|inc|r(a|p)m|kom|iso|sys|[ap]sf|ms[i|u|f]|dat|msi|cab|psf|dvr-ms|ace|asx|qt|xt|esd)(?.
    )?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    refresh_pattern -i (.|-)(ico(.)?|pn[pg]|css|(g|t)iff?|jpe?g(2|3|4)?|psd|c(d|b)r|cad|bmp|img)(?.)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    refresh_pattern -i (.|-)(webm|(x-)?swf|mp(eg)?(3|4)|mpe?g(av)?|(x-)?f(l|4)v|divx?|rmvb?|mov|trp|ts|avi|m38u|wmv|wmp|m4v|mkv|asf|dv|vob|3gp?2?)(?.)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    refresh_pattern -i (.|-)(docx?|xlsx?|pptx?|rtf|xml|pdf|tiff?|txt)(?.
    )?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    #Website
    refresh_pattern -i (.|-)(xml|js|jsp|txt|css)(?.*)?$ 360 40% 1440 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    ###################
    refresh_pattern -i .index.(html|htm)$ 0 40% 1440
    refresh_pattern . 30 25% 1440



  • updated, thanks



  • added

    
    #new refresh patterns 2
    refresh_pattern -i (\.|-)(mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    #end new refresh patterns 2
    
    


  • added deajan refresh patterns

    
    refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern -i avg.com/.*\.(bin) 4320 100% 43200 reload-into-ims
    refresh_pattern -i symantecliveupdate.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims
    refresh_pattern -i avast.com/.*\.(vpu|vpaa) 4320 100% 43200 reload-into-ims
    refresh_pattern -i avira-update.com/.*\.* 720 100% 10800 reload-into-ims
    refresh_pattern -i (download|adcdownload).apple.com/.*\.(pkg|dmg) 4320 100% 43200 reload-into-ims
    
    


  • Hi guys, where I can put this refresh_patterns in pfsense 2.3? sorry im new here. Thanks!



  • Services -> Squid Proxy Server -> Local Cache Tab - > Look at bottom of page under Custom refresh_patterns



  • Windows 10 update use also .esd files



  • thanks! good to have a decent pattern to start with.



  • FYI, Squid -k parse is complaining that 'ignore-no-cache' is obsolete.. So it has no effect.

    Running 2.3.3-DEVELOPMENT (amd64) built on Sat Feb 04 16:18:22 CST 2017 with Squid 3.5.23

    Processing: refresh_pattern -i (\.|-)(mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern -i (\.|-)(mp3|m4a|aa?c3?|wm?av?|og(x|v|a|g)|ape|mka|au|aiff|zip|flac|m4(b|r)|m1v|m2(v|p)|mo(d|v)|arj|appx|lha|lzh|on2)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern -i (\.|-)(exe|bin|(n|t)ar|acv|(r|j)ar|t?gz|(g|b)z(ip)?2?|7?z(ip)?|wm[v|a]|patch|diff|mar|vpu|inc|r(a|p)m|kom|iso|sys|[ap]sf|ms[i|u|f]|dat|msi|cab|psf|dvr-ms|ace|asx|qt|xt|esd)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern -i (\.|-)(ico(.*)?|pn[pg]|css|(g|t)iff?|jpe?g(2|3|4)?|psd|c(d|b)r|cad|bmp|img)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern -i (\.|-)(webm|(x-)?swf|mp(eg)?(3|4)|mpe?g(av)?|(x-)?f(l|4)v|divx?|rmvb?|mov|trp|ts|avi|m38u|wmv|wmp|m4v|mkv|asf|dv|vob|3gp?2?)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern -i (\.|-)(docx?|xlsx?|pptx?|rtf|xml|pdf|tiff?|txt)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern -i (\.|-)(xml|js|jsp|txt|css)(\?.*)?$ 360 40% 1440 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern -i .index.(html|htm)$ 0 40% 1440
    Processing: refresh_pattern . 30 25% 1440
    Processing: refresh_pattern -i \.(3gp|7z|ace|asx|avi|bin|cab|dat|deb|rpm|divx|dvr-ms)      129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|(x-|)flv) 129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    /usr/local/etc/squid/squid.conf line 52: refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|(x-|)flv) 129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    refreshAddToList: Invalid regular expression '\.(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|(x-|)flv)': empty (sub)expression
    Processing: refresh_pattern -i \.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js)         129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern -i \.(mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p))                   129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern -i \.(og(x|v|a|g)|rar|rm|r(a|p)m|snd|vob|wav)                  129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern -i \.(pp(s|t)|wax|wm(a|v)|wmx|wpl|zip|cb(r|z|t))               129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern ^gopher:  1440  0%  1440
    Processing: refresh_pattern ^ftp:    10080 95% 43200 override-lastmod reload-into-ims
    Processing: refresh_pattern -i \.(doc|pdf)$           100080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private reload-into-ims
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern -i \.(html|htm)$          1440   40% 40320 ignore-no-cache ignore-no-store ignore-private override-expire reload-into-ims
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern (Release|Packages(.gz)*)$    0   20%  2880
    Processing: refresh_pattern .                          180   95% 43200 override-lastmod reload-into-ims
    Processing: refresh_pattern -i (/cgi-bin/|\?)         0      0%      0
    Processing: refresh_pattern \.(ico|video-stats)$ 129600 100% 129600 override-expire ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth override-lastmod ignore-must-revalidate
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern imeem.*\.flv$                           0     0%         0 override-lastmod override-expire
    Processing: refresh_pattern \.rapidshare.*\/[0-9]*\/.*\/[^\/]* 161280    90%    161280 ignore-reload
    Processing: refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?) 129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.flv?) 129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern ^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 129600 20% 129600 ignore-no-cache ignore-no-store ignore-private override-expire ignore-reload ignore-auth ignore-must-revalidate
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern ^.*safebrowsing.*google                                  129600 100% 129600 override-expire ignore-reload ignore-no-cache ignore-private ignore-auth ignore-must-revalidate
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern ^http://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.uk)     129600 100% 129600 override-expire ignore-reload ignore-private
    Processing: refresh_pattern ytimg\.com.*\.jpg                                        129600 100% 129600 override-expire ignore-reload
    Processing: refresh_pattern images\.friendster\.com.*\.(png|gif)                     129600 100% 129600 override-expire ignore-reload
    Processing: refresh_pattern garena\.com                                              129600 100% 129600 override-expire reload-into-ims
    Processing: refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png)           129600 100% 129600 override-expire ignore-reload
    Processing: refresh_pattern vid\.akm\.dailymotion\.com.*\.on2\?                      129600 100% 129600 ignore-no-cache override-expire override-lastmod
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 129600 100% 129600 reload-into-ims override-expire ignore-private
    Processing: refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\.                      129600 100% 129600 reload-into-ims ignore-no-cache ignore-no-store ignore-reload override-expire
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern ^http:\/\/www.onemanga.com.*\/                           129600 100% 129600 reload-into-ims ignore-no-cache ignore-no-store ignore-reload override-expire
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern guru.avg.com/.*\.(bin)                              43200 100% 43200  ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern (avgate|avira).*(idx|gz)$                           43200 100% 43200  ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern kaspersky.*\.avc$                                   43200 100% 43200  ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern kaspersky                                           43200 100% 43200  ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern update.nai.com/.*\.(gem|zip|mcs)                    43200 100% 43200  ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern ^http:\/\/liveupdate.symantecliveupdate.com.*\(zip) 43200 100% 43200  ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern -i symantecliveupdate.com/.*\.(zip|exe)             43200 100% 43200 reload-into-ims
    Processing: refresh_pattern -i avast.com/.*\.(vpu|vpaa) 4320 100% 43200 reload-into-ims
    Processing: refresh_pattern -i avira-update.com/.*\.* 720 100% 10800 reload-into-ims
    Processing: refresh_pattern windowsupdate.com/.*\.(cab|exe)                     43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern update.microsoft.com/.*\.(cab|exe)                  43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern download.microsoft.com/.*\.(cab|exe)                43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    Processing: refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    Processing: refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    Processing: refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf) 4320 100% 43200 reload-into-ims
    Processing: refresh_pattern update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    Processing: refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    Processing: refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    Processing: refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    Processing: refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims
    Processing: refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims
    Processing: refresh_pattern -i (download|adcdownload).apple.com/.*\.(pkg|dmg) 4320 100% 43200 reload-into-ims
    Processing: refresh_pattern ((facebook.com)|(85.131.151.39)).*\.(jpg|png|gif) 129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|swf|mp3)           129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern static\.ak\.fbcdn\.net*\.(jpg|gif|png)            129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern ^http:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png)  129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern ^http:\/\/openx.*\.(jp(e?g|e|2)|gif|pn[pg]|swf|ico|css|tiff?) 129600 100% 129600 reload-into-ims ignore-reload override-expire ignore-no-cache ignore-no-store
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern ^http:\/\/ads(1|2|3).kompas.com.*\/                           43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-cache ignore-no-store
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern ^http:\/\/img.ads.kompas.com.*\/                              43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-cache ignore-no-store
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern .kompasimages.com.*\.(jpg|gif|png|swf)                        43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-cache ignore-no-store
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern ^http:\/\/openx.kompas.com.*\/                                43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-cache ignore-no-store
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern kaskus.\us.*\.(jp(e?g|e|2)|gif|png|swf)                       43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-cache ignore-no-store
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern ^http:\/\/img.kaskus.us.*\.(jpg|gif|png|swf)                  43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-cache ignore-no-store
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern ^http:\/\/\.www[0-9][0-9]\.indowebster\.com\/(.*)(mp3|rar|zip|flv|wmv|3gp|mp(4|3)|exe|msi|zip) 43200 100% 129600 reload-into-ims  ignore-reload override-expire ignore-no-cache ignore-no-store  ignore-auth
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    Processing: refresh_pattern -i ^http://(khm?)([^/]*?)\.google\.(de|com)     129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    /usr/local/etc/squid/squid.conf line 136: refresh_pattern -i ^http://(khm?)([^/]*?)\.google\.(de|com)     129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    refreshAddToList: Invalid regular expression '^http://(khm?)([^/]*?)\.google\.(de|com)': repetition-operator operand invalid
    Processing: refresh_pattern -i ^http://ecn\.t\d\.tiles\.virtualearth\.net/tiles/\w*\.jpeg     129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    UPGRADE: refresh_pattern option 'ignore-no-cache' is obsolete. Remove it.
    
    


  • Two invalid expressions.
    Somebody can fix them?

    /usr/local/etc/squid/squid.conf line 52: refresh_pattern -i .(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|(x-|)flv) 129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    refreshAddToList: Invalid regular expression '.(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|(x-|)flv)': empty (sub)expression

    _  kid1| /usr/local/etc/squid/squid.conf line 136: refresh_pattern -i ^http://(khm?)([^/]?).google.(de|com)    129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    refreshAddToList: Invalid regular expression '^http://(khm?)([^/]
    ?).google.(de|com)': repetition-operator operand invalid_



  • For the first one, I think the problem is here:

    (x-|)

    There is nothing on the other side of the OR.  I don't know what file type would be handled by the extension x-.  To fix it, just remove the entire (x-|).

    For the second one, I don't know for sure, but it seems like a lot of regex just to handle .google..


  • Banned

    I would strongly urge everyone to ditch the pointless idea of trying to cache dynamic sites. It doesn't work in like 99% of cases, plus breaks Squid. And please ditch ALL those antivirus patterns. None of them works. The AVs are streaming updates with new sigs to clients every couple of minutes these days. There's nothing to cache.



  • So why use squid? Because most sites are changing to https.

    Various web content are in videos, overloading the network.



  • So why use squid?

    I use it as a filtering platform for squidguard.

    Various web content are in videos

    Caching videos is pointless unless everyone is watching the same ones.

    I have seen commercial products that take squid and heavily modify it with all kinds of custom stuff, and they claim they can cache almost everything.



  • I use squid for caching, it saves bandwidth for Microsoft Updates and Apple updates + SquidGuard as Ad+Malware+porn filter



  • it saves bandwidth for Microsoft Updates

    I have yet to see a working config with squid and WU.  Most people give up and use WSUS.



  • I cannt use WSUS, PCs are not under my control.



  • If you have a working squid config that caches WU for Win7-10 clients, I'd love to see it.



  • These refresh pattern made a big difference on my WI-fi network.
    Since I started using it my clients have been very happy with the speed.
    Even though, now it seems like they need some update, I still appreciate this well done job.
    Thanks



  • You can use squidlite to check if the refresh pattens are improving the hit rate. I have found that the comodo antivirus updates were caching ( cannot retest at the moment) and so did window updates. The best thing is to test if the hit ratio is improving.





  • @KOM

    Here are changes made to squid.conf (or through GUI on pfsense in corresponding custom fields)
    that allows caching of Windows Update for me:

    
    acl Windows_Update dstdomain windowsupdate.microsoft.com
    acl Windows_Update dstdomain .update.microsoft.com
    acl Windows_Update dstdomain download.windowsupdate.com
    acl Windows_Update dstdomain www.download.windowsupdate.com
    acl Windows_Update dstdomain au.download.windowsupdate.com
    acl Windows_Update dstdomain bg.v4.pr.dl.ws.microsoft.com
    range_offset_limit 200 MB Windows_Update
    quick_abort_min 1000 KB
    quick_abort_max 5000 KB
    quick_abort_pct 80
    request_body_max_size 0 KB
    
    refresh_pattern -i .*windowsupdate.com/.*\.(cab|exe)                     259200 100% 259200 ignore-no-store ignore-reload reload-into-ims
    refresh_pattern -i .*update.microsoft.com/.*\.(cab|exe|dll|msi|psf)                  259200 100% 259200 ignore-no-store ignore-reload reload-into-ims
    refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims
    refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims
    
    cache_mem 15000 MB 
    # Should be less than 50% of avail. RAM
    
    maximum_object_size_in_memory 5000000 KB 
    #should be less than cache_mem 
    
    memory_replacement_policy heap LFUDA
    cache_replacement_policy heap LFUDA
    
    minimum_object_size 1000 KB
    #Do not store very small object on disk
    
    maximum_object_size 5000 MB 
    #should be less or equal to maximum_object_size_in_memory
    
    

    Here is output for today access.log of squid (number of HITS):
    cat  /var/squid/logs/access.log | grep HIT | grep windowsupdate -c
    2860
    cat /var/squid/logs/access.log | grep HIT | grep microsoft.com -c
    19128

    Here is output for today access.log of squid (number of MISSes):
    cat /var/squid/logs/access.log | grep MISS | grep windowsupdate -c
    61834
    cat /var/squid/logs/access.log | grep MISS | grep microsoft.com -c
    55567

    I had to rebuild my disk cache 2 days ago, so I assume not everything is cached yet. Or some of my settings needs improvement…

    It might not be ideal setup.
    If you have some comments, improvements, please let me know.

    Regards,

    Valeriy



  • How did your dynamic cache get now?



  • @saluto:

    How did your dynamic cache get now?

    It is good.
    This is result for yesterday:

    
    [2.3.3-DEVELOPMENT]/var/squid/logs: cat /var/squid/logs/access.log.2 | grep MISS | grep microsoft.com -c
    96270
    [2.3.3-DEVELOPMENT]/var/squid/logs: cat /var/squid/logs/access.log.2 | grep HIT | grep microsoft.com -c
    21177
    (18% HITS vs. 82% MISS)
    [2.3.3-DEVELOPMENT]/var/squid/logs: cat /var/squid/logs/access.log.2 | grep HIT | grep windowsupdate -c
    8352
    [2.3.3-DEVELOPMENT]/var/squid/logs: cat /var/squid/logs/access.log.2 | grep MISS | grep windowsupdate -c
    34794
    (19% HITS vs. 81% MISS)
    
    


  • :o

    81% miss rate seems awfully high for something that's supposedly working good.



  • Should look at the hit/miss ratio of file sizes. If you miss 99% of 1kB files it does not matter however it can through your ratio out.





  • KOM: "19% hit rate seems awfully high for something that's supposedly not working at all." (;

    Even 19 % hit rate is a benefit if you have to work with a slow/congested line. I think hit rate can be optimized by setting up a machine that will download updates before the standard time frame windows is using to check for/download updates, so that updates are already cached when other machines check for them (I'm going to implement this on one of the systems I'm managing as soon as I find the time). Also, like aGeekHere said, you'd have to look at the byte ratio to be able to see how effective this form of caching actually is. Not sure how this could be done, maybe with a tool like Calamaris and a pre-filtered Squid log file?

    Bottom line: caching Windows updates does indeed work, and is actually useful for (at least) some people.



  • Bottom line: caching Windows updates does indeed work

    I still have yet to see any evidence to back this claim.  A 19% hit rate for WU is crap.  And no, there aren't a zillion little 1K files skewing the ratio.

    Here is a proper test:

    take two Windows 7 clients that you installed fresh and put them both behind the proxy, leaving only 1 powered on.
    Install all updates until there are none left
    Power on the second unit and then do all the same updates
    Once complete, on pfSense run:

    squidclient -h LAN_IP -p 3128 mgr:info
    

    and post the output.  That will give a lot more detail about the recent session.



  • updated, thanks to Valeriy

    if there are any mistakes in the refresh pattern let me know



  • KOM was right, in fact it was not working properly. Also, the results of bare counting of MISSes/HITs from not fully populated cache was not good measurement. Right now, any windowsupdate downloaded file (even partial) is fetched from cache, verified manually and monitored in access.log.

    Few days ago I have made some improvements to utilize store_rewrite module.
    They were tested, as suggested above by KOM, successfully.

    Indeed, some of WU were not cached properly since they were received from different CDN hosts.

    Config files are below (squid.conf is only part that is related to this issue)

    Apple and Symantec updates are cached well, too.

    Steam/Xbox (cs.steampowered.com / llnwd.net respective CDN) are not confirmed yet.

    The rest of refresh patterns and URL rewrites are still in experimental mode. Needs a lot of testing and research in order to make it done properly.

    Attached is screenshot from Lightsquid, you can clearly see changes after 3rd of March.

    P.S. Questions:

    • not quite sure if I need to list all possible domains under acl dstdomain directive, or just first level domain will be good enough?

    Comments:

    • I have total 1.3TB of swap available and 32GB on server running Pfsense+Squid+SquidGuard+Snort+PfBlocker. Configuration is optimized for ~2000 hosts on LAN.

    Current byte/hit rate is 20-45%

    /usr/local/etc/squid/squid.conf

    
    .......
    # No range offset limit for windowsupdate sites
    acl Windows_Update dstdomain windowsupdate.microsoft.com
    acl Windows_Update dstdomain update.microsoft.com
    acl Windows_Update dstdomain download.windowsupdate.com
    acl Windows_Update dstdomain windowsupdate.com
    acl Windows_Update dstdomain au.download.windowsupdate.com
    acl Windows_Update dstdomain bg.v4.pr.dl.ws.microsoft.com
    acl Windows_Update dstdomain wsus.ds.download.windowsupdate.com
    acl Windows_Update dstdomain au.b1.download.windowsupdate.com
    range_offset_limit -1  Windows_Update
    
    # Storeid_rewrite configuration
    
    acl updatesites dstdom_regex "/usr/local/etc/squid/updatesites.txt"
    
    store_id_access allow updatesites
    store_id_access deny all
    store_id_program /usr/local/libexec/squid/storeid_file_rewrite /usr/local/etc/squid/storeid_rewrite.conf
    store_id_children 200 startup=60 idle=1 concurrency=0
    
    # Refresh patterns
    # 525600 min is one year
    
    # Gaming CDN
    
    refresh_pattern -i \.llnwd.net 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
    
    refresh_pattern -i \.cs.steampowered.com 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
    
    #windows update
    
    refresh_pattern -i \.update.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
    
    refresh_pattern -i \.windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
    
    refresh_pattern -i \.download.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
    
    refresh_pattern -i \.ws.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
    
    # Symantec Updates
    # Only reload-into-ims works properly, other flags break update process 
    
    refresh_pattern -i \.symantecliveupdate.com/.*\.(zip|exe|z7)          525600 100% 525600 reload-into-ims
    
    

    /usr/local/etc/squid/updatesites.txt

    
    \.adobe.com
    \.java.com
    \.sun.com
    \.oracle.com
    \.apple.com
    \.microsoft.com
    \.windowsupdate.com
    \.ubuntu.com
    \.steampowered.com
    \.llnwd.net
    \.symantecliveupdate.com
    
    

    /usr/local/etc/squid/storeid_rewrite.conf

    
    ^http:\/\/.+?\.microsoft\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx)     http://wupdate.squid.local/$1
    ^http:\/\/.+?\.windowsupdate\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx) http://wupdate.squid.local/$1
    ^http:\/\/.+?\.cs\.steampowered\.com\/(.*)      http://steamupdates.squid.internal/$1
    ^http:\/\/.+?\.apple\.com\/(.*) http://appupdates.apple.squid.internal/$1
    ^http:\/\/.+?\.llnwd\.net\/(.*)                 http://llnwd.net.squid.internal/$1
    
    

    Regards,

    Valeriy



  • Just a note: manually modifying squid.conf is not good idea, since it can and will be rewritten by PfSense later. With next post I ll try to summarize all the changes and how to incorporate them using GUI of pfSense.



  • This is interesting, I wish I had the time to test this thoroughly. Will do that as soon as I find the time.

    FWIW, I had massive problems with the "range_offset_limit -1" directive and Windows 10 updates, with "psf" added to the "refresh_pattern". Update download was extremely slow and used up all available bandwidth after a while. It looked like Win10 was aborting and restarting the update download every few seconds, while Squid was still downloading from previous requests. Commented out the "range_offset_limit -1", and updates were downloading "normally" again. The "-1" value isn't documented in the current Squid docs, but it's documented for Squid 2.7. I wonder if "range_offset_limit -1" is still valid/needed with recent Squid versions.



  • It worked very well.



  • No caches of video, can you help?



  • Forget about video caching…
    Most video services push content via HTTPS, which we cannt cache.

    If you really need to have some video supplied to your users, then just download it and make is available via PLEX service, for instance or just regular network file system (SMB, NFS, etc.)



  • @Valeriy:

    Forget about video caching…
    Most video services push content via HTTPS, which we cannt cache.

    If you really need to have some video supplied to your users, then just download it and make is available via PLEX service, for instance or just regular network file system (SMB, NFS, etc.)

    Unless…You do MITM inspection.



  • @pfsensation:

    @Valeriy:

    Forget about video caching…
    Most video services push content via HTTPS, which we cannt cache.

    If you really need to have some video supplied to your users, then just download it and make is available via PLEX service, for instance or just regular network file system (SMB, NFS, etc.)

    Unless…You do MITM inspection.

    I do not know the MITM inspection.



  • @saluto:

    @pfsensation:

    @Valeriy:

    Forget about video caching…
    Most video services push content via HTTPS, which we cannt cache.

    If you really need to have some video supplied to your users, then just download it and make is available via PLEX service, for instance or just regular network file system (SMB, NFS, etc.)

    Unless…You do MITM inspection.

    I do not know the MITM inspection.

    I'm currently having issues with it after the latest Squid update, unsure as to why. However, you can do MITM inspection by enabling SSL filtering and deploying out a CA certificate to your clients.