Any Recommendations this Custom refresh_pattern?

JonathanLee · May 12, 2022, 6:08 PM

#WINDOWS
refresh_pattern -i windowsupdate.com/..(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
refresh_pattern -i microsoft.com/..(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
refresh_pattern -i windows.com/..(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
refresh_pattern -i microsoft.com.akadns.net/..(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
refresh_pattern -i deploy.akamaitechnologies.com/..(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
refresh_pattern -i msedge.b.tlu.dl.delivery.mp.microsoft.com/..(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims

#FACEBOOK
refresh_pattern ^http.facebook.com. 720 100% 4320

#REGULAR FACEBOOK STUFF

#FACEBOOK IMAGES
refresh_pattern -i pixel.facebook.com..(jpg|png|gif|ico|css|js) 241920 80% 241920
refresh_pattern -i .akamaihd.net..(jpg|png|gif|ico|css|js) 241920 80% 241920
refresh_pattern -i ((facebook.com)|(85.131.151.39)).(jpg|png|gif) 241920 99% 241920 store-stale
refresh_pattern static.(xx|ak).fbcdn.net.(jpg|gif|png) 241920 99% 241920
refresh_pattern ^http.profile.ak.fbcdn.net.(jpg|gif|png) 241920 99% 241920

#YAHOO
refresh_pattern ^http.mail.yahoo.com. 720 100% 4320 # YAHOO MAIL
refresh_pattern ^http.yahoo. 720 100% 4320 # YAHOO ITSELF
refresh_pattern ^http.yimg. 720 100% 4320 # YAHOO IMAGES

#apple update
refresh_pattern -i apple.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip|dist)$ 0 80% 43200 refresh-ims
#refresh_pattern -i (download|adcdownload).apple.com/.*.(pkg|dmg) 4320 100% 43200
refresh_pattern -i appldnld.apple.com 129600 100% 129600
refresh_pattern -i phobos.apple.com 129600 100% 129600
refresh_pattern -i iosapps.itunes.apple.com 129600 100% 129600

#GOOGLE STUFF
refresh_pattern ^http.gmail. 720 100% 4320 # GMAIL
refresh_pattern ^http.google. 720 100% 4320 # GOOGLE

#AntiVirus
refresh_pattern ^http.kaspersky. 43200 100% 43200
refresh_pattern ^http.geo.kaspersky.com. 43200 100% 43200

#Office/Windows
refresh_pattern ^http.officecdn.microsoft.com. 43200 100% 43200
refresh_pattern ^http.officecdn.microsoft.com.edgesuite.net. 43200 100% 43200
refresh_pattern ^http.delivery.mp.microsoft.com/filestreamingservice/files. 43200 100% 43200
refresh_pattern ^http.download.windowsupdate.com. 43200 100% 43200
refresh_pattern ^http.download.windowsupdate.com/. 43200 100% 43200
refresh_pattern ^http.au.download.windowsupdate.com/. 43200 100% 43200
refresh_pattern ^http.tlu.dl.delivery.mp.microsoft.com/filestreamingservice/files. 43200 100% 43200

#Website
refresh_pattern -i (.|-)(xml|js|jsp|txt|css)(?.*)?$ 360 40% 1440

#MISC FILE CACHING HERE
refresh_pattern -i .(3gp|7z|ace|asx|avi|bin|cab|dat|deb|rpm|divx|dvr-ms)(?|$) 43800 100% 129600 # 3GP | 7Z | ACE | ASX | AVI | BIN | CAB | DAT | DEB | RPM | DIVX | DVR-MS
refresh_pattern -i .(rar|jar|gz|tgz|tar|bz2|iso)(?|$) 43800 100% 129600 # RAR | JAR | GZ | TGZ | TAR | BZ2 | ISO
refresh_pattern -i .(m1v|M2V|M2P|MOD|MOV|FLV)(?|$) 43800 100% 129600 # M1V | M2V | M2P | MOD | MOV | FLV
refresh_pattern -i .(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js)(?|$) 43800 100% 129600 # JPG | JPEG | JPE | JP2 | GIF | PNG | BMP | TIFF | ICO | SWF
refresh_pattern -i .(mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p))(?|$) 43800 100% 129600 # MPEG STYLE CACHING, VIDEO AND MUSIC | MPG MPEG | MP1-2-3-4 | MK-A/V | MS-I-U-P
refresh_pattern -i .(og(x|v|a|g)|rar|rm|r(a|p)m|snd|vob|wav)(?|$) 43800 100% 129600 # OGX | OGV | OGA | OGG | RAR | RM | RAM | RPM | SND | VOB | WAV
refresh_pattern -i .(pp(s|t)|wax|wm(a|v)|wmx|wpl|zip|cb(r|z|t))(?|$) 43800 100% 129600 # PPS | PPT | WAX | WMA | WMV | WMX | WPL | ZIP | CBR | CBZ | CBT
refresh_pattern -i .(woff|txt|exe|dmg|webm)(?|$) 43800 100% 129600 # WOFF | TXT | EXE | DMG | WEBM
refresh_pattern -i .(css)(?|$) 10080 60% 43800 # CSS
refresh_pattern -i .(js)(?|$) 10080 60% 10080 # JS
refresh_pattern -i .(doc|pdf)(?|$) 10080 90% 43200 # DOC | PDF
refresh_pattern -i .(html|htm)(?|$) 1440 60% 10080 # HTML | HTM
refresh_pattern -i .(iso|avi|wav|mp3|mp4|mpeg|swf|flv|x-flv)$ 43200 90% 432000 #THIS SHOULD BE DOCUMENTED/DONE ABOVE, BUT LEAVING HERE JUST IN CASE
refresh_pattern -i .(deb|rpm|exe|zip|tar|tgz|ram|rar|bin|ppt|doc|docx|tiff)$ 10080 90% 43200 # DEB | RPM | EXE | ZIP | TAR | TGZ | RAM | RAR | BIN | PPT | DOC | TIFF | DOCX
refresh_pattern -i .(html|htm|css|js)$ 1440 40% 40320
refresh_pattern -i .index.(html|htm)$ 0 40% 10080
refresh_pattern -i .(ppt|pptx|doc|docx|docm|docb|dot|pdf|pub|ps)$ 100000 90% 200000 refresh-ims
refresh_pattern -i .(xls|xlsx|xlt|xlm|xlsm|xltm|xlw|csv|txt)$ 100000 90% 200000 refresh-ims
refresh_pattern -i .(app|bin|deb|rpm|drpm|exe|zip|zipx|tar|tgz|tbz2|tlz|iso|arj|cfs|dar|jar)$ 100000 90% 200000 refresh-ims
refresh_pattern -i .(bz|bz2|ipa|ram|rar|uxx|gz|msi|dll|lz|lzma|7z|s7z|Z|z|zz|sz)$ 100000 90% 200000 refresh-ims
refresh_pattern -i .(exe|msi)$ 0 90% 200000 refresh-ims
refresh_pattern -i .(cab|psf|vidt|apk|wtex|hz|ova|ovf)$ 100000 90% 200000 refresh-ims
refresh_pattern -i .(xml|flow|asp|aspx)$ 0 90% 200000 refresh-ims
refresh_pattern -i .(json)$ 0 90% 200000 refresh-ims
refresh_pattern -i .(asx|mp2|mp3|mp4|mp5|wmv|flv|mts|f4v|f4|pls|midi|mid)$ 100000 90% 200000 refresh-ims
refresh_pattern -i .(mpa|m2a|mpe|avi|mov|mpg|mpeg|mpg3|mpg4|mpg5)$ 100000 90% 200000 refresh-ims
refresh_pattern -i .(m1s|mp2v|m2v|m2s|m2ts|mp2t|wmx|rm|rmvb|3pg|3gpp|omg|ogm|asf|war)$ 100000 90% 200000 refresh-ims
refresh_pattern -i .(swf|js|ejs)$ 100000 90% 200000 refresh-ims
refresh_pattern -i .(wav|css|class|dat|zsci|ver|advcs)$ 100000 90% 200000 refresh-ims
refresh_pattern -i .(gif|png|ico|jpg|jpeg|jp2|webp)$ 100000 90% 200000 refresh-ims
refresh_pattern -i .(jpx|j2k|j2c|fpx|bmp|tif|tiff|bif)$ 100000 90% 20000 refresh-ims
refresh_pattern -i .(pcd|pict|rif|exif|hdr|bpg|img|jif|jfif)$ 100000 90% 200000 refresh-ims
refresh_pattern -i .(woff|woff2|eps|ttf|otf|svg|svgi|svgz|ps|ps1|acsm|eot)$ 100000 90% 200000 refresh-ims
refresh_pattern -i (.|-)(mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t)(?.*)?$ 43200 100% 432000
refresh_pattern -i (.|-)(mp3|m4a|aa?c3?|wm?av?|og(x|v|a|g)|ape|mka|au|aiff|zip|flac|m4(b|r)|m1v|m2(v|p)|mo(d|v)|arj|appx|lha|lzh|on2) 43200 100% 432000
refresh_pattern -i (.|-)(exe|bin|(n|t)ar|acv|(r|j)ar|t?gz|(g|b)z(ip)?2?|7?z(ip)?|wm[v|a]|patch|diff|mar|vpu|inc|r(a|p)m|kom|iso|sys|[ap]sf|ms[i|u|f]|dat|msi|cab|psf|dvr-ms|ace|asx|qt|xt|esd) 43200 100% 432000
refresh_pattern -i (.|-)(ico(.)?|pn[pg]|css|(g|t)iff?|jpe?g(2|3|4)?|psd|c(d|b)r|cad|bmp|img) 43200 100% 432000
refresh_pattern -i (.|-)(webm|(x-)?swf|mp(eg)?(3|4)|mpe?g(av)?|(x-)?f(l|4)v|divx?|rmvb?|mov|trp|ts|avi|m38u|wmv|wmp|m4v|mkv|asf|dv|vob|3gp?2?) 43200 100% 432000
refresh_pattern -i (.|-)(docx?|xlsx?|pptx?|rtf|xml|pdf|tiff?|txt) 43200 100% 432000

range_offset_limit 0

This is what I have so far originally it would had around 20 percent hits until a couple days ago its now down to zero so I deleted the catch and added all in all the #MISC FILE CACHING HERE section and disabled the adcdownload and the squid.internal hope that brings the disk hits back up again.

Keep in mind a lot of this was from the works cited below some is from Netgate.

Works Cited:

mmd123. (n.d.). MMD123/squid-cache-dynamic_refresh-list: List of dynamic_refresh patterns to use with Squid3, I had a hard time finding things online, so I figured I'd make a collaborative github repo for just this purpose. GitHub. Retrieved May 12, 2022, from https://github.com/mmd123/squid-cache-dynamic_refresh-list

aGeekhere · May 14, 2022, 5:42 AM

@jonathanlee I have a pull request that you can try
https://github.com/mmd123/squid-cache-dynamic_refresh-list/pulls

JonathanLee · May 14, 2022, 6:21 AM

@ageekhere most of the good items are from that GitHub repository :) thanks for sharing this.

It runs good. I added Kaspersky on mine

aGeekhere · May 14, 2022, 6:30 AM

@jonathanlee Kaspersky has already been added to the refresh pattern

see lines 84 to 85
refresh_pattern kaspersky.*.avc$ 43200 100% 43200
refresh_pattern kaspersky 43200 100% 43200

JonathanLee · May 15, 2022, 4:14 PM

@ageekhere

20-35 percent disk hit rates from storage in the last hour now WOW!! You do have to have certificates installed for this to work good.

(Image: Cache Information)

JonathanLee · May 15, 2022, 4:25 PM

@ageekhere

I am going to try the games section thank you. My son he plays the XBOX ONE and the tiles that load are the same when it goes online maybe the firewall will get more hits.

microsoft Games

refresh_pattern -i lancache-microsoft 525600 100% 525600
refresh_pattern -i .download.windowsupdate.com 525600 100% 525600
refresh_pattern -i download.windowsupdate.com 525600 100% 525600
refresh_pattern -i dlassets.xboxlive.com 525600 100% 525600
refresh_pattern -i .xboxone.loris.llnwd.net 525600 100% 525600
refresh_pattern -i xboxone.vo.llnwd.net 525600 100% 525600
refresh_pattern -i images-eds.xboxlive.com 525600 100% 525600
refresh_pattern -i xbox-mbr.xboxlive.com 525600 100% 525600
refresh_pattern -i assets1.xboxlive.com.nsatc.net 525600 100% 525600
refresh_pattern -i assets1.xboxlive.com 525600 100% 525600

JonathanLee · Aug 22, 2022, 3:04 PM

@ageekhere

I noticed some small items on the github

There are multiples of the default.

(Image: Default Refresh)

I noticed that it should only use one of them. Maybe if it added #360 mins fresh 1 day store
#180 mins fresh 30 days store
#30 mins fresh 1 day store

and note above
#only choose one

If you run all three of them, it slows the cache down.

(Image: Reference Squid Website Why choose one)

aGeekhere · Aug 22, 2022, 11:12 PM

@jonathanlee Agreed there are still many improvements to be made to the refresh pattern list, if you see improvements you can create a pull request or submit an issue at https://github.com/mmd123/squid-cache-dynamic_refresh-list

JonathanLee · Aug 22, 2022, 11:30 PM

@ageekhere

Done

Opened