Improve Custom refresh pattern
-
I have this for Apple iOS updates and iOS apps.
refresh_pattern -i appldnld\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate refresh_pattern -i phobos\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate -
Thanks updated
-
how about youtube?
-
I have this for Apple iOS updates and iOS apps.
refresh_pattern -i appldnld\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate refresh_pattern -i phobos\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidateDoesn't apple block MITM using certificate pinning? Since their app store is over HTTPS.
-
KOM was right, in fact it was not working properly. Also, the results of bare counting of MISSes/HITs from not fully populated cache was not good measurement. Right now, any windowsupdate downloaded file (even partial) is fetched from cache, verified manually and monitored in access.log.
Few days ago I have made some improvements to utilize store_rewrite module.
They were tested, as suggested above by KOM, successfully.Indeed, some of WU were not cached properly since they were received from different CDN hosts.
Config files are below (squid.conf is only part that is related to this issue)
Apple and Symantec updates are cached well, too.
Steam/Xbox (cs.steampowered.com / llnwd.net respective CDN) are not confirmed yet.
The rest of refresh patterns and URL rewrites are still in experimental mode. Needs a lot of testing and research in order to make it done properly.
Attached is screenshot from Lightsquid, you can clearly see changes after 3rd of March.
P.S. Questions:
- not quite sure if I need to list all possible domains under acl dstdomain directive, or just first level domain will be good enough?
Comments:
- I have total 1.3TB of swap available and 32GB on server running Pfsense+Squid+SquidGuard+Snort+PfBlocker. Configuration is optimized for ~2000 hosts on LAN.
Current byte/hit rate is 20-45%
/usr/local/etc/squid/squid.conf
....... # No range offset limit for windowsupdate sites acl Windows_Update dstdomain windowsupdate.microsoft.com acl Windows_Update dstdomain update.microsoft.com acl Windows_Update dstdomain download.windowsupdate.com acl Windows_Update dstdomain windowsupdate.com acl Windows_Update dstdomain au.download.windowsupdate.com acl Windows_Update dstdomain bg.v4.pr.dl.ws.microsoft.com acl Windows_Update dstdomain wsus.ds.download.windowsupdate.com acl Windows_Update dstdomain au.b1.download.windowsupdate.com range_offset_limit -1 Windows_Update # Storeid_rewrite configuration acl updatesites dstdom_regex "/usr/local/etc/squid/updatesites.txt" store_id_access allow updatesites store_id_access deny all store_id_program /usr/local/libexec/squid/storeid_file_rewrite /usr/local/etc/squid/storeid_rewrite.conf store_id_children 200 startup=60 idle=1 concurrency=0 # Refresh patterns # 525600 min is one year # Gaming CDN refresh_pattern -i \.llnwd.net 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth refresh_pattern -i \.cs.steampowered.com 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth #windows update refresh_pattern -i \.update.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth refresh_pattern -i \.windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth refresh_pattern -i \.download.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth refresh_pattern -i \.ws.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth # Symantec Updates # Only reload-into-ims works properly, other flags break update process refresh_pattern -i \.symantecliveupdate.com/.*\.(zip|exe|z7) 525600 100% 525600 reload-into-ims/usr/local/etc/squid/updatesites.txt
\.adobe.com \.java.com \.sun.com \.oracle.com \.apple.com \.microsoft.com \.windowsupdate.com \.ubuntu.com \.steampowered.com \.llnwd.net \.symantecliveupdate.com/usr/local/etc/squid/storeid_rewrite.conf
^http:\/\/.+?\.microsoft\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx) http://wupdate.squid.local/$1 ^http:\/\/.+?\.windowsupdate\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx) http://wupdate.squid.local/$1 ^http:\/\/.+?\.cs\.steampowered\.com\/(.*) http://steamupdates.squid.internal/$1 ^http:\/\/.+?\.apple\.com\/(.*) http://appupdates.apple.squid.internal/$1 ^http:\/\/.+?\.llnwd\.net\/(.*) http://llnwd.net.squid.internal/$1Regards,
Valeriy
May I know.
1. Is it just copy CODE in Table 1(/usr/local/etc/squid/squid.conf) and paste to "Custom refresh_patterns" of squid on Pfsense ? Or any other changes need on Table 1.
2. How useful Table 2(/usr/local/etc/squid/updatesites.txt ) and Table 3(/usr/local/etc/squid/storeid_rewrite.conf)? Need to insert these tables on Table 1? If needed, guide me pls(I am Beginner)
My aim is to get steam update cache(dota2) -
I have this for Apple iOS updates and iOS apps.
refresh_pattern -i appldnld\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate refresh_pattern -i phobos\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidateDoesn't apple block MITM using certificate pinning? Since their app store is over HTTPS.
Apps and iOS updates are over HTTP
-
Hi,
Sorry for the stupid question..
But where does the following code goes:#new refresh patterns 3 acl Windows_Update dstdomain windowsupdate.microsoft.com acl Windows_Update dstdomain .update.microsoft.com acl Windows_Update dstdomain download.windowsupdate.com acl Windows_Update dstdomain www.download.windowsupdate.com acl Windows_Update dstdomain au.download.windowsupdate.com acl Windows_Update dstdomain bg.v4.pr.dl.ws.microsoft.comDoes it go to the ACLs tab?
Is it white list or does it go in local cache -> Custom refresh_patternsThank you
-
Is it white list or does it go in local cache -> Custom refresh_patterns
Custom refresh_patterns
-
Any refresh for videos in http?
-
-
do i just need to copy this codes and paste it into my custom pattern_refresh box? soory im a newbie here.
-
Yes, did it improve your hit rate?
-
Two invalid expressions.
Somebody can fix them?/usr/local/etc/squid/squid.conf line 52: refresh_pattern -i .(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|(x-|)flv) 129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
refreshAddToList: Invalid regular expression '.(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|(x-|)flv)': empty (sub)expression_ kid1| /usr/local/etc/squid/squid.conf line 136: refresh_pattern -i ^http://(khm?)([^/]?).google.(de|com) 129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
refreshAddToList: Invalid regular expression '^http://(khm?)([^/]?).google.(de|com)': repetition-operator operand invalid_I just change it to ^http://(khm?)([^/]?).google.(de|com) by removing the *
-
So replace
refresh_pattern -i ^http://(khm?)([^/]*?).google.(de|com) 129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
with ?
refresh_pattern -i ^http://(khm?)([^/]?).google.(de|com) 129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reloadreplace
refresh_pattern -i .(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|(x-|)flv) 129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reloadwith ?
refresh_pattern -i .(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|flv) 129600 100% 129600 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
Can someone confirm the changes are correct.
-
Facebook is a heavy traffic site for me which I could probably benefit from caching. I'm curious what the ip is?
#images facebook
refresh_pattern ((facebook.com)|(85.131.151.39))..(jpg|png|gif) 129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
refresh_pattern -i .fbcdn.net..(jpg|gif|png|swf|mp3) 129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
refresh_pattern static.ak.fbcdn.net*.(jpg|gif|png) 129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
refresh_pattern ^http://profile.ak.fbcdn.net*.(jpg|gif|png) 129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store -
Just a question has anyone found an Improvement in there hit rate using this custom refresh pattern?
I wounder if there is a simple refresh_pattern * that sates cache all instead of defining all these file types and domains.
-
A simple refresh pattern will break HTTP heavily.
You can increase number of minutes in default pattern and see yourself. -
Facebook is a heavy traffic site for me which I could probably benefit from caching. I'm curious what the ip is?
#images facebook
refresh_pattern ((facebook.com)|(85.131.151.39))..(jpg|png|gif) 129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
refresh_pattern -i .fbcdn.net..(jpg|gif|png|swf|mp3) 129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
refresh_pattern static.ak.fbcdn.net*.(jpg|gif|png) 129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store
refresh_pattern ^http://profile.ak.fbcdn.net*.(jpg|gif|png) 129600 100% 129600 ignore-reload override-expire ignore-no-cache ignore-no-storesomething in the link below, but I was not able to implement in pfsense.
http://comastnet.blogspot.com.br/2016/06/squid-youtube-cache-2016-terbaru-suport.html -
any idea of caching playstore apps
-
something in the link below, but I was not able to implement in pfsense.
http://comastnet.blogspot.com.br/2016/06/squid-youtube-cache-2016-terbaru-suport.htmlI cannot believe that this guy puts up this web page with a massive spew of commands & config, and then sets the page so you can't copy the text. What a tool.
