Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Squid StoreID and Facebook plus caching Windows updates

    Scheduled Pinned Locked Moved Cache/Proxy
    13 Posts 1 Posters 1.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • JonathanLeeJ
      JonathanLee
      last edited by JonathanLee

      Hello fellow Netgate community members,

      After lots of trial and error while researching this I have gotten StoreID to start working finally correctly. Well I think I have there is always bugs that pop up. I wanted to share the with you incase other hardcore Squid users are still out there.

      "....ICAP + StoreID helper idea works in production with more then one site for quite some time but it has some overheads and I would rate this kind of a setup as an Expert only" (wiki.squid-cache.org).

      This does require SSL certificates to work so you can not just make it work without owning the devices and hardware to install the certs.

      You can't see the data in the cache you can only see hits as it is inside of the cache encrypted so no bad guys also...

      Squid does come with a very nice useable StoreID program that is built into the package and it has all you need to do is create your database for it.

      "This program acts as a store_id helper program, rewriting URLs passed by Squid into storage-ids that can be used to achieve better caching for websites that use different URLs for the same content.
      It takes a text file with two tab separated columns. Column 1: Regular expression to match against the URL Column 2: Rewrite rule to generate a Store-ID Eg: ^http://[^.]+.dl.sourceforge.net/(.*) http://dl.sourceforge.net.squid.internal/$1
      Rewrite rules are matched in the same order as they appear in the rules file. So for best performance, sort it in order of frequency of occurrence.
      This program will automatically detect the existence of a concurrecy channel-ID and adjust appropriately. It may be used with any value 0 or above for the store_id_children concurrency= parameter" (systutorials).

      I placed this under custom refresh_patterns

      acl getmethod method GET
acl rewritedoms dstdomain .facebook.com .akamaihd.net .fbcdn.net .google.com .static.com .apple.com .oracle.com .sun.com .java.com .adobe.com .steamstatic.com .steampowered.com .steamcontent.com .google.com
always_direct allow !getmethod
store_id_access deny connect
store_id_access deny !getmethod
store_id_access allow rewritedoms
store_id_program /usr/local/libexec/squid/storeid_file_rewrite /var/squid/storeid/storeid_rewrite.txt
store_id_children 10 startup=5 idle=1 concurrency=0

refresh_pattern ([^.]+.|)(download|adcdownload).(apple.|)com/.*\.(pkg|dmg) 4320 100% 43200 reload-into-ims

# Updates: Windows
refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
refresh_pattern -i microsoft.com.akadns.net/.*\.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims
refresh_pattern -i deploy.akamaitechnologies.com/.*\.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims

refresh_pattern ([^.]+\.)?(cs|content[1-9]|hsar|content-origin|client-download).[steampowered|steamcontent].com/.*\.* 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern ([^.]+\.)?.akamai.steamstatic.com/.*\.* 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod

refresh_pattern -i ([^.]+\.)?.adobe.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.java.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.sun.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.oracle.com/.*\.(zip|exe|tar.gz) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod

refresh_pattern -i appldnld\.apple\.com 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?apple.com/.*\.(ipa) 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
 
refresh_pattern -i  ([^.]+\.)?.google.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern -i ([^.]+\.)?g.static.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK
refresh_pattern ^http?://*.facebook.com/*  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK IMAGES  
refresh_pattern -i pixel.facebook.com..(jpg|png|gif|ico|css|js)  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern -i .akamaihd.net..(jpg|png|gif|ico|css|js)  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private   
refresh_pattern -i (facebook.com).(jpg|png|gif)  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private 
refresh_pattern static.(xx|ak).fbcdn.net.(jpg|gif|png) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern ^https?://profile.ak.fbcdn.net*.(jpg|gif|png)  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK VIDEO
refresh_pattern -i .video.ak.fbcdn.net.*.(mp4|flv|mp3|amf)  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private   
refresh_pattern (audio|video)/(webm|mp4)  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

refresh_pattern -i squid\.internal	10080	80%	79900 override-lastmod override-expire ignore-reload ignore-no-store ignore-must-revalidate ignore-private ignore-auth

range_offset_limit 0
quick_abort_min -1 KB

      The path to Squids store id program is here, it is part of Squid package and will be on your system already just waiting for someone to dust off that code and activate it.

      /usr/local/libexec/squid/storeid_file_rewrite

      Keep in mind you are required to use a database. I created one with this location and just saved my needs in it for testing
      /var/squid/storeid/storeid_rewrite.txt

      I created my db and saved it in the same location as a txt file.

      Mine is a very basic setup as I am new to StoreID however Squid lists a useable database here for many sites. I am sure the major ISP systems have monster cache systems.

      https://wiki.squid-cache.org/Features/StoreID/DB

      Keep in mind there is risks if you do not store items correctly it can display items on unwanted terminals. So don't mess this up on anyone. Know the risks associated with this.

       # Facebook
^https?:\/\/(fbcdn|scontent).*(akamaihd|fbcdn)\.net\/.*\/v\/.*\/(.*\.mp4)	http://facebook.squid.internal/$3
^https?:\/\/fbcdn\-(static|profile)\-a\.akamaihd\.net\/static\-ak\/rsrc\.php\/((?!.*\.(?:js|css|swf)).*)	http://facebook.squid.internal/static/$2
^https?:\/\/(fbcdn|scontent).*(akamaihd|fbcdn)\.net\/(h|s)(profile|photos).*\/(.*\.(png|gif|jpg))(\?.+)? 	http://facebook.squid.internal/$5
^https?:\/\/fbstatic\-a\.akamaihd\.net\/rsrc\.php\/((?!.*\.(?:js|css|swf)).*) 	http://facebook.squid.internal/static/$1
^http:\/\/.*[steampowered|steamcontent]\.com\/([^?]*)	http://steamupdates.squid.internal/$1
^https?\:\/\/download\.oracle\.com\/((otn\-pub|otn)\/[\d\w]+\/[\d\w]+\/[\w\d\-]+\/[\w\d\-]+\.(exe|dmg|rpm|msi|tar\.(gz|Z)))\?	http://java.oracle.otn.ngtech.squid.internal/$1
^https?\:\/\/([\d\w\-]+)\.oracle\.com\/(([\d\w]+)\/[\d\w]+\/[\d\w]+\/([\d\w\-]+)\/([\d\w]+\/)?[\d\w\-\.\_]+\.(dmg|msi|exe|tar\.gz|tar\.Z))\?	http://java.oracle.download.ngtech.squid.internal/$2
^http:\/\/[^\.]+\.phobos\.apple\.com\/(.*)	http://appupdates.apple.squid.internal/$1
^http:\/\/[^\.]+\.c\.android\.clients\.google\.com\/(.*)	http://androidupdates.google.squid.internal/$1

      Screenshot 2024-03-18 at 23.30.25.png

      Now watch the hits come on in and after a while you will have your cache working great with other CDN s just add them to the database. Soon they say they Squid will also add something called Metalink support into squid. Ref http://www.metalinker.org to simplify this process and help cut down on energy use for downloading the same thing over and over and over across networks. There is no point when the videos and images are the same why not just redeliver it locally and securely.

      This really does accelerate traffic great.

      ref:
      https://wiki.squid-cache.org/Features/StoreID
      https://www.systutorials.com/docs/linux/man/8-storeid_file_rewrite/
      https://wiki.squid-cache.org/ConfigExamples/Caching/WindowsUpdates

      I hope you enjoyed this research of the tools already built into this package. If anyone is out there that knows how to make this more secure or wants to add anything please comment below. I am amazed it started working for me. Please if you see something wrong let me know. I have been excited to see this work from 2019 until today and this was the first time I have seen it in action and it did not display the same add over every photo this time.

      I also wanted to note I have a custom Squid option installed here. So the system works with both splice only systems and the systems I want the cache/AV running on inspecting traffic.

      acl manager proto cache_object
acl localhost src 192.168.1.1/32
http_access allow manager localhost
http_access deny manager

acl BrokenButTrustedServers dstdomain "/usr/local/pkg/dstdom.broken"
acl DomainMismatch ssl_error SQUID_X509_V_ERR_DOMAIN_MISMATCH
sslproxy_cert_error allow BrokenButTrustedServers DomainMismatch
sslproxy_cert_error deny all

acl splice_only src 192.168.1.8 #Tasha iPhone
acl splice_only src 192.168.1.10 #Jon iPhone
acl splice_only src 192.168.1.11 #Amazon Fire
acl splice_only src 192.168.1.15 #Tasha HP
acl splice_only src 192.168.1.16 #iPad

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"

acl markBumped annotate_client bumped=true
acl bump_only src 192.168.1.3 #webtv
acl bump_only src 192.168.1.4 #toshiba
acl bump_only src 192.168.1.5 #imac
acl bump_only src 192.168.1.9 #macbook
acl bump_only src 192.168.1.13 #dell

ssl_bump peek step1
ssl_bump splice splice_only
ssl_bump splice NoSSLIntercept
ssl_bump bump bump_only markBumped
ssl_bump stare all

acl markedBumped note bumped true
url_rewrite_access deny markedBumped
http_access deny all

#acl SSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.bump"
#ssl_bump bump SSLIntercept

      Also here is a copy of my auto splice sites

      /usr/local/pkg/url.nobump

      #Sites to be splice
^.*conviva\.com.*
license\.adrise\.tv.*
c2r\.ts\.cdn\.office\.net
^.*cdn\.office\.net
^.*bitdefender\.net
config\.teams\.microsoft\.com
^.*.azure-devices\.net
substrate\.office\.com.*
hulu\.playback\.edge\.bamgrid\.com
assetshuluimcom-a\.akamaihd\.net
hulu\.sc\.omtrdc\.net
infinity-c33\.youboranqs01\.com
beacons\.extremereach\.io
^.*tubi\.video
^.*tubi\.io
a-fds\.youborafds01\.com
youboranqs01\.com
amzpvxrayasset-a\.akamaihd\.net
pv-cdn.net
^.*media-amazon\.com
aiv-delivery\.net
unagi\.amazon\.com
atv-ps\.amazon\.com
pv-cdn\.net
fls-na\.amazon\.com
^.*aiv-cdn\.net
c0a299900000\.local
update\.microsoft\.com
update\.microsoft\.com\.akadns\.net
delivery\.mp\.microsoft\.com
appldnld\.apple\.com
configuration\.apple\.com
gdmf\.apple\.com
mesu\.apple\.com
oscdn\.apple\.com
osrecovery\.apple\.com
skl\.apple\.com
swcdn\.apple\.com
swdist\.apple\.com
swscan\.apple\.com
updates-http\.cdn-apple\.com
updates\.cdn-apple\.com
appldnld\.apple\.com\.edgesuite\.net
entrust\.net
digicert\.com
apple-cloudkit\.com
apple-livephotoskit\.com
gc\.apple\.com
icloud-content\.com
apple\.com
cdn-apple\.com
icloud\.com
api\.apple-cloudkit\.com
^.*appattest\.apple\.com
^.*itunes\.apple\.com
^.*mzstatic\.com
itunes\.com
music\.apple\.com
app-site-association\.cdn-apple\.com
app-site-association\.networking\.apple\.com
xp\.apple\.com
play\.google\.com
android\.com
^((alt[0-9]-mtalk\.)|(mtalk\.)|(mtalk-(staging|dev)\.))google\.com
google-analytics\.com
googleusercontent\.com
^((gvt)([0-9]))\.com
ggpht\.com
dl\.google\.com
dl-ssl\.google\.com
android\.clients\.google\.com
^(((clients)[0-9])|accounts)\.google\.(com|us)
connectivitycheck\.android\.com
android\.clients\.google\.com
device-provisioning\.googleapis\.com
omahaproxy\.appspot\.com
payments\.google\.com
googleapis\.com
notifications\.google\.com
^(pki|(crl|ocsp)\.pki)\.google\.com
ogs\.google\.com
googleapis\.com
androidmanagement\.googleapis\.com
mservice\.bankofamerica\.com
privacyportal-bofa\.my\.onetrust\.com
bankofamerica\.com
mcafee\.com
kaspersky\.com
kaspersky-labs\.com
dc1-st\.ksn\.kaspersky-labs\.com
dc1-file\.ksn\.kaspersky-labs\.com
dc1\.ksn\.kaspersky-labs\.com
olui2m\.fs\.ml\.com
ml\.com
^.*zoom\.us
^.*teams\.microsoft\.com
teams\.events\.data\.microsoft\.com
statics\.teams\.cdn\.office\.net
^.*(outlook\.)(office365|office)\.com
edge-chat\.facebook\.com
internet\.speedpay\.com
^.*hulustream\.com
cws-hulu\.conviva\.com
^.*hulu\.com
hulu\.hb\.omtrdc\.net
^.*dssott\.com
prod-ripcut-delivery\.disney-plus\.net
^(disney\.(content|connections))\.edge\.bamgrid\.com
disney\.api\.edge\.bamgrid\.com
disney\.playback\.edge\.bamgrid\.com
disney\.my\.sentry\.io
^.*amazonvideo\.com
unagi-na\.amazon\.com
events\.data\.microsoft\.com
tubi\.io
production-public\.tubi\.io
tubitv\.com
caauthservice\.state\.gov
studentaid\.gov
mohela\.com
www\.whitehouse\.gov
www\.rcsdk8\.org
rcsdk8\.powerschool\.com
www\.weaveinc\.org
^.*cdn\.nintendo\.net
^.*bitdefender\.net

      /usr/local/pkg/dstdom.broken

      download.microsoft.com
update.microsoft.com
update.microsoft.com.akadns.net
update.microsoft.com.nsatc.net

      Enjoy please message below if something is completely off.

      I am looking for any info on the $number part of the StoreID and if possible, the risks of using all with StoreID access. It seems to work very well with all and slows down when you use the desdomain acl.

      It took a lot of time to get it to work right years, plus a degree to understand it. I still feel I only brushed the surface. This will store the updates and deliver them again. The goal here is to not redownload things multiple times for requests. It caches the requests just like a web browser does when you refresh. It speeds up delivery and lessens the loads on networks.

      My biggest fear with this configuration is that it could store a container and data marshal the network card from inside the cache. I worry about that, so I only want get requests not connect right? So, all should be fine. There is not much information on this, and to improve it requires conversations. To my understanding the cache inside pfSense is encrypted so users can only access it for use of the cache and acceleration outside of that it is blocked.

      Imagin a future in which web caching occurs on a consumer grade AP or Netgate firewall and it helps to drastically lowering energy consumption and speeding up web content delivery. Think about how much energy is wasted on redownloading the same thing over and over each day over WAN connections, it must be huge. Most users say we don't need that we have GB fiber connections, again with the whole green energy push the appliances we use for Facebook and Streaming video might be the sector that can be drastically improved on. Again, it needs development and testing plus a quest to get it to even halfway work right. I was amazed when it functioned for me the first time, I went to a website, and it had every single photo picture loaded at one click, it wasn't slow loads, it was the fastest I have ever seen my old DSL system run.

      If anyone has the amount of energy that is consumed by repeated downloads each day for major web sites please reply.

      Make sure to upvote

      1 Reply Last reply Reply Quote 0
      • JonathanLeeJ
        JonathanLee
        last edited by

        Small addition you must change the owership of the file so that Squid can use it.

        chgrp -Rf proxy /var/squid/storeid
        chown -Rf squid /var/squid/storeid
        chmod -Rf 775 /var/squid/storeid

        Or else it will say helper program exiting to fast and kill the cache

        Make sure to upvote

        1 Reply Last reply Reply Quote 0
        • JonathanLeeJ
          JonathanLee
          last edited by

          does anyone know what the /$6 means at the end of the text files Squid does not cover this information I know it means in reg ex to count over however how can it could over if it is reading a file it would need a variable to store it after so that is where I get confused here.

          Make sure to upvote

          1 Reply Last reply Reply Quote 0
          • JonathanLeeJ
            JonathanLee
            last edited by

            Please if anyone else knows how to improve this please yet me know this configuration project has been going on for a long time for me. It works well now however I feel I could improve on the security side.

            Goal I do not want a container to be able to be downloaded into the cache and escape into the firewall file system.

            Make sure to upvote

            1 Reply Last reply Reply Quote 0
            • JonathanLeeJ
              JonathanLee
              last edited by

              update:
              Dynamic and Update Content
              Custom refresh_patterns

              acl getmethod method GET
acl rewritedoms dstdomain .facebook.com .akamaihd.net .fbcdn.net .google.com .static.com .apple.com .oracle.com .sun.com .java.com .adobe.com .steamstatic.com .steampowered.com .steamcontent.com .google.com

always_direct allow !getmethod
store_id_access deny connect
store_id_access deny !getmethod
store_id_access allow rewritedoms
store_id_program /usr/local/libexec/squid/storeid_file_rewrite /var/squid/storeid/storeid_rewrite.txt
store_id_children 10 startup=5 idle=1 concurrency=0

#APPLE STUFF
refresh_pattern -i apple.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip|dist)$ 0 80% 43200  refresh-ims

#apple update
refresh_pattern -i (download|adcdownload).apple.com/.*\.(pkg|dmg) 4320 100% 43200 
refresh_pattern -i appldnld\.apple\.com 129600 100% 129600     
refresh_pattern -i phobos\.apple\.com 129600 100% 129600     
refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600     

# Updates: Windows
refresh_pattern -i microsoft.com/..(cab|exe|msi|msu|msf|asf|wma|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i windowsupdate.com/..(cab|exe|msi|msu|msf|asf|wma|wmv)|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i windows.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i .*windowsupdate.com/.*\.(cab|exe)                     259200 100% 259200   
refresh_pattern -i .*update.microsoft.com/.*\.(cab|exe|dll|msi|psf)                  259200 100% 259200   
refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 
refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200
#windows update NEW UPDATE 0.04
refresh_pattern update.microsoft.com/.*\.(cab|exe)                  43200 100% 129600    
refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf) 4320 100% 43200  
refresh_pattern update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern -i \.update.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.download.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.ws.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       

refresh_pattern ([^.]+\.)?(cs|content[1-9]|hsar|content-origin|client-download).[steampowered|steamcontent].com/.*\.* 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern ([^.]+\.)?.akamai.steamstatic.com/.*\.* 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod

refresh_pattern -i ([^.]+\.)?.adobe.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.java.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.sun.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.oracle.com/.*\.(zip|exe|tar.gz) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod

refresh_pattern -i appldnld\.apple\.com 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?apple.com/.*\.(ipa) 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
 
refresh_pattern -i ([^.]+\.)?.google.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern -i ([^.]+\.)?g.static.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK
refresh_pattern ^http?://*.facebook.com/*  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK IMAGES  
refresh_pattern -i pixel.facebook.com..(jpg|png|gif|ico|css|js)  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern -i .akamaihd.net..(jpg|png|gif|ico|css|js)  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private   
refresh_pattern -i (facebook.com).(jpg|png|gif)  10080 80% 43200 store-stale override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private 
refresh_pattern static.(xx|ak).fbcdn.net.(jpg|gif|png) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern ^https?://profile.ak.fbcdn.net*.(jpg|gif|png)  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK VIDEO
refresh_pattern -i .video.ak.fbcdn.net.*.(mp4|flv|mp3|amf) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private   
refresh_pattern (audio|video)/(webm|mp4) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

refresh_pattern -i squid\.internal 10080 80% 79900 override-lastmod override-expire ignore-reload ignore-no-store ignore-must-revalidate ignore-private ignore-auth

range_offset_limit 200 MB windowsupdate
maximum_object_size 200 MB windowsupdate
range_offset_limit 0
quick_abort_min -1 KB

              update
              Custom Options (SSL/MITM)

              acl manager proto cache_object
acl windowsupdate dstdomain windowsupdate.microsoft.com
acl windowsupdate dstdomain .update.microsoft.com
acl windowsupdate dstdomain download.windowsupdate.com
acl windowsupdate dstdomain redir.metaservices.microsoft.com
acl windowsupdate dstdomain images.metaservices.microsoft.com
acl windowsupdate dstdomain c.microsoft.com
acl windowsupdate dstdomain www.download.windowsupdate.com
acl windowsupdate dstdomain wustat.windows.com
acl windowsupdate dstdomain crl.microsoft.com
acl windowsupdate dstdomain sls.microsoft.com
acl windowsupdate dstdomain productactivation.one.microsoft.com
acl windowsupdate dstdomain ntservicepack.microsoft.com
acl localhost src 192.168.1.1/32

acl CONNECT method CONNECT
acl wuCONNECT dstdomain www.update.microsoft.com
acl wuCONNECT dstdomain sls.microsoft.com

http_access allow CONNECT wuCONNECT localnet
http_access allow CONNECT wuCONNECT localhost
http_access allow windowsupdate localnet
http_access allow windowsupdate localhost
http_access deny manager

acl BrokenButTrustedServers dstdomain "/usr/local/pkg/dstdom.broken"
acl DomainMismatch ssl_error SQUID_X509_V_ERR_DOMAIN_MISMATCH
sslproxy_cert_error allow BrokenButTrustedServers DomainMismatch
sslproxy_cert_error deny all

acl splice_only src 192.168.1.8 #Tasha iPhone
acl splice_only src 192.168.1.10 #Jon iPhone
acl splice_only src 192.168.1.11 #Amazon Fire
acl splice_only src 192.168.1.15 #Tasha HP
acl splice_only src 192.168.1.16 #iPad

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"

acl markBumped annotate_client bumped=true
acl bump_only src 192.168.1.3 #webtv
acl bump_only src 192.168.1.4 #toshiba
acl bump_only src 192.168.1.5 #imac
acl bump_only src 192.168.1.9 #macbook
acl bump_only src 192.168.1.13 #dell

ssl_bump peek step1
ssl_bump splice splice_only
ssl_bump splice NoSSLIntercept
ssl_bump bump bump_only markBumped
ssl_bump stare all

acl markedBumped note bumped true
url_rewrite_access deny markedBumped
http_access deny all

#acl SSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.bump"
#ssl_bump bump SSLIntercept

              Theses changes made it work better.

              Please if anyone else like to research this let me know if you see anything off?

              Make sure to upvote

              1 Reply Last reply Reply Quote 0
              • JonathanLeeJ
                JonathanLee
                last edited by JonathanLee

                Update to custom options new items tried today caused increased performance

                cache deny https_login
                read_ahead_gap 32 KB
                negative_ttl 1 second
                connect_timeout 30 seconds
                request_timeout 60 seconds
                half_closed_clients off
                shutdown_lifetime 10 seconds
                negative_dns_ttl 1 seconds
                ignore_unknown_nameservers on
                pipeline_prefetch 100

                I have been testing the above options and they seem to increase performance drastically.

                acl manager proto cache_object
acl localhost src 192.168.1.1/32
acl https_login url_regex -i ^https.*(login|Login).*
acl CONNECT method CONNECT
acl wuCONNECT dstdomain www.update.microsoft.com
acl wuCONNECT dstdomain sls.microsoft.com
http_access allow CONNECT wuCONNECT localnet
http_access allow CONNECT wuCONNECT localhost
http_access allow windowsupdate localnet
http_access allow windowsupdate localhost
http_access deny manager

acl BrokenButTrustedServers dstdomain "/usr/local/pkg/dstdom.broken"
acl DomainMismatch ssl_error SQUID_X509_V_ERR_DOMAIN_MISMATCH
sslproxy_cert_error allow BrokenButTrustedServers DomainMismatch
sslproxy_cert_error deny all

acl splice_only src 192.168.1.8 #Tasha iPhone
acl splice_only src 192.168.1.10 #Jon iPhone
acl splice_only src 192.168.1.11 #Amazon Fire
acl splice_only src 192.168.1.15 #Tasha HP
acl splice_only src 192.168.1.16 #iPad

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"

acl markBumped annotate_client bumped=true
acl bump_only src 192.168.1.3 #webtv
acl bump_only src 192.168.1.4 #toshiba
acl bump_only src 192.168.1.5 #imac
acl bump_only src 192.168.1.9 #macbook
acl bump_only src 192.168.1.13 #dell

cache deny https_login
read_ahead_gap 32 KB
negative_ttl 1 second
connect_timeout 30 seconds
request_timeout 60 seconds
half_closed_clients off
shutdown_lifetime 10 seconds
negative_dns_ttl 1 seconds
ignore_unknown_nameservers on
pipeline_prefetch 100

ssl_bump peek step1
ssl_bump none https_login
ssl_bump splice splice_only
ssl_bump splice NoSSLIntercept
ssl_bump bump bump_only markBumped
ssl_bump stare all

acl markedBumped note bumped true
url_rewrite_access deny markedBumped
http_access deny all

#acl SSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.bump"
#ssl_bump bump SSLIntercept

                update Custom refresh_patterns this area is in the config first so acl windows update can be used in general configuration also.

                acl getmethod method GET

acl windowsupdate dstdomain windowsupdate.microsoft.com
acl windowsupdate dstdomain .update.microsoft.com
acl windowsupdate dstdomain download.windowsupdate.com
acl windowsupdate dstdomain redir.metaservices.microsoft.com
acl windowsupdate dstdomain images.metaservices.microsoft.com
acl windowsupdate dstdomain c.microsoft.com
acl windowsupdate dstdomain www.download.windowsupdate.com
acl windowsupdate dstdomain wustat.windows.com
acl windowsupdate dstdomain crl.microsoft.com
acl windowsupdate dstdomain sls.microsoft.com
acl windowsupdate dstdomain productactivation.one.microsoft.com
acl windowsupdate dstdomain ntservicepack.microsoft.com
acl windowsupdate dstdomain dc1-st.ksn.kaspersky-labs.com
acl windowsupdate dstdomain dc1-file.ksn.kaspersky-labs.com
acl windowsupdate dstdomain dc1.ksn.kaspersky-labs.com

acl rewritedoms dstdomain .facebook.com .akamaihd.net .fbcdn.net .google.com .static.com .apple.com .oracle.com .sun.com .java.com .adobe.com .steamstatic.com .steampowered.com .steamcontent.com .google.com

always_direct allow !getmethod
store_id_access deny connect
store_id_access deny !getmethod
store_id_access allow rewritedoms
store_id_program /usr/local/libexec/squid/storeid_file_rewrite /var/squid/storeid/storeid_rewrite.txt
store_id_children 10 startup=5 idle=1 concurrency=0

#APPLE STUFF
refresh_pattern -i apple.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip|dist)$ 0 80% 43200  refresh-ims

#apple update
refresh_pattern -i (download|adcdownload).apple.com/.*\.(pkg|dmg) 4320 100% 43200 
refresh_pattern -i appldnld\.apple\.com 129600 100% 129600     
refresh_pattern -i phobos\.apple\.com 129600 100% 129600     
refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600     

# Updates: Windows
refresh_pattern -i microsoft.com/..(cab|exe|msi|msu|msf|asf|wma|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i windowsupdate.com/..(cab|exe|msi|msu|msf|asf|wma|wmv)|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i windows.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i .*windowsupdate.com/.*\.(cab|exe) 259200 100% 259200   
refresh_pattern -i .*update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 259200 100% 259200   
refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 
refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200
#windows update NEW UPDATE 0.04
refresh_pattern update.microsoft.com/.*\.(cab|exe) 43200 100% 129600    
refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf) 4320 100% 43200  
refresh_pattern update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern -i \.update.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.download.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.ws.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       

refresh_pattern ([^.]+\.)?(cs|content[1-9]|hsar|content-origin|client-download).[steampowered|steamcontent].com/.*\.* 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern ([^.]+\.)?.akamai.steamstatic.com/.*\.* 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod

refresh_pattern -i ([^.]+\.)?.adobe.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.java.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.sun.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.oracle.com/.*\.(zip|exe|tar.gz) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod

refresh_pattern -i appldnld\.apple\.com 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?apple.com/.*\.(ipa) 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
 
refresh_pattern -i ([^.]+\.)?.google.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern -i ([^.]+\.)?g.static.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK
refresh_pattern ^http?://*.facebook.com/*  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK IMAGES  
refresh_pattern -i pixel.facebook.com..(jpg|png|gif|ico|css|js)  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern -i .akamaihd.net..(jpg|png|gif|ico|css|js) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private   
refresh_pattern -i (facebook.com).(jpg|png|gif) 10080 80% 43200 store-stale override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private 
refresh_pattern static.(xx|ak).fbcdn.net.(jpg|gif|png) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern ^https?://profile.ak.fbcdn.net*.(jpg|gif|png) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK VIDEO
refresh_pattern -i .video.ak.fbcdn.net.*.(mp4|flv|mp3|amf) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private   
refresh_pattern (audio|video)/(webm|mp4) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

refresh_pattern -i squid\.internal 10080 80% 79900 override-lastmod override-expire ignore-reload ignore-no-store ignore-must-revalidate ignore-private ignore-auth

range_offset_limit 512 MB windowsupdate
maximum_object_size 512 MB windowsupdate
range_offset_limit 0
quick_abort_min -1 KB

                Update to non bump list

                ^.*gateway.facebook.com/ws/realtime?

                This addition corrects many issues with delays and auto log outs this must be spliced and marked as non bump to function effectively on the web cache accelerator system.

                #Sites to be splice
^.*gateway\.facebook\.com\/ws\/realtime\?
^.*conviva\.com.*
license\.adrise\.tv.*
c2r\.ts\.cdn\.office\.net
^.*cdn\.office\.net
^.*bitdefender\.net
config\.teams\.microsoft\.com
^.*.azure-devices\.net
substrate\.office\.com.*
hulu\.playback\.edge\.bamgrid\.com
assetshuluimcom-a\.akamaihd\.net
hulu\.sc\.omtrdc\.net
infinity-c33\.youboranqs01\.com
beacons\.extremereach\.io
^.*tubi\.video
^.*tubi\.io
a-fds\.youborafds01\.com
youboranqs01\.com
amzpvxrayasset-a\.akamaihd\.net
pv-cdn.net
^.*media-amazon\.com
aiv-delivery\.net
unagi\.amazon\.com
atv-ps\.amazon\.com
pv-cdn\.net
fls-na\.amazon\.com
^.*aiv-cdn\.net
c0a299900000\.local
update\.microsoft\.com
update\.microsoft\.com\.akadns\.net
delivery\.mp\.microsoft\.com
appldnld\.apple\.com
configuration\.apple\.com
gdmf\.apple\.com
mesu\.apple\.com
oscdn\.apple\.com
osrecovery\.apple\.com
skl\.apple\.com
swcdn\.apple\.com
swdist\.apple\.com
swscan\.apple\.com
updates-http\.cdn-apple\.com
updates\.cdn-apple\.com
appldnld\.apple\.com\.edgesuite\.net
entrust\.net
digicert\.com
apple-cloudkit\.com
apple-livephotoskit\.com
gc\.apple\.com
icloud-content\.com
apple\.com
cdn-apple\.com
icloud\.com
api\.apple-cloudkit\.com
^.*appattest\.apple\.com
^.*itunes\.apple\.com
^.*mzstatic\.com
itunes\.com
music\.apple\.com
app-site-association\.cdn-apple\.com
app-site-association\.networking\.apple\.com
xp\.apple\.com
play\.google\.com
android\.com
^((alt[0-9]-mtalk\.)|(mtalk\.)|(mtalk-(staging|dev)\.))google\.com
google-analytics\.com
googleusercontent\.com
^((gvt)([0-9]))\.com
ggpht\.com
dl\.google\.com
dl-ssl\.google\.com
android\.clients\.google\.com
^(((clients)[0-9])|accounts)\.google\.(com|us)
connectivitycheck\.android\.com
android\.clients\.google\.com
device-provisioning\.googleapis\.com
omahaproxy\.appspot\.com
payments\.google\.com
googleapis\.com
notifications\.google\.com
^(pki|(crl|ocsp)\.pki)\.google\.com
ogs\.google\.com
googleapis\.com
androidmanagement\.googleapis\.com
mservice\.bankofamerica\.com
privacyportal-bofa\.my\.onetrust\.com
bankofamerica\.com
mcafee\.com
kaspersky\.com
kaspersky-labs\.com
dc1-st\.ksn\.kaspersky-labs\.com
dc1-file\.ksn\.kaspersky-labs\.com
dc1\.ksn\.kaspersky-labs\.com
olui2m\.fs\.ml\.com
ml\.com
^.*zoom\.us
^.*teams\.microsoft\.com
teams\.events\.data\.microsoft\.com
statics\.teams\.cdn\.office\.net
^.*(outlook\.)(office365|office)\.com
edge-chat\.facebook\.com
internet\.speedpay\.com
^.*hulustream\.com
cws-hulu\.conviva\.com
^.*hulu\.com
hulu\.hb\.omtrdc\.net
^.*dssott\.com
prod-ripcut-delivery\.disney-plus\.net
^(disney\.(content|connections))\.edge\.bamgrid\.com
disney\.api\.edge\.bamgrid\.com
disney\.playback\.edge\.bamgrid\.com
disney\.my\.sentry\.io
^.*amazonvideo\.com
unagi-na\.amazon\.com
events\.data\.microsoft\.com
tubi\.io
production-public\.tubi\.io
tubitv\.com
caauthservice\.state\.gov
studentaid\.gov
mohela\.com
www\.whitehouse\.gov
www\.rcsdk8\.org
rcsdk8\.powerschool\.com
www\.weaveinc\.org
^.*cdn\.nintendo\.net
^.*bitdefender\.net

                Make sure to upvote

                1 Reply Last reply Reply Quote 0
                • JonathanLeeJ
                  JonathanLee
                  last edited by JonathanLee

                  update researching

                  added to stop storing misses for real time checks. It is really not needed its a cache item as it is in real time so why store in the cache.

                  acl no_miss url_regex -i ^.*gateway\.facebook\.com\/ws\/realtime\?
miss_access deny no_miss

                  researching vary expired header issues config seems to have negative effect on performance I have sense removed this test

                  vary_ignore_expire on

                  acl manager proto cache_object
acl localhost src 192.168.1.1/32
acl https_login url_regex -i ^https.*(login|Login).*
acl no_miss url_regex -i ^.*gateway\.facebook\.com\/ws\/realtime\?
acl CONNECT method CONNECT
acl wuCONNECT dstdomain www.update.microsoft.com
acl wuCONNECT dstdomain sls.microsoft.com
http_access allow CONNECT wuCONNECT localnet
http_access allow CONNECT wuCONNECT localhost
http_access allow windowsupdate localnet
http_access allow windowsupdate localhost
http_access deny manager

acl BrokenButTrustedServers dstdomain "/usr/local/pkg/dstdom.broken"
acl DomainMismatch ssl_error SQUID_X509_V_ERR_DOMAIN_MISMATCH
sslproxy_cert_error allow BrokenButTrustedServers DomainMismatch
sslproxy_cert_error deny all

acl splice_only src 192.168.1.8 #Tasha iPhone
acl splice_only src 192.168.1.10 #Jon iPhone
acl splice_only src 192.168.1.11 #Amazon Fire
acl splice_only src 192.168.1.15 #Tasha HP
acl splice_only src 192.168.1.16 #iPad

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"

acl markBumped annotate_client bumped=true
acl bump_only src 192.168.1.3 #webtv
acl bump_only src 192.168.1.4 #toshiba
acl bump_only src 192.168.1.5 #imac
acl bump_only src 192.168.1.9 #macbook
acl bump_only src 192.168.1.13 #dell

cache deny https_login
read_ahead_gap 32 KB
negative_ttl 1 second
connect_timeout 30 seconds
request_timeout 60 seconds
half_closed_clients off
shutdown_lifetime 10 seconds
negative_dns_ttl 1 seconds
ignore_unknown_nameservers on
pipeline_prefetch 100
vary_ignore_expire on


ssl_bump peek step1
miss_access deny no_miss 
ssl_bump splice https_login
ssl_bump splice splice_only
ssl_bump splice NoSSLIntercept
ssl_bump bump bump_only markBumped
ssl_bump stare all

acl markedBumped note bumped true
url_rewrite_access deny markedBumped
http_access deny all

#acl SSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.bump"
#ssl_bump bump SSLIntercept

                  Make sure to upvote

                  1 Reply Last reply Reply Quote 0
                  • JonathanLeeJ
                    JonathanLee
                    last edited by

                    Adaptions made ordering of placement of
                    refresh_pattern -i squid.internal 10080 80% 79900 override-lastmod override-expire ignore-reload ignore-no-store ignore-must-revalidate ignore-private ignore-auth

                    testing :
                    refresh_all_ims on
                    reload_into_ims on

                    This seems to get a lot more hits this way.

                    #cachemgr_passwd disable offline_toggle reconfigure shutdown
                    #cachemgr_passwd Secret all

                    ability to control what can be accessed inside of cachemgr.cgi if you have this enabled

                    acl getmethod method GET

acl windowsupdate dstdomain windowsupdate.microsoft.com
acl windowsupdate dstdomain .update.microsoft.com
acl windowsupdate dstdomain download.windowsupdate.com
acl windowsupdate dstdomain redir.metaservices.microsoft.com
acl windowsupdate dstdomain images.metaservices.microsoft.com
acl windowsupdate dstdomain c.microsoft.com
acl windowsupdate dstdomain www.download.windowsupdate.com
acl windowsupdate dstdomain wustat.windows.com
acl windowsupdate dstdomain crl.microsoft.com
acl windowsupdate dstdomain sls.microsoft.com
acl windowsupdate dstdomain productactivation.one.microsoft.com
acl windowsupdate dstdomain ntservicepack.microsoft.com
acl windowsupdate dstdomain dc1-st.ksn.kaspersky-labs.com
acl windowsupdate dstdomain dc1-file.ksn.kaspersky-labs.com
acl windowsupdate dstdomain dc1.ksn.kaspersky-labs.com

acl rewritedoms dstdomain .facebook.com .akamaihd.net .fbcdn.net .google.com .static.com .apple.com .oracle.com .sun.com .java.com .adobe.com .steamstatic.com .steampowered.com .steamcontent.com .google.com

store_id_program /usr/local/libexec/squid/storeid_file_rewrite /var/squid/storeid/storeid_rewrite.txt
store_id_children 10 startup=5 idle=1 concurrency=0
always_direct allow !getmethod
store_id_access deny connect
store_id_access deny !getmethod
store_id_access allow rewritedoms
refresh_all_ims on
reload_into_ims on

refresh_pattern -i squid\.internal 10080 80% 79900 override-lastmod override-expire ignore-reload ignore-no-store ignore-must-revalidate ignore-private ignore-auth

#APPLE STUFF
refresh_pattern -i apple.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip|dist)$ 0 80% 43200  refresh-ims

#apple update
refresh_pattern -i (download|adcdownload).apple.com/.*\.(pkg|dmg) 4320 100% 43200 
refresh_pattern -i appldnld\.apple\.com 129600 100% 129600     
refresh_pattern -i phobos\.apple\.com 129600 100% 129600     
refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600     

# Updates: Windows
refresh_pattern -i microsoft.com/..(cab|exe|msi|msu|msf|asf|wma|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i windowsupdate.com/..(cab|exe|msi|msu|msf|asf|wma|wmv)|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i windows.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i .*windowsupdate.com/.*\.(cab|exe) 259200 100% 259200   
refresh_pattern -i .*update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 259200 100% 259200   
refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 
refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200
#windows update NEW UPDATE 0.04
refresh_pattern update.microsoft.com/.*\.(cab|exe) 43200 100% 129600    
refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf) 4320 100% 43200  
refresh_pattern update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern -i \.update.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.download.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.ws.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       

refresh_pattern ([^.]+\.)?(cs|content[1-9]|hsar|content-origin|client-download).[steampowered|steamcontent].com/.*\.* 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern ([^.]+\.)?.akamai.steamstatic.com/.*\.* 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod

refresh_pattern -i ([^.]+\.)?.adobe.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.java.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.sun.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.oracle.com/.*\.(zip|exe|tar.gz) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod

refresh_pattern -i appldnld\.apple\.com 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?apple.com/.*\.(ipa) 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
 
refresh_pattern -i ([^.]+\.)?.google.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern -i ([^.]+\.)?g.static.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK
refresh_pattern ^http?://*.facebook.com/*  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK IMAGES  
refresh_pattern -i pixel.facebook.com..(jpg|png|gif|ico|css|js)  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern -i .akamaihd.net..(jpg|png|gif|ico|css|js) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private   
refresh_pattern -i (facebook.com).(jpg|png|gif) 10080 80% 43200 store-stale override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private 
refresh_pattern static.(xx|ak).fbcdn.net.(jpg|gif|png) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern ^https?://profile.ak.fbcdn.net*.(jpg|gif|png) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK VIDEO
refresh_pattern -i .video.ak.fbcdn.net.*.(mp4|flv|mp3|amf) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private   
refresh_pattern (audio|video)/(webm|mp4) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private


range_offset_limit 512 MB windowsupdate
maximum_object_size 512 MB windowsupdate
range_offset_limit 0
quick_abort_min -1 KB

                    acl manager proto cache_object
acl localhost src 192.168.1.1/32
#cachemgr_passwd disable offline_toggle reconfigure shutdown
#cachemgr_passwd secret all
acl https_login url_regex -i ^https.*(login|Login).*
acl no_miss url_regex -i ^.*gateway\.facebook\.com\/ws\/realtime\?
acl CONNECT method CONNECT
acl wuCONNECT dstdomain www.update.microsoft.com
acl wuCONNECT dstdomain sls.microsoft.com
http_access allow CONNECT wuCONNECT localnet
http_access allow CONNECT wuCONNECT localhost
http_access allow windowsupdate localnet
http_access allow windowsupdate localhost
http_access deny manager

acl BrokenButTrustedServers dstdomain "/usr/local/pkg/dstdom.broken"
acl DomainMismatch ssl_error SQUID_X509_V_ERR_DOMAIN_MISMATCH
sslproxy_cert_error allow BrokenButTrustedServers DomainMismatch
sslproxy_cert_error deny all

acl splice_only src 192.168.1.8 #Tasha iPhone
acl splice_only src 192.168.1.10 #Jon iPhone
acl splice_only src 192.168.1.11 #Amazon Fire
acl splice_only src 192.168.1.15 #Tasha HP
acl splice_only src 192.168.1.16 #iPad

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"

acl markBumped annotate_client bumped=true
acl bump_only src 192.168.1.3 #webtv
acl bump_only src 192.168.1.4 #toshiba
acl bump_only src 192.168.1.5 #imac
acl bump_only src 192.168.1.9 #macbook
acl bump_only src 192.168.1.13 #dell

cache deny https_login
read_ahead_gap 32 KB
negative_ttl 1 second
connect_timeout 30 seconds
request_timeout 60 seconds
half_closed_clients off
shutdown_lifetime 10 seconds
negative_dns_ttl 1 seconds
ignore_unknown_nameservers on
pipeline_prefetch 100

ssl_bump peek step1
miss_access deny no_miss 
ssl_bump splice https_login
ssl_bump splice splice_only
ssl_bump splice NoSSLIntercept
ssl_bump bump bump_only markBumped
ssl_bump stare all

acl markedBumped note bumped true
url_rewrite_access deny markedBumped
http_access deny all

#acl SSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.bump"
#ssl_bump bump SSLIntercept

                    Make sure to upvote

                    JonathanLeeJ 1 Reply Last reply Reply Quote 0
                    • JonathanLeeJ
                      JonathanLee @JonathanLee
                      last edited by

                      @JonathanLee

                      ssl_bump peek step1
miss_access deny no_miss 
ssl_bump splice https_login markBumped
ssl_bump splice splice_only markBumped
ssl_bump splice NoSSLIntercept markBumped
ssl_bump bump bump_only
ssl_bump stare all

acl markedBumped note bumped true
url_rewrite_access deny markedBumped

                      This seems to have more hit 304s

                      Make sure to upvote

                      1 Reply Last reply Reply Quote 0
                      • JonathanLeeJ
                        JonathanLee
                        last edited by 

                        refresh_pattern -i .(video-lax\d\-\d\.xx|video\.ak)\.fbcdn.net.*\.(mp4|flv|mp3|amf) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

                        This works better for cache of videos and acceleration the old pattern no longer worked as of a couple days ago they adapted it to have lax for my area and added numerical info in the URL this radically increased cache and acceleration times for me with use of their new URL scheme

                        Make sure to upvote

                        1 Reply Last reply Reply Quote 0
                        • JonathanLeeJ
                          JonathanLee
                          last edited by JonathanLee

                          Fix for speed issues was to use domain acl for most of the no bump splice items, this drastically speeds up system also.

                          Researching CIPHERs also with this please ignore cipher changes this was my tests with cipher testing site for more use of high ciphers

                          acl localhost src 192.168.1.1/32
#cachemgr_passwd disable offline_toggle reconfigure shutdown
#cachemgr_passwd REDACTED! all
acl no_miss url_regex -i gateway\.facebook\.com\/ws\/realtime\?
acl no_miss url_regex -i web-chat-e2ee\.facebook\.com\/ws\/chat	
acl CONNECT method CONNECT
acl wuCONNECT dstdomain www.update.microsoft.com
acl wuCONNECT dstdomain sls.microsoft.com
http_access allow CONNECT wuCONNECT localnet
http_access allow CONNECT wuCONNECT localhost
http_access allow windowsupdate localnet
http_access allow windowsupdate localhost
http_access deny manager

acl BrokenButTrustedServers dstdomain "/usr/local/pkg/dstdom.broken"
acl DomainMismatch ssl_error SQUID_X509_V_ERR_DOMAIN_MISMATCH
sslproxy_cert_error allow BrokenButTrustedServers DomainMismatch
sslproxy_cert_error deny all

acl splice_only src 192.168.1.8 #Tasha iPhone
acl splice_only src 192.168.1.10 #Jon iPhone
acl splice_only src 192.168.1.11 #Amazon Fire
acl splice_only src 192.168.1.15 #Tasha HP
acl splice_only src 192.168.1.16 #iPad

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/reg.url.nobump"
acl NoBumpDNS dstdomain "/usr/local/pkg/dns.nobump"

acl markBumped annotate_client bumped=true
acl bump_only src 192.168.1.3 #webtv
acl bump_only src 192.168.1.4 #toshiba
acl bump_only src 192.168.1.5 #imac
acl bump_only src 192.168.1.9 #macbook
acl bump_only src 192.168.1.13 #dell

ssl_bump peek step1
miss_access deny no_miss 
ssl_bump splice https_login
ssl_bump splice splice_only
ssl_bump splice NoBumpDNS
ssl_bump splice NoSSLIntercept
ssl_bump bump bump_only markBumped
ssl_bump stare all

acl markedBumped note bumped true
url_rewrite_access deny markedBumped

read_ahead_gap 64 KB
negative_ttl 1 second
connect_timeout 30 seconds
request_timeout 60 seconds
half_closed_clients off
shutdown_lifetime 10 seconds
negative_dns_ttl 1 seconds
ignore_unknown_nameservers on
pipeline_prefetch 100

#acl SSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.bump"
#ssl_bump bump SSLIntercept

                          acl getmethod method GET

tls_outgoing_options cipher=HIGH:MEDIUM:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS

tls_outgoing_options options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE

acl windowsupdate dstdomain windowsupdate.microsoft.com
acl windowsupdate dstdomain .update.microsoft.com
acl windowsupdate dstdomain download.windowsupdate.com
acl windowsupdate dstdomain redir.metaservices.microsoft.com
acl windowsupdate dstdomain images.metaservices.microsoft.com
acl windowsupdate dstdomain c.microsoft.com
acl windowsupdate dstdomain www.download.windowsupdate.com
acl windowsupdate dstdomain wustat.windows.com
acl windowsupdate dstdomain crl.microsoft.com
acl windowsupdate dstdomain sls.microsoft.com
acl windowsupdate dstdomain productactivation.one.microsoft.com
acl windowsupdate dstdomain ntservicepack.microsoft.com
acl windowsupdate dstdomain dc1-st.ksn.kaspersky-labs.com
acl windowsupdate dstdomain dc1-file.ksn.kaspersky-labs.com
acl windowsupdate dstdomain dc1.ksn.kaspersky-labs.com

acl rewritedoms dstdomain .facebook.com .akamaihd.net .fbcdn.net .google.com .static.com .apple.com .oracle.com .sun.com .java.com .adobe.com .steamstatic.com .steampowered.com .steamcontent.com .google.com

store_id_program /usr/local/libexec/squid/storeid_file_rewrite /var/squid/storeid/storeid_rewrite.txt
store_id_children 10 startup=5 idle=1 concurrency=0
always_direct allow !getmethod
store_id_access deny connect
store_id_access deny !getmethod
store_id_access allow rewritedoms
reload_into_ims on
max_stale 20 years
minimum_expiry_time 0


refresh_pattern -i squid\.internal 10080 80% 79900 override-lastmod override-expire ignore-reload ignore-no-store ignore-must-revalidate ignore-private ignore-auth

#APPLE STUFF
refresh_pattern -i apple.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip|dist)$ 0 80% 43200  refresh-ims

#apple update
refresh_pattern -i (download|adcdownload).apple.com/.*\.(pkg|dmg) 4320 100% 43200 
refresh_pattern -i appldnld\.apple\.com 129600 100% 129600     
refresh_pattern -i phobos\.apple\.com 129600 100% 129600     
refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600     

# Updates: Windows
refresh_pattern -i microsoft.com/..(cab|exe|msi|msu|msf|asf|wma|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i windowsupdate.com/..(cab|exe|msi|msu|msf|asf|wma|wmv)|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i windows.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i .*windowsupdate.com/.*\.(cab|exe) 259200 100% 259200   
refresh_pattern -i .*update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 259200 100% 259200   
refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 
refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200
#windows update NEW UPDATE 0.04
refresh_pattern update.microsoft.com/.*\.(cab|exe) 43200 100% 129600    
refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf) 4320 100% 43200  
refresh_pattern update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern -i \.update.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.download.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.ws.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       

refresh_pattern ([^.]+\.)?(cs|content[1-9]|hsar|content-origin|client-download).[steampowered|steamcontent].com/.*\.* 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern ([^.]+\.)?.akamai.steamstatic.com/.*\.* 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod

refresh_pattern -i ([^.]+\.)?.adobe.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.java.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.sun.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.oracle.com/.*\.(zip|exe|tar.gz) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod

refresh_pattern -i appldnld\.apple\.com 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?apple.com/.*\.(ipa) 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
 
refresh_pattern -i ([^.]+\.)?.google.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern -i ([^.]+\.)?g.static.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK
refresh_pattern ^http?://*.facebook.com/*  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK IMAGES  
refresh_pattern -i pixel.facebook.com..(jpg|png|gif|ico|css|js)  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern -i .akamaihd.net..(jpg|png|gif|ico|css|js) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private   
refresh_pattern -i (facebook.com).(jpg|png|gif) 10080 80% 43200 store-stale override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private 
refresh_pattern static.(xx|ak).fbcdn.net.(jpg|gif|png) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern (scontent\-lax\d\-\d\.xx|.ak)\.fbcdn.net.*(jpg|gif|png) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

refresh_pattern ^https?://profile.ak.fbcdn.net*.(jpg|gif|png) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK VIDEO
refresh_pattern -i .(video-lax\d\-\d\.xx|video\.ak)\.fbcdn.net.*\.(mp4|flv|mp3|amf) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern (audio|video)/(webm|mp4) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
acl https_login url_regex -i ^https.*(login|Login).*
cache deny https_login

range_offset_limit 512 MB windowsupdate
range_offset_limit 4 MB
range_offset_limit 0

                          quick_abort_min -1 KB

                          Files that go with this as an example

                          .dssott.com
.prod-ripcut-delivery.disney-plus.net
.disney.api.edge.bamgrid.com
.disney.playback.edge.bamgrid.com
.disney.my.sentry.io
.hulustream.com
.hulu.com
.hulu.hb.omtrdc.net
.hulu.playback.edge.bamgrid.com
.assetshuluimcom-a.akamaihd.net
.hulu.sc.omtrdc.net
.beacons.extremereach.io
.tubi.video
.tubi.io
.tubitv.com
.a-fds.youborafds01.com
.license.adrise.tv
.amzpvxrayasset-a.akamaihd.net
.pv-cdn.net
.media-amazon.com
.aiv-delivery.net
.unagi.amazon.com
.atv-ps.amazon.com
.pv-cdn.net
.fls-na.amazon.com
.aiv-cdn.net
.c0a299900000.local
.conviva.com
.cdn.office.net
.bitdefender.net
.azure-devices.net
.substrate.office.com
.update.microsoft.com
.update.microsoft.com.akadns.net
.delivery.mp.microsoft.com
.appldnld.apple.com
.configuration.apple.com
.gdmf.apple.com
.mesu.apple.com
.oscdn.apple.com
.osrecovery.apple.com
.skl.apple.com
.swcdn.apple.com
.swdist.apple.com
.swscan.apple.com
.appldnld.apple.com.edgesuite.net
.entrust.net
.digicert.com
.apple-cloudkit.com
.apple-livephotoskit.com
.gc.apple.com
.icloud-content.com
.cdn-apple.com
.icloud.com
.appattest.apple.com
.itunes.apple.com
.mzstatic.com
.itunes.com
.music.apple.com
.app-site-association.networking.apple.com
.xp.apple.com
.play.google.com
.android.com
.google-analytics.com
.googleusercontent.com
.ggpht.com
.dl.google.com
.dl-ssl.google.com
.android.clients.google.com
.android.clients.google.com
.omahaproxy.appspot.com
.payments.google.com
.googleapis.com
.notifications.google.com
.ogs.google.com
.googleapis.com
.privacyportal-bofa.my.onetrust.com
.bankofamerica.com
.mcafee.com
.kaspersky.com
.kaspersky-labs.com
.ml.com
.zoom.us
.teams.microsoft.com
.edge-chat.facebook.com
.internet.speedpay.com
.amazonvideo.com
.unagi-na.amazon.com
.events.data.microsoft.com
.caauthservice.state.gov
.studentaid.gov
.mohela.com
www.whitehouse.gov
www.rcsdk8.org
.rcsdk8.powerschool.com
www.weaveinc.org
.cdn.nintendo.net

                          regular expression file

                          #Sites to be splice
(disney\.(content|connections))\.edge\.bamgrid\.com
web-chat-e2ee\.facebook\.com\/ws\/chat	
gateway\.facebook\.com\/ws\/realtime\?
^((alt[0-9]-mtalk\.)|(mtalk\.)|(mtalk-(staging|dev)\.))google\.com
^((gvt)([0-9]))\.com
^(((clients)[0-9])|accounts)\.google\.(com|us)
^(pki|(crl|ocsp)\.pki)\.google\.com
(outlook\.)(office365|office)\.com
infinity-c[0-9][0-9]\.youboranqs[0-9][0-9]\.com

                          This change has a major improvement

                          Use of command

                          squid -k parse

                          helped direct me to use dstdomain acts over the hundreds of regex items that was causing performance issues.

                          Make sure to upvote

                          1 Reply Last reply Reply Quote 0
                          • JonathanLeeJ JonathanLee referenced this topic on
                          • JonathanLeeJ
                            JonathanLee
                            last edited by

                            Continued Research:

                            Changes to following have a massive increase in hit ratios:

                            Local Cache

                            acl block_hours time 00:30-05:00
ssl_bump terminate all block_hours
http_access deny all block_hours
acl getmethod method GET
acl to_ipv6 dst ipv6
acl from_ipv6 src ipv6

tls_outgoing_options cipher=HIGH:MEDIUM:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS
tls_outgoing_options options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE

acl HttpAccess dstdomain "/usr/local/pkg/http.access"
acl windowsupdate dstdomain "/usr/local/pkg/windowsupdate"
acl rewritedoms dstdomain "/usr/local/pkg/desdom"

store_id_program /usr/local/libexec/squid/storeid_file_rewrite /var/squid/storeid/storeid_rewrite.txt
store_id_children 10 startup=5 idle=1 concurrency=0


#always_direct allow !getmethod #CHANGE HERE NOT USING SQUID WITH PEERS


#store_id_access deny connect #CHANGE HERE


store_id_access deny !getmethod
store_id_access allow rewritedoms


#store_id_access deny all #CHANGE HERE

refresh_all_ims on
reload_into_ims on
max_stale 20 years
minimum_expiry_time 0

refresh_pattern -i squid\.internal 10080 80% 79900 override-lastmod override-expire ignore-reload ignore-no-store ignore-private

#FACEBOOK
refresh_pattern ^https.*.facebook.com/* 10080 80% 43200

#FACEBOOK IMAGES  
refresh_pattern -i pixel.facebook.com..(jpg|png|gif|ico|css|js|jpg?) 10080 80% 43200
refresh_pattern -i .akamaihd.net..(jpg|png|gif|ico|css|js|jpg?) 10080 80% 43200 
refresh_pattern -i facebook.com.(jpg|png|gif|jpg?) 10080 80% 43200 store-stale
refresh_pattern static.(xx|ak).fbcdn.net.(jpg|gif|png|jpg?) 10080 80% 43200
refresh_pattern ^https.*profile.ak.fbcdn.net.*(jpg|gif|png|jpg?) 10080 80% 43200
refresh_pattern ^https.*fbcdn.net.*(jpg|gif|png|jpg?) 10080 80% 43200

#FACEBOOK VIDEO
refresh_pattern -i .video.ak.fbcdn.net.*.(mp4|flv|mp3|amf) 10080 80% 43200
refresh_pattern (audio|video)/(webm|mp4) 10080 80% 43200

#APPLE STUFF
refresh_pattern -i apple.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip|dist)$ 0 80% 43200  refresh-ims

#apple update
refresh_pattern -i (download|adcdownload).apple.com/.*\.(pkg|dmg) 4320 100% 43200
refresh_pattern -i appldnld\.apple\.com 129600 100% 129600
refresh_pattern -i phobos\.apple\.com 129600 100% 129600
refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600

# Updates: Windows
refresh_pattern -i microsoft.com/..(cab|exe|msi|msu|msf|asf|wma|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i windowsupdate.com/..(cab|exe|msi|msu|msf|asf|wma|wmv)|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i windows.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i .*windowsupdate.com/.*\.(cab|exe) 259200 100% 259200   
refresh_pattern -i .*update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 259200 100% 259200   
refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 
refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200
#windows update NEW UPDATE 0.04
refresh_pattern update.microsoft.com/.*\.(cab|exe) 43200 100% 129600    
refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf) 4320 100% 43200  
refresh_pattern update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern -i \.update.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.download.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.ws.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
    
refresh_pattern ([^.]+\.)?(cs|content[1-9]|hsar|content-origin|client-download).[steampowered|steamcontent].com/.*\.* 43200 100% 43200     
refresh_pattern ([^.]+\.)?.akamai.steamstatic.com/.*\.* 43200 100% 43200

refresh_pattern -i ([^.]+\.)?.adobe.com/.*\.(zip|exe) 43200 100% 43200
refresh_pattern -i ([^.]+\.)?.java.com/.*\.(zip|exe) 43200 100% 43200
refresh_pattern -i ([^.]+\.)?.sun.com/.*\.(zip|exe) 43200 100% 43200
refresh_pattern -i ([^.]+\.)?.oracle.com/.*\.(zip|exe|tar.gz) 43200 100% 43200

refresh_pattern -i appldnld\.apple\.com 43200 100% 43200
refresh_pattern -i ([^.]+\.)?apple.com/.*\.(ipa) 43200 100% 43200
 
refresh_pattern -i ([^.]+\.)?.google.com/.*\.(exe|crx) 10080 80% 43200
refresh_pattern -i ([^.]+\.)?g.static.com/.*\.(exe|crx) 10080 80% 43200

acl https_login url_regex -i ^https.*(login|Login).*
cache deny https_login

range_offset_limit 512 MB windowsupdate
range_offset_limit 4 MB
range_offset_limit 0
quick_abort_min -1 KB

                            cachemgr_passwd disable offline_toggle reconfigure shutdown
cachemgr_passwd CLASSFIED_REDACTED all
eui_lookup on
acl no_miss url_regex -i gateway\.facebook\.com\/ws\/realtime\?
acl no_miss url_regex -i web-chat-e2ee\.facebook\.com\/ws\/chat
acl CONNECT method CONNECT
acl wuCONNECT dstdomain www.update.microsoft.com
acl wuCONNECT dstdomain sls.microsoft.com
http_access allow CONNECT wuCONNECT localnet
http_access allow CONNECT wuCONNECT localhost
http_access allow windowsupdate localnet
http_access allow windowsupdate localhost
http_access allow HttpAccess localnet
http_access allow HttpAccess localhost
http_access deny manager
http_access deny to_ipv6
http_access deny from_ipv6

acl BrokenButTrustedServers dstdomain "/usr/local/pkg/dstdom.broken"
acl DomainMismatch ssl_error SQUID_X509_V_ERR_DOMAIN_MISMATCH
sslproxy_cert_error allow BrokenButTrustedServers DomainMismatch
sslproxy_cert_error deny all

acl splice_only src 192.168.1.8 #Tasha iPhone
acl splice_only src 192.168.1.10 #Jon iPhone
acl splice_only src 192.168.1.11 #Amazon Fire
acl splice_only src 192.168.1.15 #Tasha HP
acl splice_only src 192.168.1.16 #iPad

acl splice_only_mac arp REDACTED MAC ADDRESS
acl splice_only_mac arp REDACTED MAC ADDRESS
acl splice_only_mac arp REDACTED MAC ADDRESS
acl splice_only_mac arp REDACTED MAC ADDRESS
acl splice_only_mac arp REDACTED MAC ADDRESS

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/reg.url.nobump"
acl NoBumpDNS dstdomain "/usr/local/pkg/dns.nobump"

acl markBumped annotate_client bumped=true
acl active_use annotate_client active=true
acl bump_only src 192.168.1.3 #webtv
acl bump_only src 192.168.1.4 #toshiba
acl bump_only src 192.168.1.5 #imac
acl bump_only src 192.168.1.9 #macbook
acl bump_only src 192.168.1.13 #dell

acl bump_only_mac arp REDACTED MAC ADDRESS
acl bump_only_mac arp REDACTED MAC ADDRESS
acl bump_only_mac arp REDACTED MAC ADDRESS
acl bump_only_mac arp REDACTED MAC ADDRESS
acl bump_only_mac arp REDACTED MAC ADDRESS

ssl_bump peek step1
miss_access deny no_miss active_use
ssl_bump splice https_login active_use
ssl_bump splice splice_only_mac splice_only active_use
ssl_bump splice NoBumpDNS active_use
ssl_bump splice NoSSLIntercept active_use
ssl_bump bump bump_only_mac bump_only active_use
acl activated note active_use true
ssl_bump terminate !activated

acl markedBumped note bumped true
url_rewrite_access deny markedBumped

#workers 3
#read_ahead_gap 32 KB

negative_ttl 1 second
connect_timeout 30 seconds
request_timeout 60 seconds

#half_closed_clients off

shutdown_lifetime 10 seconds
negative_dns_ttl 1 seconds

#ignore_unknown_nameservers on
#client_persistent_connections off
#server_persistent_connections off

pipeline_prefetch 100

#acl SSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.bump"
#ssl_bump bump SSLIntercept

                            Also changes were made to utilize a SWAP partition I created a FREEBSD based swap on an external drive and or you can use a SSD drive.

                            WARNING IF YOU DO NOT KNOW HOW TO CORRECTLY PARTITION A DRVIE DO NOT ATTEMPT THIS AS YOU CAN DESTROY ALL SOFTWARE.

                            I had to use the SWAP on the SSD and or use an external drive as a swap. This was done to help with updates to ClamAV as it will start to swap until update is competed.

                            /etc/fstab

                            # Device		Mountpoint	FStype	Options		Dump	Pass#
/dev/msdosfs/EFISYS	/boot/efi	msdosfs	rw,noatime,noauto	0	0
/dev/msdosfs/DTBFAT0	/boot/msdos	msdosfs	rw,noatime,noauto	0	0
/dev/da0		none	swap	sw		0	0

                            Make sure to upvote

                            1 Reply Last reply Reply Quote 0
                            • JonathanLeeJ
                              JonathanLee
                              last edited by

                              This seems to improve speeds

                              http_upgrade_request_protocols websocket allow all 
accept_filter httpready
accept_filter dataready
collapsed_forwarding on
half_closed_clients off
pipeline_prefetch 6

                              Make sure to upvote

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post
                              Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.