Unofficial Squid Custom Refresh Patterns
-
After researching I finally found a great set of custom refresh-patterns for Windows updates and they work. This saves time and energy as it is only downloading the huge updates once over the WAN and after it is on your LAN. If anyone wants to check them out I am getting good hits with this list. This is great for networks that have a-lot of Windows devices, all the major updates are the same so you only really need it to download one time for each LAN you are using.
This took years to get this right and it is always adapting, that along side purchasing some great Squid reference books and I still was at times bugging the Squid developers via email to glean more information about this.
Local Cache/ Dynamic and Update Content/Custom refresh_patterns
#---for testing multiple squid cache processes on 2100-MAX must Squid -Z the path to the second cache before useable Macro usage #if ${process_number} = 2 #cache_dir diskd /nvme/LOGS_Optane/Squid_Cache_B 32000 64 256 #endif # --- Block access during off hours (e.g. maintenance) --- acl block_hours time 00:30-05:00 ssl_bump terminate all block_hours http_access deny all block_hours deny_info TCP_RESET block_hours # --- Disable unused services --- icp_port 0 htcp_port 0 snmp_port 0 icp_access deny all htcp_access deny all snmp_access deny all # --- HTTP Method and IPv6 ACLs --- acl getmethod method GET acl to_ipv6 dst ipv6 acl from_ipv6 src ipv6 # --- TLS Outgoing Options --- tls_outgoing_options options=NO_SSLv3,NO_TLSv1,NO_TLSv1_1,SINGLE_DH_USE,SINGLE_ECDH_USE tls_outgoing_options min-version=1.2 tls_outgoing_options cipher=ECDHE+AESGCM:ECDHE+CHACHA20 tls_outgoing_options default-ca=on # --- Domain ACLs --- acl HttpAccess dstdomain "/usr/local/pkg/http.access" acl windowsupdate dstdomain "/usr/local/pkg/windowsupdate" #for use with allowing only windows update domains accessing the 512MB allowed cache # --- Store ID Config (if needed) --- #acl rewritedoms dstdomain "/usr/local/pkg/desdom" #store_id_program /usr/local/libexec/squid/storeid_file_rewrite /var/squid/storeid/storeid_rewrite.txt #store_id_children 2 startup=1 idle=1 concurrency=5 #store_id_access deny connect #store_id_access deny !getmethod #store_id_access allow rewritedoms #store_id_access deny all # --- Cache Behavior --- always_direct allow all refresh_all_ims off reload_into_ims on max_stale 1 day minimum_expiry_time 0 #----timeouts to help with large downloads windows updates read_timeout 15 minutes request_timeout 10 minutes connect_timeout 45 seconds client_lifetime 15 minutes pconn_lifetime 5 minutes # ---- SOFTWARE UPDATES ---- # Microsoft Windows Updates, Office 365, Defender, and common update files refresh_pattern -i (download\.microsoft\.com|update\.microsoft\.com|windowsupdate\.com|windows\.com|officecdn\.microsoft\.com|officecdn\.microsoftonline\.com|officeclient\.officeapps\.live\.com|akamaiedge\.net|download\.windowsupdate\.com|download\.windowsupdate\.com\.msedge\.net|wustat\.windows\.com|microsoftedge\.com|edgecastcdn\.net|azureedge\.net)/.*\.(cab|exe|msi|msu|msf|psf|zip|dat|mum|manifest|manifest\.xml|json|cat|xml) 4320 80% 43200 reload-into-ims # Microsoft Store app updates and edge CDN domains refresh_pattern -i (storeedge\.fd\.dsp\.ms|storeedgefd\.azureedge\.net|dl\.delivery\.mp\.microsoft\.com)/.*\.(appx|appxbundle|msixbundle|msix|eappx) 4320 80% 43200 reload-into-ims # Microsoft telemetry and diagnostics domains - do NOT cache refresh_pattern -i (telemetry\.microsoft\.com|vortex\.data\.microsoft\.com|watson\.telemetry\.microsoft\.com|settings\.win\.microsoft\.com|watson\.microsoft\.com|feedback\.microsoft\.com|sqm\.users\.office\.com|compatexchange\.cloudapp\.net|diagnostics\.support\.microsoft\.com|telemetry\.office\.com|telemetry\.cdn\.microsoft\.com|.*\.cdn\.microsoft\.com\/telemetry|mobile\.settings\.windows\.com|sqm\.telemetry\.microsoft\.com|ntservicepacks\.blob\.core\.windows\.net)/ 0 0% 0 reload-into-ims # --- Block caching for login URLs --- acl https_login url_regex -i login(\.php|\.aspx|\.html|\/|$) cache deny https_login # --- Range Requests Limits --- range_offset_limit 512 MB windowsupdate range_offset_limit 0 !windowsupdate # --- Abort thresholds --- quick_abort_min -1domains stored in /usr/local/pkg/windowsupdate
microsoft.com windowsupdate.com windows.com office.net xboxlive.com edgesuite.netdomains storred in /usr/local/pkg/http.access
you must also create a rule for allowing http access or this won't work sooffice.com office.net microsoft.com windowsupdate.com windows.com office.net xboxlive.com edgesuite.net apple.com entrust.net digicert.com msftconnecttest.com msftncsi.comAdvanced Features/Custom Options
#ssl_engine devcrypto #only works for older squid and devices that use a crypto chip http_access deny to_ipv6 http_access deny from_ipv6 acl local_nets dst 10.0.0.0/24 192.168.8.0/24 192.168.1.14 http_access deny local_nets request_header_access Proxy-Auto-Config deny all request_header_access Proxy-Authorization deny all request_header_access Proxy-Authenticate deny all request_header_access Proxy-Connection deny all request_header_access X-Forwarded-For deny all http_access deny !safeports http_access deny CONNECT !sslports http_access allow manager localhost http_access deny manager cachemgr_passwd disable offline_toggle reconfigure shutdown cachemgr_passwd PASSWORD HERE all #this is cleartext warning eui_lookup on acl no_miss url_regex -i gateway\.facebook\.com\/ws\/realtime\? acl no_miss url_regex -i web-chat-e2ee\.facebook\.com\/ws\/chat acl CONNECT method CONNECT acl wuCONNECT dstdomain www.update.microsoft.com acl wuCONNECT dstdomain sls.microsoft.com http_access allow CONNECT wuCONNECT localnet http_access allow CONNECT wuCONNECT localhost http_access allow CONNECT windowsupdate localnet http_access allow CONNECT windowsupdate localhost http_access allow CONNECT HttpAccess localnet http_access allow CONNECT HttpAccess localhost acl deny_rep_mime_doh rep_mime_type application/dns-message acl deny_rep_mime_doh rep_mime_type text/dns acl deny_rep_mime_doh rep_mime_type application/dns+json http_reply_access deny deny_rep_mime_doh acl doh_rfc8484 urlpath_regex -i ^/dns-query acl doh_rfc8484 urlpath_regex -i dns= acl doh_rfc8484 urlpath_regex -i ^/resolve acl doh_group any-of deny_rep_mime_doh doh_rfc8484 http_access deny doh_group acl BrokenButTrustedServers dstdomain "/usr/local/pkg/dstdom.broken" acl DomainMismatch ssl_error SQUID_X509_V_ERR_DOMAIN_MISMATCH sslproxy_cert_error allow BrokenButTrustedServers DomainMismatch sslproxy_cert_error deny all acl splice_only_ip src (ip address here) #Amazon Fire acl splice_only_mac arp (mac address here)#Amazon Fire acl bump_only_ip src (IP was here) #macbook acl bump_only_mac arp (mac address) #MacBook #===SSL Intercept Rules=== acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/reg.url.nobump" acl NoBumpDNS dstdomain -n "/usr/local/pkg/dns.nobump" #acl SSL_Intercept_Terminate dstdomain -n "/usr/local/pkg/url.bump" acl active_use annotate_client active=true #acl terminate_group any-of deny_rep_mime_doh doh_rfc8484 to_ipv6 from_ipv6 acl terminate_group any-of to_ipv6 from_ipv6 acl splice_group any-of https_login NoBumpDNS NoSSLIntercept acl splice_only_local_group all-of splice_only_mac splice_only_ip acl splice_main any-of splice_group splice_only_local_group acl bump_main all-of bump_only_mac bump_only_ip ssl_bump peek step1 ssl_bump terminate terminate_group miss_access deny no_miss active_use ssl_bump splice splice_main active_use ssl_bump bump bump_main active_use acl activated note active_use true ssl_bump terminate !activated #debug_options 28,9,33,2 #for storeid testing #===Performance== server_persistent_connections on client_persistent_connections on client_idle_pconn_timeout 1 minutes server_idle_pconn_timeout 1 minutes collapsed_forwarding on dns_timeout 5 seconds connect_timeout 10 seconds shutdown_lifetime 5 seconds negative_dns_ttl 1 minutes read_ahead_gap 128 KB pipeline_prefetch 5 #this adds some risk warning happy_eyeballs_connect_timeout 5 memory_pools on memory_pools_limit 128 MB tcp_outgoing_tos 0x10 localnet coredump_dir /nvme/LOGS_Optane/Squid_Dump #for use with my nvme drive for core dumps if you have a second drive mountedwhat I have in /usr/local/pkg/dstdom.broken
microsoft.com windowsupdate.com msftconnecttest.com msftncsi.com msedge.net aka.ms live.com akamaiedge.net akadns.net akamai.net update.microsoft.com.nsatc.net statics.teams.cdn.live.net oneclient.sfx.ms microsoftonline.com teams.microsoft.com office.com office.net bitdefender.net gsa.apple.com apple.com icloud.com icloud-content.comAgain this will only work if you have a CA installed and allow windows update advanced settings to update from other devices on your LAN.
Let me know if it works you will have to use your own mac and ip address combos and have to install certificates in your LAN systems, this will only work for devices that have a certificate authority on. This also has a lock out time.
-
Here is it running with some update memory hits it also shows refresh modified etc the longer it runs.
Statistics of use for 2 days of uptime with 4 devices.
Cachecgi
Used Checks % Matches Pattern 431 2912 14.80 -i (download\.microsoft\.com|update\.microsoft\.com|windowsupdate\.com|windows\.com|officecdn\.microsoft\.com|officecdn\.microsoftonline\.com|officeclient\.officeapps\.live\.com|akamaiedge\.net|download\.windowsupdate\.com|download\.windowsupdate\.com\.msedge\.net|wustat\.windows\.com|microsoftedge\.com|edgecastcdn\.net|azureedge\.net)/.*\.(cab|exe|msi|msu|msf|psf|zip|dat|mum|manifest|manifest\.xml|json|cat|xml) 0 2481 0.00 -i (storeedge\.fd\.dsp\.ms|storeedgefd\.azureedge\.net|dl\.delivery\.mp\.microsoft\.com)/.*\.(appx|appxbundle|msixbundle|msix|eappx) 0 2481 0.00 -i (telemetry\.microsoft\.com|vortex\.data\.microsoft\.com|watson\.telemetry\.microsoft\.com|settings\.win\.microsoft\.com|watson\.microsoft\.com|feedback\.microsoft\.com|sqm\.users\.office\.com|compatexchange\.cloudapp\.net|diagnostics\.support\.microsoft\.com|telemetry\.office\.com|telemetry\.cdn\.microsoft\.com|.*\.cdn\.microsoft\.com\/telemetry|mobile\.settings\.windows\.com|sqm\.telemetry\.microsoft\.com|ntservicepacks\.blob\.core\.windows\.net)/ 0 2481 0.00 ^ftp: 0 2481 0.00 ^gopher: 1198 2481 48.29 -i (/cgi-bin/|\?) 1283 1283 100.00 . RefreshCheck calls per protocol Protocol #Calls %Calls HTTP 331 8.77 ICP 0 0.00 HTCP 0 0.00 Cache Digests 0 0.00 On Store 3442 91.23 RefreshCheck histograms for various protocols HTTP histogram: Count %Total Category 0 0.00 Fresh: request max-stale wildcard 0 0.00 Fresh: request max-stale value 265 80.06 Fresh: expires time not reached 12 3.63 Fresh: refresh_pattern last-mod factor percentage 0 0.00 Fresh: refresh_pattern min value 0 0.00 Fresh: refresh_pattern override-expires 0 0.00 Fresh: refresh_pattern override-lastmod 14 4.23 Stale: response has must-revalidate 0 0.00 Stale: changed reload into IMS 0 0.00 Stale: request has no-cache directive 9 2.72 Stale: age exceeds request max-age value 0 0.00 Stale: expires time reached 5 1.51 Stale: refresh_pattern max age rule 0 0.00 Stale: refresh_pattern last-mod factor percentage 4 1.21 Stale: by default On Store histogram: Count %Total Category 0 0.00 Fresh: request max-stale wildcard 0 0.00 Fresh: request max-stale value 823 23.91 Fresh: expires time not reached 16 0.46 Fresh: refresh_pattern last-mod factor percentage 0 0.00 Fresh: refresh_pattern min value 0 0.00 Fresh: refresh_pattern override-expires 0 0.00 Fresh: refresh_pattern override-lastmod 0 0.00 Stale: response has must-revalidate 0 0.00 Stale: changed reload into IMS 0 0.00 Stale: request has no-cache directive 0 0.00 Stale: age exceeds request max-age value 14 0.41 Stale: expires time reached 16 0.46 Stale: refresh_pattern max age rule 51 1.48 Stale: refresh_pattern last-mod factor percentage 2519 73.18 Stale: by defaultI have got about 14% being allowed to be used. StoreID had me with a lot more but it had some lag when it was running for some reason or another that is why it is commented out. I have not have many updated over Microsoft store will update you after that runs for about a month that might require storeid to work. Telemetry is set to 0 0 0 do not cache on purpose.
-
Can you check https://github.com/mmd123/squid-cache-dynamic_refresh-list and push your updates
-
@aGeekhere said in Unofficial Squid Custom Refresh Patterns:
https://github.com/mmd123/squid-cache-dynamic_refresh-list
I added them thanks.
Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.