Dynamic cache not work

Ninno · Oct 14, 2016, 6:35 AM

Hi,
I enabled the dynamic cache by following the guides on the net, this is my configuration.

# This file is automatically generated by pfSense
# Do not edit manually !

http_port 172.16.0.1:3128
http_port 127.0.0.1:3128 intercept
icp_port 0
dns_v4_first off
pid_filename /var/run/squid/squid.pid
cache_effective_user squid
cache_effective_group proxy
error_default_language it
icon_directory /usr/local/etc/squid/icons
visible_hostname PROXY
cache_mgr admin@localhost
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none
netdb_filename /var/squid/logs/netdb.state
pinger_enable off
pinger_program /usr/local/libexec/squid/pinger

logfile_rotate 30
debug_options rotate=30
shutdown_lifetime 3 seconds
# Allow local network(s) on interface(s)
acl localnet src  172.16.0.0/16
forwarded_for on
httpd_suppress_version_string on
uri_whitespace strip

#windows update
refresh_pattern windowsupdate.com/.*\.(cab|exe)                     43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
refresh_pattern update.microsoft.com/.*\.(cab|exe)                  43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
refresh_pattern download.microsoft.com/.*\.(cab|exe)                43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip|esd) 4320 80% 43200 reload-into-ims
refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf|esd) 4320 100% 43200 reload-into-ims 
refresh_pattern update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf|esd) 4320 100% 43200 reload-into-ims
refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 4320 100% 43200 reload-into-ims

#ALL
refresh_pattern -i (\.|-)(mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t|esd)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth

cache_mem 1024 MB
maximum_object_size_in_memory 512 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
minimum_object_size 0 KB
maximum_object_size 5000 MB
cache_dir ufs /var/squid/cache 80000 16 256
offline_mode off
cache_swap_low 90
cache_swap_high 95
cache allow all
# Add any of your own refresh_pattern entries above these.
refresh_pattern ^ftp:    1440  20%  10080
refresh_pattern ^gopher:  1440  0%  1440
refresh_pattern -i (/cgi-bin/|\?) 0  0%  0
refresh_pattern .    0  20%  4320

#Remote proxies

# Setup some default acls
# From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.
# acl localhost src 127.0.0.1/32
acl allsrc src all
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 3129 1025-65535 
acl sslports port 443 563  

# From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.
#acl manager proto cache_object

acl purge method PURGE
acl connect method CONNECT

# Define protocols used for redirects
acl HTTP proto HTTP
acl HTTPS proto HTTPS
http_access allow manager localhost

http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

# Always allow localhost connections
# From 3.2 further configuration cleanups have been done to make things easier and safer.
# The manager, localhost, and to_localhost ACL definitions are now built-in.
# http_access allow localhost

quick_abort_min -1 KB
quick_abort_max 0 KB
request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc

# Reverse Proxy settings

# Custom options before auth

# Setup allowed ACLs
# Allow local network(s) on interface(s)
http_access allow localnet
# Default block all to be sure
http_access deny allsrc

It all seems ok, but does not work …
I'm installing 30 new PCs and updates are always downloaded from the internet.

Someone asks I can help you understand where I'm wrong?
Thank you

KOM · Oct 14, 2016, 1:22 PM

Someone asks I can help you understand where I'm wrong?

Trying to use squid to cache Windows Updates. You're better off running WSUS.

Kababayan · Nov 3, 2023, 4:53 AM

@Ninno:

Hi,
I enabled the dynamic cache by following the guides on the net, this is my configuration.

# This file is automatically generated by pfSense
# Do not edit manually !

http_port 172.16.0.1:3128
http_port 127.0.0.1:3128 intercept
icp_port 0
dns_v4_first off
pid_filename /var/run/squid/squid.pid
cache_effective_user squid
cache_effective_group proxy
error_default_language it
icon_directory /usr/local/etc/squid/icons
visible_hostname PROXY
cache_mgr admin@localhost
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none
netdb_filename /var/squid/logs/netdb.state
pinger_enable off
pinger_program /usr/local/libexec/squid/pinger

logfile_rotate 30
debug_options rotate=30
shutdown_lifetime 3 seconds
# Allow local network(s) on interface(s)
acl localnet src  172.16.0.0/16
forwarded_for on
httpd_suppress_version_string on
uri_whitespace strip

#windows update
refresh_pattern windowsupdate.com/.*\.(cab|exe)                     43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
refresh_pattern update.microsoft.com/.*\.(cab|exe)                  43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
refresh_pattern download.microsoft.com/.*\.(cab|exe)                43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip|esd) 4320 80% 43200 reload-into-ims
refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf|esd) 4320 100% 43200 reload-into-ims 
refresh_pattern update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf|esd) 4320 100% 43200 reload-into-ims
refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 4320 100% 43200 reload-into-ims

#ALL
refresh_pattern -i (\.|-)(mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t|esd)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth

cache_mem 1024 MB
maximum_object_size_in_memory 512 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
minimum_object_size 0 KB
maximum_object_size 5000 MB
cache_dir ufs /var/squid/cache 80000 16 256
offline_mode off
cache_swap_low 90
cache_swap_high 95
cache allow all
# Add any of your own refresh_pattern entries above these.
refresh_pattern ^ftp:    1440  20%  10080
refresh_pattern ^gopher:  1440  0%  1440
refresh_pattern -i (/cgi-bin/|\?) 0  0%  0
refresh_pattern .    0  20%  4320

#Remote proxies

# Setup some default acls
# From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.
# acl localhost src 127.0.0.1/32
acl allsrc src all
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 3129 1025-65535 
acl sslports port 443 563  

# From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.
#acl manager proto cache_object

acl purge method PURGE
acl connect method CONNECT

# Define protocols used for redirects
acl HTTP proto HTTP
acl HTTPS proto HTTPS
http_access allow manager localhost

http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

# Always allow localhost connections
# From 3.2 further configuration cleanups have been done to make things easier and safer.
# The manager, localhost, and to_localhost ACL definitions are now built-in.
# http_access allow localhost

quick_abort_min -1 KB
quick_abort_max 0 KB
request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc

# Reverse Proxy settings

# Custom options before auth

# Setup allowed ACLs
# Allow local network(s) on interface(s)
http_access allow localnet
# Default block all to be sure
http_access deny allsrc

It all seems ok, but does not work …
I'm installing 30 new PCs and updates are always downloaded from the internet.

Someone asks I can help you understand where I'm wrong?
Thank you

You need rewriter helper. add this to config of squid.

acl getmethod method GET
acl dontrewrite url_regex redbot.org
always_direct allow !getmethod

store_id_access deny connect
store_id_access deny !getmethod
store_id_access deny dontrewrite
store_id_access allow all
store_id_program '/usr/local/libexec/squid/storeid_file_rewrite' /path/to/db_file # Searcch google how to use storeid_file_rewrite it is included in squid 3.5.19
store_id_children 25 startup=15 idle=5 concurrency=0

add that to squid config. that should cache most cdn but it's a db base just google you will be able to find some db example in the web.
store_id_rewrite:

[code]#!/usr/local/bin/perl

use strict;
use warnings;
use Pod::Usage;

=pod

=head1 NAME

 storeid_file_rewrite - File based Store-ID helper for Squid

=head1 SYNOPSIS

 storeid_file_rewrite filepath

=head1 DESCRIPTION

This program acts as a store_id helper program, rewriting URLs passed
by Squid into storage-ids that can be used to achieve better caching
for websites that use different URLs for the same content.

It takes a text file with two tab separated columns.
Column 1: Regular expression to match against the URL
Column 2: Rewrite rule to generate a Store-ID
Eg:
^http:\/\/[^\.]+\.dl\.sourceforge\.net\/(.*)	http://dl.sourceforge.net.squid.internal/$1

Rewrite rules are matched in the same order as they appear in the rules file.
So for best performance, sort it in order of frequency of occurrence.

This program will automatically detect the existence of a concurrecy channel-ID and adjust appropriately.
It may be used with any value 0 or above for the store_id_children concurrency= parameter.

=head1 OPTIONS

The only command line parameter this helper takes is the regex rules file name.

=head1 AUTHOR

This program and documentation was written by I<alan mizrahi="" <alan@mizrahi.com.ve="">>

Based on prior work by I<eliezer croitoru="" <eliezer@ngtech.co.il="">>

=head1 COPYRIGHT

 * Copyright (C) 1996-2016 The Squid Software Foundation and contributors
 *
 * Squid software is distributed under GPLv2+ license and includes
 * contributions from numerous individuals and organizations.
 * Please see the COPYING and CONTRIBUTORS files for details.

 Copyright (C) 2013 Alan Mizrahi <alan@mizrahi.com.ve>
 Based on code from Eliezer Croitoru <eliezer@ngtech.co.il>

 This program is free software; you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
 the Free Software Foundation; either version 2 of the License, or
 (at your option) any later version.

 This program is distributed in the hope that it will be useful,
 but WITHOUT ANY WARRANTY; without even the implied warranty of
 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 GNU General Public License for more details.

 You should have received a copy of the GNU General Public License
 along with this program; if not, write to the Free Software
 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307, USA.

=head1 QUESTIONS

Questions on the usage of this program can be sent to the I<squid users="" mailing="" list="" <squid-users@squid-cache.org="">>

=head1 REPORTING BUGS

Bug reports need to be made in English.
See http://wiki.squid-cache.org/SquidFaq/BugReporting for details of what you need to include with your bug report.

Report bugs or bug fixes using http://bugs.squid-cache.org/

Report serious security bugs to I<squid bugs="" <squid-bugs@squid-cache.org="">>

Report ideas for new improvements to the I<squid developers="" mailing="" list="" <squid-dev@squid-cache.org="">>

=head1 SEE ALSO

squid (8), GPL (7),

The Squid wiki http://wiki.squid-cache.org/Features/StoreID

The Squid Configuration Manual http://www.squid-cache.org/Doc/config/

=cut

my @rules; # array of [regex, replacement string]

die "Usage: $0 <rewrite-file>\n" unless $#ARGV == 0;

# read config file
open RULES, $ARGV[0] or die "Error opening $ARGV[0]: $!";
while (<rules>) {
	chomp;
	next if /^\s*#?$/;
	if (/^\s*([^\t]+?)\s*\t+\s*([^\t]+?)\s*$/) {
		push(@rules, [qr/$1/, $2]);
	} else {
		print STDERR "$0: Parse error in $ARGV[0] (line $.)\n";
	}
}
close RULES;

$|=1;
# read urls from squid and do the replacement
URL: while (<stdin>) {
	chomp;
	last if $_ eq 'quit';

  my $channel = "";
  if (s/^(\d+\s+)//o) {
    $channel = $1;
  }

	foreach my $rule (@rules) {
		if (my @match = /$rule->[0]/) {
			$_ = $rule->[1];

			for (my $i=1; $i<=scalar(@match); $i++) {
				s/\$i/$match[$i-1]/g;
			}
			print $channel, "OK store-id=$_\n";
			next URL;
		}
	}
	print $channel, "ERR\n";
}
[/code]</stdin></rules></rewrite-file></squid></squid></squid></eliezer@ngtech.co.il></alan@mizrahi.com.ve></eliezer></alan>

here some db examoles you might need: http://wiki.squid-cache.org/Features/StoreID/DB

JonathanLee · Nov 3, 2023, 4:53 AM

@Kababayan

I did not know about the built in store id helper program.

I was testing this one out from git hub

https://github.com/rudiservo/pfsense_storeid/tree/master

Quick Question . . .

What are you doing with the acl that points to redbot.org?

acl dontrewrite url_regex redbot.org

Why create a one time ACL just to block it?

per Squid's website

refresh_pattern ^http://(youtube|ytimg|vimeo|[a-zA-Z0-9\-]+)\.squid\.internal/.*  10080 80%  79900 override-lastmod override-expire ignore-reload ignore-must-revalidate ignore-private

acl rewritedoms dstdomain .dailymotion.com .video-http.media-imdb.com .c.youtube.com av.vimeo.com .dl.sourceforge.net .ytimg.com .vid.ec.dmcdn.net .videoslasher.com

store_id_program /usr/local/squid/bin/new_format.rb
store_id_children 40 startup=10 idle=5 concurrency=0
store_id_access allow rewritedoms !banned_methods
store_id_access deny all

They just use the refresh and one acl with all the domains in it that are needed.