Dynamic cache not work



  • Hi,
    I enabled the dynamic cache by following the guides on the net, this is my configuration.

    # This file is automatically generated by pfSense
    # Do not edit manually !
    
    http_port 172.16.0.1:3128
    http_port 127.0.0.1:3128 intercept
    icp_port 0
    dns_v4_first off
    pid_filename /var/run/squid/squid.pid
    cache_effective_user squid
    cache_effective_group proxy
    error_default_language it
    icon_directory /usr/local/etc/squid/icons
    visible_hostname PROXY
    cache_mgr admin@localhost
    access_log /var/squid/logs/access.log
    cache_log /var/squid/logs/cache.log
    cache_store_log none
    netdb_filename /var/squid/logs/netdb.state
    pinger_enable off
    pinger_program /usr/local/libexec/squid/pinger
    
    logfile_rotate 30
    debug_options rotate=30
    shutdown_lifetime 3 seconds
    # Allow local network(s) on interface(s)
    acl localnet src  172.16.0.0/16
    forwarded_for on
    httpd_suppress_version_string on
    uri_whitespace strip
    
    #windows update
    refresh_pattern windowsupdate.com/.*\.(cab|exe)                     43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    refresh_pattern update.microsoft.com/.*\.(cab|exe)                  43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    refresh_pattern download.microsoft.com/.*\.(cab|exe)                43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip|esd) 4320 80% 43200 reload-into-ims
    refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf|esd) 4320 100% 43200 reload-into-ims 
    refresh_pattern update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
    refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
    refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
    refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf|esd) 4320 100% 43200 reload-into-ims
    refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 4320 100% 43200 reload-into-ims
    
    #ALL
    refresh_pattern -i (\.|-)(mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t|esd)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    
    cache_mem 1024 MB
    maximum_object_size_in_memory 512 KB
    memory_replacement_policy heap GDSF
    cache_replacement_policy heap LFUDA
    minimum_object_size 0 KB
    maximum_object_size 5000 MB
    cache_dir ufs /var/squid/cache 80000 16 256
    offline_mode off
    cache_swap_low 90
    cache_swap_high 95
    cache allow all
    # Add any of your own refresh_pattern entries above these.
    refresh_pattern ^ftp:    1440  20%  10080
    refresh_pattern ^gopher:  1440  0%  1440
    refresh_pattern -i (/cgi-bin/|\?) 0  0%  0
    refresh_pattern .    0  20%  4320
    
    #Remote proxies
    
    # Setup some default acls
    # From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.
    # acl localhost src 127.0.0.1/32
    acl allsrc src all
    acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 3129 1025-65535 
    acl sslports port 443 563  
    
    # From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.
    #acl manager proto cache_object
    
    acl purge method PURGE
    acl connect method CONNECT
    
    # Define protocols used for redirects
    acl HTTP proto HTTP
    acl HTTPS proto HTTPS
    http_access allow manager localhost
    
    http_access deny manager
    http_access allow purge localhost
    http_access deny purge
    http_access deny !safeports
    http_access deny CONNECT !sslports
    
    # Always allow localhost connections
    # From 3.2 further configuration cleanups have been done to make things easier and safer.
    # The manager, localhost, and to_localhost ACL definitions are now built-in.
    # http_access allow localhost
    
    quick_abort_min -1 KB
    quick_abort_max 0 KB
    request_body_max_size 0 KB
    delay_pools 1
    delay_class 1 2
    delay_parameters 1 -1/-1 -1/-1
    delay_initial_bucket_level 100
    delay_access 1 allow allsrc
    
    # Reverse Proxy settings
    
    # Custom options before auth
    
    # Setup allowed ACLs
    # Allow local network(s) on interface(s)
    http_access allow localnet
    # Default block all to be sure
    http_access deny allsrc
    
    

    It all seems ok, but does not work …
    I'm installing 30 new PCs and updates are always downloaded from the internet.

    Someone asks I can help you understand where I'm wrong?
    Thank you



  • Someone asks I can help you understand where I'm wrong?

    Trying to use squid to cache Windows Updates.  You're better off running WSUS.



  • @Ninno:

    Hi,
    I enabled the dynamic cache by following the guides on the net, this is my configuration.

    # This file is automatically generated by pfSense
    # Do not edit manually !
    
    http_port 172.16.0.1:3128
    http_port 127.0.0.1:3128 intercept
    icp_port 0
    dns_v4_first off
    pid_filename /var/run/squid/squid.pid
    cache_effective_user squid
    cache_effective_group proxy
    error_default_language it
    icon_directory /usr/local/etc/squid/icons
    visible_hostname PROXY
    cache_mgr admin@localhost
    access_log /var/squid/logs/access.log
    cache_log /var/squid/logs/cache.log
    cache_store_log none
    netdb_filename /var/squid/logs/netdb.state
    pinger_enable off
    pinger_program /usr/local/libexec/squid/pinger
    
    logfile_rotate 30
    debug_options rotate=30
    shutdown_lifetime 3 seconds
    # Allow local network(s) on interface(s)
    acl localnet src  172.16.0.0/16
    forwarded_for on
    httpd_suppress_version_string on
    uri_whitespace strip
    
    #windows update
    refresh_pattern windowsupdate.com/.*\.(cab|exe)                     43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    refresh_pattern update.microsoft.com/.*\.(cab|exe)                  43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    refresh_pattern download.microsoft.com/.*\.(cab|exe)                43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
    refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip|esd) 4320 80% 43200 reload-into-ims
    refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf|esd) 4320 100% 43200 reload-into-ims 
    refresh_pattern update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
    refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
    refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
    refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf|esd) 4320 100% 43200 reload-into-ims
    refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 4320 100% 43200 reload-into-ims
    
    #ALL
    refresh_pattern -i (\.|-)(mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t|esd)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    
    cache_mem 1024 MB
    maximum_object_size_in_memory 512 KB
    memory_replacement_policy heap GDSF
    cache_replacement_policy heap LFUDA
    minimum_object_size 0 KB
    maximum_object_size 5000 MB
    cache_dir ufs /var/squid/cache 80000 16 256
    offline_mode off
    cache_swap_low 90
    cache_swap_high 95
    cache allow all
    # Add any of your own refresh_pattern entries above these.
    refresh_pattern ^ftp:    1440  20%  10080
    refresh_pattern ^gopher:  1440  0%  1440
    refresh_pattern -i (/cgi-bin/|\?) 0  0%  0
    refresh_pattern .    0  20%  4320
    
    #Remote proxies
    
    # Setup some default acls
    # From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.
    # acl localhost src 127.0.0.1/32
    acl allsrc src all
    acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 3129 1025-65535 
    acl sslports port 443 563  
    
    # From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.
    #acl manager proto cache_object
    
    acl purge method PURGE
    acl connect method CONNECT
    
    # Define protocols used for redirects
    acl HTTP proto HTTP
    acl HTTPS proto HTTPS
    http_access allow manager localhost
    
    http_access deny manager
    http_access allow purge localhost
    http_access deny purge
    http_access deny !safeports
    http_access deny CONNECT !sslports
    
    # Always allow localhost connections
    # From 3.2 further configuration cleanups have been done to make things easier and safer.
    # The manager, localhost, and to_localhost ACL definitions are now built-in.
    # http_access allow localhost
    
    quick_abort_min -1 KB
    quick_abort_max 0 KB
    request_body_max_size 0 KB
    delay_pools 1
    delay_class 1 2
    delay_parameters 1 -1/-1 -1/-1
    delay_initial_bucket_level 100
    delay_access 1 allow allsrc
    
    # Reverse Proxy settings
    
    # Custom options before auth
    
    # Setup allowed ACLs
    # Allow local network(s) on interface(s)
    http_access allow localnet
    # Default block all to be sure
    http_access deny allsrc
    
    

    It all seems ok, but does not work …
    I'm installing 30 new PCs and updates are always downloaded from the internet.

    Someone asks I can help you understand where I'm wrong?
    Thank you

    You need  rewriter helper.  add this to config of squid.

    acl getmethod method GET
    acl dontrewrite url_regex redbot.org
    always_direct allow !getmethod

    store_id_access deny connect
    store_id_access deny !getmethod
    store_id_access deny dontrewrite
    store_id_access allow all
    store_id_program '/usr/local/libexec/squid/storeid_file_rewrite'  /path/to/db_file  # Searcch google how to use storeid_file_rewrite it is included in squid 3.5.19
    store_id_children 25 startup=15 idle=5 concurrency=0

    add that to squid config. that should cache most cdn but  it's a db base just google you will be able to find some db example in the web.
    store_id_rewrite:

    [code]#!/usr/local/bin/perl
    
    use strict;
    use warnings;
    use Pod::Usage;
    
    =pod
    
    =head1 NAME
    
     storeid_file_rewrite - File based Store-ID helper for Squid
    
    =head1 SYNOPSIS
    
     storeid_file_rewrite filepath
    
    =head1 DESCRIPTION
    
    This program acts as a store_id helper program, rewriting URLs passed
    by Squid into storage-ids that can be used to achieve better caching
    for websites that use different URLs for the same content.
    
    It takes a text file with two tab separated columns.
    Column 1: Regular expression to match against the URL
    Column 2: Rewrite rule to generate a Store-ID
    Eg:
    ^http:\/\/[^\.]+\.dl\.sourceforge\.net\/(.*)	http://dl.sourceforge.net.squid.internal/$1
    
    Rewrite rules are matched in the same order as they appear in the rules file.
    So for best performance, sort it in order of frequency of occurrence.
    
    This program will automatically detect the existence of a concurrecy channel-ID and adjust appropriately.
    It may be used with any value 0 or above for the store_id_children concurrency= parameter.
    
    =head1 OPTIONS
    
    The only command line parameter this helper takes is the regex rules file name.
    
    =head1 AUTHOR
    
    This program and documentation was written by I<alan mizrahi="" <alan@mizrahi.com.ve="">>
    
    Based on prior work by I<eliezer croitoru="" <eliezer@ngtech.co.il="">>
    
    =head1 COPYRIGHT
    
     * Copyright (C) 1996-2016 The Squid Software Foundation and contributors
     *
     * Squid software is distributed under GPLv2+ license and includes
     * contributions from numerous individuals and organizations.
     * Please see the COPYING and CONTRIBUTORS files for details.
    
     Copyright (C) 2013 Alan Mizrahi <alan@mizrahi.com.ve>
     Based on code from Eliezer Croitoru <eliezer@ngtech.co.il>
    
     This program is free software; you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation; either version 2 of the License, or
     (at your option) any later version.
    
     This program is distributed in the hope that it will be useful,
     but WITHOUT ANY WARRANTY; without even the implied warranty of
     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
     GNU General Public License for more details.
    
     You should have received a copy of the GNU General Public License
     along with this program; if not, write to the Free Software
     Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307, USA.
    
    =head1 QUESTIONS
    
    Questions on the usage of this program can be sent to the I<squid users="" mailing="" list="" <squid-users@squid-cache.org="">>
    
    =head1 REPORTING BUGS
    
    Bug reports need to be made in English.
    See http://wiki.squid-cache.org/SquidFaq/BugReporting for details of what you need to include with your bug report.
    
    Report bugs or bug fixes using http://bugs.squid-cache.org/
    
    Report serious security bugs to I<squid bugs="" <squid-bugs@squid-cache.org="">>
    
    Report ideas for new improvements to the I<squid developers="" mailing="" list="" <squid-dev@squid-cache.org="">>
    
    =head1 SEE ALSO
    
    squid (8), GPL (7),
    
    The Squid wiki http://wiki.squid-cache.org/Features/StoreID
    
    The Squid Configuration Manual http://www.squid-cache.org/Doc/config/
    
    =cut
    
    my @rules; # array of [regex, replacement string]
    
    die "Usage: $0 <rewrite-file>\n" unless $#ARGV == 0;
    
    # read config file
    open RULES, $ARGV[0] or die "Error opening $ARGV[0]: $!";
    while (<rules>) {
    	chomp;
    	next if /^\s*#?$/;
    	if (/^\s*([^\t]+?)\s*\t+\s*([^\t]+?)\s*$/) {
    		push(@rules, [qr/$1/, $2]);
    	} else {
    		print STDERR "$0: Parse error in $ARGV[0] (line $.)\n";
    	}
    }
    close RULES;
    
    $|=1;
    # read urls from squid and do the replacement
    URL: while (<stdin>) {
    	chomp;
    	last if $_ eq 'quit';
    
      my $channel = "";
      if (s/^(\d+\s+)//o) {
        $channel = $1;
      }
    
    	foreach my $rule (@rules) {
    		if (my @match = /$rule->[0]/) {
    			$_ = $rule->[1];
    
    			for (my $i=1; $i<=scalar(@match); $i++) {
    				s/\$i/$match[$i-1]/g;
    			}
    			print $channel, "OK store-id=$_\n";
    			next URL;
    		}
    	}
    	print $channel, "ERR\n";
    }
    [/code]</stdin></rules></rewrite-file></squid></squid></squid></eliezer@ngtech.co.il></alan@mizrahi.com.ve></eliezer></alan>
    

    here some db examoles you might need: http://wiki.squid-cache.org/Features/StoreID/DB