Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Dynamic cache not work

    Scheduled Pinned Locked Moved Cache/Proxy
    4 Posts 4 Posters 1.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      Ninno
      last edited by

      Hi,
      I enabled the dynamic cache by following the guides on the net, this is my configuration.

      # This file is automatically generated by pfSense
      # Do not edit manually !
      
      http_port 172.16.0.1:3128
      http_port 127.0.0.1:3128 intercept
      icp_port 0
      dns_v4_first off
      pid_filename /var/run/squid/squid.pid
      cache_effective_user squid
      cache_effective_group proxy
      error_default_language it
      icon_directory /usr/local/etc/squid/icons
      visible_hostname PROXY
      cache_mgr admin@localhost
      access_log /var/squid/logs/access.log
      cache_log /var/squid/logs/cache.log
      cache_store_log none
      netdb_filename /var/squid/logs/netdb.state
      pinger_enable off
      pinger_program /usr/local/libexec/squid/pinger
      
      logfile_rotate 30
      debug_options rotate=30
      shutdown_lifetime 3 seconds
      # Allow local network(s) on interface(s)
      acl localnet src  172.16.0.0/16
      forwarded_for on
      httpd_suppress_version_string on
      uri_whitespace strip
      
      #windows update
      refresh_pattern windowsupdate.com/.*\.(cab|exe)                     43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
      refresh_pattern update.microsoft.com/.*\.(cab|exe)                  43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
      refresh_pattern download.microsoft.com/.*\.(cab|exe)                43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
      refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
      refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
      refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip|esd) 4320 80% 43200 reload-into-ims
      refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf|esd) 4320 100% 43200 reload-into-ims 
      refresh_pattern update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
      refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
      refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
      refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
      refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf|esd) 4320 100% 43200 reload-into-ims
      refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 4320 100% 43200 reload-into-ims
      
      #ALL
      refresh_pattern -i (\.|-)(mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t|esd)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
      
      cache_mem 1024 MB
      maximum_object_size_in_memory 512 KB
      memory_replacement_policy heap GDSF
      cache_replacement_policy heap LFUDA
      minimum_object_size 0 KB
      maximum_object_size 5000 MB
      cache_dir ufs /var/squid/cache 80000 16 256
      offline_mode off
      cache_swap_low 90
      cache_swap_high 95
      cache allow all
      # Add any of your own refresh_pattern entries above these.
      refresh_pattern ^ftp:    1440  20%  10080
      refresh_pattern ^gopher:  1440  0%  1440
      refresh_pattern -i (/cgi-bin/|\?) 0  0%  0
      refresh_pattern .    0  20%  4320
      
      #Remote proxies
      
      # Setup some default acls
      # From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.
      # acl localhost src 127.0.0.1/32
      acl allsrc src all
      acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 3129 1025-65535 
      acl sslports port 443 563  
      
      # From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.
      #acl manager proto cache_object
      
      acl purge method PURGE
      acl connect method CONNECT
      
      # Define protocols used for redirects
      acl HTTP proto HTTP
      acl HTTPS proto HTTPS
      http_access allow manager localhost
      
      http_access deny manager
      http_access allow purge localhost
      http_access deny purge
      http_access deny !safeports
      http_access deny CONNECT !sslports
      
      # Always allow localhost connections
      # From 3.2 further configuration cleanups have been done to make things easier and safer.
      # The manager, localhost, and to_localhost ACL definitions are now built-in.
      # http_access allow localhost
      
      quick_abort_min -1 KB
      quick_abort_max 0 KB
      request_body_max_size 0 KB
      delay_pools 1
      delay_class 1 2
      delay_parameters 1 -1/-1 -1/-1
      delay_initial_bucket_level 100
      delay_access 1 allow allsrc
      
      # Reverse Proxy settings
      
      # Custom options before auth
      
      # Setup allowed ACLs
      # Allow local network(s) on interface(s)
      http_access allow localnet
      # Default block all to be sure
      http_access deny allsrc
      
      

      It all seems ok, but does not work …
      I'm installing 30 new PCs and updates are always downloaded from the internet.

      Someone asks I can help you understand where I'm wrong?
      Thank you

      1 Reply Last reply Reply Quote 0
      • KOMK
        KOM
        last edited by

        Someone asks I can help you understand where I'm wrong?

        Trying to use squid to cache Windows Updates.  You're better off running WSUS.

        1 Reply Last reply Reply Quote 0
        • K
          Kababayan
          last edited by

          @Ninno:

          Hi,
          I enabled the dynamic cache by following the guides on the net, this is my configuration.

          # This file is automatically generated by pfSense
          # Do not edit manually !
          
          http_port 172.16.0.1:3128
          http_port 127.0.0.1:3128 intercept
          icp_port 0
          dns_v4_first off
          pid_filename /var/run/squid/squid.pid
          cache_effective_user squid
          cache_effective_group proxy
          error_default_language it
          icon_directory /usr/local/etc/squid/icons
          visible_hostname PROXY
          cache_mgr admin@localhost
          access_log /var/squid/logs/access.log
          cache_log /var/squid/logs/cache.log
          cache_store_log none
          netdb_filename /var/squid/logs/netdb.state
          pinger_enable off
          pinger_program /usr/local/libexec/squid/pinger
          
          logfile_rotate 30
          debug_options rotate=30
          shutdown_lifetime 3 seconds
          # Allow local network(s) on interface(s)
          acl localnet src  172.16.0.0/16
          forwarded_for on
          httpd_suppress_version_string on
          uri_whitespace strip
          
          #windows update
          refresh_pattern windowsupdate.com/.*\.(cab|exe)                     43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
          refresh_pattern update.microsoft.com/.*\.(cab|exe)                  43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
          refresh_pattern download.microsoft.com/.*\.(cab|exe)                43200 100% 129600 ignore-no-cache ignore-no-store ignore-reload reload-into-ims
          refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
          refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
          refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip|esd) 4320 80% 43200 reload-into-ims
          refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf|esd) 4320 100% 43200 reload-into-ims 
          refresh_pattern update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
          refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
          refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
          refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 10080 100% 43200 reload-into-ims
          refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf|esd) 4320 100% 43200 reload-into-ims
          refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf|esd) 4320 100% 43200 reload-into-ims
          
          #ALL
          refresh_pattern -i (\.|-)(mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t|esd)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
          
          cache_mem 1024 MB
          maximum_object_size_in_memory 512 KB
          memory_replacement_policy heap GDSF
          cache_replacement_policy heap LFUDA
          minimum_object_size 0 KB
          maximum_object_size 5000 MB
          cache_dir ufs /var/squid/cache 80000 16 256
          offline_mode off
          cache_swap_low 90
          cache_swap_high 95
          cache allow all
          # Add any of your own refresh_pattern entries above these.
          refresh_pattern ^ftp:    1440  20%  10080
          refresh_pattern ^gopher:  1440  0%  1440
          refresh_pattern -i (/cgi-bin/|\?) 0  0%  0
          refresh_pattern .    0  20%  4320
          
          #Remote proxies
          
          # Setup some default acls
          # From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.
          # acl localhost src 127.0.0.1/32
          acl allsrc src all
          acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 3129 1025-65535 
          acl sslports port 443 563  
          
          # From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.
          #acl manager proto cache_object
          
          acl purge method PURGE
          acl connect method CONNECT
          
          # Define protocols used for redirects
          acl HTTP proto HTTP
          acl HTTPS proto HTTPS
          http_access allow manager localhost
          
          http_access deny manager
          http_access allow purge localhost
          http_access deny purge
          http_access deny !safeports
          http_access deny CONNECT !sslports
          
          # Always allow localhost connections
          # From 3.2 further configuration cleanups have been done to make things easier and safer.
          # The manager, localhost, and to_localhost ACL definitions are now built-in.
          # http_access allow localhost
          
          quick_abort_min -1 KB
          quick_abort_max 0 KB
          request_body_max_size 0 KB
          delay_pools 1
          delay_class 1 2
          delay_parameters 1 -1/-1 -1/-1
          delay_initial_bucket_level 100
          delay_access 1 allow allsrc
          
          # Reverse Proxy settings
          
          # Custom options before auth
          
          # Setup allowed ACLs
          # Allow local network(s) on interface(s)
          http_access allow localnet
          # Default block all to be sure
          http_access deny allsrc
          
          

          It all seems ok, but does not work …
          I'm installing 30 new PCs and updates are always downloaded from the internet.

          Someone asks I can help you understand where I'm wrong?
          Thank you

          You need  rewriter helper.  add this to config of squid.

          acl getmethod method GET
          acl dontrewrite url_regex redbot.org
          always_direct allow !getmethod

          store_id_access deny connect
          store_id_access deny !getmethod
          store_id_access deny dontrewrite
          store_id_access allow all
          store_id_program '/usr/local/libexec/squid/storeid_file_rewrite'  /path/to/db_file  # Searcch google how to use storeid_file_rewrite it is included in squid 3.5.19
          store_id_children 25 startup=15 idle=5 concurrency=0

          add that to squid config. that should cache most cdn but  it's a db base just google you will be able to find some db example in the web.
          store_id_rewrite:

          [code]#!/usr/local/bin/perl
          
          use strict;
          use warnings;
          use Pod::Usage;
          
          =pod
          
          =head1 NAME
          
           storeid_file_rewrite - File based Store-ID helper for Squid
          
          =head1 SYNOPSIS
          
           storeid_file_rewrite filepath
          
          =head1 DESCRIPTION
          
          This program acts as a store_id helper program, rewriting URLs passed
          by Squid into storage-ids that can be used to achieve better caching
          for websites that use different URLs for the same content.
          
          It takes a text file with two tab separated columns.
          Column 1: Regular expression to match against the URL
          Column 2: Rewrite rule to generate a Store-ID
          Eg:
          ^http:\/\/[^\.]+\.dl\.sourceforge\.net\/(.*)	http://dl.sourceforge.net.squid.internal/$1
          
          Rewrite rules are matched in the same order as they appear in the rules file.
          So for best performance, sort it in order of frequency of occurrence.
          
          This program will automatically detect the existence of a concurrecy channel-ID and adjust appropriately.
          It may be used with any value 0 or above for the store_id_children concurrency= parameter.
          
          =head1 OPTIONS
          
          The only command line parameter this helper takes is the regex rules file name.
          
          =head1 AUTHOR
          
          This program and documentation was written by I<alan mizrahi="" <alan@mizrahi.com.ve="">>
          
          Based on prior work by I<eliezer croitoru="" <eliezer@ngtech.co.il="">>
          
          =head1 COPYRIGHT
          
           * Copyright (C) 1996-2016 The Squid Software Foundation and contributors
           *
           * Squid software is distributed under GPLv2+ license and includes
           * contributions from numerous individuals and organizations.
           * Please see the COPYING and CONTRIBUTORS files for details.
          
           Copyright (C) 2013 Alan Mizrahi <alan@mizrahi.com.ve>
           Based on code from Eliezer Croitoru <eliezer@ngtech.co.il>
          
           This program is free software; you can redistribute it and/or modify
           it under the terms of the GNU General Public License as published by
           the Free Software Foundation; either version 2 of the License, or
           (at your option) any later version.
          
           This program is distributed in the hope that it will be useful,
           but WITHOUT ANY WARRANTY; without even the implied warranty of
           MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
           GNU General Public License for more details.
          
           You should have received a copy of the GNU General Public License
           along with this program; if not, write to the Free Software
           Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307, USA.
          
          =head1 QUESTIONS
          
          Questions on the usage of this program can be sent to the I<squid users="" mailing="" list="" <squid-users@squid-cache.org="">>
          
          =head1 REPORTING BUGS
          
          Bug reports need to be made in English.
          See http://wiki.squid-cache.org/SquidFaq/BugReporting for details of what you need to include with your bug report.
          
          Report bugs or bug fixes using http://bugs.squid-cache.org/
          
          Report serious security bugs to I<squid bugs="" <squid-bugs@squid-cache.org="">>
          
          Report ideas for new improvements to the I<squid developers="" mailing="" list="" <squid-dev@squid-cache.org="">>
          
          =head1 SEE ALSO
          
          squid (8), GPL (7),
          
          The Squid wiki http://wiki.squid-cache.org/Features/StoreID
          
          The Squid Configuration Manual http://www.squid-cache.org/Doc/config/
          
          =cut
          
          my @rules; # array of [regex, replacement string]
          
          die "Usage: $0 <rewrite-file>\n" unless $#ARGV == 0;
          
          # read config file
          open RULES, $ARGV[0] or die "Error opening $ARGV[0]: $!";
          while (<rules>) {
          	chomp;
          	next if /^\s*#?$/;
          	if (/^\s*([^\t]+?)\s*\t+\s*([^\t]+?)\s*$/) {
          		push(@rules, [qr/$1/, $2]);
          	} else {
          		print STDERR "$0: Parse error in $ARGV[0] (line $.)\n";
          	}
          }
          close RULES;
          
          $|=1;
          # read urls from squid and do the replacement
          URL: while (<stdin>) {
          	chomp;
          	last if $_ eq 'quit';
          
            my $channel = "";
            if (s/^(\d+\s+)//o) {
              $channel = $1;
            }
          
          	foreach my $rule (@rules) {
          		if (my @match = /$rule->[0]/) {
          			$_ = $rule->[1];
          
          			for (my $i=1; $i<=scalar(@match); $i++) {
          				s/\$i/$match[$i-1]/g;
          			}
          			print $channel, "OK store-id=$_\n";
          			next URL;
          		}
          	}
          	print $channel, "ERR\n";
          }
          [/code]</stdin></rules></rewrite-file></squid></squid></squid></eliezer@ngtech.co.il></alan@mizrahi.com.ve></eliezer></alan>
          

          here some db examoles you might need: http://wiki.squid-cache.org/Features/StoreID/DB

          JonathanLeeJ 1 Reply Last reply Reply Quote 1
          • JonathanLeeJ
            JonathanLee @Kababayan
            last edited by

            @Kababayan

            I did not know about the built in store id helper program.

            I was testing this one out from git hub

            https://github.com/rudiservo/pfsense_storeid/tree/master

            Quick Question . . .

            What are you doing with the acl that points to redbot.org?

            acl dontrewrite url_regex redbot.org
            
            

            Why create a one time ACL just to block it?

            per Squid's website

            refresh_pattern ^http://(youtube|ytimg|vimeo|[a-zA-Z0-9\-]+)\.squid\.internal/.*  10080 80%  79900 override-lastmod override-expire ignore-reload ignore-must-revalidate ignore-private
            
            acl rewritedoms dstdomain .dailymotion.com .video-http.media-imdb.com .c.youtube.com av.vimeo.com .dl.sourceforge.net .ytimg.com .vid.ec.dmcdn.net .videoslasher.com
            
            store_id_program /usr/local/squid/bin/new_format.rb
            store_id_children 40 startup=10 idle=5 concurrency=0
            store_id_access allow rewritedoms !banned_methods
            store_id_access deny all
            

            They just use the refresh and one acl with all the domains in it that are needed.

            Make sure to upvote

            1 Reply Last reply Reply Quote 0
            • JonathanLeeJ JonathanLee referenced this topic on
            • First post
              Last post
            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.