Squid's StoreID and Squid's built in StoreID helper usr/local/libexec/squid/storeid_file_rewrite

JonathanLee

Has anyone else attempted to use this built in StoreID helper program file with dynamic refresh patterns? I got StoreID working for Facebook but it just shows my one photo over and over for everything and doesn't work correctly for content acceleration or for use with items that keep downloading over and over. I am doing something wrong. /$2.$3 or /$1 is not really explained any where. Does anyone know what this means in the configuration with Regex. Another configuration I attempted got OK for all the images and items it was placing into the cache however would never retrieve them second time arround only restored them. I used the helper program on GitHub and I got it working for Ubuntu very well. Does anyone have any configuration examples to share with me for the built in StoreID helper program that comes with the Squid package?

usr/local/libexec/squid/storeid_file_rewrite
What database/configuration examples are you using?

Ref:
docs.oracle.com/cd/E88353_01/html/E72487/storeid-file-rewrite-8.html

wiki.squid-cache.org/Features/StoreID

wiki.squid-cache.org/Features/StoreID/DB

forum.netgate.com/topic/106614/dynamic-cache-not-work

github.com/rudiservo/pfsense_storeid

http://ca2.squid-cache.org/Doc/config/store_id_program/

JonathanLee

Update I asked Squid email support about this and got a good reply with some help about the built in program

"On 1/01/25 21:27, Robin Wood wrote:
I've not got time to read your whole email, but you are asking about regular expressions.
^http://[^.]+.dl.sourceforge.net/(.) http:// dl.sourceforge.net.squid.internal/$1
What this means is to match the first URL and "capture" the bit at the end, the bit in brackets. This then gets rewritten to the second URL with the captured bit added on to the end, that is $1. If you captured two things in brackets the first would be $1, the second $2.
Do some reading on regex and regular expressions. The basics are relatively easy to understand, beyond that, it can get very complicated very quickly.*

Correct. Also, this is a configuration file for the particular helper performing StoreID changes. The pattern style and language may/will differ based on what the custom helper is doing.

The OP one (Squid provided storeid_file_rewrite) is written in Perl language, and passes the list from the file almost directly to the "Substitute Regular Expression - s///" function of Perl. As such, the documentation of that function is what you need to read for specific answers.
(https://www.tutorialspoint.com/perl/perl_regular_expressions.htm)

Amos"

This program uses Perl Regular Expressions something called Substitute Regular Expressions. Long story short learn more about it with the following URL.
https://www.tutorialspoint.com/perl/perl_regular_expressions.htm

Hope this helps any other students attempting to learn and research this stuff.

JonathanLee

Store ID program:

I am using the built in program attached here..
/usr/local/libexec/squid/storeid_file_rewrite

#!/usr/local/bin/perl

use strict;
use warnings;
use Pod::Usage;

=pod

=head1 NAME

 storeid_file_rewrite - File based Store-ID helper for Squid

=head1 SYNOPSIS

 storeid_file_rewrite filepath

=head1 DESCRIPTION

This program acts as a store_id helper program, rewriting URLs passed
by Squid into storage-ids that can be used to achieve better caching
for websites that use different URLs for the same content.

It takes a text file with two tab separated columns.
Column 1: Regular expression to match against the URL
Column 2: Rewrite rule to generate a Store-ID
Eg:
^http:\/\/[^\.]+\.dl\.sourceforge\.net\/(.*)    http://dl.sourceforge.net.squid.internal/$1

Rewrite rules are matched in the same order as they appear in the rules file.
So for best performance, sort it in order of frequency of occurrence.

This program will automatically detect the existence of a concurrency channel-ID and adjust appropriately.
It may be used with any value 0 or above for the store_id_children concurrency= parameter.

=head1 OPTIONS

The only command line parameter this helper takes is the regex rules file name.

=head1 AUTHOR

This program and documentation was written by I<Alan Mizrahi <alan@mizrahi.com.ve>>

Based on prior work by I<Eliezer Croitoru <eliezer@ngtech.co.il>>

=head1 COPYRIGHT

 * Copyright (C) 1996-2023 The Squid Software Foundation and contributors
 *
 * Squid software is distributed under GPLv2+ license and includes
 * contributions from numerous individuals and organizations.
 * Please see the COPYING and CONTRIBUTORS files for details.

 Copyright (C) 2013 Alan Mizrahi <alan@mizrahi.com.ve>
 Based on code from Eliezer Croitoru <eliezer@ngtech.co.il>

 This program is free software; you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
 the Free Software Foundation; either version 2 of the License, or
 (at your option) any later version.

 This program is distributed in the hope that it will be useful,
 but WITHOUT ANY WARRANTY; without even the implied warranty of
 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 GNU General Public License for more details.

 You should have received a copy of the GNU General Public License
 along with this program; if not, write to the Free Software
 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307, USA.

=head1 QUESTIONS

Questions on the usage of this program can be sent to the I<Squid Users mailing list <squid-users@lists.squid-cache.org>>

=head1 REPORTING BUGS

Bug reports need to be made in English.
See http://wiki.squid-cache.org/SquidFaq/BugReporting for details of what you need to include with your bug report.

Report bugs or bug fixes using http://bugs.squid-cache.org/

Report serious security bugs to I<Squid Bugs <squid-bugs@lists.squid-cache.org>>

Report ideas for new improvements to the I<Squid Developers mailing list <squid-dev@lists.squid-cache.org>>

=head1 SEE ALSO

squid (8), GPL (7),

The Squid wiki http://wiki.squid-cache.org/Features/StoreID

The Squid Configuration Manual http://www.squid-cache.org/Doc/config/

=cut

my @rules; # array of [regex, replacement string]

die "Usage: $0 <rewrite-file>\n" unless $#ARGV == 0;

# read config file
open RULES, $ARGV[0] or die "Error opening $ARGV[0]: $!";
while (<RULES>) {
    chomp;
    next if /^\s*#?$/;
    if (/^\s*([^\t]+?)\s*\t+\s*([^\t]+?)\s*$/) {
        push(@rules, [qr/$1/, $2]);
    } else {
        print STDERR "$0: Parse error in $ARGV[0] (line $.)\n";
    }
}
close RULES;

$|=1;
# read urls from squid and do the replacement
URL: while (<STDIN>) {
    chomp;
    last if $_ eq 'quit';

    my $channel = "";
    if (s/^(\d+\s+)//o) {
        $channel = $1;
    }

    foreach my $rule (@rules) {
        if (my @match = /$rule->[0]/) {
            $_ = $rule->[1];

            for (my $i=1; $i<=scalar(@match); $i++) {
                s/\$$i/$match[$i-1]/g;
            }
            print $channel, "OK store-id=$_\n";
            next URL;
        }
    }
    print $channel, "ERR\n";
}