Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    No squid packages will start (user 'squid' not found) on 2.1-DEVELOPMENT

    Scheduled Pinned Locked Moved
    2.1 Snapshot Feedback and Problems - RETIRED
    5
    22
    28.7k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      phil.davis
      last edited by

      The difference between 2.0.1 and 2.1-DEVELOPMENT is that the package is installed using a PBI. The "squid" program in /usr/local/sbin is now just a link to:
      /usr/pbi/squid-i386/.sbin/squid

      There is a default squid.conf in:
      /usr/pbi/squid-i386/.etc/squid/squid.conf

      The system seems to be using this conf file, which specifies cache_effective_user squid - and from that point all the /var/squid file owner issues occur.

      The conf file that is supposed to be used is /usr/local/etc/squid/squid.conf

      I modified /usr/local/pkg/squid.inc - on the end of all places that run "/usr/local/sbin/squid -D" add " -f /usr/local/etc/squid/squid.conf"
      That makes it use the pfSense-specific squid.conf file.

      There are still places that do "squid -k" commands to get Squid to reread its conf file, and I get some messages about 'squid: ERROR: No running copy' - I think that adding the "-f" parameter means that other checks for the squid process might need to be modified.

      An easier solution might be to put an actual copy of the squid program into /usr/local/sbin rather than a link, then it might find its conf file OK?

      As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
      If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

      1 Reply Last reply Reply Quote 0
      • P
        phil.davis
        last edited by

        I tried putting a real copy of the squid program in /usr/local/sbin
        That doesn't work, it still uses /usr/pbi/squid-i386/etc/squid/squid.conf
        It seems that the default squid.conf location is an absolute path hard-coded into the program. I was hoping that it would be a relative path (relative to the location that the squid program was run from), but not so.

        I have got Squid and SquidGuard running nicely on 2.0.1-DEVELOPMENT by editing /usr/local/pkg/squid.inc
        (a) Change all the occurrences of "squid -D" to "squid -D -f /usr/local/etc/squid/squid.conf"
        (b) Change all the occurrences of "squid -k *" to "squid -k * -f /usr/local/etc/squid/squid.conf"
           (where * is reconfigure, rotate, shutdown, kill)

        (a) makes it use the correct conf file at startup.
        (b) makes it find the squid process to change its configuration, rotate log files or stop it.

        These changes are also needed in:
        squid_ng.xml
        squidguard_configurator.inc
        swapstate_check.php

        Whoever maintains the squid package, can they make this change (or another better designed one) to squid.inc for 2.1?

        As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
        If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

        1 Reply Last reply Reply Quote 0
        • P
          phil.davis
          last edited by

          I suspect that Squid Traffic Management will not work (but I haven't tested it).
          /var/squid/logs/cache.log reports unrecognized parameters on squid startup, lines like
          parseConfigFile: squid.conf:58 unrecognized: 'delay_pools'
          This happens for delay_pools, delay_class, delay_parameters, delay_initial and delay_access.
          It looks like squid needs to be compiled with –enable_delay_pools - the supplied version in the pbi maybe does not have this compiler flag set?

          None of these parseConfigFile messages are in the log on my 2.0.1 nanobsd system.

          As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
          If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

          1 Reply Last reply Reply Quote 0
          • P
            phil.davis
            last edited by

            SquidGuard timed rules work on 2.1-DEVELOPMENT.
            I tried a rule that turned on and off every 10 minutes for a hour or so.
            /var/squidGuard/log/squidGuard.log contained regular "Info: recalculating alarm in nn seconds" messages.
            The blocked website became available and blocked as the time changed.
            (Note that you often have to be careful to clear the browser cache when doing this testing, otherwise you can just be looking at locally-cached data in the client.)
            On my 2.0.1 nanobsd system, I get "Info: recalculating alarm in nn seconds" messages a couple of times, then they just stop appearing in the log file. It seems to just forget that there are timed rules to calculate.
            So, it looks like this problem in 2.0.1 is fixed in 2.1

            As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
            If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

            1 Reply Last reply Reply Quote 0
            • jimpJ
              jimp Rebel Alliance Developer Netgate
              last edited by

              @phil.davis:

              I have got Squid and SquidGuard running nicely on 2.0.1-DEVELOPMENT by editing /usr/local/pkg/squid.inc
              (a) Change all the occurrences of "squid -D" to "squid -D -f /usr/local/etc/squid/squid.conf"
              (b) Change all the occurrences of "squid -k *" to "squid -k * -f /usr/local/etc/squid/squid.conf"
                 (where * is reconfigure, rotate, shutdown, kill)

              (a) makes it use the correct conf file at startup.
              (b) makes it find the squid process to change its configuration, rotate log files or stop it.

              These changes are also needed in:
              squid_ng.xml
              squidguard_configurator.inc
              swapstate_check.php

              Whoever maintains the squid package, can they make this change (or another better designed one) to squid.inc for 2.1?

              I can do that but I won't have time to get to that today. That should be a safe change to make both on 2.0 and 2.1 though, but it would need to be tested. If someone wants to do that and make a merge request on github we can pull it in, otherwise it'll be sometime next week before I can get to it.

              @phil.davis:

              I suspect that Squid Traffic Management will not work (but I haven't tested it).
              /var/squid/logs/cache.log reports unrecognized parameters on squid startup, lines like
              parseConfigFile: squid.conf:58 unrecognized: 'delay_pools'
              This happens for delay_pools, delay_class, delay_parameters, delay_initial and delay_access.
              It looks like squid needs to be compiled with –enable_delay_pools - the supplied version in the pbi maybe does not have this compiler flag set?

              None of these parseConfigFile messages are in the log on my 2.0.1 nanobsd system.

              Yeah that would suggest it's not honoring the build flags in the file. I opened a ticket for that here: http://redmine.pfsense.org/issues/2274

              Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

              Need help fast? Netgate Global Support!

              Do not Chat/PM for help!

              1 Reply Last reply Reply Quote 0
              • P
                phil.davis
                last edited by

                I just put the latest 2G nanobsd image http://iserv.nl/files/pfsense/releng83/i386/pfSense-2.1-DEVELOPMENT-2g-i386-nanobsd-20120319-1526.img.gz onto a CF, ran the wizard and loaded Squid.

                I get the following warnings in /tmp/PHP_errors.txt

                [19-Mar-2012 16:57:23 UTC] PHP Warning:  unlink(/etc/squid/squid_radius_auth.conf): No such file or directory in /etc/inc/pkg-utils.inc on line 802
[19-Mar-2012 16:57:23 UTC] PHP Warning:  symlink(): No such file or directory in /etc/inc/pkg-utils.inc on line 803
[19-Mar-2012 16:57:23 UTC] PHP Warning:  unlink(/etc/squid/mime.conf): No such file or directory in /etc/inc/pkg-utils.inc on line 802
[19-Mar-2012 16:57:23 UTC] PHP Warning:  symlink(): No such file or directory in /etc/inc/pkg-utils.inc on line 803
[19-Mar-2012 16:57:23 UTC] PHP Warning:  unlink(/etc/squid/squid.conf): No such file or directory in /etc/inc/pkg-utils.inc on line 802
[19-Mar-2012 16:57:23 UTC] PHP Warning:  symlink(): No such file or directory in /etc/inc/pkg-utils.inc on line 803

                This comes from the unlink and symlink calls in /etc/inc/pkg-utils.inc

                exec("/usr/local/sbin/pbi_info | grep {$pkg} | xargs /usr/local/sbin/pbi_info | awk '/Prefix/ {print $2}'",$pbidir);
$pbidir = $pbidir[0];
exec("find /usr/local/etc/ -name *.conf | grep {$pkg}",$files);
foreach($files as $f) {
	$pbiconf = str_replace('/usr/local',$pbidir,$f);
	unlink($pbiconf);
	symlink($f,$pbiconf);
}

                Perhaps this is part of the reason for the problems finding the squid.conf file?

                The system log complains about not finding the user 'squid'. It should be using username 'proxy'. This is because the proper conf file is not being used. I will apply the edits in my post above to get squid starting again. But maybe getting the above pkg_utils.inc code fragment working successfully will put symlinks in from the pbi dirs to point at the conf files we want to use in /usr/local/etc/squid - then adding the "-f" parameter to all the squid commands in scripts would not be necessary.

                As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                1 Reply Last reply Reply Quote 0
                • P
                  phil.davis
                  last edited by

                  On rebooting the squid now comes up OK (after adding the "-f" parameter to all the squid commands in scripts). The system log has the odd-looking message:

                  php: : Not calling package sync code for dependency squid of squid because some include files are missing

                  This seems like not a good thing. I looked in squid.xml but can't see a file there that is not in the dirs on disk. Squid has still come up.

                  Also, there are 2 squid processes:

                  59573  ??  INs    0:00.00 /usr/pbi/squid-i386/sbin/squid -D -f /usr/local/etc/s
60077  ??  SN     0:00.27 (squid) -D -f /usr/local/etc/squid/squid.conf (squid)

                  But maybe getting symlinks to the conf file right in the installation will prevent the 2 processes?

                  As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                  If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                  1 Reply Last reply Reply Quote 0
                  • P
                    phil.davis
                    last edited by

                    The main problem turned out to be that squid also includes squid_radius_auth (and libwww). When the code in /etc/inc/pkg-utils.inc uses pbi_info to find packages that are called squid* it finds 2 packages. The xargs pbi_info code doesn't work for 2 package names. And in any case we only want to deal with "squid" in that place.
                    As a side-issue, the output of the exec goes to $pbidir. The PHP exec doc says that if the output array is non-empty, then the output will be appended to the array. This is a possible problem, because $pbidir is used in other places in pkg-utils.inc and might have text in it already left-over from elsewhere. So it would be safer to use different variable names. You could also do isset() and unset() code before using $pbidir, to make sure it is empty.

                    Here is some code that worked for me:

                    exec("/usr/local/sbin/pbi_info | grep {$pkg}- | xargs /usr/local/sbin/pbi_info | awk '/Prefix/ {print $2}'",$pbidirarray);
$pbidir0 = $pbidirarray[0];
exec("find /usr/local/etc/ -name *.conf | grep {$pkg}",$files);
foreach($files as $f) {
	$pbiconf = str_replace('/usr/local',$pbidir0,$f);
	unlink($pbiconf);
	symlink($f,$pbiconf);
}

                    The changes to pkg-utils.inc are"

                    a) "grep {$pkg}-" : add the "-" to the package name being looked for. This prevents "squid" matching "squid_radius_auth". In general, the PBI package name is always followed by a dash and then other version, platform etc text. So this will add safety for all PBI installs. This is the 1-character addition that really makes it work!

                    b) Use unused variables $pbidirarray and $pbidir0 to prevent any possible side-effects of $pbidir that is used elsewhere.

                    Now I get just 1 squid process started once the system has booted. There is no need to add "-f /usr/local/etc/squid/squid.conf" to a lot of squid scripts. The symlink to squid.conf now gets setup correctly and squid finds the proper pfSense-generated squid.conf. This means that it runs as proxy:proxy and can find its cache OK (or know not to use a cache in the nanobsd case).

                    Note that there will still be issues for some packages who's names are substring of each other - e.g. if there is a package "auth" and "squid_radius_auth" then looking for "auth-" will all find "squid_radius_auth-". I suspect that this is a real pest all through this sort of code already! At least adding the "-" reduces these cross-package name issues. Someone who has lots of spare time can try and make sub-string selection bullet-proof through the whole package system.

                    I will put something in RedMine and GitHub about this.

                    As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                    If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                    1 Reply Last reply Reply Quote 0
                    • jimpJ
                      jimp Rebel Alliance Developer Netgate
                      last edited by

                      You can anchor the grep.

                      "^foo-"

                      Would match only if the line started with foo

                      So it may work better with:

                      grep '^{$pkg}-'

                      Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                      Need help fast? Netgate Global Support!

                      Do not Chat/PM for help!

                      1 Reply Last reply Reply Quote 0
                      • P
                        phil.davis
                        last edited by

                        I just added a pull request to add the "^" plus a few other extra checks adding/removing symlinks that tidies up the sequence: install squid, install squidGuard, remove squidGuard, remove squid. It resolves all the package install/remove interactions that I can see, particularly those caused by "squid" being a substring of "squidGuard" and "squidGuard" being a mixed-case package name. Hopefully the changes to /etc/inc/pkg-utils.inc will also fixup generic issues for these cases for other packages.
                        I have tested with the new 22 March 2012 2G nanobsd FreeBSD 8.3-RC2 snapshot.

                        As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                        If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post