No squid packages will start (user 'squid' not found) on 2.1-DEVELOPMENT

kchr · Jan 25, 2012, 2:39 PM

Running the squid-reverse package right now, and the squid binary starts as I have added a "squid" user, but something seem to change the owner of the /var/squid directories when it starts:

ls -l /var/squid/logs/access.log

-rw-r–--- 1 proxy proxy 59985 Jan 25 15:38 /var/squid/logs/access.log

Which show up like the following in the system logs:

Jan 25 15:37:14 squid[8697]: Squid Parent: child process 55094 started
Jan 25 15:37:14 squid[55094]: Cannot open '/var/squid/logs/access.log' for writing. The parent directory must be writeable by the user 'squid', which is the cache_effective_user set in squid.conf.
Jan 25 15:37:14 squid[8697]: Squid Parent: child process 55094 exited due to signal 6
Jan 25 14:37:14 kernel: pid 55094 (squid), uid 1003: exited on signal 6
Jan 25 15:37:14 squid[8697]: Exiting due to repeated, frequent failures

eXtermia · Feb 12, 2012, 3:43 PM

I have the exact same errors. I can chown or even squid -z what ever the next time it starts i lose permission to the logs and the cache. Then it dies.

With Squid 2.x and 3.x as well as the Reverse squid package. This is using 2.1-DEVELOPMENT (amd64)
built on Mon Dec 12 18:16:13 EST 2011

/usr/local/sbin(8): squid -z 2012/02/12 03:39:22| Creating Swap Directories
FATAL: Failed to make swap directory /var/squid/cache/01/00: (13) Permission denied
Squid Cache (Version 2.7.STABLE9): Terminated abnormally.
CPU Usage: 0.002 seconds = 0.000 user + 0.002 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 0

[2.1-DEVELOPMENT][root@pfsense.]/usr/local/sbin(28): chown squid /var/squid/cache/0A
[2.1-DEVELOPMENT][root@pfsense.]/usr/local/sbin(29): chown squid /var/squid/cache/0B
[2.1-DEVELOPMENT][root@pfsense.]/usr/local/sbin(30): chown squid /var/squid/cache/0C
[2.1-DEVELOPMENT][root@pfsense.]/usr/local/sbin(31): chown squid /var/squid/cache/0D
[2.1-DEVELOPMENT][root@pfsense.]/usr/local/sbin(32): chown squid /var/quid/cache/0E
[2.1-DEVELOPMENT][root@pfsense.]/usr/local/sbin(33): chown squid /var/quid/cache/0F
[2.1-DEVELOPMENT][root@pfsense.]/usr/local/sbin(34): squid -z 2012/02/12 03:41:02| Creating Swap Directories
[2.1-DEVELOPMENT][root@pfsense.]/usr/local/sbin(35): chown squid /var/squid/logs/
[2.1-DEVELOPMENT][root@pfsense.]/usr/local/sbin(36): chown squid /var/squid/logs/access.log
[2.1-DEVELOPMENT][root@pfsense.]/usr/local/sbin(37): pw useradd -g proxy -s /sbin/nologin -d /var/squid -n squid
pw: login name `squid' already exists
[2.1-DEVELOPMENT][root@pfsense.]/usr/local/sbin(38): squid -z 2012/02/12 03:47:18| Creating Swap Directories
[2.1-DEVELOPMENT][root@pfsense.]/usr/local/sbin(39): squid -k shutdown
squid: ERROR: No running copy
[2.1-DEVELOPMENT][root@]/usr/local/sbin(40): squid -k rotate
squid: ERROR: No running copy
[2.1-DEVELOPMENT][root@pfsense.]/usr/local/sbin(42): chown squid /var/squid/logs/store.log
[2.1-DEVELOPMENT][root@pfsense.]/usr/local/sbin(43):
[2.1-DEVELOPMENT][root@pfsense.]/usr/local/sbin(43): 2012/02/12 03:40:24| Creating Swap Directories
FATAL: Failed to make swap directory /var/squid/cache/09/00: (13) Permission denied

this goes for the logs too ( i fixed those first then had problems with the cache)

after I got everything running the service started but the first time I made a change all persmissions were removed again and I lost all

marcelloc · Feb 13, 2012, 4:38 PM

chown -R squid /var/squid/cache/

or

rm -rf /var/squid/cache/*

then try to start squid.

eXtermia · Feb 22, 2012, 2:07 AM

no go, service starts once but any configuration changes at all results back to

Feb 22 02:41:24 squid[28812]: Squid Parent: child process 26120 started
Feb 22 02:41:24 squid[26120]: Cannot open '/var/squid/logs/access.log' for writing. The parent directory must be writeable by the user 'squid', which is the cache_effective_user set in squid.conf.
Feb 22 02:41:24 squid[28812]: Squid Parent: child process 26120 exited due to signal 6
Feb 22 02:41:24 kernel: pid 26120 (squid), uid 100: exited on signal 6
Feb 22 02:41:27 squid[28812]: Squid Parent: child process 26494 started
Feb 22 02:41:27 squid[26494]: Cannot open '/var/squid/logs/access.log' for writing. The parent directory must be writeable by the user 'squid', which is the cache_effective_user set in squid.conf.
Feb 22 02:41:27 squid[28812]: Squid Parent: child process 26494 exited due to signal 6
Feb 22 02:41:27 kernel: pid 26494 (squid), uid 100: exited on signal 6
Feb 22 02:41:27 squid[28812]: Exiting due to repeated, frequent failures

strangest thing is I changed the logs to '/var/squid/logs123' but the error still shows up as Cannot open '/var/squid/logs/access.log' for writing. even although the /usr/local/etc/squid/squid.conf
says

icon_directory /usr/local/etc/squid/icons
visible_hostname localhost
cache_mgr admin@localhost
access_log /var/squid/log123/access.log
cache_log /var/squid/log123/cache.log
cache_store_log none
I have repeatedly chown the log folder
but as soon as the sevices starts, stops, or reconfigures it goes back to failing

even disabling the logging makes no change

eXtermia · Feb 28, 2012, 11:30 PM

even more strange news.

Still same problems from the web interface but if I

/usr/local/sbin(127): squid -s

/usr/local/sbin(128): ps ax | grep squid
3130 ?? Is 0:00.00 /usr/pbi/squid-amd64/sbin/squid -s
3284 ?? S 0:00.02 (squid) -s (squid)
55633 0 R+ 0:00.00 grep squid

/usr/local/sbin(129): squid -k reconfigure

squid runs but as soon as I change ANYTHING in the web GUI I again have to
chown -R squid /var/squid/

and then
squid -s to get it to run again

restarting from the GUI it always fails and always lets the permissons on the access.log unaccessible.

but running the command from shell it works again like above.. a wth moment or what?

phil.davis · Mar 6, 2012, 4:43 AM

I have all the same issues on a 1G nanobsd system running 2.1-DEVELOPMENT

After installing Squid from the package installer web interface I had to:

pw useradd -g proxy -s /sbin/nologin -d /var/squid -n squid
chown -R squid /var/squid
mkdir /var/squid/cache
squid -z

It looks like it starts OK from the WebGUI, /var/squid/logs/cache.log has good looking stuff in it, the system log looks like it has started a process for the service. But "ps ax | grep squid" doesn't find a process any more! It disappears for some reason that I haven't worked out yet.

squid -s

starts it happily and it runs.

So there are issues with the Squid installation scripts and startup mechanism on 2.1-DEVELOPMENT.

Note: 2.1-DEVELOPMENT uses the PBI package system. It fetches the squid 2.7.9-1 pbi ffile and installs it.

phil.davis · Mar 6, 2012, 5:27 AM

Since the /var filesystem is only transient on the nanobsd, the /var/squid stuff does not get recreated after a reboot. So, on 2.1-DEVELOPMENT, after startup, to get Squid going, the following is done from a command prompt:

chown -R squid /var/squid
mkdir /var/squid/cache
chown -R squid /var/squid/cache
squid -z
squid -s

The squid username is preserved - that lives in /etc/passwd on the CF card.
The /var/squid dir got created by something, so there must be some script that is trying to setup things for squid, but doesn't get too far.

jimp · Mar 7, 2012, 4:25 PM

That should be all done by squid_resync() that should be run when squid starts at bootup.

Next time you reboot, try to do something like this in Diag > Command, PHP exec box:
include 'squid.inc';
squid_resync();

then see if it works.

phil.davis · Mar 8, 2012, 5:54 AM

As suggested, after rebooting, I did:
include 'squid.inc';
squid_resync();

No joy, the system log complains that there is no /var/squid/cache dir.

ls -ld /var/squid
drwxr-xr-x 5 proxy proxy 512 Mar 8 11:19 /var/squid

The squid dir is owned by proxy, not by squid.

After manually resetting the owner, creating /var/squid/cache and doing "squid -z", "squid -s" it is fine. But then after a while (I think after doing other stuff in the web GUI) /var/squid goes back to being owned by proxy and squid does not work any more. So it seems that there is code in webGUI php scripts somewhere that doesn't set the squid owner correctly - if that is fixed then maybe all the downstream effects/problems will be resolved.

phil.davis · Mar 8, 2012, 12:01 PM

The difference between 2.0.1 and 2.1-DEVELOPMENT is that the package is installed using a PBI. The "squid" program in /usr/local/sbin is now just a link to:
/usr/pbi/squid-i386/.sbin/squid

There is a default squid.conf in:
/usr/pbi/squid-i386/.etc/squid/squid.conf

The system seems to be using this conf file, which specifies cache_effective_user squid - and from that point all the /var/squid file owner issues occur.

The conf file that is supposed to be used is /usr/local/etc/squid/squid.conf

I modified /usr/local/pkg/squid.inc - on the end of all places that run "/usr/local/sbin/squid -D" add " -f /usr/local/etc/squid/squid.conf"
That makes it use the pfSense-specific squid.conf file.

There are still places that do "squid -k" commands to get Squid to reread its conf file, and I get some messages about 'squid: ERROR: No running copy' - I think that adding the "-f" parameter means that other checks for the squid process might need to be modified.

An easier solution might be to put an actual copy of the squid program into /usr/local/sbin rather than a link, then it might find its conf file OK?

phil.davis · Mar 9, 2012, 5:31 AM

I tried putting a real copy of the squid program in /usr/local/sbin
That doesn't work, it still uses /usr/pbi/squid-i386/etc/squid/squid.conf
It seems that the default squid.conf location is an absolute path hard-coded into the program. I was hoping that it would be a relative path (relative to the location that the squid program was run from), but not so.

I have got Squid and SquidGuard running nicely on 2.0.1-DEVELOPMENT by editing /usr/local/pkg/squid.inc
(a) Change all the occurrences of "squid -D" to "squid -D -f /usr/local/etc/squid/squid.conf"
(b) Change all the occurrences of "squid -k *" to "squid -k * -f /usr/local/etc/squid/squid.conf"
(where * is reconfigure, rotate, shutdown, kill)

(a) makes it use the correct conf file at startup.
(b) makes it find the squid process to change its configuration, rotate log files or stop it.

These changes are also needed in:
squid_ng.xml
squidguard_configurator.inc
swapstate_check.php

Whoever maintains the squid package, can they make this change (or another better designed one) to squid.inc for 2.1?

phil.davis · Mar 9, 2012, 5:55 AM

I suspect that Squid Traffic Management will not work (but I haven't tested it).
/var/squid/logs/cache.log reports unrecognized parameters on squid startup, lines like
parseConfigFile: squid.conf:58 unrecognized: 'delay_pools'
This happens for delay_pools, delay_class, delay_parameters, delay_initial and delay_access.
It looks like squid needs to be compiled with –enable_delay_pools - the supplied version in the pbi maybe does not have this compiler flag set?

None of these parseConfigFile messages are in the log on my 2.0.1 nanobsd system.

phil.davis · Mar 9, 2012, 6:01 AM

SquidGuard timed rules work on 2.1-DEVELOPMENT.
I tried a rule that turned on and off every 10 minutes for a hour or so.
/var/squidGuard/log/squidGuard.log contained regular "Info: recalculating alarm in nn seconds" messages.
The blocked website became available and blocked as the time changed.
(Note that you often have to be careful to clear the browser cache when doing this testing, otherwise you can just be looking at locally-cached data in the client.)
On my 2.0.1 nanobsd system, I get "Info: recalculating alarm in nn seconds" messages a couple of times, then they just stop appearing in the log file. It seems to just forget that there are timed rules to calculate.
So, it looks like this problem in 2.0.1 is fixed in 2.1

jimp · Mar 9, 2012, 5:15 PM

@phil.davis:

I have got Squid and SquidGuard running nicely on 2.0.1-DEVELOPMENT by editing /usr/local/pkg/squid.inc
(a) Change all the occurrences of "squid -D" to "squid -D -f /usr/local/etc/squid/squid.conf"
(b) Change all the occurrences of "squid -k *" to "squid -k * -f /usr/local/etc/squid/squid.conf"
(where * is reconfigure, rotate, shutdown, kill)

(a) makes it use the correct conf file at startup.
(b) makes it find the squid process to change its configuration, rotate log files or stop it.

These changes are also needed in:
squid_ng.xml
squidguard_configurator.inc
swapstate_check.php

Whoever maintains the squid package, can they make this change (or another better designed one) to squid.inc for 2.1?

I can do that but I won't have time to get to that today. That should be a safe change to make both on 2.0 and 2.1 though, but it would need to be tested. If someone wants to do that and make a merge request on github we can pull it in, otherwise it'll be sometime next week before I can get to it.

@phil.davis:

I suspect that Squid Traffic Management will not work (but I haven't tested it).
/var/squid/logs/cache.log reports unrecognized parameters on squid startup, lines like
parseConfigFile: squid.conf:58 unrecognized: 'delay_pools'
This happens for delay_pools, delay_class, delay_parameters, delay_initial and delay_access.
It looks like squid needs to be compiled with –enable_delay_pools - the supplied version in the pbi maybe does not have this compiler flag set?

None of these parseConfigFile messages are in the log on my 2.0.1 nanobsd system.

Yeah that would suggest it's not honoring the build flags in the file. I opened a ticket for that here: http://redmine.pfsense.org/issues/2274

phil.davis · Mar 19, 2012, 6:35 PM

I just put the latest 2G nanobsd image http://iserv.nl/files/pfsense/releng83/i386/pfSense-2.1-DEVELOPMENT-2g-i386-nanobsd-20120319-1526.img.gz onto a CF, ran the wizard and loaded Squid.

I get the following warnings in /tmp/PHP_errors.txt

[19-Mar-2012 16:57:23 UTC] PHP Warning:  unlink(/etc/squid/squid_radius_auth.conf): No such file or directory in /etc/inc/pkg-utils.inc on line 802
[19-Mar-2012 16:57:23 UTC] PHP Warning:  symlink(): No such file or directory in /etc/inc/pkg-utils.inc on line 803
[19-Mar-2012 16:57:23 UTC] PHP Warning:  unlink(/etc/squid/mime.conf): No such file or directory in /etc/inc/pkg-utils.inc on line 802
[19-Mar-2012 16:57:23 UTC] PHP Warning:  symlink(): No such file or directory in /etc/inc/pkg-utils.inc on line 803
[19-Mar-2012 16:57:23 UTC] PHP Warning:  unlink(/etc/squid/squid.conf): No such file or directory in /etc/inc/pkg-utils.inc on line 802
[19-Mar-2012 16:57:23 UTC] PHP Warning:  symlink(): No such file or directory in /etc/inc/pkg-utils.inc on line 803

This comes from the unlink and symlink calls in /etc/inc/pkg-utils.inc

exec("/usr/local/sbin/pbi_info | grep {$pkg} | xargs /usr/local/sbin/pbi_info | awk '/Prefix/ {print $2}'",$pbidir);
$pbidir = $pbidir[0];
exec("find /usr/local/etc/ -name *.conf | grep {$pkg}",$files);
foreach($files as $f) {
	$pbiconf = str_replace('/usr/local',$pbidir,$f);
	unlink($pbiconf);
	symlink($f,$pbiconf);
}

Perhaps this is part of the reason for the problems finding the squid.conf file?

The system log complains about not finding the user 'squid'. It should be using username 'proxy'. This is because the proper conf file is not being used. I will apply the edits in my post above to get squid starting again. But maybe getting the above pkg_utils.inc code fragment working successfully will put symlinks in from the pbi dirs to point at the conf files we want to use in /usr/local/etc/squid - then adding the "-f" parameter to all the squid commands in scripts would not be necessary.

phil.davis · Mar 19, 2012, 7:15 PM

On rebooting the squid now comes up OK (after adding the "-f" parameter to all the squid commands in scripts). The system log has the odd-looking message:

php: : Not calling package sync code for dependency squid of squid because some include files are missing

This seems like not a good thing. I looked in squid.xml but can't see a file there that is not in the dirs on disk. Squid has still come up.

Also, there are 2 squid processes:

59573  ??  INs    0:00.00 /usr/pbi/squid-i386/sbin/squid -D -f /usr/local/etc/s
60077  ??  SN     0:00.27 (squid) -D -f /usr/local/etc/squid/squid.conf (squid)

But maybe getting symlinks to the conf file right in the installation will prevent the 2 processes?

phil.davis · Mar 20, 2012, 10:39 AM

The main problem turned out to be that squid also includes squid_radius_auth (and libwww). When the code in /etc/inc/pkg-utils.inc uses pbi_info to find packages that are called squid* it finds 2 packages. The xargs pbi_info code doesn't work for 2 package names. And in any case we only want to deal with "squid" in that place.
As a side-issue, the output of the exec goes to $pbidir. The PHP exec doc says that if the output array is non-empty, then the output will be appended to the array. This is a possible problem, because $pbidir is used in other places in pkg-utils.inc and might have text in it already left-over from elsewhere. So it would be safer to use different variable names. You could also do isset() and unset() code before using $pbidir, to make sure it is empty.

Here is some code that worked for me:

exec("/usr/local/sbin/pbi_info | grep {$pkg}- | xargs /usr/local/sbin/pbi_info | awk '/Prefix/ {print $2}'",$pbidirarray);
$pbidir0 = $pbidirarray[0];
exec("find /usr/local/etc/ -name *.conf | grep {$pkg}",$files);
foreach($files as $f) {
	$pbiconf = str_replace('/usr/local',$pbidir0,$f);
	unlink($pbiconf);
	symlink($f,$pbiconf);
}

The changes to pkg-utils.inc are"

a) "grep {$pkg}-" : add the "-" to the package name being looked for. This prevents "squid" matching "squid_radius_auth". In general, the PBI package name is always followed by a dash and then other version, platform etc text. So this will add safety for all PBI installs. This is the 1-character addition that really makes it work!

b) Use unused variables $pbidirarray and $pbidir0 to prevent any possible side-effects of $pbidir that is used elsewhere.

Now I get just 1 squid process started once the system has booted. There is no need to add "-f /usr/local/etc/squid/squid.conf" to a lot of squid scripts. The symlink to squid.conf now gets setup correctly and squid finds the proper pfSense-generated squid.conf. This means that it runs as proxy:proxy and can find its cache OK (or know not to use a cache in the nanobsd case).

Note that there will still be issues for some packages who's names are substring of each other - e.g. if there is a package "auth" and "squid_radius_auth" then looking for "auth-" will all find "squid_radius_auth-". I suspect that this is a real pest all through this sort of code already! At least adding the "-" reduces these cross-package name issues. Someone who has lots of spare time can try and make sub-string selection bullet-proof through the whole package system.

I will put something in RedMine and GitHub about this.

jimp · Mar 22, 2012, 4:56 PM

You can anchor the grep.

"^foo-"

Would match only if the line started with foo

So it may work better with:

grep '^{$pkg}-'

phil.davis · Mar 23, 2012, 7:36 AM

I just added a pull request to add the "^" plus a few other extra checks adding/removing symlinks that tidies up the sequence: install squid, install squidGuard, remove squidGuard, remove squid. It resolves all the package install/remove interactions that I can see, particularly those caused by "squid" being a substring of "squidGuard" and "squidGuard" being a mixed-case package name. Hopefully the changes to /etc/inc/pkg-utils.inc will also fixup generic issues for these cases for other packages.
I have tested with the new 22 March 2012 2G nanobsd FreeBSD 8.3-RC2 snapshot.