Squid and SquidGuard does not start after reboot

lsoltero

I just thought we were done…

all the subsystems are working fine... however, i can't modify any of the cache settings for squid.

when I try to set the hard disk cache size, memory cache size, and maximum object size under proxy server->cache management the web page resets the value after you hit save. I have confirmed that the values are not being updated in squid.conf

cache_mem 8 MB
maximum_object_size_in_memory 32 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir aufs /var/squid/cache 100 16 256
minimum_object_size 0 KB
maximum_object_size 10 KB
offline_mode off

so what causes this?

oh man.... take a look at the entry for squidcache in config.xlm

<squidcache><config><config><harddisk_cache_size>25600</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>128</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>2465792</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config>
<config><harddisk_cache_size>25600</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>128</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>4194304</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config>
<config><harddisk_cache_size>15360</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>128</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>1048576</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config>
<config><harddisk_cache_size>1000</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>80</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>4</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config>
<config><harddisk_cache_size>25600</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>128</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>4194304</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config></config></squidcache>

I would say there are a few too many entries here...what would cause this? It seems that every time I hit add a new config gets written but the old one is not removed!

the

<squidcache><config>extra <config>right at the head of this configuration seems particularly disturbing...

I will edit the config.xml manually to see if I can get this to work....

--luis</config></config></squidcache>

lsoltero

OK… that worked... here is squid.conf

cache_mem 128 MB
maximum_object_size_in_memory 32 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir aufs /var/squid/cache 25600 16 256
minimum_object_size 0 KB
maximum_object_size 2465792 KB
offline_mode off
cache_swap_low 90
cache_swap_high 95

and here is the entry in config.xml
<squidcache><config><harddisk_cache_size>25600</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>128</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>2465792</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config></squidcache>

so something went awol with webinterface/xml editor.

any ideas?

--luis

jimp

Yeah that is a little odd. I committed a change that makes squid's default log dir match. That seemed like the better choice.

lsoltero

I take it that this is a bug in the webconfigurator and not the squid package… I will search the forum to see if anyone else has reported this issue.

--luis

lsoltero

Here is another bug in /usr/local/pkg/squid.inc which causes an incompatibility in log rotation with lightsquid.

Lightsquid has a much more flexible squid log rotation facility. When log rotation is disabled in squid (as it should be) and enabled in lightsquid, the squid package removes all the squid -k rotate entries from the crontabs.

here is the sequence of events.

1. disable log rotation is squid
2. enable it in lightsquid and hit save.
3. lightsquid correctly updates cron registering an entry which looks like

<task_name>lightsquid_squid_rotate</task_name>
<minute>0</minute>
<hour>0</hour>
<mday></mday>
<month></month>
<wday>*/1</wday>
<who>root</who>
<command></command>/usr/local/sbin/squid -k rotate > /dev/null

with an appropriate task_name…

4. after the lightsquid config files are saved the system invokes a resync squid which executes squid_install_cron(false) which is correct since squid itself is not doing log rotation...

However, looking at the code in squid.inc we notice

foreach($config['cron']['item'] as $item) {
if(strstr($item['command'], "/usr/local/sbin/squid")) {
$is_installed = true;
break;
}
$x++;
}

which basically means that squid is tromping through the crontab looking for any entry with /usr/local/sbin/squid in it and zaps it including the entry just added by lightsquid!!!!

If you enable log rotation in squid and then again in lightsquid you get 2 entries in the crontab as expected. one created for each package.

*/140 * * * * root /usr/local/sbin/reset_slbd.sh
0 0 * * * root /usr/local/sbin/squid -k rotate
0 */2 * * * root /usr/bin/perl /usr/local/www/lightsquid/lightparser.pl today
15 0 * * * root /usr/bin/perl /usr/local/www/lightsquid/lightparser.pl yesterday
0 0 * * */1 root /usr/local/sbin/squid -k rotate > /dev/null

looking at /conf/config.xml you find the following 2 entries in cron

the one added by squid

<minute>0</minute>
<hour>0</hour>
<mday></mday>
<month></month>
<wday>*</wday>
<who>root</who>
<command></command>/usr/local/sbin/squid -k rotate

and the one added by light squid

<task_name>lightsquid_squid_rotate</task_name>
<minute>0</minute>
<hour>0</hour>
<mday></mday>
<month></month>
<wday>*/1</wday>
<who>root</who>
<command></command>/usr/local/sbin/squid -k rotate > /dev/null

note that the entry added by squid does not have a task_name.

If you now disable log rotation in squid then ALL entries for "squid -k rotate" are removed including the one added by lightsquid. The net result is that it is impossible to get log rotation to work in lightsquid without enabling it in squid. However, enabling it in squid results in 2 entries in cron the daily one added by squid overriding the one set in lightsquid. So basically, lightsquid rotation of squid logs does not currently work.

squid.inc should be modified to add a task_name to the squid -k rotate entry it creates. When removing entries from cron it should only remove the entries it added… not ** ALL ** the entries it finds with squid in them.

the solution is pretty simple... if instead of searching for squid squid_install_cron() is modified to use a unique "task_name" then everything plays together well.

Here is a patch..

diff squid.inc squid.inc.orig

557c557
< if(strstr($item['task_name'], "squid_rotate_logs")) {
–-

if(strstr($item['command'], "/usr/local/sbin/squid")) {
567d566
< $cron_item['task_name'] = "squid_rotate_logs";

After the patch squid adds/removes its own cron entries without bothering the ones managed by light squid. Now you can enable log rotation in lightsquid as expected.

–luis

jimp

Good find!

I'll commit that shortly.

jimp

It should be fixed now. I bumped the version on the package so once it's in the repo (about 5 minutes or so) it will show that it needs updated in the gui.

lsoltero

Working great… Here is the new crontab

more /etc/crontab

SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin
HOME=/var/log
#minute hour mday month wday who command

pfSense specific crontab entries

Created: July 11, 2010, 6:06 pm

0 * * * * root /usr/bin/nice -n20 newsyslog
1,31 0-5 * * * root /usr/bin/nice -n20 adjkerntz -a
1 3 1 * * root /usr/bin/nice -n20 /etc/rc.update_bogons.sh
*/60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 sshlockout
1 1 * * * root /usr/bin/nice -n20 /etc/rc.dyndns.update
*/60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot
*/5 * * * * root /usr/local/bin/checkreload.sh
*/5 * * * * root /etc/ping_hosts.sh
*/140 * * * * root /usr/local/sbin/reset_slbd.sh
0 */2 * * * root /usr/bin/perl /usr/local/www/lightsquid/lightparser.pl today
15 0 * * * root /usr/bin/perl /usr/local/www/lightsquid/lightparser.pl yesterday
0 0 */1 * * root /usr/local/sbin/squid -k rotate > /dev/null

If possible do not add items to this file manually.

If you do so, this file must be terminated with a blank line (e.g. new line)

Notice the single squid -k rotate with the weekly rotation added by lightsquid.

A few notes on upgrading.

1. Turn of log rotation in squid before upgrading. If squid is configured with logrotation and the upgrade is done then the new squid will not remove (and can't) remove the squid -k rotate entry since it no longer owns. It doesn't matter if lightsquid rotation is left on when the upgrade is done.

2. after the upgrade lightsquid stops working.... so you need to reinstall lightsquid... no big deal... i reinstalled squidGuard as well just to make darn sure.

3. the xml config files for squidGuard and lightSquid get zapped in the reinstall so PHP errors are displayed when the services are started. These are benign and nothing to worry about.

4. after reinstalling squidGuard and lightSquid you must go to the corresponding config pages in the webadmin gui and save and restart the configuration.

After that check /etc/cron to make sure that log rotation is taking place.

one last note... since squid -k rotate is being done by lightsquid and not squid this means there is no logfile_rotate entry in squid.conf specifying the number of logs to keep. The default is 10.... so squid.log will rotate through 0-9 with the oldest being removed. It probably is not a good idea (or maybe it is) to have lightsquid edit squid.conf although there are examples where it does.... For example, the lightsquid service modifies the log path (/var/squid/log) for squid... so maybe lightsquid should also modify the logile_rotate field squid.conf as well. If this is to be done then the webadmin page for lightsquid would have to be updated to get the number of logs to keep.

Here is the code in lightsquid_resync() updates the squid.conf file

// update squid conf
if (isset($config['installedpackages']['squid']['config'][0])) {
$config['installedpackages']['squid']['config'][0]['log_enabled'] = 'on';
$config['installedpackages']['squid']['config'][0]['log_dir'] = LS_SQUIDLOGPATH;
write_config();
squid_resync();
}

Maybe this should include a

$config['installedpackages']['squid']['config'][0]['logile_rotate'] = X;

where X is the user specified number of copies of the log file to keep.

However, having said all that 10 if fine for me.

And BTW… the language in the lightsquid admin which describes the logrotation probably should be cleaned up.. Here is how it currently reads.

Select squid log rotate period. System will execute task every XX time as from 00:00 hours.
This option will allow the updating of the faster
For example: if selected '2 day' - system wil start task every 2 day of month.
This option will allow the updating of the faster
Note: You must choose from that the rate of filling dialogue access.log squid;
The more customers, the more often it should be the job.

You definitely don't want to "choose from that the rate of filling dialogue access.log squid;"... in the squid config page... By enabling log rotation and the number of log files in the squid admin page you endup turning on squid log rotation in squid resulting in 2 "squid -k rotate" entries in the cronfile!

Anyway... you might let the lightsquid maintainer know that it would be good to clean up the description to have explain how this all works.

Take care and thanks for your prompt response on this issue.

--luis

lsoltero

hm… it just occurred to me maybe by doing

$config['installedpackages']['squid']['config'][0]['logile_rotate'] = X;

followed by a squid_sync(); might cause squid to add its on cron entry for log rotation… anyway.. you get the general idea. Some experimentation is required here to get this to work correctly.

--luis

jimp

You might want to start a new thread for that issue, since this is getting pretty far from the original package. The lightsquid maintainer isn't likely to notice it this far in.

If the normal maintainer doesn't notice it, I might have time to take a look later on this week.

lsoltero

OK. the discussion has now been moved to

http://forum.pfsense.org/index.php/topic,26604.0.html

Take care.

–luis