Squid and SquidGuard does not start after reboot
-
certainly getting the fix into the tree seems like a reasonable thing to do…
Squid users having problems hopefully will find this form thread and will then know that to fix the problem the need to edit /etc/rc
go to line 322 and delete it.i.e.
make this
Remove stale files that have already been processed by bootup
scripts
rm -f /tmp/filter_dirty
rm -f /tmp/rc.linkuplook like this
Remove stale files that have already been processed by bootup
scripts
rm -f /tmp/rc.linkup
--luis
-
I just added a package to apply a simple patch that removes the line in question.
Give it a spin and see if it helps. It's called "Patch rc to leave filter_dirty" (for lack of a better name :-)
-
it failed!
Remove stale files that have already been processed by bootup
scripts
rm -f /tmp/filter_dirty
rm -f /tmp/rc.linkup
nohup /usr/bin/nice -n20 /usr/local/sbin/check_reload_status &Here is the list of installed packages on my system
Patch rc to leave filter_dirty System No info, check the forum 0.1 Patch to stop /etc/rc from removing /tmp/filter_dirty on boot. Fixes boot issues with some packages on certain platforms.
squid Network No info, check the forum Current: 2.7.9
Installed: 2.7.8_1 High performance web proxy cache.squidGuard Network Management No info, check the forum 1.3-2 High perfomance web proxy URL filter. Required proxy Squid package.
Let me know when you want me to try it again.
–luis
-
Apparently I messed up the path in the patch. I fixed it, should be OK now.
-
-
Did you break squid trying to fix this issue per chance?
Here is my current situation.
1. brand new 1.2.3-RELEASE installation with a few configuration chances as required to get the router working on a LAN/WAN
2. installed the filter_dirty package
3. installed squid
4. turn on transparent proxysquid will not run… proxy_monitor fires it up and it shuts down right away... On reboot the only mesg I see when syncing packages is the rc filter_dirty fix and squid that is it...
ps aux | grep squid
shows that squid is not running... proxy_monitor is running and continually tries to start squid.if you try to go to any website the pf rule redirects you to the pfsense box which asks for a username and password. You enter the pfsense admin username and password and then it shows you the main admin website.
So... again the proxy is not running.
any ideas?
Here is a list of the packages that are installed.
Patch rc to leave filter_dirty System No info, check the forum 0.1 Patch to stop /etc/rc from removing /tmp/filter_dirty on boot. Fixes boot issues with some packages on certain platforms.squid Network No info, check the forum 2.7.9 High performance web proxy cache.
--luis
-
Well I updated squid to 2.7.9 separately from trying to fix this, but I thought I had tested it on nanobsd at the time. That was late last week though, and I don't recall the specifics.
Is there nothing in the system log about squid at all?
-
OK found the problem…
in a clean installation /var/squid is not setup with the correct permissions.
/var/squid is owned by root:wheel with permission 755
many of the files such as access.log and owned by squid:squid with permission 640 when they should be owned by proxy.
chown -R proxy /var/squid
solves the problem.
So... this issue has nothing to do with the filter_dirty patch... However, some time in the recent past some changes must have been made to the squid package to break the permissions in /var/squid...
--luis
-
ok, I committed a fix for that. It should be in the package repo in about 2 minutes.
-
OK… it all seems to work now... squid, lightsquid, and squidGuard all install and play nicely.
One very small thing...
i notice that now by default squid install its log files into /var/squid/logs [note the (s)]. When you install and configure lightsquid the package automatically chances the logging directory for squid to /var/squid/log [note the absence of (s)]. This is not really a problem because it doen't break anything but the access files in the original /var/squid/logs folder never get rotated or deleted. They stay there forever… This occupies resources if the log file happens to be big before you turn on lightSquid.
Thanks again for your help.
--luis
-
I just thought we were done…
all the subsystems are working fine... however, i can't modify any of the cache settings for squid.
when I try to set the hard disk cache size, memory cache size, and maximum object size under proxy server->cache management the web page resets the value after you hit save. I have confirmed that the values are not being updated in squid.conf
cache_mem 8 MB
maximum_object_size_in_memory 32 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir aufs /var/squid/cache 100 16 256
minimum_object_size 0 KB
maximum_object_size 10 KB
offline_mode offso what causes this?
oh man.... take a look at the entry for squidcache in config.xlm
<squidcache><config><config><harddisk_cache_size>25600</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>128</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>2465792</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config>
<config><harddisk_cache_size>25600</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>128</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>4194304</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config>
<config><harddisk_cache_size>15360</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>128</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>1048576</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config>
<config><harddisk_cache_size>1000</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>80</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>4</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config>
<config><harddisk_cache_size>25600</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>128</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>4194304</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config></config></squidcache>I would say there are a few too many entries here...what would cause this? It seems that every time I hit add a new config gets written but the old one is not removed!
the
<squidcache><config>extra <config>right at the head of this configuration seems particularly disturbing...
I will edit the config.xml manually to see if I can get this to work....
--luis</config></config></squidcache>
-
OK… that worked... here is squid.conf
cache_mem 128 MB
maximum_object_size_in_memory 32 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir aufs /var/squid/cache 25600 16 256
minimum_object_size 0 KB
maximum_object_size 2465792 KB
offline_mode off
cache_swap_low 90
cache_swap_high 95and here is the entry in config.xml
<squidcache><config><harddisk_cache_size>25600</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>128</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>2465792</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config></squidcache>so something went awol with webinterface/xml editor.
any ideas?
--luis
-
Yeah that is a little odd. I committed a change that makes squid's default log dir match. That seemed like the better choice.
-
I take it that this is a bug in the webconfigurator and not the squid package… I will search the forum to see if anyone else has reported this issue.
--luis
-
Here is another bug in /usr/local/pkg/squid.inc which causes an incompatibility in log rotation with lightsquid.
Lightsquid has a much more flexible squid log rotation facility. When log rotation is disabled in squid (as it should be) and enabled in lightsquid, the squid package removes all the squid -k rotate entries from the crontabs.
here is the sequence of events.
1. disable log rotation is squid
2. enable it in lightsquid and hit save.
3. lightsquid correctly updates cron registering an entry which looks like<task_name>lightsquid_squid_rotate</task_name>
<minute>0</minute>
<hour>0</hour>
<mday></mday>
<month></month>
<wday>*/1</wday>
<who>root</who>
<command></command>/usr/local/sbin/squid -k rotate > /dev/nullwith an appropriate task_name…
4. after the lightsquid config files are saved the system invokes a resync squid which executes squid_install_cron(false) which is correct since squid itself is not doing log rotation...
However, looking at the code in squid.inc we notice
foreach($config['cron']['item'] as $item) {
if(strstr($item['command'], "/usr/local/sbin/squid")) {
$is_installed = true;
break;
}
$x++;
}which basically means that squid is tromping through the crontab looking for any entry with /usr/local/sbin/squid in it and zaps it including the entry just added by lightsquid!!!!
If you enable log rotation in squid and then again in lightsquid you get 2 entries in the crontab as expected. one created for each package.
*/140 * * * * root /usr/local/sbin/reset_slbd.sh
0 0 * * * root /usr/local/sbin/squid -k rotate
0 */2 * * * root /usr/bin/perl /usr/local/www/lightsquid/lightparser.pl today
15 0 * * * root /usr/bin/perl /usr/local/www/lightsquid/lightparser.pl yesterday
0 0 * * */1 root /usr/local/sbin/squid -k rotate > /dev/nulllooking at /conf/config.xml you find the following 2 entries in cron
the one added by squid
<minute>0</minute>
<hour>0</hour>
<mday></mday>
<month></month>
<wday>*</wday>
<who>root</who>
<command></command>/usr/local/sbin/squid -k rotateand the one added by light squid
<task_name>lightsquid_squid_rotate</task_name>
<minute>0</minute>
<hour>0</hour>
<mday></mday>
<month></month>
<wday>*/1</wday>
<who>root</who>
<command></command>/usr/local/sbin/squid -k rotate > /dev/nullnote that the entry added by squid does not have a task_name.
If you now disable log rotation in squid then ALL entries for "squid -k rotate" are removed including the one added by lightsquid. The net result is that it is impossible to get log rotation to work in lightsquid without enabling it in squid. However, enabling it in squid results in 2 entries in cron the daily one added by squid overriding the one set in lightsquid. So basically, lightsquid rotation of squid logs does not currently work.
squid.inc should be modified to add a task_name to the squid -k rotate entry it creates. When removing entries from cron it should only remove the entries it added… not ** ALL ** the entries it finds with squid in them.
the solution is pretty simple... if instead of searching for squid squid_install_cron() is modified to use a unique "task_name" then everything plays together well.
Here is a patch..
diff squid.inc squid.inc.orig
557c557
< if(strstr($item['task_name'], "squid_rotate_logs")) {
–-if(strstr($item['command'], "/usr/local/sbin/squid")) {
567d566
< $cron_item['task_name'] = "squid_rotate_logs";After the patch squid adds/removes its own cron entries without bothering the ones managed by light squid. Now you can enable log rotation in lightsquid as expected.
–luis
-
Good find!
I'll commit that shortly.
-
It should be fixed now. I bumped the version on the package so once it's in the repo (about 5 minutes or so) it will show that it needs updated in the gui.
-
Working great… Here is the new crontab
more /etc/crontab
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin
HOME=/var/log
#minute hour mday month wday who commandpfSense specific crontab entries
Created: July 11, 2010, 6:06 pm
0 * * * * root /usr/bin/nice -n20 newsyslog
1,31 0-5 * * * root /usr/bin/nice -n20 adjkerntz -a
1 3 1 * * root /usr/bin/nice -n20 /etc/rc.update_bogons.sh
*/60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 sshlockout
1 1 * * * root /usr/bin/nice -n20 /etc/rc.dyndns.update
*/60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot
*/5 * * * * root /usr/local/bin/checkreload.sh
*/5 * * * * root /etc/ping_hosts.sh
*/140 * * * * root /usr/local/sbin/reset_slbd.sh
0 */2 * * * root /usr/bin/perl /usr/local/www/lightsquid/lightparser.pl today
15 0 * * * root /usr/bin/perl /usr/local/www/lightsquid/lightparser.pl yesterday
0 0 */1 * * root /usr/local/sbin/squid -k rotate > /dev/nullIf possible do not add items to this file manually.
If you do so, this file must be terminated with a blank line (e.g. new line)
Notice the single squid -k rotate with the weekly rotation added by lightsquid.
A few notes on upgrading.
1. Turn of log rotation in squid before upgrading. If squid is configured with logrotation and the upgrade is done then the new squid will not remove (and can't) remove the squid -k rotate entry since it no longer owns. It doesn't matter if lightsquid rotation is left on when the upgrade is done.
2. after the upgrade lightsquid stops working.... so you need to reinstall lightsquid... no big deal... i reinstalled squidGuard as well just to make darn sure.
3. the xml config files for squidGuard and lightSquid get zapped in the reinstall so PHP errors are displayed when the services are started. These are benign and nothing to worry about.
4. after reinstalling squidGuard and lightSquid you must go to the corresponding config pages in the webadmin gui and save and restart the configuration.
After that check /etc/cron to make sure that log rotation is taking place.
one last note... since squid -k rotate is being done by lightsquid and not squid this means there is no logfile_rotate entry in squid.conf specifying the number of logs to keep. The default is 10.... so squid.log will rotate through 0-9 with the oldest being removed. It probably is not a good idea (or maybe it is) to have lightsquid edit squid.conf although there are examples where it does.... For example, the lightsquid service modifies the log path (/var/squid/log) for squid... so maybe lightsquid should also modify the logile_rotate field squid.conf as well. If this is to be done then the webadmin page for lightsquid would have to be updated to get the number of logs to keep.
Here is the code in lightsquid_resync() updates the squid.conf file
// update squid conf
if (isset($config['installedpackages']['squid']['config'][0])) {
$config['installedpackages']['squid']['config'][0]['log_enabled'] = 'on';
$config['installedpackages']['squid']['config'][0]['log_dir'] = LS_SQUIDLOGPATH;
write_config();
squid_resync();
}Maybe this should include a
$config['installedpackages']['squid']['config'][0]['logile_rotate'] = X;
where X is the user specified number of copies of the log file to keep.
However, having said all that 10 if fine for me.
And BTW… the language in the lightsquid admin which describes the logrotation probably should be cleaned up.. Here is how it currently reads.
Select squid log rotate period. System will execute task every XX time as from 00:00 hours.
This option will allow the updating of the faster
For example: if selected '2 day' - system wil start task every 2 day of month.
This option will allow the updating of the faster
Note: You must choose from that the rate of filling dialogue access.log squid;
The more customers, the more often it should be the job.You definitely don't want to "choose from that the rate of filling dialogue access.log squid;"... in the squid config page... By enabling log rotation and the number of log files in the squid admin page you endup turning on squid log rotation in squid resulting in 2 "squid -k rotate" entries in the cronfile!
Anyway... you might let the lightsquid maintainer know that it would be good to clean up the description to have explain how this all works.
Take care and thanks for your prompt response on this issue.
--luis
-
hm… it just occurred to me maybe by doing
$config['installedpackages']['squid']['config'][0]['logile_rotate'] = X;
followed by a squid_sync(); might cause squid to add its on cron entry for log rotation… anyway.. you get the general idea. Some experimentation is required here to get this to work correctly.
--luis
-
You might want to start a new thread for that issue, since this is getting pretty far from the original package. The lightsquid maintainer isn't likely to notice it this far in.
If the normal maintainer doesn't notice it, I might have time to take a look later on this week.
