Squid and SquidGuard does not start after reboot
-
question
the following is an excerpt from /etc/rc
–----
....
echo -n "Starting CRON... "
cd /tmp && /usr/sbin/cron -s 2>/dev/null
echo "done."Start packages
/etc/rc.start_packages
rm -rf /usr/local/pkg/pf/CVS
Remove stale files that have already been processed by bootup
scripts
rm -f /tmp/filter_dirty
rm -f /tmp/rc.linkup
nohup /usr/bin/nice -n20 /usr/local/sbin/check_reload_status &
....
is there any chance that rm -f /tmp/filter_dirty
might happen before the syncing of the rules when spawning squid? This would cause filter_config() in squid.inc to be ignored. It would explain why sleep(5) works. the sleep would cause the rm in rc to happen before filter_config() is called.let me test that by commenting out the rm in rc....
hey guess what
pfctl -sn | grep http
rdr on vr0 inet proto tcp from any to ! (vr0) port = http -> 127.0.0.1 port 80
the problem has ** NOTHING TO DO WITH THE SQUID PACKAGE ***
its a race condition cause in /etc/rc.
this is a a serious bug that could have repercussions for any package that rules that get created right about the time rc
removes /tmp/filter_dirty.comment out
rm -f /tmp/filter_dirtyand everything works like expected. no sleeps required in before filter_config() to make things work.
this no longer needed.
// Sleep for a couple seconds to give squid a chance to fire up fully.
/*
for ($i=0; $i < 10; $i++) {
sleep(1);
if (is_service_running('squid')) {
break;
}
}
*/i bet you a $1 that if you put back the mwexec_bg things would still work since the creation of the rules happens asynchronously.
HOWEVER... i still think its a good idea to leave the synchronous spawn and the loop in there just to make DARN sure that squid is up before you try to update the filters.
--luis
-
Good find there. I'm not sure why that rm is there. I asked around and at most that would mean another filter reload, which is harmless, and in this case necessary. It must still be working for me again on this test ALIX now because my WAN is wireless and rc.newwanip doesn't run as quickly.
So removing that rm is safe, but I'm not sure how best to handle that long-term. I'm not sure it should be patched out for everyone.
-
The way I see it the rm in rc is bad… the reason for this is that there are many programs that set /tmp/filter_dirty expecting to have some process later on detect it and then sync up the filters. The issue is that the rm in rc can happen at any time while subsystems are booting up. This is bad... The rc script should 'wait' until the system is fully booted before removing this file. But it doesn't. It just removes it willy-nilly at some random time without considering the effect on the boot process.
This is your classic multi-processing synchronization problem. You have one process removing a resource that another program needs without checking first.
From everything I have looked at it seems that having rc remove the file is unnecessary. Think about it... reloading process set /tmp/filter_dirty and remove it after the synchronization is complete. So... why should removing it once in rc matter? On my system, for example, currently the file does not exist even though the rm in rc is commented out. This is because the file is unlinked after the filter synchronization was completed. The house keeping in rc is unnecessary.
IMHO 1 of two things needs to be done.
1. rm needs to be removed from rc
or
2. some synchronization mechanism needs to be added to rc so that rc does not attempt to remove this file until after all processes have been started up.Please check again with the powers that be... I believe that removing the rm in rc will not negatively impact the operation of the system for anyone and therefore should be done since there are now documented examples where in some instances it breaks the boot process. No telling what other users are having weird unexplained behavior because of this. In our case it was squid. but for some other user it might be.... Also, keep in mind that this thread has been open since Oct 2009... so this is a long standing problem. Not something that just cropped up.
summing up... having rm -f /tmp/filter_dirty in rc causes inconsistent behavior in pfSense and it is therefore detrimental to the project.
--luis
PS... here is another thread where boot up problems are probably caused by removal of /tmp/filter_dirty in /etc/rc
http://forum.pfsense.org/index.php?topic=19381.0#lastPost
this one was reported in Sept 2009. -
Taking the rm out is the solution, but since that is a system file and not a package file, I'm hesitant to remove that as part of the package installation - that's what I was referring to.
/tmp/filter_dirty is removed automatically by check_reload_status after it reloads the filter – nothing needs to remove that file manually.
-
Taking the rm out is the solution, but since that is a system file and not a package file, I'm hesitant to remove that as part of the package installation - that's what I was referring to.
/tmp/filter_dirty is removed automatically by check_reload_status after it reloads the filter – nothing needs to remove that file manually.
Yes… i agree with you 100% this should not be removed by the package....
This needs to be patched in the distribution.
Who takes care of that?
--luis
-
Well 1.2.3 won't see any more updates to the base system. I can remove that line on 2.0, though (it's still there), after running it by a few others.
-
OK. thanks…
Its a pity that there are no more updates to 1.2.3...
I will add this to our list of things to do when building a new distribution...
I will let the folks at the applianceshop know as well.
Take care and thanks for your help.
--luis
-
It might be possible to get it committed into the 1.2.3 tree so a rebranded release for a customer could pick it up, failing that, an overlay for the rc file could be setup for a custom build.
Or I could just make a little patch package that removes it, too, but not bundle it with squid (since it doesn't affect everyone)
-
certainly getting the fix into the tree seems like a reasonable thing to do…
Squid users having problems hopefully will find this form thread and will then know that to fix the problem the need to edit /etc/rc
go to line 322 and delete it.i.e.
make this
Remove stale files that have already been processed by bootup
scripts
rm -f /tmp/filter_dirty
rm -f /tmp/rc.linkuplook like this
Remove stale files that have already been processed by bootup
scripts
rm -f /tmp/rc.linkup
--luis
-
I just added a package to apply a simple patch that removes the line in question.
Give it a spin and see if it helps. It's called "Patch rc to leave filter_dirty" (for lack of a better name :-)
-
it failed!
Remove stale files that have already been processed by bootup
scripts
rm -f /tmp/filter_dirty
rm -f /tmp/rc.linkup
nohup /usr/bin/nice -n20 /usr/local/sbin/check_reload_status &Here is the list of installed packages on my system
Patch rc to leave filter_dirty System No info, check the forum 0.1 Patch to stop /etc/rc from removing /tmp/filter_dirty on boot. Fixes boot issues with some packages on certain platforms.
squid Network No info, check the forum Current: 2.7.9
Installed: 2.7.8_1 High performance web proxy cache.squidGuard Network Management No info, check the forum 1.3-2 High perfomance web proxy URL filter. Required proxy Squid package.
Let me know when you want me to try it again.
–luis
-
Apparently I messed up the path in the patch. I fixed it, should be OK now.
-
-
Did you break squid trying to fix this issue per chance?
Here is my current situation.
1. brand new 1.2.3-RELEASE installation with a few configuration chances as required to get the router working on a LAN/WAN
2. installed the filter_dirty package
3. installed squid
4. turn on transparent proxysquid will not run… proxy_monitor fires it up and it shuts down right away... On reboot the only mesg I see when syncing packages is the rc filter_dirty fix and squid that is it...
ps aux | grep squid
shows that squid is not running... proxy_monitor is running and continually tries to start squid.if you try to go to any website the pf rule redirects you to the pfsense box which asks for a username and password. You enter the pfsense admin username and password and then it shows you the main admin website.
So... again the proxy is not running.
any ideas?
Here is a list of the packages that are installed.
Patch rc to leave filter_dirty System No info, check the forum 0.1 Patch to stop /etc/rc from removing /tmp/filter_dirty on boot. Fixes boot issues with some packages on certain platforms.squid Network No info, check the forum 2.7.9 High performance web proxy cache.
--luis
-
Well I updated squid to 2.7.9 separately from trying to fix this, but I thought I had tested it on nanobsd at the time. That was late last week though, and I don't recall the specifics.
Is there nothing in the system log about squid at all?
-
OK found the problem…
in a clean installation /var/squid is not setup with the correct permissions.
/var/squid is owned by root:wheel with permission 755
many of the files such as access.log and owned by squid:squid with permission 640 when they should be owned by proxy.
chown -R proxy /var/squid
solves the problem.
So... this issue has nothing to do with the filter_dirty patch... However, some time in the recent past some changes must have been made to the squid package to break the permissions in /var/squid...
--luis
-
ok, I committed a fix for that. It should be in the package repo in about 2 minutes.
-
OK… it all seems to work now... squid, lightsquid, and squidGuard all install and play nicely.
One very small thing...
i notice that now by default squid install its log files into /var/squid/logs [note the (s)]. When you install and configure lightsquid the package automatically chances the logging directory for squid to /var/squid/log [note the absence of (s)]. This is not really a problem because it doen't break anything but the access files in the original /var/squid/logs folder never get rotated or deleted. They stay there forever… This occupies resources if the log file happens to be big before you turn on lightSquid.
Thanks again for your help.
--luis
-
I just thought we were done…
all the subsystems are working fine... however, i can't modify any of the cache settings for squid.
when I try to set the hard disk cache size, memory cache size, and maximum object size under proxy server->cache management the web page resets the value after you hit save. I have confirmed that the values are not being updated in squid.conf
cache_mem 8 MB
maximum_object_size_in_memory 32 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir aufs /var/squid/cache 100 16 256
minimum_object_size 0 KB
maximum_object_size 10 KB
offline_mode offso what causes this?
oh man.... take a look at the entry for squidcache in config.xlm
<squidcache><config><config><harddisk_cache_size>25600</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>128</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>2465792</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config>
<config><harddisk_cache_size>25600</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>128</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>4194304</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config>
<config><harddisk_cache_size>15360</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>128</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>1048576</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config>
<config><harddisk_cache_size>1000</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>80</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>4</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config>
<config><harddisk_cache_size>25600</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>128</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>4194304</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config></config></squidcache>I would say there are a few too many entries here...what would cause this? It seems that every time I hit add a new config gets written but the old one is not removed!
the
<squidcache><config>extra <config>right at the head of this configuration seems particularly disturbing...
I will edit the config.xml manually to see if I can get this to work....
--luis</config></config></squidcache>
-
OK… that worked... here is squid.conf
cache_mem 128 MB
maximum_object_size_in_memory 32 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir aufs /var/squid/cache 25600 16 256
minimum_object_size 0 KB
maximum_object_size 2465792 KB
offline_mode off
cache_swap_low 90
cache_swap_high 95and here is the entry in config.xml
<squidcache><config><harddisk_cache_size>25600</harddisk_cache_size>
<harddisk_cache_system>aufs</harddisk_cache_system>
<harddisk_cache_location>/var/squid/cache</harddisk_cache_location>
<memory_cache_size>128</memory_cache_size>
<minimum_object_size>0</minimum_object_size>
<maximum_object_size>2465792</maximum_object_size>
<level1_subdirs>16</level1_subdirs>
<memory_replacement_policy>heap GDSF</memory_replacement_policy>
<cache_replacement_policy>heap LFUDA</cache_replacement_policy>
<cache_swap_low>90</cache_swap_low>
<cache_swap_high>95</cache_swap_high>
<donotcache><enable_offline></enable_offline></donotcache></config></squidcache>so something went awol with webinterface/xml editor.
any ideas?
--luis
