Squid - rotate nightmare



  • Tested with squid3-dev 3.3.10 pkg 2.2.1 package

    Previous threats that I found
    http://forum.pfsense.org/index.php?topic=45777.0
    http://forum.pfsense.org/index.php?topic=47569.0

    I'm not logging access.log but I want to rotate cache.log

    I put one day to rotate cache.log. It doesn't rotate.

    cat /usr/pbi/squid-amd64/etc/squid/squid.conf | grep log
    access_log /dev/null
    cache_log /var/squid/logs/cache.log
    cache_store_log none
    logfile_rotate 1
    

    It's squid rotate working? Must be logs activated at WebGUI for rotating cache.log?



  • Can you check if this entry exists on your cron?

    At least on squid3-dev logs(access and cache) are rotating fine.

    This may need another fix for squid cache log rotation if still present on squid 3.3

    Changes in 3.1 logfile_rotate
    No longer controls cache.log rotation. Use debug_options rotate=N instead.




  • I didn't have Cron Package installed. Now, I have it. And I see the configured job for squid3.

    I will wait until tomorrow to see that it happens at 00:00

    Thanks!

    http://www.squid-cache.org/Doc/config/logfile_rotate/



  • After 00:00 no swap-state file and cache.log not rotated

    ls -l /var/squid/logs/cache.log
    -rw-r-----  1 proxy  proxy  38140597 Apr  2 23:42 /var/squid/logs/cache.log
    


  • 0  	0  	*  	*  	*  	root  	/bin/rm /var/squid/cache/swap.state; /usr/pbi/squid-amd64/sbin/squid -k rotate -f /usr/pbi/squid-amd64/etc/squid/squid.conf
    

    swap.state shouldn't be erased? It's the index for the cache!

    http://www.squid-cache.org/Doc/config/cache_swap_state/

    Since this is the index for the whole object list you CANNOT periodically rotate it!

    http://wiki.squid-cache.org/SquidFaq/SquidLogs

    If you accidentally delete swap.state while Squid is running, you can recover it by following the instructions in the previous questions.

    http://wiki.squid-cache.org/SquidFaq/SquidLogs#swap.state



  • Log rotate	
    Defines how many days of logfiles will be kept. Rotation is disabled if left empty.
    

    It should say "files" instead than "days". In fact are days because cron jo it's done every day at 00:00.

    With rotate disabled (box empty) I continue having the cron job!

    This means erasing swap.state for nothing (?).

    squid rotate is always a nightmare. At my proxies FreeBSD based I finished writing my own daily maintenance script to run at midnight. It stoppes squid, runs sarg, does the rotates, updates squidGuard blacklists and starts squid a new time.



  • I saw there is a cron job at past quarter of each hour controlling the swap.state file:

    */15  	*  	*  	*  	*  	root  	/usr/local/pkg/swapstate_check.php
    

    The code is written by Jim Pingle. Perhaps he could clarify a little bit what to do with swap.state.

    Squid documentation has been always confused. I'm using squid before year 2000 and it's hard to find good updated documentation.



  • So do I edit the cron job from

    0  	0  	*  	*  	*  	root  	/bin/rm /var/squid/cache/swap.state; /usr/pbi/squid-amd64/sbin/squid -k rotate -f /usr/pbi/squid-amd64/etc/squid/squid.conf
    

    into

    0  	0  	*  	*  	*  	root  	/usr/pbi/squid-amd64/sbin/squid -k rotate -f /usr/pbi/squid-amd64/etc/squid/squid.conf
    

    this?
    To prevent the swap.state to be lost?



  • How do you suggest to implement on squid3-dev? An option on gui that removes entire cache when swap.state is removed?

    my cache.log is rotating fine on squid3-dev but I can include the debug_options rotate=N on squid.conf



  • @ Tikimotel,

    Of course, but I didn't have time tot experiment if these crons are generated each time you configure squid with the WebGUI.

    Squid documentation has been always confused. I'm using squid before year 2000 and it's hard to find good updated documentation.

    I also want to propose a debate about what to do with swap.state

    Looking at official documentation it seems that is better don't touch it. In fact, I never touch it for maintenance at my squid 2.x proxies.



  • @marcelloc:

    How do you suggest to implement on squid3-dev? An option on gui that removes entire cache when swap.state is removed?

    No. I think it doesn't help and cache is a very good value.

    swap.state is removed at 00:00. It's supposed that squid -k rotate builds a new time. But I don't have logs activated at WebGUI. And I see swap.state at 0 bytes for many hours (until I go to sleep  ;D). At morning, however, swap.state has been built. But I suspect that it's only partially build with the user accesses because no information about at cache.log. So, the index should be incomplete and I'm not sure the cache is optimized.

    I will investigate more about it…

    my cache.log is rotating fine on squid3-dev but I can include the debug_options rotate=N on squid.conf

    Are you logging activated at WebGUI? I think yes, and you're logging also access.log. So, you are rotating. But in my case (as I said above) I don't want access.log and I'm not rotating cache.log at 00:00. Cron job does nothing about for me.

    Thanks! See you later about this!



  • wiki on "swap.state"

    The file is a binary format that includes MD5 checksums, and StoreEntry fields. Please see the Programmers' Guide for information on the contents and format of that file.
    
    Squid can rebuild it from the original files, but that procedure can take a long time as every file in the cache must be fully scanned for meta data.
    

    I think, while reading the wiki, that cleaning the (/bin/rm) "swap.state" should only occur when you want to re-install squid or change the cache dir policy (aufs / diskd) or cachesize.
    It contains the checksums of all cached items in the cache, so rebuilding ever 24h seems unnecessary and slow.


  • Banned

    @Tikimotel:

    rebuilding ever 24h seems unnecessary and slow.

    The file should only get deleted when

    
    	// If the swap.state file is taking up more than 75% disk space, 
    	//	or the drive is 90% full and swap.state is larger than 1GB, 
    	//	kill it and initiate a rotate to write a fresh copy.
    
    

    If it happens every day for you, that'd strongly suggest you should just get a suitable (read: bigger) HDD for your box.



  • @doktornotor:

    If it happens every day for you, that'd strongly suggest you should just get a suitable (read: bigger) HDD for your box.

    It happens because cron job for rotate has a rm command for it! As we discussed…

    0  	0  	*  	*  	*  	root  	/bin/rm /var/squid/cache/swap.state; /usr/pbi/squid-amd64/sbin/squid -k rotate -f /usr/pbi/squid-amd64/etc/squid/squid.conf
    


  • @Tikimotel:

    I think, while reading the wiki, that cleaning the (/bin/rm) "swap.state" should only occur when you want to re-install squid or change the cache dir policy (aufs / diskd) or cachesize.
    It contains the checksums of all cached items in the cache, so rebuilding ever 24h seems unnecessary and slow.

    I agree.

    In fact, the swap.state is controlled each hour by

    */15  	*  	*  	*  	*  	root  	/usr/local/pkg/swapstate_check.php
    

    and it's completely built at squid restarts.

    So, If you want to be sure /bin/rm /var/squid/cache/swap.state is erroneous and will be better a squid stop/start. But this (stop/start) can be a problem for environments with full 24 hours service.

    That I'm wondering it's if erasing swap.state is a real problem or not for the cache. In other words, it's squid capable to find the objects without an initial index? Can we trust to the index if has been "manually" erased and it's built a new time "on the fly"?


  • Banned

    @bellera:

    It happens because cron job for rotate has a rm command for it! As we discussed…

    Uh. Why?

    @bellera:

    That I'm wondering it's if erasing swap.state is a real problem or not for the cache.

    It takes damn ages to rebuild the index… Like, hours / or even days for a reasonably big / huge cache - and causes huge I/O meanwhile as a "bonus". It is a completely retarded idea to delete the thing every 24 hours.



  • Uh. Why?
    

    Because squid pfSense installation (or configuration) generates these two cron jobs:

    0  	0  	*  	*  	*  	root  	/bin/rm /var/squid/cache/swap.state; /usr/pbi/squid-amd64/sbin/squid -k rotate -f /usr/pbi/squid-amd64/etc/squid/squid.conf
    */15  	*  	*  	*  	*  	root  	/usr/local/pkg/swapstate_check.php
    

    You can see them if you have Cron Job package installed.

    Do you use squid with pfSense?


  • Banned

    No, not with pfSense. Using squid on a couple of dedicated boxes. Never occured to me someone could some with an "idea" of wiping the cache index daily. If you are running out of disk space, you are running the proxy on a wrong HW and/or you should sanitize the cache size settings. Not delete indexes.



  • @doktornotor:

    If you are running out of disk space, you are running the proxy on a wrong HW and/or you should sanitize the cache size settings. Not delete indexes.

    I agree.

    /usr/local/pkg/swapstate_check.php is controlling this each hour.

    I'm using also squid outside pfSense. But now I'm migrating one installation having squid into pfSense.

    First cron should be without rm command:

    @Tikimotel:

    So do I edit the cron job from

    0  	0  	*  	*  	*  	root  	/bin/rm /var/squid/cache/swap.state; /usr/pbi/squid-amd64/sbin/squid -k rotate -f /usr/pbi/squid-amd64/etc/squid/squid.conf
    

    into

    0  	0  	*  	*  	*  	root  	/usr/pbi/squid-amd64/sbin/squid -k rotate -f /usr/pbi/squid-amd64/etc/squid/squid.conf
    

    this?
    To prevent the swap.state to be lost?





  • http://wiki.squid-cache.org/SquidFaq/SquidLogs#swap.state

    If you remove this file while squid IS running, you can easily recreate it. The safest way is to simply shutdown the running process:

    % squid -k shutdown

    This will disrupt service, but at least you will have your swap log back. Alternatively, you can tell squid to rotate its log files. This also causes a clean swap log to be written.

    % squid -k rotate

    A clean swap log to be written –> At 00:00 I see swap.state at 0 bytes. And squid works well…

    http://wiki.squid-cache.org/SquidFaq/OperatingSquid#I_want_to_restart_Squid_with_a_clean_cache

    Squid-2.6 and later contain mechanisms which will automatically detect dirty information in both the cache directories and swap.state file. When squid starts up it runs these validation and security checks. The objects which fail for any reason are automatically purged from the cache.

    The above mechanisms can be triggered manually to force squid into a full cache_dir scan and re-load all objects from disk by simply shuttign down Squid and deleting the swap.state journal from each cache_dir before restarting.

    NP: Deleting the swap.state before shutting down will cause Squid to generate new ones and fail to do the re-scan you wanted.

    The NP is very interesting. squid will not spend time rebuilding the entire swap.state



  • @bellera:

    http://wiki.squid-cache.org/SquidFaq/OperatingSquid#I_want_to_restart_Squid_with_a_clean_cache

    If you remove the swap.state file while squid is not running, it will have to completely rescan your cache folder to rebuild it once squid is started back up. This can be a lengthy and time consuming process. It may be better to remove the contents of the existing cache folder, and rebuild the structure again by running:

    I've updated the package to include a new option to admin chose if cache is clean on log rotates o clean if hd reaches 90% of disk space.



  • I'll bump package version after these tests, please reinstall the package, test and feedback.



  • @marcelloc:

    I've updated the package to include a new option to admin chose if cache is clean on log rotates o clean if hd reaches 90% of disk space.

    It seems a good idea.

    I'll bump package version after these tests, please reinstall the package, test and feedback.

    Ok, but I need some days. I'm very busy with this big change at my main networking site.



  • @bellera:

    Ok, but I need some days. I'm very busy with this big change at my main networking site.

    You can change cron execution time to test it faster  :)



  • rotate of cache.log and swap.state correct!

    [2.1-RELEASE][admin@pfsense.localdomain]/root(36): ./see_squid_logs.sh 
    -rw-r-----  1 proxy  proxy  21839 Apr  5 16:03 /var/squid/logs/cache.log
    -rw-rw-rw-  1 proxy  proxy  338844 Apr  5 16:27 /var/squid/cache/swap.state
    [2.1-RELEASE][admin@pfsense.localdomain]/root(37): ./see_squid_logs.sh
    -rw-r-----  1 proxy  proxy      0 Apr  5 16:55 /var/squid/logs/cache.log
    -rw-r-----  1 proxy  proxy  21839 Apr  5 16:03 /var/squid/logs/cache.log.0
    -rw-rw-rw-  1 proxy  proxy  337212 Apr  5 16:55 /var/squid/cache/swap.state
    -rw-r-----  1 proxy  proxy       0 Apr  5 16:55 /var/squid/cache/swap.state.last-clean
    

    but

    SSL Bump not working with 2.2.2 package.

    Reported at https://forum.pfsense.org/index.php?topic=62256.msg407762#msg407762



  • @exograpix asked for explaining the new option about rotate cron.

    squid3-dev 3.3.10 pkg 2.2.1

    0  	0  	*  	*  	*  	root  	/bin/rm /var/squid/cache/swap.state; /usr/pbi/squid-amd64/sbin/squid -k rotate -f /usr/pbi/squid-amd64/etc/squid/squid.conf
    */15  	*  	*  	*  	*  	root  	/usr/local/pkg/swapstate_check.php
    

    squid3-dev 3.3.10 pkg 2.2.2 version default

    0  	0  	*  	*  	*  	root  	/usr/pbi/squid-amd64/sbin/squid -k rotate -f /usr/pbi/squid-amd64/etc/squid/squid.conf
    */15  	*  	*  	*  	*  	root  	/usr/local/pkg/swapstate_check.php
    

    squid3-dev 3.3.10 pkg 2.2.2 version with reseting the cache –--> Proxy server: Cache management: Clear cache on log rotate

    0  	0  	*  	*  	*  	root  	/usr/local/pkg/swapstate_check.php clean; /usr/pbi/squid-amd64/sbin/squid -k rotate -f /usr/pbi/squid-amd64/etc/squid/squid.conf
    */15  	*  	*  	*  	*  	root  	/usr/local/pkg/swapstate_check.php
    

    So, if you activate the new option you will reset your cache every night.

    At 2.2.1 cache.log file wasn't rotate because squid.conf didn't have any debug_options rotate=N (squid3 syntax) line. And rm for swap.state is not necessary, as discussed in this thread.


Log in to reply