Should squid empty its cache directory on restart?



  • Just noticed after restarting one of my pfsense servers that the squid cache directory is now empty, is this expected behavior or can it be made persistent (its main use is for widnows updates).

    config:

    # This file is automatically generated by pfSense
    # Do not edit manually !
    http_port 192.168.10.254:3128
    http_port 127.0.0.1:3128 intercept
    icp_port 7
    dns_v4_first off
    pid_filename /var/run/squid.pid
    cache_effective_user proxy
    cache_effective_group proxy
    error_default_language en
    icon_directory /usr/pbi/squid-amd64/etc/squid/icons
    visible_hostname localhost
    cache_mgr admin@localhost
    access_log /var/squid/logs/access.log
    cache_log /var/squid/logs/cache.log
    cache_store_log none
    sslcrtd_children 0
    logfile_rotate 0
    shutdown_lifetime 3 seconds
    # Allow local network(s) on interface(s)
    acl localnet src  192.168.10.0/24
    uri_whitespace strip
    
    acl dynamic urlpath_regex cgi-bin \?
    cache deny dynamic
    cache_mem 2048 MB
    maximum_object_size_in_memory 128 KB
    memory_replacement_policy heap GDSF
    cache_replacement_policy heap LFUDA
    cache_dir ufs /var/squid/cache 240000 16 256
    minimum_object_size 0 KB
    maximum_object_size 5120000 KB
    offline_mode offcache_swap_low 90
    cache_swap_high 95
    
    # No redirector configured
    
    #Remote proxies
    
    # Setup some default acls
    acl allsrc src all
    acl localhost src 127.0.0.1/32
    acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 1025-65535 
    acl sslports port 443 563  
    acl manager proto cache_object
    acl purge method PURGE
    acl connect method CONNECT
    
    # Define protocols used for redirects
    acl HTTP proto HTTP
    acl HTTPS proto HTTPS
    
    http_access allow manager localhost
    
    # Allow external cache managers
    acl ext_manager src 192.168.10.254
    http_access allow manager ext_manager
    
    http_access deny manager
    http_access allow purge localhost
    http_access deny purge
    http_access deny !safeports
    http_access deny CONNECT !sslports
    
    # Always allow localhost connections
    http_access allow localhost
    
    request_body_max_size 0 KB
    delay_pools 1
    delay_class 1 2
    delay_parameters 1 -1/-1 -1/-1
    delay_initial_bucket_level 100
    delay_access 1 allow allsrc
    
    # Reverse Proxy settings
    
    # Package Integration
    never_direct allow all
    cache_peer 127.0.0.1 parent 3125 0 name=havp no-query no-digest no-netdb-exchange default
    
    # Custom options
    acl aclname snmp_community public 
    snmp_access allow aclname
    
    # Setup allowed acls
    # Allow local network(s) on interface(s)
    http_access allow localnet
    # Default block all to be sure
    http_access deny allsrc
    

    cache.log snippet:

    2015/01/13 00:32:51| Preparing for shutdown after 130895 requests
    2015/01/13 00:32:51| Waiting 3 seconds for active connections to finish
    2015/01/13 00:32:51| FD 32 Closing HTTP connection
    2015/01/13 00:32:51| FD 33 Closing HTTP connection
    2015/01/13 00:32:57| Starting Squid Cache version 3.1.22 for amd64-portbld-freebsd8.3...
    2015/01/13 00:32:57| Process ID 98530
    2015/01/13 00:32:57| With 11095 file descriptors available
    2015/01/13 00:32:57| Initializing IP Cache...
    2015/01/13 00:32:57| DNS Socket created at [::], FD 14
    2015/01/13 00:32:57| DNS Socket created at 0.0.0.0, FD 15
    2015/01/13 00:32:57| Adding domain cnlcomputers from /etc/resolv.conf
    2015/01/13 00:32:57| Adding nameserver 8.8.8.8 from /etc/resolv.conf
    2015/01/13 00:32:57| Adding nameserver 8.8.4.4 from /etc/resolv.conf
    2015/01/13 00:32:57| User-Agent logging is disabled.
    2015/01/13 00:32:57| Referer logging is disabled.
    2015/01/13 00:32:57| Unlinkd pipe opened on FD 22
    2015/01/13 00:32:57| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
    2015/01/13 00:32:57| Store logging disabled
    2015/01/13 00:32:57| Swap maxSize 245760000 + 2097152 KB, estimated 19065934 objects
    2015/01/13 00:32:57| Target number of buckets: 953296
    2015/01/13 00:32:57| Using 1048576 Store buckets
    2015/01/13 00:32:57| Max Mem  size: 2097152 KB
    2015/01/13 00:32:57| Max Swap size: 245760000 KB
    2015/01/13 00:32:57| Rebuilding storage in /var/squid/cache (DIRTY)
    2015/01/13 00:32:57| Using Least Load store dir selection
    2015/01/13 00:32:57| Current Directory is /etc
    2015/01/13 00:32:57| Loaded Icons.
    2015/01/13 00:32:57| helperOpenServers: Starting 0/0 'ssl_crtd' processes
    2015/01/13 00:32:57| helperOpenServers: No 'ssl_crtd' processes needed.
    2015/01/13 00:32:57| Accepting  HTTP connections at 192.168.10.254:3128, FD 24.
    2015/01/13 00:32:57| Accepting  intercepted HTTP connections at 127.0.0.1:3128, FD 25.
    2015/01/13 00:32:57| Accepting ICP messages at [::]:7, FD 26.
    2015/01/13 00:32:57| HTCP Disabled.
    2015/01/13 00:32:57| Configuring Parent 127.0.0.1/3125/0
    2015/01/13 00:32:57| Ready to serve requests.
    2015/01/13 00:47:57| Done scanning /var/squid/cache swaplog (0 entries)
    2015/01/13 00:47:57| Finished rebuilding storage from disk.
    2015/01/13 00:47:57|    207762 Entries scanned
    2015/01/13 00:47:57|         0 Invalid entries.
    2015/01/13 00:47:57|         0 With invalid flags.
    2015/01/13 00:47:57|    207762 Objects loaded.
    2015/01/13 00:47:57|         0 Objects expired.
    2015/01/13 00:47:57|         0 Objects cancelled.
    2015/01/13 00:47:57|         0 Duplicate URLs purged.
    2015/01/13 00:47:57|         0 Swapfile clashes avoided.
    2015/01/13 00:47:57|   Took 900.03 seconds (230.84 objects/sec).
    2015/01/13 00:47:57| Beginning Validation Procedure
    2015/01/13 00:47:57|   Completed Validation Procedure
    2015/01/13 00:47:57|   Validated 415549 Entries
    2015/01/13 00:47:57|   store_swap_size = 47803814
    2015/01/13 00:47:57| storeLateRelease: released 0 objects
    2015/01/13 00:47:57| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/13 00:47:57| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/13 00:47:57| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/13 00:47:57| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/13 00:47:57| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/13 00:47:57| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/13 00:47:57| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/13 00:47:57| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/13 00:47:57| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/13 00:47:57| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/13 00:47:57| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/13 00:47:57| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/13 00:47:57| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/13 00:47:57| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/13 00:47:57| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/13 00:47:57| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/13 00:47:57| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/13 00:47:57| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/13 00:47:57| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/13 00:47:57| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/13 00:47:57| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/13 00:47:57| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/13 00:47:57| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/13 00:47:57| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/13 00:47:57| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/13 00:47:57| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/13 00:47:57| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/13 00:47:57| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/13 00:47:57| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/13 00:47:57| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/13 01:28:49| logfileOpen: /var/log/squid/netdb.state: (2) No such file or directory
    2015/01/13 01:28:49| netdbSaveState: /var/log/squid/netdb.state: (2) No such file or directory
    2015/01/14 14:15:43| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/14 14:15:43| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/14 14:15:43| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/14 14:15:43| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/14 14:15:43| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/14 14:15:43| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/14 14:15:43| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/14 14:15:43| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/14 14:15:43| comm_old_accept: FD 25: (53) Software caused connection abort
    2015/01/14 14:15:43| httpAccept: FD 25: accept failure: (53) Software caused connection abort
    2015/01/14 16:06:05| Reconfiguring Squid Cache (version 3.1.22)...
    2015/01/14 16:06:05| FD 24 Closing HTTP connection
    2015/01/14 16:06:05| FD 25 Closing HTTP connection
    2015/01/14 16:06:05| FD 26 Closing ICP connection
    2015/01/14 16:06:05| Processing Configuration File: /usr/pbi/squid-amd64/etc/squid/squid.conf (depth 0)
    2015/01/14 16:06:05| Starting Authentication on port 127.0.0.1:3128
    2015/01/14 16:06:05| Disabling Authentication on port 127.0.0.1:3128 (interception enabled)
    2015/01/14 16:06:05| Disabling IPv6 on port 127.0.0.1:3128 (interception enabled)
    2015/01/14 16:06:05| Initializing https proxy context
    2015/01/14 16:06:05| Store logging disabled
    2015/01/14 16:06:05| User-Agent logging is disabled.
    2015/01/14 16:06:05| Referer logging is disabled.
    2015/01/14 16:06:05| DNS Socket created at [::], FD 15
    2015/01/14 16:06:05| DNS Socket created at 0.0.0.0, FD 16
    2015/01/14 16:06:05| Adding domain cnlcomputers from /etc/resolv.conf
    2015/01/14 16:06:05| Adding nameserver 127.0.0.1 from /etc/resolv.conf
    2015/01/14 16:06:05| Adding nameserver 8.8.8.8 from /etc/resolv.conf
    2015/01/14 16:06:05| Adding nameserver 8.8.4.4 from /etc/resolv.conf
    2015/01/14 16:06:05| helperOpenServers: Starting 0/0 'ssl_crtd' processes
    2015/01/14 16:06:05| helperOpenServers: No 'ssl_crtd' processes needed.
    2015/01/14 16:06:05| Accepting  HTTP connections at 192.168.10.254:3128, FD 18.
    2015/01/14 16:06:05| Accepting  intercepted HTTP connections at 127.0.0.1:3128, FD 24.
    2015/01/14 16:06:05| Accepting ICP messages at [::]:7, FD 25.
    2015/01/14 16:06:05| HTCP Disabled.
    2015/01/14 16:06:05| Configuring Parent 127.0.0.1/3125/0
    2015/01/14 16:06:05| Loaded Icons.
    2015/01/14 16:06:05| Ready to serve requests.
    2015/01/14 16:11:53| Reconfiguring Squid Cache (version 3.1.22)...
    2015/01/14 16:11:53| FD 18 Closing HTTP connection
    2015/01/14 16:11:53| FD 24 Closing HTTP connection
    2015/01/14 16:11:53| FD 25 Closing ICP connection
    2015/01/14 16:11:53| Processing Configuration File: /usr/pbi/squid-amd64/etc/squid/squid.conf (depth 0)
    2015/01/14 16:11:53| Starting Authentication on port 127.0.0.1:3128
    2015/01/14 16:11:53| Disabling Authentication on port 127.0.0.1:3128 (interception enabled)
    2015/01/14 16:11:53| Disabling IPv6 on port 127.0.0.1:3128 (interception enabled)
    2015/01/14 16:11:53| Initializing https proxy context
    2015/01/14 16:11:53| Store logging disabled
    2015/01/14 16:11:53| User-Agent logging is disabled.
    2015/01/14 16:11:53| Referer logging is disabled.
    2015/01/14 16:11:53| DNS Socket created at [::], FD 15
    2015/01/14 16:11:53| DNS Socket created at 0.0.0.0, FD 16
    2015/01/14 16:11:53| Adding domain cnlcomputers from /etc/resolv.conf
    2015/01/14 16:11:53| Adding nameserver 127.0.0.1 from /etc/resolv.conf
    2015/01/14 16:11:53| Adding nameserver 8.8.8.8 from /etc/resolv.conf
    2015/01/14 16:11:53| Adding nameserver 8.8.4.4 from /etc/resolv.conf
    2015/01/14 16:11:53| helperOpenServers: Starting 0/0 'ssl_crtd' processes
    2015/01/14 16:11:53| helperOpenServers: No 'ssl_crtd' processes needed.
    2015/01/14 16:11:53| Accepting  HTTP connections at 192.168.10.254:3128, FD 18.
    2015/01/14 16:11:53| Accepting  intercepted HTTP connections at 127.0.0.1:3128, FD 24.
    2015/01/14 16:11:53| Accepting ICP messages at [::]:7, FD 25.
    2015/01/14 16:11:53| HTCP Disabled.
    2015/01/14 16:11:53| Configuring Parent 127.0.0.1/3125/0
    2015/01/14 16:11:53| Loaded Icons.
    2015/01/14 16:11:53| Ready to serve requests.
    2015/01/14 16:19:10| Preparing for shutdown after 75753 requests
    2015/01/14 16:19:10| Waiting 3 seconds for active connections to finish
    2015/01/14 16:19:10| FD 18 Closing HTTP connection
    2015/01/14 16:19:10| FD 24 Closing HTTP connection
    2015/01/14 16:19:22| Starting Squid Cache version 3.1.22 for amd64-portbld-freebsd8.3...
    2015/01/14 16:19:22| Process ID 76037
    2015/01/14 16:19:22| With 11095 file descriptors available
    2015/01/14 16:19:22| Initializing IP Cache...
    2015/01/14 16:19:22| DNS Socket created at [::], FD 11
    2015/01/14 16:19:22| DNS Socket created at 0.0.0.0, FD 12
    2015/01/14 16:19:22| Adding domain cnlcomputers from /etc/resolv.conf
    2015/01/14 16:19:22| Adding nameserver 127.0.0.1 from /etc/resolv.conf
    2015/01/14 16:19:22| Adding nameserver 8.8.8.8 from /etc/resolv.conf
    2015/01/14 16:19:22| Adding nameserver 8.8.4.4 from /etc/resolv.conf
    2015/01/14 16:19:22| User-Agent logging is disabled.
    2015/01/14 16:19:22| Referer logging is disabled.
    2015/01/14 16:19:22| Unlinkd pipe opened on FD 18
    2015/01/14 16:19:22| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
    2015/01/14 16:19:22| Store logging disabled
    2015/01/14 16:19:22| Swap maxSize 245760000 + 2097152 KB, estimated 19065934 objects
    2015/01/14 16:19:22| Target number of buckets: 953296
    2015/01/14 16:19:22| Using 1048576 Store buckets
    2015/01/14 16:19:22| Max Mem  size: 2097152 KB
    2015/01/14 16:19:22| Max Swap size: 245760000 KB
    2015/01/14 16:19:22| Old swap file detected...
    2015/01/14 16:19:22| Rebuilding storage in /var/squid/cache (DIRTY)
    2015/01/14 16:19:22| Using Least Load store dir selection
    2015/01/14 16:19:22| Current Directory is /usr/local/www
    2015/01/14 16:19:22| Loaded Icons.
    2015/01/14 16:19:22| helperOpenServers: Starting 0/0 'ssl_crtd' processes
    2015/01/14 16:19:22| helperOpenServers: No 'ssl_crtd' processes needed.
    2015/01/14 16:19:22| Accepting  HTTP connections at 192.168.10.254:3128, FD 23.
    2015/01/14 16:19:22| Accepting  intercepted HTTP connections at 127.0.0.1:3128, FD 24.
    2015/01/14 16:19:22| Accepting ICP messages at [::]:7, FD 25.
    2015/01/14 16:19:22| HTCP Disabled.
    2015/01/14 16:19:22| Configuring Parent 127.0.0.1/3125/0
    2015/01/14 16:19:22| Ready to serve requests.
    2015/01/14 16:19:22| Done reading /var/squid/cache swaplog (49 entries)
    2015/01/14 16:19:22| Finished rebuilding storage from disk.
    2015/01/14 16:19:22|         5 Entries scanned
    2015/01/14 16:19:22|        43 Invalid entries.
    2015/01/14 16:19:22|         0 With invalid flags.
    2015/01/14 16:19:22|         5 Objects loaded.
    2015/01/14 16:19:22|         0 Objects expired.
    2015/01/14 16:19:22|         0 Objects cancelled.
    2015/01/14 16:19:22|         0 Duplicate URLs purged.
    2015/01/14 16:19:22|         0 Swapfile clashes avoided.
    2015/01/14 16:19:22|   Took 0.03 seconds (192.26 objects/sec).
    2015/01/14 16:19:22| Beginning Validation Procedure
    2015/01/14 16:19:22|   Completed Validation Procedure
    2015/01/14 16:19:22|   Validated 35 Entries
    2015/01/14 16:19:22|   store_swap_size = 54
    2015/01/14 16:19:23| storeLateRelease: released 0 objects
    2015/01/14 16:20:46| Preparing for shutdown after 11 requests
    2015/01/14 16:20:46| Waiting 3 seconds for active connections to finish
    

    Thanks



  • You have nothing under /var/squid/cache??  I haven't noticed this myself.



  • When I restart the persistent cache on disk stays untouched. Only thing lost, obviously, is the cached objects in RAM.



  • I enabled cache manager only to then see 5mb used under the /cache directory when it should be gigabytes with all the windows cab and PSF files, I had been getting cache hits so I know they were there. Also have a look at the cache.log it went from tens of thousands of objects to just tens.



  • Hi,

    I have had this issue for quite some time, I spent few days going through squid confg and start-up files butfound nothing suspicious. I then started thinking maybe it is pfSense fault not squid, I changed squid cache directory from the default /var/squid/cache to /home/squid_cache and voila!! All my cached downloads are now persistent after reboot.

    After changing the cache directory in the GUI, I restarted squid from an ssh terminal  "/usr/local/etc/rc.d/squid.sh stop" then "/usr/local/etc/rc.d/squid.sh start". Squid then created the new cache directory and restarted properly. I checked the old cache directory and it was still there (squid did not delete), however, after reboot the old cache directory in /var was deleted!!

    I am guessing this is what's happening: user Start the machine –-> ... --> pfSense for some reason deletes /var/squid/cache --> ... --> squid starts and find no cache directory --> squid creates a new empty directory as specified in its confg file --> user ends up with an empty cache.

    Give this a try, I hop it helps.



  • Do you have any packages installed like Sarg or Lightsquid?



  • Squid is the only package I have installed, a box that will do DHCP, firewall and DNS & web caching is what I was after when I built this.

    By moving the cache out of /var, squid is finally caching (into a persistent cache). I have had this running for like a week now caching updates for different machines, I got it rebooted few times and the cache is ~4GB and keeps growing  :)


Log in to reply