• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Kid1| Select loop Error. Retry 1

Scheduled Pinned Locked Moved Cache/Proxy
17 Posts 3 Posters 3.9k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • V
    Valeriy
    last edited by Feb 15, 2017, 3:32 PM

    kid1| Select loop Error. Retry 1

    What does this error mean?
    What debug options might help to trace it down to event that causes it?

    This error is happening randomly, sometimes every 2-3 minutes, sometimes once per 1-3 hours.
    During this event squid becomes very slow to accept new HTTP requests.

    Squid version is 3.5.23.
    Config:

    
    http_port x.x.x.x:3128
    http_port 127.0.0.1:3128 intercept
    icp_port 0
    dns_v4_first off
    pid_filename /var/run/squid/squid.pid
    cache_effective_user squid
    cache_effective_group proxy
    error_default_language en
    icon_directory /usr/local/etc/squid/icons
    visible_hostname localhost
    cache_mgr admin@localhost
    access_log /var/squid/logs/access.log
    cache_log /var/squid/logs/cache.log
    cache_store_log none
    netdb_filename /var/squid/logs/netdb.state
    pinger_enable off
    pinger_program /usr/local/libexec/squid/pinger
    
    logfile_rotate 365
    debug_options rotate=365
    shutdown_lifetime 3 seconds
    # Allow local network(s) on interface(s)
    acl localnet src  x.x.x.x/30
    forwarded_for delete
    via off
    httpd_suppress_version_string on
    uri_whitespace strip
    
    #Refresh patterns
    #All File V 0.03
    
    #new refresh patterns 2
    refresh_pattern -i (\.|-)(mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-store ignore-private ignore-auth
    #end new refresh patterns 2
    
    #new refresh patterns
    refresh_pattern -i (\.|-)(mp3|m4a|aa?c3?|wm?av?|og(x|v|a|g)|ape|mka|au|aiff|zip|flac|m4(b|r)|m1v|m2(v|p)|mo(d|v)|arj|appx|lha|lzh|on2)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-store ignore-private ignore-auth
    refresh_pattern -i (\.|-)(exe|bin|(n|t)ar|acv|(r|j)ar|t?gz|(g|b)z(ip)?2?|7?z(ip)?|wm[v|a]|patch|diff|mar|vpu|inc|r(a|p)m|kom|iso|sys|[ap]sf|ms[i|u|f]|dat|msi|cab|psf|dvr-ms|ace|asx|qt|xt|esd)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-store ignore-private ignore-auth
    refresh_pattern -i (\.|-)(ico(.*)?|pn[pg]|css|(g|t)iff?|jpe?g(2|3|4)?|psd|c(d|b)r|cad|bmp|img)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-store ignore-private ignore-auth
    refresh_pattern -i (\.|-)(webm|(x-)?swf|mp(eg)?(3|4)|mpe?g(av)?|(x-)?f(l|4)v|divx?|rmvb?|mov|trp|ts|avi|m38u|wmv|wmp|m4v|mkv|asf|dv|vob|3gp?2?)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-store ignore-private ignore-auth
    refresh_pattern -i (\.|-)(docx?|xlsx?|pptx?|rtf|xml|pdf|tiff?|txt)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-store ignore-private ignore-auth
    #Website
    refresh_pattern -i (\.|-)(xml|js|jsp|txt|css)(\?.*)?$ 360 40% 1440 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-store ignore-private ignore-auth
    refresh_pattern -i .index.(html|htm)$ 0 40% 1440
    refresh_pattern . 30 25% 1440
    #end new refresh patterns
    
    refresh_pattern -i \.(3gp|7z|ace|asx|avi|bin|cab|dat|deb|rpm|divx|dvr-ms)      129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    #refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|(x-|)flv) 129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    refresh_pattern -i \.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js)         129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    refresh_pattern -i \.(mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p))                   129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    refresh_pattern -i \.(og(x|v|a|g)|rar|rm|r(a|p)m|snd|vob|wav)                  129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    refresh_pattern -i \.(pp(s|t)|wax|wm(a|v)|wmx|wpl|zip|cb(r|z|t))               129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    
    refresh_pattern ^gopher:  1440  0%  1440
    refresh_pattern ^ftp:    10080 95% 43200 override-lastmod reload-into-ims
    
    refresh_pattern -i \.(doc|pdf)$           100080 90% 43200 override-expire ignore-no-store ignore-private reload-into-ims
    refresh_pattern -i \.(html|htm)$          1440   40% 40320 ignore-no-store ignore-private override-expire reload-into-ims
    refresh_pattern (Release|Packages(.gz)*)$    0   20%  2880
    refresh_pattern .                          180   95% 43200 override-lastmod reload-into-ims
    
    # 1 year = 525600 mins, 1 month = 43800 mins
    refresh_pattern -i (/cgi-bin/|\?)         0      0%      0
    refresh_pattern \.(ico|video-stats)$ 129600 100% 129600 override-expire ignore-reload ignore-no-store ignore-private ignore-auth override-lastmod ignore-must-revalidate
    
    refresh_pattern imeem.*\.flv$                           0     0%         0 override-lastmod override-expire
    refresh_pattern \.rapidshare.*\/[0-9]*\/.*\/[^\/]* 161280    90%    161280 ignore-reload
    
    refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?) 129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims
    refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.flv?) 129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims
    refresh_pattern ^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 129600 20% 129600 ignore-no-store ignore-private override-expire ignore-reload ignore-auth ignore-must-revalidate
    
    refresh_pattern ^.*safebrowsing.*google                                  129600 100% 129600 override-expire ignore-reload ignore-private ignore-auth ignore-must-revalidate
    refresh_pattern ^http://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.uk)     129600 100% 129600 override-expire ignore-reload ignore-private
    refresh_pattern ytimg\.com.*\.jpg                                        129600 100% 129600 override-expire ignore-reload
    refresh_pattern images\.friendster\.com.*\.(png|gif)                     129600 100% 129600 override-expire ignore-reload
    refresh_pattern garena\.com                                              129600 100% 129600 override-expire reload-into-ims
    refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png)           129600 100% 129600 override-expire ignore-reload
    refresh_pattern vid\.akm\.dailymotion\.com.*\.on2\?                      129600 100% 129600 override-expire override-lastmod
    refresh_pattern mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 129600 100% 129600 reload-into-ims override-expire ignore-private
    refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\.                      129600 100% 129600 reload-into-ims ignore-no-store ignore-reload override-expire
    refresh_pattern ^http:\/\/www.onemanga.com.*\/                           129600 100% 129600 reload-into-ims ignore-no-store ignore-reload override-expire
    
    # ANTI VIRUS
    refresh_pattern guru.avg.com/.*\.(bin)                              43200 100% 43200  ignore-no-store ignore-reload reload-into-ims
    refresh_pattern (avgate|avira).*(idx|gz)$                           43200 100% 43200  ignore-no-store ignore-reload reload-into-ims
    refresh_pattern kaspersky.*\.avc$                                   43200 100% 43200  ignore-no-store ignore-reload reload-into-ims
    refresh_pattern kaspersky                                           43200 100% 43200  ignore-no-store ignore-reload reload-into-ims
    refresh_pattern update.nai.com/.*\.(gem|zip|mcs)                    43200 100% 43200  ignore-no-store ignore-reload reload-into-ims
    refresh_pattern ^http:\/\/liveupdate.symantecliveupdate.com.*\(zip) 43200 100% 43200  ignore-no-store ignore-reload reload-into-ims
    refresh_pattern -i symantecliveupdate.com/.*\.(zip|exe)             43200 100% 43200 reload-into-ims
    refresh_pattern -i avast.com/.*\.(vpu|vpaa) 4320 100% 43200 reload-into-ims
    refresh_pattern -i avira-update.com/.*\.* 720 100% 10800 reload-into-ims
    
    #windows update
    refresh_pattern windowsupdate.com/.*\.(cab|exe)                     43200 100% 129600 ignore-no-store ignore-reload reload-into-ims
    refresh_pattern update.microsoft.com/.*\.(cab|exe)                  43200 100% 129600 ignore-no-store ignore-reload reload-into-ims
    refresh_pattern download.microsoft.com/.*\.(cab|exe)                43200 100% 129600 ignore-no-store ignore-reload reload-into-ims
    refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf) 4320 100% 43200 reload-into-ims
    refresh_pattern update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims
    refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims
    refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims
    
    #apple update
    refresh_pattern -i (download|adcdownload).apple.com/.*\.(pkg|dmg) 4320 100% 43200 reload-into-ims
    
    #images facebook
    refresh_pattern ((facebook.com)|(85.131.151.39)).*\.(jpg|png|gif) 129600 100% 129600 ignore-reload override-expire ignore-no-store
    refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|swf|mp3)           129600 100% 129600 ignore-reload override-expire ignore-no-store
    refresh_pattern static\.ak\.fbcdn\.net*\.(jpg|gif|png)            129600 100% 129600 ignore-reload override-expire ignore-no-store
    refresh_pattern ^http:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png)  129600 100% 129600 ignore-reload override-expire ignore-no-store
    
    #banner IIX
    refresh_pattern ^http:\/\/openx.*\.(jp(e?g|e|2)|gif|pn[pg]|swf|ico|css|tiff?) 129600 100% 129600 reload-into-ims ignore-reload override-expire ignore-no-store
    refresh_pattern ^http:\/\/ads(1|2|3).kompas.com.*\/                           43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-store
    refresh_pattern ^http:\/\/img.ads.kompas.com.*\/                              43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-store
    refresh_pattern .kompasimages.com.*\.(jpg|gif|png|swf)                        43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-store
    refresh_pattern ^http:\/\/openx.kompas.com.*\/                                43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-store
    refresh_pattern kaskus.\us.*\.(jp(e?g|e|2)|gif|png|swf)                       43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-store
    refresh_pattern ^http:\/\/img.kaskus.us.*\.(jpg|gif|png|swf)                  43200  100% 129600 reload-into-ims ignore-reload override-expire ignore-no-store
    
    #IIX DOWNLOAD
    refresh_pattern ^http:\/\/\.www[0-9][0-9]\.indowebster\.com\/(.*)(mp3|rar|zip|flv|wmv|3gp|mp(4|3)|exe|msi|zip) 43200 100% 129600 reload-into-ims  ignore-reload override-expire ignore-no-store  ignore-auth
    
    #refresh_pattern -i ^http://(khm?)([^/]*?)\.google\.(de|com)     129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    
    refresh_pattern -i ^http://ecn\.t\d\.tiles\.virtualearth\.net/tiles/\w*\.jpeg     129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload
    
    url_rewrite_access allow all
    
    #RAM size is 32GB
    cache_mem 15000 MB
    maximum_object_size_in_memory 100000 KB
    memory_replacement_policy heap GDSF
    cache_replacement_policy heap LFUDA
    minimum_object_size 0 KB
    maximum_object_size 5000 MB
    cache_dir ufs /var/squid/cache 150000 32 256
    offline_mode off
    cache_swap_low 94
    cache_swap_high 95
    acl donotcache dstdomain "/var/squid/acl/donotcache.acl"
    cache deny donotcache
    cache allow all
    
    # Add any of your own refresh_pattern entries above these.
    refresh_pattern ^ftp:    1440  20%  10080
    refresh_pattern ^gopher:  1440  0%  1440
    refresh_pattern -i (/cgi-bin/|\?) 0  0%  0
    refresh_pattern .    0  20%  4320
    
    #Remote proxies
    
    # Setup some default acls
    # ACLs all, manager, localhost, and to_localhost are predefined.
    acl allsrc src all
    acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 3129 1025-65535
    acl sslports port 443 563
    
    acl purge method PURGE
    acl connect method CONNECT
    
    # Define protocols used for redirects
    acl HTTP proto HTTP
    acl HTTPS proto HTTPS
    acl allowed_subnets src 192.168.111.2/32 172.16.0.0/12 10.0.0.0/8
    acl whitelist dstdom_regex -i "/var/squid/acl/whitelist.acl"
    http_access allow manager localhost
    
    http_access deny manager
    http_access allow purge localhost
    http_access deny purge
    http_access deny !safeports
    http_access deny CONNECT !sslports
    
    # Always allow localhost connections
    http_access allow localhost
    
    quick_abort_min 1000 KB
    quick_abort_max 5000 KB
    quick_abort_pct 80
    request_body_max_size 0 KB
    delay_pools 1
    delay_class 1 2
    delay_parameters 1 -1/-1 -1/-1
    delay_initial_bucket_level 100
    delay_access 1 allow allsrc
    
    # Reverse Proxy settings
    
    # Custom options before auth
    icap_service_failure_limit -1
    debug_options 1,5
    debug_options 5,4
    debug_options 6,2
    qos_flows local-hit=0x30
    
    # Always allow access to whitelist domains
    http_access allow whitelist
    # Setup allowed ACLs
    # Allow local network(s) on interface(s)
    http_access allow allowed_subnets
    http_access allow localnet
    # Default block all to be sure
    http_access deny allsrc
    
    
    1 Reply Last reply Reply Quote 0
    • D
      doktornotor Banned
      last edited by Feb 16, 2017, 12:33 PM

      Remove all the evil refresh patterns before debugging anything else.

      1 Reply Last reply Reply Quote 0
      • V
        Valeriy
        last edited by Feb 16, 2017, 1:29 PM

        Of course!

        Refresh patterns has been  removed for debugging purposes, it did not influence the outcome…

        1 Reply Last reply Reply Quote 0
        • V
          Valeriy
          last edited by Feb 16, 2017, 1:33 PM

          Problem seems to disappear when I made the following change today:

          http_port 127.0.0.1:3128 intercept

          http_port 3128 transparent

          Seems to be a fix!
          though, while playing with debug options before that change earlier, I could not find any trace of originated "Select Loop Error", not the meaning of it.

          1 Reply Last reply Reply Quote 0
          • D
            doktornotor Banned
            last edited by Feb 16, 2017, 1:47 PM

            Perhaps you should just either stick with the GUI for configuration or deploy your own Squid appliance elsewhere and do whatever you want with that.

            1 Reply Last reply Reply Quote 0
            • V
              Valeriy
              last edited by Feb 16, 2017, 1:57 PM

              Are you saying that on forum  I should not share solution that does not involve GUI? Please elaborate, Sir!

              1 Reply Last reply Reply Quote 0
              • D
                doktornotor Banned
                last edited by Feb 16, 2017, 2:06 PM

                1/ There is no such solution. Any manual messing with the configuration files will get overwritten on next package resync. (Triggered by anything from WAN IP change, saving configuration, reconfiguring firewall, upgrading other packages or god knows what.)

                2/ transparent and intercept are exact same things, transparent is a deprecated backwards-compatibility name, since Squid 3.1.

                1 Reply Last reply Reply Quote 0
                • V
                  Valeriy
                  last edited by Feb 16, 2017, 2:08 PM

                  In regards to deploying squid appliance elsewhere, I am thinking about that.
                  Is there a way to deploy transparent proxy solution without proxy sitting on NAT/edge firewall, that is doable via Pfsense GUI?
                  Wccp is anywhere in plans?

                  I need limiters/transparent proxy functionality + local hits (from cache) to avoid being capped by limiters.

                  Look at this topic, I started recently that involves Squid marking local hits with TOS value and a little bit in depth explanation of network setup
                  https://forum.pfsense.org/index.php?topic=125646.0

                  1 Reply Last reply Reply Quote 0
                  • D
                    doktornotor Banned
                    last edited by Feb 16, 2017, 2:15 PM

                    pfSense is not a proxy appliance. Cannot assist with anything like that, sorry.

                    http://wiki.squid-cache.org/ConfigExamples/Intercept/FreeBsdWccp2Receiver - outdated, unmaintained, good luck.

                    1 Reply Last reply Reply Quote 0
                    • V
                      Valeriy
                      last edited by Feb 16, 2017, 2:21 PM

                      Thanks for insights, anyway.

                      changed directive to
                      http_port 3128 intercept
                      So far seems to be running good.
                      What are implications of that?

                      Default pfsense GUI will make change back to http_port 127.0.0.1:3128 intercept that was causing that kid1| Select loop Error. Retry 1

                      1 Reply Last reply Reply Quote 0
                      • V
                        Valeriy
                        last edited by Feb 16, 2017, 2:24 PM

                        @doktornotor:

                        1/ There is no such solution. Any manual messing with the configuration files will get overwritten on next package resync. (Triggered by anything from WAN IP change, saving configuration, reconfiguring firewall, upgrading other packages or god knows what.)

                        2/ transparent and intercept are exact same things, transparent is a deprecated backwards-compatibility name, since Squid 3.1.

                        1/Completely aware of it, thanks for mentioning.
                        2/OK

                        1 Reply Last reply Reply Quote 0
                        • D
                          doktornotor Banned
                          last edited by Feb 16, 2017, 2:39 PM

                          As noted, intercept and transparent are the same thing. If you simply restarted Squid, you'd likely get the same result.

                          1 Reply Last reply Reply Quote 0
                          • D
                            doktornotor Banned
                            last edited by Feb 16, 2017, 9:35 PM

                            http://bugs.squid-cache.org/show_bug.cgi?id=2816 - maybe if you wait another 8 years, they'll eventually fix it. :P

                            1 Reply Last reply Reply Quote 0
                            • V
                              Valeriy
                              last edited by Feb 22, 2017, 4:12 PM

                              [SOLVED]

                              After complete reinstall of Squid and running clean configuration (optimized for current load) and
                              by running in heavy debug mode (generated gigs of logs), I could finally figure out what was the cause of that error.. At least in my case it was related to "digest rebuild" process that was occurring every hour.

                              
                              2017/02/18 03:51:56.806 kid1| 71,2| store_digest.cc(310) storeDigestRebuildStart: storeDigestRebuildStart: rebuild #1
                              2017/02/18 03:51:56.806 kid1| 71,2| store_digest.cc(95) storeDigestCalcCap: have: 5394, want 5394 entries; limits: [19709, 11815384]
                              2017/02/18 03:51:56.806 kid1| 71,2| store_digest.cc(332) storeDigestResize: 11815384 -> 19709; change: 11795675 (100%)
                              2017/02/18 03:51:56.806 kid1| 71,2| store_digest.cc(339) storeDigestResize: big change, resizing.
                              2017/02/18 03:51:56.806 kid1| 70,2| CacheDigest.cc(49) cacheDigestInit: cacheDigestInit: capacity: 19709 entries, bpe: ; size: 12319 bytes
                              2017/02/18 03:51:56.806 kid1| 71,2| store_digest.cc(415) storeDigestRewriteStart: storeDigestRewrite: start rewrite #1
                              2017/02/18 03:51:56.806 kid1| 71,2| store_digest.cc(429) storeDigestRewriteStart: storeDigestRewriteStart: waiting for rebuild to finish.
                              2017/02/18 03:51:56.951 kid1| 71,2| store_digest.cc(369) storeDigestRebuildFinish: storeDigestRebuildFinish: done.
                              
                              

                              Digest is used to communicate cached object information with other caches (Squid servers).
                              Every time when digest was rebuilt, Squid would stop serving requests for 2-3 minutes with all kind of DNS/network related issues.

                              I have used custom directive to disable digest generation and it served the purpose.

                              digest_generation off

                              Of course, if you are running other caches that rely on yours as peer or parent, you should find other solution.

                              P.S.

                              I am not sure why with heavy loaded cache (ten of millions of objects, >150GB) that loop Error happened more often.. (every 5 minutes sometimes)
                              And why sometimes it would not have happened for few hours?

                              Now the only (2-5 min) interruption of Squid happens only after midnight

                              
                              storeDirWriteCleanLogs: Starting...
                              2017/02/21 00:00:00 kid1|     65536 entries written so far.
                              2017/02/21 00:00:01 kid1|    131072 entries written so far.
                              2017/02/21 00:00:01 kid1|    196608 entries written so far.
                              2017/02/21 00:00:01 kid1|    262144 entries written so far.
                              2017/02/21 00:00:01 kid1|    327680 entries written so far.
                              2017/02/21 00:00:01 kid1|    393216 entries written so far.
                              2017/02/21 00:00:01 kid1|    458752 entries written so far.
                              2017/02/21 00:00:01 kid1|    524288 entries written so far.
                              2017/02/21 00:00:01 kid1|    589824 entries written so far.
                              2017/02/21 00:00:01 kid1|    655360 entries written so far.
                              2017/02/21 00:00:01 kid1|    720896 entries written so far.
                              2017/02/21 00:00:01 kid1|    786432 entries written so far.
                              2017/02/21 00:00:01 kid1|    851968 entries written so far.
                              2017/02/21 00:00:01 kid1|    917504 entries written so far.
                              2017/02/21 00:00:01 kid1|    983040 entries written so far.
                              2017/02/21 00:00:01 kid1|   1048576 entries written so far.
                              2017/02/21 00:00:01 kid1|   1114112 entries written so far.
                              2017/02/21 00:00:01 kid1|   1179648 entries written so far.
                              2017/02/21 00:00:01 kid1|   1245184 entries written so far.
                              2017/02/21 00:00:01 kid1|   1310720 entries written so far.
                              2017/02/21 00:00:02 kid1|   1376256 entries written so far.
                              2017/02/21 00:00:02 kid1|   1441792 entries written so far.
                              2017/02/21 00:00:02 kid1|   Finished.  Wrote 1500499 entries.
                              2017/02/21 00:00:02 kid1|   Took 2.03 seconds (737535.86 entries/sec).
                              2017/02/21 00:00:02 kid1| logfileRotate: stdio:/var/squid/logs/access.log
                              2017/02/21 00:00:02 kid1| Rotate log file stdio:/var/squid/logs/access.log
                              2017/02/21 00:00:03 kid1| helperOpenServers: Starting 20/200 'squidGuard' processes
                              2017/02/21 00:02:57 kid1|  FD 365, [::] [Stopped, reason:Listener socket closed job2092676]: (53) Software caused connection abort
                              2017/02/21 00:02:57 kid1|  FD 365, [::] [Stopped, reason:Listener socket closed job2092676]: (53) Software caused connection abort
                              2017/02/21 00:02:57 kid1|  FD 365, [::] [Stopped, reason:Listener socket closed job2092676]: (53) Software caused connection abort
                              ...... (~300 more)
                              
                              

                              Regards,

                              V.

                              1 Reply Last reply Reply Quote 0
                              • D
                                doktornotor Banned
                                last edited by Feb 22, 2017, 5:24 PM

                                Yeah, no idea either. Whatever is wrong with digests and upstream caches needs to be taken upstream and solved there. There's nothing pfSense-specific here.

                                http://lists.squid-cache.org/listinfo/squid-users would be a good starting point, I guess.

                                1 Reply Last reply Reply Quote 0
                                • K
                                  KOM
                                  last edited by Feb 22, 2017, 5:45 PM

                                  Good work.

                                  1 Reply Last reply Reply Quote 0
                                  • D
                                    doktornotor Banned
                                    last edited by Feb 25, 2017, 7:24 PM

                                    Made the digest_generation junk off by default in 0.4.36_1. (No GUI option, not worth it.)

                                    https://github.com/pfsense/FreeBSD-ports/pull/313

                                    1 Reply Last reply Reply Quote 0
                                    17 out of 17
                                    • First post
                                      17/17
                                      Last post
                                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                      This community forum collects and processes your personal information.
                                      consent.not_received