Kid1| Select loop Error. Retry 1
-
kid1| Select loop Error. Retry 1
What does this error mean?
What debug options might help to trace it down to event that causes it?This error is happening randomly, sometimes every 2-3 minutes, sometimes once per 1-3 hours.
During this event squid becomes very slow to accept new HTTP requests.Squid version is 3.5.23.
Config:http_port x.x.x.x:3128 http_port 127.0.0.1:3128 intercept icp_port 0 dns_v4_first off pid_filename /var/run/squid/squid.pid cache_effective_user squid cache_effective_group proxy error_default_language en icon_directory /usr/local/etc/squid/icons visible_hostname localhost cache_mgr admin@localhost access_log /var/squid/logs/access.log cache_log /var/squid/logs/cache.log cache_store_log none netdb_filename /var/squid/logs/netdb.state pinger_enable off pinger_program /usr/local/libexec/squid/pinger logfile_rotate 365 debug_options rotate=365 shutdown_lifetime 3 seconds # Allow local network(s) on interface(s) acl localnet src x.x.x.x/30 forwarded_for delete via off httpd_suppress_version_string on uri_whitespace strip #Refresh patterns #All File V 0.03 #new refresh patterns 2 refresh_pattern -i (\.|-)(mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-store ignore-private ignore-auth #end new refresh patterns 2 #new refresh patterns refresh_pattern -i (\.|-)(mp3|m4a|aa?c3?|wm?av?|og(x|v|a|g)|ape|mka|au|aiff|zip|flac|m4(b|r)|m1v|m2(v|p)|mo(d|v)|arj|appx|lha|lzh|on2)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-store ignore-private ignore-auth refresh_pattern -i (\.|-)(exe|bin|(n|t)ar|acv|(r|j)ar|t?gz|(g|b)z(ip)?2?|7?z(ip)?|wm[v|a]|patch|diff|mar|vpu|inc|r(a|p)m|kom|iso|sys|[ap]sf|ms[i|u|f]|dat|msi|cab|psf|dvr-ms|ace|asx|qt|xt|esd)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-store ignore-private ignore-auth refresh_pattern -i (\.|-)(ico(.*)?|pn[pg]|css|(g|t)iff?|jpe?g(2|3|4)?|psd|c(d|b)r|cad|bmp|img)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-store ignore-private ignore-auth refresh_pattern -i (\.|-)(webm|(x-)?swf|mp(eg)?(3|4)|mpe?g(av)?|(x-)?f(l|4)v|divx?|rmvb?|mov|trp|ts|avi|m38u|wmv|wmp|m4v|mkv|asf|dv|vob|3gp?2?)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-store ignore-private ignore-auth refresh_pattern -i (\.|-)(docx?|xlsx?|pptx?|rtf|xml|pdf|tiff?|txt)(\?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-store ignore-private ignore-auth #Website refresh_pattern -i (\.|-)(xml|js|jsp|txt|css)(\?.*)?$ 360 40% 1440 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-store ignore-private ignore-auth refresh_pattern -i .index.(html|htm)$ 0 40% 1440 refresh_pattern . 30 25% 1440 #end new refresh patterns refresh_pattern -i \.(3gp|7z|ace|asx|avi|bin|cab|dat|deb|rpm|divx|dvr-ms) 129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload #refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|(x-|)flv) 129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload refresh_pattern -i \.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js) 129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload refresh_pattern -i \.(mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p)) 129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload refresh_pattern -i \.(og(x|v|a|g)|rar|rm|r(a|p)m|snd|vob|wav) 129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload refresh_pattern -i \.(pp(s|t)|wax|wm(a|v)|wmx|wpl|zip|cb(r|z|t)) 129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern ^ftp: 10080 95% 43200 override-lastmod reload-into-ims refresh_pattern -i \.(doc|pdf)$ 100080 90% 43200 override-expire ignore-no-store ignore-private reload-into-ims refresh_pattern -i \.(html|htm)$ 1440 40% 40320 ignore-no-store ignore-private override-expire reload-into-ims refresh_pattern (Release|Packages(.gz)*)$ 0 20% 2880 refresh_pattern . 180 95% 43200 override-lastmod reload-into-ims # 1 year = 525600 mins, 1 month = 43800 mins refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern \.(ico|video-stats)$ 129600 100% 129600 override-expire ignore-reload ignore-no-store ignore-private ignore-auth override-lastmod ignore-must-revalidate refresh_pattern imeem.*\.flv$ 0 0% 0 override-lastmod override-expire refresh_pattern \.rapidshare.*\/[0-9]*\/.*\/[^\/]* 161280 90% 161280 ignore-reload refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?) 129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.flv?) 129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims refresh_pattern ^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 129600 20% 129600 ignore-no-store ignore-private override-expire ignore-reload ignore-auth ignore-must-revalidate refresh_pattern ^.*safebrowsing.*google 129600 100% 129600 override-expire ignore-reload ignore-private ignore-auth ignore-must-revalidate refresh_pattern ^http://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.uk) 129600 100% 129600 override-expire ignore-reload ignore-private refresh_pattern ytimg\.com.*\.jpg 129600 100% 129600 override-expire ignore-reload refresh_pattern images\.friendster\.com.*\.(png|gif) 129600 100% 129600 override-expire ignore-reload refresh_pattern garena\.com 129600 100% 129600 override-expire reload-into-ims refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 129600 100% 129600 override-expire ignore-reload refresh_pattern vid\.akm\.dailymotion\.com.*\.on2\? 129600 100% 129600 override-expire override-lastmod refresh_pattern mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 129600 100% 129600 reload-into-ims override-expire ignore-private refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\. 129600 100% 129600 reload-into-ims ignore-no-store ignore-reload override-expire refresh_pattern ^http:\/\/www.onemanga.com.*\/ 129600 100% 129600 reload-into-ims ignore-no-store ignore-reload override-expire # ANTI VIRUS refresh_pattern guru.avg.com/.*\.(bin) 43200 100% 43200 ignore-no-store ignore-reload reload-into-ims refresh_pattern (avgate|avira).*(idx|gz)$ 43200 100% 43200 ignore-no-store ignore-reload reload-into-ims refresh_pattern kaspersky.*\.avc$ 43200 100% 43200 ignore-no-store ignore-reload reload-into-ims refresh_pattern kaspersky 43200 100% 43200 ignore-no-store ignore-reload reload-into-ims refresh_pattern update.nai.com/.*\.(gem|zip|mcs) 43200 100% 43200 ignore-no-store ignore-reload reload-into-ims refresh_pattern ^http:\/\/liveupdate.symantecliveupdate.com.*\(zip) 43200 100% 43200 ignore-no-store ignore-reload reload-into-ims refresh_pattern -i symantecliveupdate.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims refresh_pattern -i avast.com/.*\.(vpu|vpaa) 4320 100% 43200 reload-into-ims refresh_pattern -i avira-update.com/.*\.* 720 100% 10800 reload-into-ims #windows update refresh_pattern windowsupdate.com/.*\.(cab|exe) 43200 100% 129600 ignore-no-store ignore-reload reload-into-ims refresh_pattern update.microsoft.com/.*\.(cab|exe) 43200 100% 129600 ignore-no-store ignore-reload reload-into-ims refresh_pattern download.microsoft.com/.*\.(cab|exe) 43200 100% 129600 ignore-no-store ignore-reload reload-into-ims refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf) 4320 100% 43200 reload-into-ims refresh_pattern update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims #apple update refresh_pattern -i (download|adcdownload).apple.com/.*\.(pkg|dmg) 4320 100% 43200 reload-into-ims #images facebook refresh_pattern ((facebook.com)|(85.131.151.39)).*\.(jpg|png|gif) 129600 100% 129600 ignore-reload override-expire ignore-no-store refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|swf|mp3) 129600 100% 129600 ignore-reload override-expire ignore-no-store refresh_pattern static\.ak\.fbcdn\.net*\.(jpg|gif|png) 129600 100% 129600 ignore-reload override-expire ignore-no-store refresh_pattern ^http:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png) 129600 100% 129600 ignore-reload override-expire ignore-no-store #banner IIX refresh_pattern ^http:\/\/openx.*\.(jp(e?g|e|2)|gif|pn[pg]|swf|ico|css|tiff?) 129600 100% 129600 reload-into-ims ignore-reload override-expire ignore-no-store refresh_pattern ^http:\/\/ads(1|2|3).kompas.com.*\/ 43200 100% 129600 reload-into-ims ignore-reload override-expire ignore-no-store refresh_pattern ^http:\/\/img.ads.kompas.com.*\/ 43200 100% 129600 reload-into-ims ignore-reload override-expire ignore-no-store refresh_pattern .kompasimages.com.*\.(jpg|gif|png|swf) 43200 100% 129600 reload-into-ims ignore-reload override-expire ignore-no-store refresh_pattern ^http:\/\/openx.kompas.com.*\/ 43200 100% 129600 reload-into-ims ignore-reload override-expire ignore-no-store refresh_pattern kaskus.\us.*\.(jp(e?g|e|2)|gif|png|swf) 43200 100% 129600 reload-into-ims ignore-reload override-expire ignore-no-store refresh_pattern ^http:\/\/img.kaskus.us.*\.(jpg|gif|png|swf) 43200 100% 129600 reload-into-ims ignore-reload override-expire ignore-no-store #IIX DOWNLOAD refresh_pattern ^http:\/\/\.www[0-9][0-9]\.indowebster\.com\/(.*)(mp3|rar|zip|flv|wmv|3gp|mp(4|3)|exe|msi|zip) 43200 100% 129600 reload-into-ims ignore-reload override-expire ignore-no-store ignore-auth #refresh_pattern -i ^http://(khm?)([^/]*?)\.google\.(de|com) 129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload refresh_pattern -i ^http://ecn\.t\d\.tiles\.virtualearth\.net/tiles/\w*\.jpeg 129600 100% 129600 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-reload url_rewrite_access allow all #RAM size is 32GB cache_mem 15000 MB maximum_object_size_in_memory 100000 KB memory_replacement_policy heap GDSF cache_replacement_policy heap LFUDA minimum_object_size 0 KB maximum_object_size 5000 MB cache_dir ufs /var/squid/cache 150000 32 256 offline_mode off cache_swap_low 94 cache_swap_high 95 acl donotcache dstdomain "/var/squid/acl/donotcache.acl" cache deny donotcache cache allow all # Add any of your own refresh_pattern entries above these. refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 #Remote proxies # Setup some default acls # ACLs all, manager, localhost, and to_localhost are predefined. acl allsrc src all acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 3128 3129 1025-65535 acl sslports port 443 563 acl purge method PURGE acl connect method CONNECT # Define protocols used for redirects acl HTTP proto HTTP acl HTTPS proto HTTPS acl allowed_subnets src 192.168.111.2/32 172.16.0.0/12 10.0.0.0/8 acl whitelist dstdom_regex -i "/var/squid/acl/whitelist.acl" http_access allow manager localhost http_access deny manager http_access allow purge localhost http_access deny purge http_access deny !safeports http_access deny CONNECT !sslports # Always allow localhost connections http_access allow localhost quick_abort_min 1000 KB quick_abort_max 5000 KB quick_abort_pct 80 request_body_max_size 0 KB delay_pools 1 delay_class 1 2 delay_parameters 1 -1/-1 -1/-1 delay_initial_bucket_level 100 delay_access 1 allow allsrc # Reverse Proxy settings # Custom options before auth icap_service_failure_limit -1 debug_options 1,5 debug_options 5,4 debug_options 6,2 qos_flows local-hit=0x30 # Always allow access to whitelist domains http_access allow whitelist # Setup allowed ACLs # Allow local network(s) on interface(s) http_access allow allowed_subnets http_access allow localnet # Default block all to be sure http_access deny allsrc
-
Remove all the evil refresh patterns before debugging anything else.
-
Of course!
Refresh patterns has been removed for debugging purposes, it did not influence the outcome…
-
Problem seems to disappear when I made the following change today:
http_port 127.0.0.1:3128 intercepthttp_port 3128 transparent
Seems to be a fix!
though, while playing with debug options before that change earlier, I could not find any trace of originated "Select Loop Error", not the meaning of it. -
Perhaps you should just either stick with the GUI for configuration or deploy your own Squid appliance elsewhere and do whatever you want with that.
-
Are you saying that on forum I should not share solution that does not involve GUI? Please elaborate, Sir!
-
1/ There is no such solution. Any manual messing with the configuration files will get overwritten on next package resync. (Triggered by anything from WAN IP change, saving configuration, reconfiguring firewall, upgrading other packages or god knows what.)
2/ transparent and intercept are exact same things, transparent is a deprecated backwards-compatibility name, since Squid 3.1.
-
In regards to deploying squid appliance elsewhere, I am thinking about that.
Is there a way to deploy transparent proxy solution without proxy sitting on NAT/edge firewall, that is doable via Pfsense GUI?
Wccp is anywhere in plans?I need limiters/transparent proxy functionality + local hits (from cache) to avoid being capped by limiters.
Look at this topic, I started recently that involves Squid marking local hits with TOS value and a little bit in depth explanation of network setup
https://forum.pfsense.org/index.php?topic=125646.0 -
pfSense is not a proxy appliance. Cannot assist with anything like that, sorry.
http://wiki.squid-cache.org/ConfigExamples/Intercept/FreeBsdWccp2Receiver - outdated, unmaintained, good luck.
-
Thanks for insights, anyway.
changed directive to
http_port 3128 intercept
So far seems to be running good.
What are implications of that?Default pfsense GUI will make change back to http_port 127.0.0.1:3128 intercept that was causing that kid1| Select loop Error. Retry 1
-
1/ There is no such solution. Any manual messing with the configuration files will get overwritten on next package resync. (Triggered by anything from WAN IP change, saving configuration, reconfiguring firewall, upgrading other packages or god knows what.)
2/ transparent and intercept are exact same things, transparent is a deprecated backwards-compatibility name, since Squid 3.1.
1/Completely aware of it, thanks for mentioning.
2/OK -
As noted, intercept and transparent are the same thing. If you simply restarted Squid, you'd likely get the same result.
-
http://bugs.squid-cache.org/show_bug.cgi?id=2816 - maybe if you wait another 8 years, they'll eventually fix it. :P
-
[SOLVED]
After complete reinstall of Squid and running clean configuration (optimized for current load) and
by running in heavy debug mode (generated gigs of logs), I could finally figure out what was the cause of that error.. At least in my case it was related to "digest rebuild" process that was occurring every hour.2017/02/18 03:51:56.806 kid1| 71,2| store_digest.cc(310) storeDigestRebuildStart: storeDigestRebuildStart: rebuild #1 2017/02/18 03:51:56.806 kid1| 71,2| store_digest.cc(95) storeDigestCalcCap: have: 5394, want 5394 entries; limits: [19709, 11815384] 2017/02/18 03:51:56.806 kid1| 71,2| store_digest.cc(332) storeDigestResize: 11815384 -> 19709; change: 11795675 (100%) 2017/02/18 03:51:56.806 kid1| 71,2| store_digest.cc(339) storeDigestResize: big change, resizing. 2017/02/18 03:51:56.806 kid1| 70,2| CacheDigest.cc(49) cacheDigestInit: cacheDigestInit: capacity: 19709 entries, bpe: ; size: 12319 bytes 2017/02/18 03:51:56.806 kid1| 71,2| store_digest.cc(415) storeDigestRewriteStart: storeDigestRewrite: start rewrite #1 2017/02/18 03:51:56.806 kid1| 71,2| store_digest.cc(429) storeDigestRewriteStart: storeDigestRewriteStart: waiting for rebuild to finish. 2017/02/18 03:51:56.951 kid1| 71,2| store_digest.cc(369) storeDigestRebuildFinish: storeDigestRebuildFinish: done.
Digest is used to communicate cached object information with other caches (Squid servers).
Every time when digest was rebuilt, Squid would stop serving requests for 2-3 minutes with all kind of DNS/network related issues.I have used custom directive to disable digest generation and it served the purpose.
digest_generation off
Of course, if you are running other caches that rely on yours as peer or parent, you should find other solution.
P.S.
I am not sure why with heavy loaded cache (ten of millions of objects, >150GB) that loop Error happened more often.. (every 5 minutes sometimes)
And why sometimes it would not have happened for few hours?Now the only (2-5 min) interruption of Squid happens only after midnight
storeDirWriteCleanLogs: Starting... 2017/02/21 00:00:00 kid1| 65536 entries written so far. 2017/02/21 00:00:01 kid1| 131072 entries written so far. 2017/02/21 00:00:01 kid1| 196608 entries written so far. 2017/02/21 00:00:01 kid1| 262144 entries written so far. 2017/02/21 00:00:01 kid1| 327680 entries written so far. 2017/02/21 00:00:01 kid1| 393216 entries written so far. 2017/02/21 00:00:01 kid1| 458752 entries written so far. 2017/02/21 00:00:01 kid1| 524288 entries written so far. 2017/02/21 00:00:01 kid1| 589824 entries written so far. 2017/02/21 00:00:01 kid1| 655360 entries written so far. 2017/02/21 00:00:01 kid1| 720896 entries written so far. 2017/02/21 00:00:01 kid1| 786432 entries written so far. 2017/02/21 00:00:01 kid1| 851968 entries written so far. 2017/02/21 00:00:01 kid1| 917504 entries written so far. 2017/02/21 00:00:01 kid1| 983040 entries written so far. 2017/02/21 00:00:01 kid1| 1048576 entries written so far. 2017/02/21 00:00:01 kid1| 1114112 entries written so far. 2017/02/21 00:00:01 kid1| 1179648 entries written so far. 2017/02/21 00:00:01 kid1| 1245184 entries written so far. 2017/02/21 00:00:01 kid1| 1310720 entries written so far. 2017/02/21 00:00:02 kid1| 1376256 entries written so far. 2017/02/21 00:00:02 kid1| 1441792 entries written so far. 2017/02/21 00:00:02 kid1| Finished. Wrote 1500499 entries. 2017/02/21 00:00:02 kid1| Took 2.03 seconds (737535.86 entries/sec). 2017/02/21 00:00:02 kid1| logfileRotate: stdio:/var/squid/logs/access.log 2017/02/21 00:00:02 kid1| Rotate log file stdio:/var/squid/logs/access.log 2017/02/21 00:00:03 kid1| helperOpenServers: Starting 20/200 'squidGuard' processes 2017/02/21 00:02:57 kid1| FD 365, [::] [Stopped, reason:Listener socket closed job2092676]: (53) Software caused connection abort 2017/02/21 00:02:57 kid1| FD 365, [::] [Stopped, reason:Listener socket closed job2092676]: (53) Software caused connection abort 2017/02/21 00:02:57 kid1| FD 365, [::] [Stopped, reason:Listener socket closed job2092676]: (53) Software caused connection abort ...... (~300 more)
Regards,
V.
-
Yeah, no idea either. Whatever is wrong with digests and upstream caches needs to be taken upstream and solved there. There's nothing pfSense-specific here.
http://lists.squid-cache.org/listinfo/squid-users would be a good starting point, I guess.
-
Good work.
-
Made the digest_generation junk off by default in 0.4.36_1. (No GUI option, not worth it.)
https://github.com/pfsense/FreeBSD-ports/pull/313