Squid3 recently very slow
So I recently updated to the squid 3.4.10_2 pkg 0.2.7 package, and I'm not sure what happened, but it seems to correlate with some really slow/bad web access.
If I disable squid, all is well. If not, I frequently get websites that load slowly, or even time out (Squid actually reports a timeout connecting to the IP). A refresh usually fixes the problem, but not always. This is happening in all browsers (Chrome/Firefox/IE) and on multiple computers, both wired and wireless. So I am pretty sure the issue lies in the pfSense system (hw or sw). Since nothing else on the system seems to be slow, I am currently ruling out hw. The main config option I use is the transparent proxy setting. Pretty much everything else is default.
I've removed and re-installed the package. I removed the package, removed every file or directory with "squid" in the name and reinstalled squid3. No go. I've changed some config settings. No go. I'm currently trying to figure out how to get back to original/default settings, since current settings are preserved in the main pfSense config and are not stored with the package.
I am not using any other Squid-related packages, all other packages I have installed are iperf, mtr-nox11, OpenVPN Client Export Utility, and Service Watchdog.
Falling back to the "squid" vs "squid3" package restores speedy web access, but does not work with many websites.
I'm not sure what I can do to either a) troubleshoot (already checked logs, nothing obvious) or b) provide devs with useful info. If anyone has ideas or suggestions, please let me know.
Get squid3 working again and then shell in and check the squid client manager:
squidclient -h pfsense_ip -p 3128 mgr:info
Look for the Median Service Times section, and check to see that nothing is totally out of whack.
Next, do a
tail -f /var/log/squid/access.log
while browsing and see what's happening in realtime.
Okay, so I went ahead and checked out those things. The Median Service Times were as follows:
Median Service Times (seconds) 5 min 60 min: HTTP Requests (All): 0.10857 0.02899 Cache Misses: 0.10857 0.03829 Cache Hits: 0.00000 0.00000 Near Hits: 0.00000 0.00000 Not-Modified Replies: 0.00000 0.00000 DNS Lookups: 0.01210 0.01046 ICP Queries: 0.00000 0.00000
Which looks fine to me. I didn't do many requests, but I did enough so I got some that went through okay, some that were slow, and some that failed. I investigated one that failed, and this is what I found in the squid access.log:
1429108291.521 60287 192.168.200.104 TCP_MISS/503 4502 GET http://here.com/traffic/usa/washington-dc - ORIGINAL_DST/22.214.171.124 text/html 1429108293.842 59 192.168.200.104 TCP_MISS/301 785 GET http://here.com/favicon.ico - ORIGINAL_DST/126.96.36.199 text/html
In the browser, Squid returned an error indicating that there was a timeout contacting the site. Any time this happens, if I just refresh it usually loads just fine. The sites are not down, since at the same time I can access them just fine from other computers or browsers not configured to use Squid.
Also, I already had the option mentioned here https://forum.pfsense.org/index.php?topic=52735.msg284810#msg284810 turned on. Doesn't seem to make a difference.
No hits at all. I wonder if your cache folder hierarchy needs to be rebuilt?
IIRC, there are some issues with squid3 in transparent mode at the moment:
You might be hitting those problems as well. I've spent some time studying squid3, squidguard, sarg and lightsquid. I've gotten everything running on a Ubuntu Server 14.10 box. I will be installing a standalone proxy once Ubuntu Server 15.04 comes out next week. I've come to the conclusion that it's best to separate extra services from the basic routing firewall, so bye-bye to all packages except reporting, like bandwidthd.
Thanks for pointing out those links. I did have transparent mode on. I just turned it off and manually configured a browser to use the proxy. Also, yesterday I deleted the entire Squid cache folder structure via shell on pfSense. Still seem to have the same problem.
After you deleted it, I assume you rebuilt it with squid3 -z?
I didn't, though after the delete, I removed the entire squid3 package and re-installed it…which I assume will do the rebuild?
Should, but you should do it just to be sure.
Okay, I stopped Squid, ran this:
[2.2.1-RELEASE][root@fw]/root: /usr/local/sbin/squid -z [2.2.1-RELEASE][root@fw]/root: 2015/04/15 16:28:15 kid1| Creating missing swap directories
And restarted Squid. Still getting the same bad performance.
I don't know what else to tell you.
Heh, np…I think my next experiment will be to set myself up to get some pcaps, on the client and on the server (both internal and WAN), to see what's happening on the network.
So I am still not sure exactly what the heck is going on. In some cases, it does appear that SYNs are not being responded to. I am not sure why. Then shortly after, it works…???
I added the following to my Squid config, on the General tab in the "Custom ACLS (Before_Auth)" section, and this is helping a lot...though still not good enough for "production":
connect_timeout 2 forward_max_tries 2 connect_retries 2