Squid3 recently very slow

lucky · Apr 15, 2015, 1:30 AM

So I recently updated to the squid 3.4.10_2 pkg 0.2.7 package, and I'm not sure what happened, but it seems to correlate with some really slow/bad web access.

If I disable squid, all is well. If not, I frequently get websites that load slowly, or even time out (Squid actually reports a timeout connecting to the IP). A refresh usually fixes the problem, but not always. This is happening in all browsers (Chrome/Firefox/IE) and on multiple computers, both wired and wireless. So I am pretty sure the issue lies in the pfSense system (hw or sw). Since nothing else on the system seems to be slow, I am currently ruling out hw. The main config option I use is the transparent proxy setting. Pretty much everything else is default.

I've removed and re-installed the package. I removed the package, removed every file or directory with "squid" in the name and reinstalled squid3. No go. I've changed some config settings. No go. I'm currently trying to figure out how to get back to original/default settings, since current settings are preserved in the main pfSense config and are not stored with the package.

I am not using any other Squid-related packages, all other packages I have installed are iperf, mtr-nox11, OpenVPN Client Export Utility, and Service Watchdog.

Falling back to the "squid" vs "squid3" package restores speedy web access, but does not work with many websites.

I'm not sure what I can do to either a) troubleshoot (already checked logs, nothing obvious) or b) provide devs with useful info. If anyone has ideas or suggestions, please let me know.

Thanks

KOM · Apr 15, 2015, 1:24 PM

Get squid3 working again and then shell in and check the squid client manager:

squidclient -h pfsense_ip -p 3128 mgr:info

Look for the Median Service Times section, and check to see that nothing is totally out of whack.

Next, do a

tail -f /var/log/squid/access.log

while browsing and see what's happening in realtime.

lucky · Apr 15, 2015, 6:46 PM

Okay, so I went ahead and checked out those things. The Median Service Times were as follows:

Median Service Times (seconds)  5 min    60 min:
        HTTP Requests (All):   0.10857  0.02899
        Cache Misses:          0.10857  0.03829
        Cache Hits:            0.00000  0.00000
        Near Hits:             0.00000  0.00000
        Not-Modified Replies:  0.00000  0.00000
        DNS Lookups:           0.01210  0.01046
        ICP Queries:           0.00000  0.00000

Which looks fine to me. I didn't do many requests, but I did enough so I got some that went through okay, some that were slow, and some that failed. I investigated one that failed, and this is what I found in the squid access.log:

1429108291.521  60287 192.168.200.104 TCP_MISS/503 4502 GET http://here.com/traffic/usa/washington-dc - ORIGINAL_DST/66.54.66.154 text/html
1429108293.842     59 192.168.200.104 TCP_MISS/301 785 GET http://here.com/favicon.ico - ORIGINAL_DST/66.54.66.154 text/html

In the browser, Squid returned an error indicating that there was a timeout contacting the site. Any time this happens, if I just refresh it usually loads just fine. The sites are not down, since at the same time I can access them just fine from other computers or browsers not configured to use Squid.

lucky · Apr 15, 2015, 6:56 PM

Also, I already had the option mentioned here https://forum.pfsense.org/index.php?topic=52735.msg284810#msg284810 turned on. Doesn't seem to make a difference.

KOM · Apr 15, 2015, 7:09 PM

No hits at all. I wonder if your cache folder hierarchy needs to be rebuilt?

IIRC, there are some issues with squid3 in transparent mode at the moment:

https://forum.pfsense.org/index.php?topic=91894.0

https://forum.pfsense.org/index.php?topic=89315.0

You might be hitting those problems as well. I've spent some time studying squid3, squidguard, sarg and lightsquid. I've gotten everything running on a Ubuntu Server 14.10 box. I will be installing a standalone proxy once Ubuntu Server 15.04 comes out next week. I've come to the conclusion that it's best to separate extra services from the basic routing firewall, so bye-bye to all packages except reporting, like bandwidthd.

lucky · Apr 15, 2015, 7:14 PM

Thanks for pointing out those links. I did have transparent mode on. I just turned it off and manually configured a browser to use the proxy. Also, yesterday I deleted the entire Squid cache folder structure via shell on pfSense. Still seem to have the same problem.

KOM · Apr 15, 2015, 7:17 PM

After you deleted it, I assume you rebuilt it with squid3 -z?

lucky · Apr 15, 2015, 7:29 PM

I didn't, though after the delete, I removed the entire squid3 package and re-installed it…which I assume will do the rebuild?

KOM · Apr 15, 2015, 8:22 PM

Should, but you should do it just to be sure.

lucky · Apr 15, 2015, 8:33 PM

Okay, I stopped Squid, ran this:

[2.2.1-RELEASE][root@fw]/root: /usr/local/sbin/squid -z
[2.2.1-RELEASE][root@fw]/root: 2015/04/15 16:28:15 kid1| Creating missing swap directories

And restarted Squid. Still getting the same bad performance.

KOM · Apr 15, 2015, 8:40 PM

I don't know what else to tell you.

lucky · Apr 15, 2015, 8:45 PM

Heh, np…I think my next experiment will be to set myself up to get some pcaps, on the client and on the server (both internal and WAN), to see what's happening on the network.

lucky · Apr 16, 2015, 12:11 AM

So I am still not sure exactly what the heck is going on. In some cases, it does appear that SYNs are not being responded to. I am not sure why. Then shortly after, it works…???

I added the following to my Squid config, on the General tab in the "Custom ACLS (Before_Auth)" section, and this is helping a lot...though still not good enough for "production":

connect_timeout 2
forward_max_tries 2
connect_retries 2