Web browsing very slow when squidguard active
-
Hello All,
I want to ask if someone here experienced slow browsing when squidguard active and have some tips to overcome this problem.
Before that, here is my pfsense server configuration :
PFSense version : 2.4.3-RELEASE (amd64)
Squid version : 0.4.43
Squidguard version : 1.16.4
Lightsquid version : 3.0.6_4Hardware (esxi virtualization)
CPU: Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz
HDD : 100 Gb (10Gb currently for squid hdd cache)
RAM : 12 Gb (10Gb set for squid memory cache)Total PC Clients : 200 max active at the same time
My Squid proxy setting is using authentication to samba 4 ldap and working perfectly.
But then my company asking me to limit social media and video streaming access on working hour, then i use squidguard for content filtering but it's really slowing down the internet access.
The squidguard itself is working as it should. The squidguard have some group ACL too which setting is depends on each company department policy.
I've done some tips like modify some lines on loader.conf etc but still no improvement on this problem.I set the integrations on squid advanced setting like this line below because when the url_rewrite_children is just 16 as default, the log always shows me to consider increasing the number because all rewrite children is busy
url_rewrite_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf;url_rewrite_bypass off;url_rewrite_children 200 startup=75 idle=10 concurrency=0
With this setting, 1 problem is gone.Clamav active or not it's no difference too.
The only error message i got is just ERROR: URL-rewrite produces invalid request: POST ERR HTTP/1.1That's it the detail, thank you for any suggestion for this problem :)
-
You say browsing is slow, then you say you bumped url_rewrite_children, then you say 1 problem is fixed.
What was the other problem???
-
Sorry if i'm not so clear.
With the setting, rewrite children busy problem is clear. But the slow browsing still there. -
It could be many things. Slow CPU, large cache, slow hard disk, slow DNS. Run this command:
squidclient -h LAN_IP -p 3128 mgr:info
where LAN_IP is the LAN IP address that squid listens on. Look at the Median Service Times section of the output. Anything look big? Here is mine for example:
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 18.48929 12.67057
Cache Misses: 0.09219 0.07014
Cache Hits: 0.00000 0.00000
Near Hits: 0.00000 0.03622
Not-Modified Replies: 0.00000 0.00000
DNS Lookups: 0.01331 0.01331
ICP Queries: 0.00000 0.00000DNS lookups should be measured in hundredths of a second. If they're tenths or larger, you have a DNS problem.
-
@KOM:
It could be many things. Slow CPU, large cache, slow hard disk, slow DNS. Run this command:
squidclient -h LAN_IP -p 3128 mgr:info
where LAN_IP is the LAN IP address that squid listens on. Look at the Median Service Times section of the output. Anything look big? Here is mine for example:
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 18.48929 12.67057
Cache Misses: 0.09219 0.07014
Cache Hits: 0.00000 0.00000
Near Hits: 0.00000 0.03622
Not-Modified Replies: 0.00000 0.00000
DNS Lookups: 0.01331 0.01331
ICP Queries: 0.00000 0.00000DNS lookups should be measured in hundredths of a second. If they're tenths or larger, you have a DNS problem.
Thanks for the suggestion, i'm back again to this squid things after finishing another project.
I changed my squid access port to 8080 and i allowed that port on PFSense LAN Rules and Squid ACL safe ports.
Squid proxy and squidguard are running now. with around 10 users, browsing are fast.
But on office hours, around 100 users logged on. A website which usually opens in 5 secs can become slower into 30 secs.
When i shut off the squidguard on the office hour, browsing become normal again but no content filtering at all.After i try your suggestion, I got some error like this :
squidclient -h MY_IP -p 8080 mgr:infoHTTP/1.1 403 Forbidden Server: squid/3.5.26 Mime-Version: 1.0 Date: Mon, 21 May 2018 11:22:04 GMT Content-Type: text/html;charset=utf-8 Content-Length: 3523 X-Squid-Error: ERR_ACCESS_DENIED 0 Vary: Accept-Language Content-Language: en X-Cache: MISS from xxx.xxx.local X-Cache-Lookup: NONE from xxx.xxx.local:8080 Via: 1.1 xxx.xxx.local (squid/3.5.26) Connection: close <title>ERROR: The requested URL could not be retrieved</title> # ERROR ## The requested URL could not be retrieved * * * The following error was encountered while trying to retrieve the URL: [cache_object://MY_IP/info](cache_object://MY_IP/info) > **Access Denied.** Access control configuration prevents your request from being allowed at this time. Please contact your service provider if you feel this is incorrect. Your cache administrator is [xxx@xxx.com](mailto:xxx@xxx.com?subject=CacheErrorInfo%20-%20ERR_ACCESS_DENIED&body=CacheHost%3A%20xxx03.xxx.local%0D%0AErrPage%3A%20ERR_ACCESS_DENIED%0D%0AErr%3A%20%5Bnone%5D%0D%0ATimeStamp%3A%20Mon,%2021%20May%202018%2011%3A22%3A04%20GMT%0D%0A%0D%0AClientIP%3A%2010.178.9.78%0D%0A%0D%0AHTTP%20Request%3A%0D%0AGET%20%2Finfo%20HTTP%2F1.0%0AUser-Agent%3A%20squidclient%2F3.5.26%0D%0AAccept%3A%20*%2F*%0D%0AConnection%3A%20close%0D%0AHost%3A%20MY_IP%0D%0A%0D%0A%0D%0A). * * * Generated Mon, 21 May 2018 11:22:04 GMT by xxx.xxx.local (squid/3.5.26)I need more suggestion, while i'm digging some logs now.
-
Ah now i can generate this :
Squid Object Cache: Version 3.5.27 Build Info: Service Name: squid Start Time: Mon, 21 May 2018 11:18:58 GMT Current Time: Tue, 22 May 2018 10:58:14 GMT Connection information for squid: Number of clients accessing cache: 208 Number of HTTP requests received: 463691 Number of ICP messages received: 0 Number of ICP messages sent: 0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 326.7 Average ICP messages per minute since start: 0.0 Select loop called: 27805416 times, 3.063 ms avg Cache information for squid: Hits as % of all requests: 5min: 0.0%, 60min: 0.0% Hits as % of bytes sent: 5min: 0.5%, 60min: 1.2% Memory hits as % of hit requests: 5min: 0.0%, 60min: 100.0% Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0% Storage Swap size: 0 KB Storage Swap capacity: 0.0% used, 0.0% free Storage Mem size: 360 KB Storage Mem capacity: 0.0% used, 100.0% free Mean Object Size: 0.00 KB Requests given to unlinkd: 0 Median Service Times (seconds) 5 min 60 min: HTTP Requests (All): 0.18699 3.11263 Cache Misses: 0.46965 0.27332 Cache Hits: 0.00000 0.00000 Near Hits: 0.00000 0.00000 Not-Modified Replies: 0.00000 0.00000 DNS Lookups: 0.00278 0.01269 ICP Queries: 0.00000 0.00000 Resource usage for squid: UP Time: 85155.572 seconds CPU Time: 691.461 seconds CPU Usage: 0.81% CPU Usage, 5 minute avg: 0.50% CPU Usage, 60 minute avg: 0.43% Maximum Resident Size: 1444736 KB Page faults with physical i/o: 0 Memory accounted for: Total accounted: 7096 KB memPoolAlloc calls: 132887173 memPoolFree calls: 135763872 File descriptor usage for squid: Maximum number of file descriptors: 293616 Largest file desc currently in use: 1426 Number of file desc currently in use: 239 Files queued for open: 0 Available number of file descriptors: 293377 Reserved number of file descriptors: 100 Store Disk files open: 0 Internal Data Structures: 54 StoreEntries 54 StoreEntries with MemObjects 51 Hot Object Cache Items 0 on-disk objectsLooks nothing wrong, or am i missing something maybe?