Web browsing very slow when squidguard active



  • Hello All,

    I want to ask if someone here experienced slow browsing when squidguard active and have some tips to overcome this problem.

    Before that, here is my pfsense server configuration :
    PFSense version : 2.4.3-RELEASE (amd64)
    Squid version : 0.4.43
    Squidguard version :  1.16.4
    Lightsquid version : 3.0.6_4

    Hardware (esxi virtualization)
    CPU: Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz
    HDD : 100 Gb (10Gb currently for squid hdd cache)
    RAM : 12 Gb (10Gb set for squid memory cache)

    Total PC Clients : 200 max active at the same time

    My Squid proxy setting is using authentication to samba 4 ldap and working perfectly.
    But then my company asking me to limit social media and video streaming access on working hour, then i use squidguard for content filtering but it's really slowing down the internet access.
    The squidguard itself is working as it should. The squidguard have some group ACL too which setting is depends on each company department policy.
    I've done some tips like modify some lines on loader.conf etc but still no improvement on this problem.

    I set the integrations on squid advanced setting like this line below because when the url_rewrite_children is just 16 as default, the log always shows me to consider increasing the number because all rewrite children is busy
    url_rewrite_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf;url_rewrite_bypass off;url_rewrite_children 200 startup=75 idle=10 concurrency=0
    With this setting, 1 problem is gone.

    Clamav active or not it's no difference too.
    The only error message i got is just ERROR: URL-rewrite produces invalid request: POST ERR HTTP/1.1

    That's it the detail, thank you for any suggestion for this problem  :)



  • You say browsing is slow, then you say you bumped url_rewrite_children, then you say 1 problem is fixed.

    What was the other problem???



  • Sorry if i'm not so clear.
    With the setting, rewrite children busy problem is clear. But the slow browsing still there.



  • It could be many things.  Slow CPU, large cache, slow hard disk, slow DNS.  Run this command:

    squidclient -h LAN_IP -p 3128 mgr:info

    where LAN_IP is the LAN IP address that squid listens on.  Look at the Median Service Times section of the output.  Anything look big?  Here is mine for example:

    Median Service Times (seconds)  5 min    60 min:
            HTTP Requests (All):  18.48929 12.67057
            Cache Misses:          0.09219  0.07014
            Cache Hits:            0.00000  0.00000
            Near Hits:            0.00000  0.03622
            Not-Modified Replies:  0.00000  0.00000
            DNS Lookups:          0.01331  0.01331
            ICP Queries:          0.00000  0.00000

    DNS lookups should be measured in hundredths of a second.  If they're tenths or larger, you have a DNS problem.



  • @KOM:

    It could be many things.  Slow CPU, large cache, slow hard disk, slow DNS.  Run this command:

    squidclient -h LAN_IP -p 3128 mgr:info

    where LAN_IP is the LAN IP address that squid listens on.  Look at the Median Service Times section of the output.  Anything look big?  Here is mine for example:

    Median Service Times (seconds)  5 min    60 min:
            HTTP Requests (All):  18.48929 12.67057
            Cache Misses:          0.09219  0.07014
            Cache Hits:            0.00000  0.00000
            Near Hits:            0.00000  0.03622
            Not-Modified Replies:  0.00000  0.00000
            DNS Lookups:          0.01331  0.01331
            ICP Queries:          0.00000  0.00000

    DNS lookups should be measured in hundredths of a second.  If they're tenths or larger, you have a DNS problem.

    Thanks for the suggestion, i'm back again to this squid things after finishing another project.

    I changed my squid access port to 8080 and i allowed that port on PFSense LAN Rules and Squid ACL safe ports.
    Squid proxy and squidguard are running now. with around 10 users, browsing are fast.
    But on office hours, around 100 users logged on. A website which usually opens in 5 secs can become slower into 30 secs.
    When i shut off the squidguard on the office hour, browsing become normal again but no content filtering at all.

    After i try your suggestion, I got some error like this :
    squidclient -h MY_IP -p 8080 mgr:info

    
    HTTP/1.1 403 Forbidden
    Server: squid/3.5.26
    Mime-Version: 1.0
    Date: Mon, 21 May 2018 11:22:04 GMT
    Content-Type: text/html;charset=utf-8
    Content-Length: 3523
    X-Squid-Error: ERR_ACCESS_DENIED 0
    Vary: Accept-Language
    Content-Language: en
    X-Cache: MISS from xxx.xxx.local 
    X-Cache-Lookup: NONE from xxx.xxx.local:8080
    Via: 1.1 xxx.xxx.local (squid/3.5.26)
    Connection: close
    
    <title>ERROR: The requested URL could not be retrieved</title>
    
    # ERROR
    
    ## The requested URL could not be retrieved
    
    * * *
    
    The following error was encountered while trying to retrieve the URL: [cache_object://MY_IP/info](cache_object://MY_IP/info)
    
    > **Access Denied.**
    
    Access control configuration prevents your request from being allowed at this time. Please contact your service provider if you feel this is incorrect.
    
    Your cache administrator is [xxx@xxx.com](mailto:xxx@xxx.com?subject=CacheErrorInfo%20-%20ERR_ACCESS_DENIED&body=CacheHost%3A%20xxx03.xxx.local%0D%0AErrPage%3A%20ERR_ACCESS_DENIED%0D%0AErr%3A%20%5Bnone%5D%0D%0ATimeStamp%3A%20Mon,%2021%20May%202018%2011%3A22%3A04%20GMT%0D%0A%0D%0AClientIP%3A%2010.178.9.78%0D%0A%0D%0AHTTP%20Request%3A%0D%0AGET%20%2Finfo%20HTTP%2F1.0%0AUser-Agent%3A%20squidclient%2F3.5.26%0D%0AAccept%3A%20*%2F*%0D%0AConnection%3A%20close%0D%0AHost%3A%20MY_IP%0D%0A%0D%0A%0D%0A).
    
    * * *
    
    Generated Mon, 21 May 2018 11:22:04 GMT by xxx.xxx.local (squid/3.5.26)
    
    

    I need more suggestion, while i'm digging some logs now.



  • Ah now i can generate this :

    
    Squid Object Cache: Version 3.5.27
    Build Info:
    Service Name: squid
    Start Time:     Mon, 21 May 2018 11:18:58 GMT
    Current Time:   Tue, 22 May 2018 10:58:14 GMT
    Connection information for squid:
            Number of clients accessing cache:      208
            Number of HTTP requests received:       463691
            Number of ICP messages received:        0
            Number of ICP messages sent:    0
            Number of queued ICP replies:   0
            Number of HTCP messages received:       0
            Number of HTCP messages sent:   0
            Request failure ratio:   0.00
            Average HTTP requests per minute since start:   326.7
            Average ICP messages per minute since start:    0.0
            Select loop called: 27805416 times, 3.063 ms avg
    Cache information for squid:
            Hits as % of all requests:      5min: 0.0%, 60min: 0.0%
            Hits as % of bytes sent:        5min: 0.5%, 60min: 1.2%
            Memory hits as % of hit requests:       5min: 0.0%, 60min: 100.0%
            Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0%
            Storage Swap size:      0 KB
            Storage Swap capacity:   0.0% used,  0.0% free
            Storage Mem size:       360 KB
            Storage Mem capacity:    0.0% used, 100.0% free
            Mean Object Size:       0.00 KB
            Requests given to unlinkd:      0
    Median Service Times (seconds)  5 min    60 min:
            HTTP Requests (All):   0.18699  3.11263
            Cache Misses:          0.46965  0.27332
            Cache Hits:            0.00000  0.00000
            Near Hits:             0.00000  0.00000
            Not-Modified Replies:  0.00000  0.00000
            DNS Lookups:           0.00278  0.01269
            ICP Queries:           0.00000  0.00000
    Resource usage for squid:
            UP Time:        85155.572 seconds
            CPU Time:       691.461 seconds
            CPU Usage:      0.81%
            CPU Usage, 5 minute avg:        0.50%
            CPU Usage, 60 minute avg:       0.43%
            Maximum Resident Size: 1444736 KB
            Page faults with physical i/o: 0
    Memory accounted for:
            Total accounted:         7096 KB
            memPoolAlloc calls: 132887173
            memPoolFree calls:  135763872
    File descriptor usage for squid:
            Maximum number of file descriptors:   293616
            Largest file desc currently in use:   1426
            Number of file desc currently in use:  239
            Files queued for open:                   0
            Available number of file descriptors: 293377
            Reserved number of file descriptors:   100
            Store Disk files open:                   0
    Internal Data Structures:
                54 StoreEntries
                54 StoreEntries with MemObjects
                51 Hot Object Cache Items
                 0 on-disk objects
    
    

    Looks nothing wrong, or am i missing something maybe?