HTTP slow and HTTPS sometimes end up with error page…



  • Hi all,

    as Suggested by Gertjan that my concern should be post it here…

    here it goes…

    I’m using pfsense 2.4.2-RELEASE-p1 (amd64)

    System    VMware Virtual Machine
    Netgate Device ID: 0b04cb9c68032f0927c2 
    BIOS    Vendor: Phoenix Technologies LTD
    Version: 6.00
    Release Date: Tue Sep 30 2014
    Version    2.4.2-RELEASE-p1 (amd64)
    built on Tue Dec 12 13:45:26 CST 2017
    FreeBSD 11.1-RELEASE-p6

    The system is on the latest version.
    Version information updated at Mon Jan 22 8:40:32 WIB 2018
    CPU Type    Intel® Xeon® CPU E5-2697 v3 @ 2.60GHz
    56 CPUs: 1 package(s) x 56 core(s)
    AES-NI CPU Crypto: Yes (inactive)

    RAM 32Gb

    having HTTPS MIM with splice all mode…
    squidguard activated

    below is my squid.conf

    This file is automatically generated by pfSense

    Do not edit manually !

    http_port x.x.x.x:3128 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=10MB cert=/usr/local/etc/squid/serverkey.pem capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!SHA1:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv2,NO_SSLv3,NO_TLSv1,SINGLE_DH_USE,SINGLE_ECDH_USE

    http_port 127.0.0.1:3128 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=10MB cert=/usr/local/etc/squid/serverkey.pem capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!SHA1:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv2,NO_SSLv3,NO_TLSv1,SINGLE_DH_USE,SINGLE_ECDH_USE

    https_port 127.0.0.1:3129 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=10MB cert=/usr/local/etc/squid/serverkey.pem capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!SHA1:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv2,NO_SSLv3,NO_TLSv1,SINGLE_DH_USE,SINGLE_ECDH_USE

    icp_port 0
    digest_generation off
    dns_v4_first on
    pid_filename /var/run/squid/squid.pid
    cache_effective_user squid
    cache_effective_group proxy
    error_default_language en
    icon_directory /usr/local/etc/squid/icons
    visible_hostname xxxxxxx
    cache_mgr xxxxxxxx
    access_log /var/squid/logs/access.log
    cache_log /var/squid/logs/cache.log
    cache_store_log none
    netdb_filename /var/squid/logs/netdb.state
    pinger_enable on
    pinger_program /usr/local/libexec/squid/pinger
    sslcrtd_program /usr/local/libexec/squid/ssl_crtd -s /var/squid/lib/ssl_db -M 4MB -b 2048
    sslcrtd_children 5
    sslproxy_capath /usr/local/share/certs/
    sslproxy_options NO_SSLv2,NO_SSLv3,NO_TLSv1,SINGLE_DH_USE,SINGLE_ECDH_USE
    sslproxy_cipher EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!SHA1:!MD5:!EXP:!PSK:!SRP:!DSS
    sslproxy_cert_error allow all
    sslproxy_cert_adapt setValidAfter all

    logfile_rotate 7
    debug_options rotate=7
    shutdown_lifetime 3 seconds

    Allow local network(s) on interface(s)

    acl localnet src  x.x.x.x/29
    forwarded_for delete
    via off
    httpd_suppress_version_string on
    uri_whitespace strip

    All Files

    ######################

    refresh_pattern -i (.|-)(exe|bin|[n|t]ar|acv|[r|j]ar|t?gz|[g|b]z[ip]?2?|7?z[ip]?|zip|wm[v|a]|patch|diff|mar|vpu|inc|r[a|p]m|kom|iso|sys|[ap]sf|ms[i|u|f]|dat|msi|cab|psf|dvr-ms|ace|asx|qt|xt|esd)[?.*]?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth

    #Apple Files
    refresh_pattern -i (.|-)(ap[k|p]|dmg|ip[a|sw]|pkg)(?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth

    #Video Audio, Flash
    refresh_pattern -i (.|-)(webm|(x-)?swf|mp(eg)?(3|4)|mpe?g(av)?|(x-)?f(l|4)v|divx?|rmvb?|mov|trp|ts|avi|m38u|wmv|wmp|m4v|mkv|asf|dv|vob|3gp?2?)(?.)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
    refresh_pattern -i (.|-)(mp(3|4)|m4a|aa?c3?|wm?av?|og(x|v|a|g)|ape|mka|au|aiff|flac|m4(b|r)|m1v|m2(v|p)|mo(d|v)|arj|appx|lha|lzh|on2)(?.
    )?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth

    #images
    refresh_pattern -i (.|-)(ico(.)?|pn[pg]|css|(g|t)iff?|jpe?g(2|3|4)?|psd|c(d|b)r|cad|bmp|img)(?.)?$ 43200 100% 432000 override-lastmod reload-into-ims ignore-no-cache ignore-no-store ignore-private ignore-auth refresh-ims

    #Office Online
    refresh_pattern -i (.|-)(docx?|xlsx?|pptx?|rtf|xml|pdf|tiff?|txt)(?.*)?$ 43200 100% 432000 refresh-ims

    #Website
    refresh_pattern -i (.|-)(xml|js|jsp|txt|css)(?.*)?$ 360 40% 1440 refresh-ims
    refresh_pattern -i .index.(html|htm)$ 0 40% 1440

    cache_mem 15000 MB
    maximum_object_size_in_memory 1024000 KB
    memory_replacement_policy heap GDSF
    cache_replacement_policy heap LFUDA
    minimum_object_size 1000 KB
    maximum_object_size 100 MB
    cache_dir ufs /var/squid/cache 50000 16 256
    offline_mode off
    cache_swap_low 80
    cache_swap_high 90
    cache allow all

    Add any of your own refresh_pattern entries above these.

    refresh_pattern ^ftp:    1440  20%  10080
    refresh_pattern ^gopher:  1440  0%  1440
    refresh_pattern -i (/cgi-bin/|?) 0  0%  0
    refresh_pattern .    0  20%  4320

    #Remote proxies

    Setup some default acls

    ACLs all, manager, localhost, and to_localhost are predefined.

    acl allsrc src all
    acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 3129 1025-65535
    acl sslports port 443 563

    acl purge method PURGE
    acl connect method CONNECT

    Define protocols used for redirects

    acl HTTP proto HTTP
    acl HTTPS proto HTTPS

    SslBump Peek and Splice

    http://wiki.squid-cache.org/Features/SslPeekAndSplice

    http://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit

    Match against the current step during ssl_bump evaluation [fast]

    Never matches and should not be used outside the ssl_bump context.

    At each SslBump step, Squid evaluates ssl_bump directives to find

    the next bumping action (e.g., peek or splice). Valid SslBump step

    values and the corresponding ssl_bump evaluation moments are:

    #  SslBump1: After getting TCP-level and HTTP CONNECT info.
    #  SslBump2: After getting TLS Client Hello info.
    #  SslBump3: After getting TLS Server Hello info.

    These ACLs exist even when ‘SSL/MITM Mode’ is set to ‘Custom’ so that

    they can be used there for custom configuration.

    acl step1 at_step SslBump1
    acl step2 at_step SslBump2
    acl step3 at_step SslBump3
    acl allowed_subnets src x.x.x.x/8
    http_access allow manager localhost

    http_access deny manager
    http_access allow purge localhost
    http_access deny purge
    http_access deny !safeports
    http_access deny CONNECT !sslports

    Always allow localhost connections

    http_access allow localhost

    request_body_max_size 0 KB
    delay_pools 1
    delay_class 1 2
    delay_parameters 1 -1/-1 -1/-1
    delay_initial_bucket_level 100
    delay_access 1 allow allsrc

    Reverse Proxy settings

    Package Integration

    url_rewrite_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf
    url_rewrite_bypass off
    url_rewrite_children 16 startup=8 idle=4 concurrency=0

    Custom options before auth

    ssl_bump peek step1
    ssl_bump splice all

    Setup allowed ACLs

    Allow local network(s) on interface(s)

    http_access allow allowed_subnets
    http_access allow localnet

    Default block all to be sure

    http_access deny allsrc

    ==============================

    I’m having intermittent slowness on HTTP and HTTPS….
    sometime HTTPS couldn’t load so I need to refresh it again…

    here is result from
    squidclient -h 127.0.0.1 -p 3128 mgr:info

    HTTP/1.1 200 OK
    Server: squid
    Mime-Version: 1.0
    Date: Mon, 22 Jan 2018 02:29:36 GMT
    Content-Type: text/plain;charset=utf-8
    Expires: Mon, 22 Jan 2018 02:29:36 GMT
    Last-Modified: Mon, 22 Jan 2018 02:29:36 GMT
    X-Cache: MISS from xxxxxxx
    X-Cache-Lookup: MISS from xxxxxxxx:3128
    Connection: close

    Squid Object Cache: Version 3.5.27
    Build Info:
    Service Name: squid
    Start Time:  Fri, 19 Jan 2018 01:41:33 GMT
    Current Time:  Mon, 22 Jan 2018 02:29:36 GMT
    Connection information for squid:
      Number of clients accessing cache:  8658
      Number of HTTP requests received:  6244979
      Number of ICP messages received:  0
      Number of ICP messages sent:  0
      Number of queued ICP replies:  0
      Number of HTCP messages received:  0
      Number of HTCP messages sent:  0
      Request failure ratio:    0.00
      Average HTTP requests per minute since start:  1429.7
      Average ICP messages per minute since start:  0.0
      Select loop called: 295699390 times, 0.886 ms avg
    Cache information for squid:
      Hits as % of all requests:  5min: 0.1%, 60min: 0.1%
      Hits as % of bytes sent:  5min: 31.2%, 60min: 29.6%
      Memory hits as % of hit requests:  5min: 94.1%, 60min: 83.2%
      Disk hits as % of hit requests:  5min: 0.0%, 60min: 3.8%
      Storage Swap size:  40931384 KB
      Storage Swap capacity:  79.9% used, 20.1% free
      Storage Mem size:  12797904 KB
      Storage Mem capacity:  83.3% used, 16.7% free
      Mean Object Size:  11648.09 KB
      Requests given to unlinkd:  1821
    Median Service Times (seconds)  5 min    60 min:
      HTTP Requests (All):  0.76407  0.64968
      Cache Misses:          0.25890  0.22004
      Cache Hits:          274.90301 28.47649
      Near Hits:            0.00000 221.51346
      Not-Modified Replies:  0.00000  0.08729
      DNS Lookups:          0.07284  0.06083
      ICP Queries:          0.00000  0.00000
    Resource usage for squid:
      UP Time:  262082.878 seconds
      CPU Time:  57314.937 seconds
      CPU Usage:  21.87%
      CPU Usage, 5 minute avg:  100.00%
      CPU Usage, 60 minute avg:  99.75%
      Maximum Resident Size: 96936368 KB
      Page faults with physical i/o: 46157
    Memory accounted for:
      Total accounted:      417220 KB
      memPoolAlloc calls: 767721337
      memPoolFree calls:  816550705
    File descriptor usage for squid:
      Maximum number of file descriptors:  942417
      Largest file desc currently in use:  6763
      Number of file desc currently in use: 5758
      Files queued for open:                  0
      Available number of file descriptors: 936659
      Reserved number of file descriptors:  100
      Store Disk files open:                  8
    Internal Data Structures:
        6225 StoreEntries
        4183 StoreEntries with MemObjects
        1519 Hot Object Cache Items
        3514 on-disk objects

    I have 56 core but I found only 1 CPU utilized by squid with 100% CPU persistently

    seems that squid only single threaded…

    but I read from https://doc.pfsense.org/index.php/Low_Throughput_Troubleshooting
    saying that pfsense 2.2 or later already use multiple cores…

    what I missed on my configuration…

    last pid: 41316;  load averages:  1.12,  1.20,  1.22                                                                                            up 8+12:07:37  09:56:43
    114 processes: 3 running, 111 sleeping
    CPU:  1.7% user,  0.1% nice,  0.4% system,  0.1% interrupt, 97.6% idle
    Mem: 10G Active, 7186M Inact, 11G Laundry, 2491M Wired, 1571M Buf, 513M Free
    Swap: 4096M Total, 251M Used, 3845M Free, 6% Inuse

    PID USERNAME    THR PRI NICE  SIZE    RES STATE  C  TIME    WCPU COMMAND
    13212 squid        1 103    0 23729M 22535M CPU46  46 982:21 100.59% squid
    11539 root          1  52    0  261M 22536K accept 23  0:00  1.30% php-fpm
    87784 root          1  52  20 13084K  2156K wait  35  1:55  0.26% sh
    8091 root          1  20    0 12700K  1888K bpf    28  7:47  0.20% filterlog
    1622 squid        1  20    0 34124K 14152K sbwait 22  0:17  0.18% squidGuard
    9237 squid        1  20    0 34124K 14148K sbwait  1  0:16  0.15% squidGuard
    11166 squid        1  20    0 34124K 14156K sbwait 17  0:10  0.15% squidGuard
    15785 squid        1  20    0 34124K 14152K sbwait 54  0:09  0.15% squidGuard
    44726 root          1  20    0 10484K  1984K select 51  3:37  0.10% syslogd
    21613 root          1  20    0 20060K  3704K CPU33  33  0:00  0.08% top
    16266 squid        1  20    0 34124K 14152K sbwait  4  0:07  0.07% squidGuard
    87330 root          1  20    0 37712K  7044K kqread  2  0:32  0.07% nginx
    22108 squid        1  20    0 34124K 14156K sbwait 12  0:06  0.05% squidGuard
    22534 squid        1  20    0 34124K 14152K sbwait 53  0:05  0.04% squidGuard
    61005 squid        1  20    0 33780K  3520K select 49  0:50  0.03% pinger
    26037 squid        1  20    0 33780K  3512K select 30  0:47  0.02% pinger
    36541 squid        1  20    0 33780K  3592K select 52  0:08  0.02% pinger
    32991 squid        1  20    0 33780K  3512K select 36  0:48  0.02% pinger
    8509 squid        1  20    0 33780K  3520K select 41  0:43  0.02% pinger
    25623 squid        1  20    0 33780K  3512K select 33  0:50  0.02% pinger
    66183 squid        1  20    0 33780K  2940K select 28  0:47  0.02% pinger
    29798 squid        1  20    0 34124K 14148K sbwait 55  0:04  0.02% squidGuard
    18928 squid        1  20    0 33780K  2940K select 31  0:45  0.02% pinger
    51648 squid        1  20    0 33780K  3512K select 19  0:47  0.02% pinger
    30062 squid        1  20    0 33780K  3852K select 20  0:03  0.02% pinger
    62063 squid        1  20    0 33780K  3512K select  2  0:49  0.02% pinger
    65590 squid        1  20    0 33780K  3512K select 22  0:49  0.02% pinger
    42315 squid        1  20    0 34124K 14148K sbwait 17  0:04  0.02% squidGuard
    80972 squid        1  20    0 33780K  3568K select  9  0:26  0.02% pinger
    20730 squid        1  20    0 33780K  3520K select  3  0:47  0.02% pinger
    75460 squid        1  20    0 33780K  2944K select  6  0:47  0.02% pinger
    66930 root          5  52    0 13032K  2060K uwait  7  1:54  0.02% dpinger
    89505 squid        1  20    0 33780K  2936K select  8  0:47  0.01% pinger
    63016 squid        1  20    0 33780K  2944K select 21  0:48  0.01% pinger
    28848 squid        1  20    0 33780K  2940K select  9  0:46  0.01% pinger
    66070 root          5  52    0 13032K  2012K uwait  20  1:54  0.01% dpinger
    66431 root          5  52    0 10984K  2016K uwait  20  1:55  0.01% dpinger
      336 root          1  20    0  9560K  488K select 55  0:30  0.01% devd
    88531 root          1  20    0 78844K  7128K select 38  0:00  0.01% sshd
    25055 root          1  20    0 24612K 12432K select 10  0:34  0.00% ntpd
    5913 root          1  20    0 43140K  5428K kqread 29  0:10  0.00% lighttpd_ls



  • I would change memory cache size to 1/4 of installed memory and also lower the size of object’s kept in memory.

    Which in theory should let squid keep all that on disk instead of memory thus freeing the memory for more important
    thing’s.

    If it doesn’t help it is easy enough to change back.



  • Hi Impatient

    I already lower the value into this, and changed the memory replacement policy to LFUDA
    now things getting better, CPU usage for 60 min already around 25%…  later I will try to enable MITM with splice all again and let see how is the performance…

    or maybe I should use pfBlockerNG?? any though ?

    cache_mem 10000 MB
    maximum_object_size_in_memory 100000 KB
    memory_replacement_policy heap LFUDA
    cache_replacement_policy heap LFUDA
    minimum_object_size 1000 KB
    maximum_object_size 2024 MB
    cache_dir aufs /var/squid/cache 100000 16 256



  • I tend to set mine to keep the smaller object’s in memory and the larger object’s on disk
    and the Splice Whitelist,Bump Otherwise option.

    Speed wise I could never notice the difference between the two except now I have free
    memory to handle the requirement’s of other package’s.

    I haven’t used squidguard in a long time my kid’s are adult’s now so the only package’s
    I use are pfBlockerNG and Snort.

    Perhap’s one day it will become multi-threaded.



  • Hi Impatient

    May I know what is your configuration on MITM looks like….

    another thing with pfblockerNG… 
    if our HTTP traffic is transparently / explicitly via proxy (Squid)…  does the pfblockerng firewall rules HIT by the users, for the rule outbound traffic going to internet if any…

    I haven’t check yet, my feeling for transparent proxy still going to firewall rule from top to down but not for explicit proxy on the browser/WPAD/PAC…
    any thought ?



  • I don’t use the transparent option I configure the browser to use the proxy.

    I know used you used to be able to force everyone to use the proxy via the firewall
    rule’s and some firewall’s still do.

    My MITM setting’s are pretty much default except for the SSL Cert. Children is set at
    16



  • @sqrobin:

    ….

    cache_dir aufs /var/squid/cache 100000 16 256

    From your initially posted configuration to here you changed your storage type from ufs to the posix threaded aufs.  This would result in a performance change due to I/O process blocking.

    See the description for aufs here:  http://www.squid-cache.org/Versions/v3/3.5/cfgman/cache_dir.html



  • SQRobin,

    I’m in the same camp.

    Did you end up fixing this?

    I have 16 Cores, 32GB RAM.  60GB Cache (DiskD - Previously AUFS with zero change), 64MB Cache Memory Size, 256K Max object, Heap GDSF.

    RAM and SWAP often go haywire after about 10 hours.  I upped the SSL Daemon Children to 64 recently to assist.

    Any other tips?


    HTTP/1.1 200 OK
    Server: squid
    Mime-Version: 1.0
    Date: Wed, 07 Feb 2018 15:30:42 GMT
    Content-Type: text/plain;charset=utf-8
    Expires: Wed, 07 Feb 2018 15:30:42 GMT
    Last-Modified: Wed, 07 Feb 2018 15:30:42 GMT
    X-Cache: MISS from localhost
    X-Cache-Lookup: MISS from localhost:3128
    Via: 1.1 localhost (squid)
    Connection: close

    Squid Object Cache: Version 3.5.27
    Build Info:
    Service Name: squid
    Start Time: Wed, 07 Feb 2018 01:26:06 GMT
    Current Time: Wed, 07 Feb 2018 15:30:42 GMT
    Connection information for squid:
    Number of clients accessing cache: 864
    Number of HTTP requests received: 289166
    Number of ICP messages received: 0
    Number of ICP messages sent: 0
    Number of queued ICP replies: 0
    Number of HTCP messages received: 0
    Number of HTCP messages sent: 0
    Request failure ratio: 0.00
    Average HTTP requests per minute since start: 342.4
    Average ICP messages per minute since start: 0.0
    Select loop called: 18051093 times, 2.807 ms avg
    Cache information for squid:
    Hits as % of all requests: 5min: 2.6%, 60min: 3.0%
    Hits as % of bytes sent: 5min: 1.1%, 60min: 1.3%
    Memory hits as % of hit requests: 5min: 51.2%, 60min: 56.1%
    Disk hits as % of hit requests: 5min: 17.8%, 60min: 23.9%
    Storage Swap size: 47263468 KB
    Storage Swap capacity: 76.9% used, 23.1% free
    Storage Mem size: 64732 KB
    Storage Mem capacity: 98.8% used,  1.2% free
    Mean Object Size: 125.91 KB
    Requests given to unlinkd: 0
    Median Service Times (seconds)  5 min    60 min:
    HTTP Requests (All):  0.05046  0.02742
    Cache Misses:          0.08265  0.07825
    Cache Hits:            0.00286  0.00091
    Near Hits:            0.05633  0.08729
    Not-Modified Replies:  0.00179  0.00091
    DNS Lookups:          0.01046  0.01331
    ICP Queries:          0.00000  0.00000
    Resource usage for squid:
    UP Time: 50675.981 seconds
    CPU Time: 2292.172 seconds
    CPU Usage: 4.52%
    CPU Usage, 5 minute avg: 29.42%
    CPU Usage, 60 minute avg: 29.02%
    Maximum Resident Size: 29684640 KB
    Page faults with physical i/o: 183
    Memory accounted for:
    Total accounted:      219900 KB
    memPoolAlloc calls:  33590776
    memPoolFree calls:  34080247
    File descriptor usage for squid:
    Maximum number of file descriptors:  939474
    Largest file desc currently in use:  9246
    Number of file desc currently in use: 9012
    Files queued for open:                  0
    Available number of file descriptors: 930462
    Reserved number of file descriptors:  100
    Store Disk files open:                  0
    Internal Data Structures:
    382429 StoreEntries
    11534 StoreEntries with MemObjects
      4520 Hot Object Cache Items
    375364 on-disk objects


 

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy