Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    HTTP slow and HTTPS sometimes end up with error page…

    General pfSense Questions
    4
    9
    1098
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      sqrobin last edited by

      Hi all,

      I'm using pfsense 2.4.2-RELEASE-p1 (amd64)

      System VMware Virtual Machine
      Netgate Device ID: 0b04cb9c68032f0927c2
      BIOS Vendor: Phoenix Technologies LTD
      Version: 6.00
      Release Date: Tue Sep 30 2014
      Version 2.4.2-RELEASE-p1 (amd64)
      built on Tue Dec 12 13:45:26 CST 2017
      FreeBSD 11.1-RELEASE-p6

      The system is on the latest version.
      Version information updated at Mon Jan 22 8:40:32 WIB 2018 
      CPU Type Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz
      56 CPUs: 1 package(s) x 56 core(s)
      AES-NI CPU Crypto: Yes (inactive)

      RAM 32Gb

      having HTTPS MIM with splice all mode…
      squidguard activated

      below is my squid.conf

      This file is automatically generated by pfSense

      Do not edit manually !

      http_port x.x.x.x:3128 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=10MB cert=/usr/local/etc/squid/serverkey.pem capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!SHA1:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv2,NO_SSLv3,NO_TLSv1,SINGLE_DH_USE,SINGLE_ECDH_USE

      http_port 127.0.0.1:3128 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=10MB cert=/usr/local/etc/squid/serverkey.pem capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!SHA1:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv2,NO_SSLv3,NO_TLSv1,SINGLE_DH_USE,SINGLE_ECDH_USE

      https_port 127.0.0.1:3129 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=10MB cert=/usr/local/etc/squid/serverkey.pem capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!SHA1:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv2,NO_SSLv3,NO_TLSv1,SINGLE_DH_USE,SINGLE_ECDH_USE

      icp_port 0
      digest_generation off
      dns_v4_first on
      pid_filename /var/run/squid/squid.pid
      cache_effective_user squid
      cache_effective_group proxy
      error_default_language en
      icon_directory /usr/local/etc/squid/icons
      visible_hostname xxxxxxx
      cache_mgr xxxxxxxx
      access_log /var/squid/logs/access.log
      cache_log /var/squid/logs/cache.log
      cache_store_log none
      netdb_filename /var/squid/logs/netdb.state
      pinger_enable on
      pinger_program /usr/local/libexec/squid/pinger
      sslcrtd_program /usr/local/libexec/squid/ssl_crtd -s /var/squid/lib/ssl_db -M 4MB -b 2048
      sslcrtd_children 5
      sslproxy_capath /usr/local/share/certs/
      sslproxy_options NO_SSLv2,NO_SSLv3,NO_TLSv1,SINGLE_DH_USE,SINGLE_ECDH_USE
      sslproxy_cipher EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!SHA1:!MD5:!EXP:!PSK:!SRP:!DSS
      sslproxy_cert_error allow all
      sslproxy_cert_adapt setValidAfter all

      logfile_rotate 7
      debug_options rotate=7
      shutdown_lifetime 3 seconds

      Allow local network(s) on interface(s)

      acl localnet src  x.x.x.x/29
      forwarded_for delete
      via off
      httpd_suppress_version_string on
      uri_whitespace strip

      All Files

      ######################

      refresh_pattern -i (.|-)(exe|bin|[n|t]ar|acv|[r|j]ar|t?gz|[g|b]z[ip]?2?|7?z[ip]?|zip|wm[v|a]|patch|diff|mar|vpu|inc|r[a|p]m|kom|iso|sys|[ap]sf|ms[i|u|f]|dat|msi|cab|psf|dvr-ms|ace|asx|qt|xt|esd)[?.*]?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth

      #Apple Files
      refresh_pattern -i (.|-)(ap[k|p]|dmg|ip[a|sw]|pkg)(?.*)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth

      #Video Audio, Flash
      refresh_pattern -i (.|-)(webm|(x-)?swf|mp(eg)?(3|4)|mpe?g(av)?|(x-)?f(l|4)v|divx?|rmvb?|mov|trp|ts|avi|m38u|wmv|wmp|m4v|mkv|asf|dv|vob|3gp?2?)(?.)?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth
      refresh_pattern -i (.|-)(mp(3|4)|m4a|aa?c3?|wm?av?|og(x|v|a|g)|ape|mka|au|aiff|flac|m4(b|r)|m1v|m2(v|p)|mo(d|v)|arj|appx|lha|lzh|on2)(?.
      )?$ 43200 100% 432000 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth

      #images
      refresh_pattern -i (.|-)(ico(.)?|pn[pg]|css|(g|t)iff?|jpe?g(2|3|4)?|psd|c(d|b)r|cad|bmp|img)(?.)?$ 43200 100% 432000 override-lastmod reload-into-ims ignore-no-cache ignore-no-store ignore-private ignore-auth refresh-ims

      #Office Online
      refresh_pattern -i (.|-)(docx?|xlsx?|pptx?|rtf|xml|pdf|tiff?|txt)(?.*)?$ 43200 100% 432000 refresh-ims

      #Website
      refresh_pattern -i (.|-)(xml|js|jsp|txt|css)(?.*)?$ 360 40% 1440 refresh-ims
      refresh_pattern -i .index.(html|htm)$ 0 40% 1440

      cache_mem 15000 MB
      maximum_object_size_in_memory 1024000 KB
      memory_replacement_policy heap GDSF
      cache_replacement_policy heap LFUDA
      minimum_object_size 1000 KB
      maximum_object_size 100 MB
      cache_dir ufs /var/squid/cache 50000 16 256
      offline_mode off
      cache_swap_low 80
      cache_swap_high 90
      cache allow all

      Add any of your own refresh_pattern entries above these.

      refresh_pattern ^ftp:    1440  20%  10080
      refresh_pattern ^gopher:  1440  0%  1440
      refresh_pattern -i (/cgi-bin/|?) 0  0%  0
      refresh_pattern .    0  20%  4320

      #Remote proxies

      Setup some default acls

      ACLs all, manager, localhost, and to_localhost are predefined.

      acl allsrc src all
      acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 3129 1025-65535
      acl sslports port 443 563

      acl purge method PURGE
      acl connect method CONNECT

      Define protocols used for redirects

      acl HTTP proto HTTP
      acl HTTPS proto HTTPS

      SslBump Peek and Splice

      http://wiki.squid-cache.org/Features/SslPeekAndSplice

      http://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit

      Match against the current step during ssl_bump evaluation [fast]

      Never matches and should not be used outside the ssl_bump context.

      At each SslBump step, Squid evaluates ssl_bump directives to find

      the next bumping action (e.g., peek or splice). Valid SslBump step

      values and the corresponding ssl_bump evaluation moments are:

      #  SslBump1: After getting TCP-level and HTTP CONNECT info.
      #  SslBump2: After getting TLS Client Hello info.
      #  SslBump3: After getting TLS Server Hello info.

      These ACLs exist even when 'SSL/MITM Mode' is set to 'Custom' so that

      they can be used there for custom configuration.

      acl step1 at_step SslBump1
      acl step2 at_step SslBump2
      acl step3 at_step SslBump3
      acl allowed_subnets src x.x.x.x/8
      http_access allow manager localhost

      http_access deny manager
      http_access allow purge localhost
      http_access deny purge
      http_access deny !safeports
      http_access deny CONNECT !sslports

      Always allow localhost connections

      http_access allow localhost

      request_body_max_size 0 KB
      delay_pools 1
      delay_class 1 2
      delay_parameters 1 -1/-1 -1/-1
      delay_initial_bucket_level 100
      delay_access 1 allow allsrc

      Reverse Proxy settings

      Package Integration

      url_rewrite_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf
      url_rewrite_bypass off
      url_rewrite_children 16 startup=8 idle=4 concurrency=0

      Custom options before auth

      ssl_bump peek step1
      ssl_bump splice all

      Setup allowed ACLs

      Allow local network(s) on interface(s)

      http_access allow allowed_subnets
      http_access allow localnet

      Default block all to be sure

      http_access deny allsrc

      ==============================

      I'm having intermittent slowness on HTTP and HTTPS….
      sometime HTTPS couldn't load so I need to refresh it again...

      here is result from
      squidclient -h 127.0.0.1 -p 3128 mgr:info

      HTTP/1.1 200 OK
      Server: squid
      Mime-Version: 1.0
      Date: Mon, 22 Jan 2018 02:29:36 GMT
      Content-Type: text/plain;charset=utf-8
      Expires: Mon, 22 Jan 2018 02:29:36 GMT
      Last-Modified: Mon, 22 Jan 2018 02:29:36 GMT
      X-Cache: MISS from xxxxxxx
      X-Cache-Lookup: MISS from xxxxxxxx:3128
      Connection: close

      Squid Object Cache: Version 3.5.27
      Build Info:
      Service Name: squid
      Start Time: Fri, 19 Jan 2018 01:41:33 GMT
      Current Time: Mon, 22 Jan 2018 02:29:36 GMT
      Connection information for squid:
      Number of clients accessing cache: 8658
      Number of HTTP requests received: 6244979
      Number of ICP messages received: 0
      Number of ICP messages sent: 0
      Number of queued ICP replies: 0
      Number of HTCP messages received: 0
      Number of HTCP messages sent: 0
      Request failure ratio: 0.00
      Average HTTP requests per minute since start: 1429.7
      Average ICP messages per minute since start: 0.0
      Select loop called: 295699390 times, 0.886 ms avg
      Cache information for squid:
      Hits as % of all requests: 5min: 0.1%, 60min: 0.1%
      Hits as % of bytes sent: 5min: 31.2%, 60min: 29.6%
      Memory hits as % of hit requests: 5min: 94.1%, 60min: 83.2%
      Disk hits as % of hit requests: 5min: 0.0%, 60min: 3.8%
      Storage Swap size: 40931384 KB
      Storage Swap capacity: 79.9% used, 20.1% free
      Storage Mem size: 12797904 KB
      Storage Mem capacity: 83.3% used, 16.7% free
      Mean Object Size: 11648.09 KB
      Requests given to unlinkd: 1821
      Median Service Times (seconds)  5 min    60 min:
      HTTP Requests (All):  0.76407  0.64968
      Cache Misses:          0.25890  0.22004
      Cache Hits:          274.90301 28.47649
      Near Hits:            0.00000 221.51346
      Not-Modified Replies:  0.00000  0.08729
      DNS Lookups:          0.07284  0.06083
      ICP Queries:          0.00000  0.00000
      Resource usage for squid:
      UP Time: 262082.878 seconds
      CPU Time: 57314.937 seconds
      CPU Usage: 21.87%
      CPU Usage, 5 minute avg: 100.00%
      CPU Usage, 60 minute avg: 99.75%
      Maximum Resident Size: 96936368 KB
      Page faults with physical i/o: 46157
      Memory accounted for:
      Total accounted:      417220 KB
      memPoolAlloc calls: 767721337
      memPoolFree calls:  816550705
      File descriptor usage for squid:
      Maximum number of file descriptors:  942417
      Largest file desc currently in use:  6763
      Number of file desc currently in use: 5758
      Files queued for open:                  0
      Available number of file descriptors: 936659
      Reserved number of file descriptors:  100
      Store Disk files open:                  8
      Internal Data Structures:
        6225 StoreEntries
        4183 StoreEntries with MemObjects
        1519 Hot Object Cache Items
        3514 on-disk objects

      I have 56 core but I found only 1 CPU utilized by squid with 100% CPU persistently

      seems that squid only single threaded...

      but I read from https://doc.pfsense.org/index.php/Low_Throughput_Troubleshooting
      saying that pfsense 2.2 or later already use multiple cores...

      what I missed on my configuration....

      last pid: 41316;  load averages:  1.12,  1.20,  1.22                                                                                            up 8+12:07:37  09:56:43
      114 processes: 3 running, 111 sleeping
      CPU:  1.7% user,  0.1% nice,  0.4% system,  0.1% interrupt, 97.6% idle
      Mem: 10G Active, 7186M Inact, 11G Laundry, 2491M Wired, 1571M Buf, 513M Free
      Swap: 4096M Total, 251M Used, 3845M Free, 6% Inuse

      PID USERNAME    THR PRI NICE  SIZE    RES STATE  C  TIME    WCPU COMMAND
      13212 squid        1 103    0 23729M 22535M CPU46  46 982:21 100.59% squid
      11539 root          1  52    0  261M 22536K accept 23  0:00  1.30% php-fpm
      87784 root          1  52  20 13084K  2156K wait  35  1:55  0.26% sh
      8091 root          1  20    0 12700K  1888K bpf    28  7:47  0.20% filterlog
      1622 squid        1  20    0 34124K 14152K sbwait 22  0:17  0.18% squidGuard
      9237 squid        1  20    0 34124K 14148K sbwait  1  0:16  0.15% squidGuard
      11166 squid        1  20    0 34124K 14156K sbwait 17  0:10  0.15% squidGuard
      15785 squid        1  20    0 34124K 14152K sbwait 54  0:09  0.15% squidGuard
      44726 root          1  20    0 10484K  1984K select 51  3:37  0.10% syslogd
      21613 root          1  20    0 20060K  3704K CPU33  33  0:00  0.08% top
      16266 squid        1  20    0 34124K 14152K sbwait  4  0:07  0.07% squidGuard
      87330 root          1  20    0 37712K  7044K kqread  2  0:32  0.07% nginx
      22108 squid        1  20    0 34124K 14156K sbwait 12  0:06  0.05% squidGuard
      22534 squid        1  20    0 34124K 14152K sbwait 53  0:05  0.04% squidGuard
      61005 squid        1  20    0 33780K  3520K select 49  0:50  0.03% pinger
      26037 squid        1  20    0 33780K  3512K select 30  0:47  0.02% pinger
      36541 squid        1  20    0 33780K  3592K select 52  0:08  0.02% pinger
      32991 squid        1  20    0 33780K  3512K select 36  0:48  0.02% pinger
      8509 squid        1  20    0 33780K  3520K select 41  0:43  0.02% pinger
      25623 squid        1  20    0 33780K  3512K select 33  0:50  0.02% pinger
      66183 squid        1  20    0 33780K  2940K select 28  0:47  0.02% pinger
      29798 squid        1  20    0 34124K 14148K sbwait 55  0:04  0.02% squidGuard
      18928 squid        1  20    0 33780K  2940K select 31  0:45  0.02% pinger
      51648 squid        1  20    0 33780K  3512K select 19  0:47  0.02% pinger
      30062 squid        1  20    0 33780K  3852K select 20  0:03  0.02% pinger
      62063 squid        1  20    0 33780K  3512K select  2  0:49  0.02% pinger
      65590 squid        1  20    0 33780K  3512K select 22  0:49  0.02% pinger
      42315 squid        1  20    0 34124K 14148K sbwait 17  0:04  0.02% squidGuard
      80972 squid        1  20    0 33780K  3568K select  9  0:26  0.02% pinger
      20730 squid        1  20    0 33780K  3520K select  3  0:47  0.02% pinger
      75460 squid        1  20    0 33780K  2944K select  6  0:47  0.02% pinger
      66930 root          5  52    0 13032K  2060K uwait  7  1:54  0.02% dpinger
      89505 squid        1  20    0 33780K  2936K select  8  0:47  0.01% pinger
      63016 squid        1  20    0 33780K  2944K select 21  0:48  0.01% pinger
      28848 squid        1  20    0 33780K  2940K select  9  0:46  0.01% pinger
      66070 root          5  52    0 13032K  2012K uwait  20  1:54  0.01% dpinger
      66431 root          5  52    0 10984K  2016K uwait  20  1:55  0.01% dpinger
        336 root          1  20    0  9560K  488K select 55  0:30  0.01% devd
      88531 root          1  20    0 78844K  7128K select 38  0:00  0.01% sshd
      25055 root          1  20    0 24612K 12432K select 10  0:34  0.00% ntpd
      5913 root          1  20    0 43140K  5428K kqread 29  0:10  0.00% lighttpd_ls

      1 Reply Last reply Reply Quote 0
      • Gertjan
        Gertjan last edited by

        Hi,

        Why 56 cores ??

        I miss something : is this squid related or not ? I mean, when you disable squid, the problem is solved - no more problems ??

        I advise you to post and read here : pfSense Forum » pfSense English Support » Packages » Cache/Proxy

        Note : not related but strange :

        AES-NI CPU Crypto: Yes (inactive) 
        

        No "help me" PM's please. Use the forum.

        1 Reply Last reply Reply Quote 0
        • S
          sqrobin last edited by

          yes this is squid related as PFSense rely with squid to perform cache and filtering…

          yes if squid turn off it will fix the issue...

          now I turn off the HTTPS MITM... and CPU usage lower a bit... but still high.....

          for the AES-NI CPU, i think its because my hardware support it but by configuration is not selected using AES-NI as I'm not yet in the phase using VPN...

          1 Reply Last reply Reply Quote 0
          • S
            sqrobin last edited by

            Hi Gertjan

            as Why 56 Cores… 
            as simple that I have the resource and I thought that I can limit it or change it later from VM....

            1 Reply Last reply Reply Quote 0
            • C
              Chrismallia last edited by

              @sqrobin:

              yes this is squid related as PFSense rely with squid to perform cache and filtering…

              yes if squid turn off it will fix the issue...

              now I turn off the HTTPS MITM... and CPU usage lower a bit... but still high.....

              for the AES-NI CPU, i think its because my hardware support it but by configuration is not selected using AES-NI as I'm not yet in the phase using VPN...

              MITM = Huge can of worms +  many apps/devices are having hard certs so you can not use your own, if you want to filter I would try PfblockNG, As for caching with todays bigger pipes and dynamic content there is not much use for it

              1 Reply Last reply Reply Quote 0
              • S
                sqrobin last edited by

                HI Chris,

                Thanks for replying…

                Could you please let me know why I should go with PfBlockNG rather than SquidGuard...

                as I ready, PFBlockNG is used if I host mail server and this will prevent IP Block Country that is known as spammer to reach our server....

                if I compare to SquidGuard, its different of purpose..... even though you can put the filter on the outbound from your internal LAN....

                So, anyone can give me a clue as why I have almost 100% CPU utilisation persistently on 1 CPU rather then spread into multiple CPU?

                CPU usage information on the Dashboard is useless  as its represent to all CPU I have... since I have many.. then if 1 CPU is high the CPU Dashboard info doesn't tell me anything..

                1 Reply Last reply Reply Quote 0
                • C
                  Chrismallia last edited by

                  PfblockerNG can use dns lists to block content that is far more efficient and far less problematic then squid  and cos its using dns lists you do not need MITM, here is a great video

                  https://www.youtube.com/watch?v=QwFpMwXEK5w&t=1066s

                  1 Reply Last reply Reply Quote 0
                  • Gertjan
                    Gertjan last edited by

                    @sqrobin:

                    So, anyone can give me a clue as why I have almost 100% CPU utilisation persistently on 1 CPU rather then spread into multiple CPU?
                    CPU usage information on the Dashboard is useless  as its represent to all CPU I have… since I have many.. then if 1 CPU is high the CPU Dashboard info doesn't tell me anything..

                    Squid is a "pfSense package". The sub parts are the pfSEnse glue-ware to add the settings and the official squid FreeBSD package - or even one level higher : check out the manual -> (example) => https://wiki.squid-cache.org/MultipleInstances

                    No "help me" PM's please. Use the forum.

                    1 Reply Last reply Reply Quote 0
                    • A
                      agixdota last edited by

                      Hi Robin,

                      im using splice all, always taking much ram time to time.
                      do you have same issue with me?

                      SOLVED
                      im try all selected option on ssl proxy

                      for your problem Robin, try disable your refresh pattern.

                      1 Reply Last reply Reply Quote 0
                      • First post
                        Last post