Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Lag on Http/Https Sites every now and then?

    Scheduled Pinned Locked Moved Cache/Proxy
    13 Posts 4 Posters 1.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • K
      killmasta93
      last edited by

      Hi,
      I was wondering if someone could assist or guide me on whats going on? Currently Running OpenVPN, squid3, squidguard, Ntopng,Sarg, lightsquid, and WPAD.The internet service has 28mbps down and up 15mbps with ping 31ms
      Many users been complaining for the past weeks about slow lag internet I really did not notice until I tried navigating a few sites and all of sudden it would not load the website. I want to first check if there is something wrong with the settings of Squid before seeing if theres an issue with the cables or the switches. All the users are forcefully navigating though WPAD because I blocked them on ports 443-80 Without transparent proxy. See pictures Below

      I ran a test running```
      squidclient -h 192.168.1.254 -p 3128 mgr:info

      and I get the median service time seems good. (maybe i might be wrong)
Thank you

      Squid Object Cache: Version 3.4.10
      Build Info:
      Start Time:    Wed, 10 Aug 2016 23:13:45 GMT
      Current Time:  Fri, 02 Sep 2016 21:26:44 GMT
      Connection information for squid:
              Number of clients accessing cache:      61
              Number of HTTP requests received:      6476964
              Number of ICP messages received:        0
              Number of ICP messages sent:    0
              Number of queued ICP replies:  0
              Number of HTCP messages received:      0
              Number of HTCP messages sent:  0
              Request failure ratio:  0.00
              Average HTTP requests per minute since start:  196.2
              Average ICP messages per minute since start:    0.0
              Select loop called: 202241612 times, 9.794 ms avg
      Cache information for squid:
              Hits as % of all requests:      5min: 0.0%, 60min: 1.1%
              Hits as % of bytes sent:        5min: 0.1%, 60min: 1.3%
              Memory hits as % of hit requests:      5min: 0.0%, 60min: 15.6%
              Disk hits as % of hit requests: 5min: 0.0%, 60min: 11.2%
              Storage Swap size:      2474120 KB
              Storage Swap capacity:  4.8% used, 95.2% free
              Storage Mem size:      1000 KB
              Storage Mem capacity:  97.7% used,  2.3% free
              Mean Object Size:      39.29 KB
              Requests given to unlinkd:      7903
      Median Service Times (seconds)  5 min    60 min:
              HTTP Requests (All):  10.77596  0.05046
              Cache Misses:        10.77596  0.05046
              Cache Hits:            0.00000  0.00000
              Near Hits:            0.00000  0.07825
              Not-Modified Replies:  0.00000  0.00000
              DNS Lookups:          0.05815  0.05815
              ICP Queries:          0.00000  0.00000
      Resource usage for squid:
              UP Time:        1980778.633 seconds
              CPU Time:      3366.378 seconds
              CPU Usage:      0.17%
              CPU Usage, 5 minute avg:        0.09%
              CPU Usage, 60 minute avg:      0.23%
              Maximum Resident Size: 326896 KB
              Page faults with physical i/o: 18
      Memory accounted for:
              Total accounted:        16482 KB
              memPoolAlloc calls: 1457121328
              memPoolFree calls:  1464779995
      File descriptor usage for squid:
              Maximum number of file descriptors:  107559
              Largest file desc currently in use:    172
              Number of file desc currently in use:  132
              Files queued for open:                  0
              Available number of file descriptors: 107427
              Reserved number of file descriptors:  100
              Store Disk files open:                  0
      Internal Data Structures:
              63083 StoreEntries
                226 StoreEntries with MemObjects
                162 Hot Object Cache Items
              62967 on-disk objects

      
and the WPAD config

      function FindProxyForURL(url, host)
      {
          if (isPlainHostName(host) ||
              shExpMatch(host, "*.local") ||
              isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
              return "DIRECT";

      if (isInNet(dnsResolve(host), "192.168.10.0",  "255.255.255.0" ))
              { return "DIRECT"; }
              if (isInNet(dnsResolve(host), "192.168.115.0",  "255.255.255.0" ))
              { return "DIRECT"; }
              if (isInNet(dnsResolve(host), "10.10.1.0",  "255.255.252.0" ))
              { return "DIRECT"; }
              if (isInNet(dnsResolve(host), "10.10.10.1",  "255.255.255.255" ))
              { return "DIRECT"; }
      if (isInNet(dnsResolve(host), "192.168.1.206",  "255.255.255.255" ))
              { return "DIRECT"; }
              if (isInNet(dnsResolve(host), "200.1.XX.XXX",  "255.255.255.255" ))
              { return "DIRECT"; }
              if (isInNet(dnsResolve(host), "181.143.XXX.XX",  "255.255.255.255" ))
              { return "DIRECT"; }
      if (dnsDomainIs(host, "dane.gov.co") || dnsDomainIs(host, "www.dane.gov.co"))
              return "DIRECT";;
      if (dnsDomainIs(host, "cafesalud.com.co") || dnsDomainIs(host, "www.cafesalud.com.co"))
              return "DIRECT";;
              if (dnsDomainIs(host, "heon.com.co") || dnsDomainIs(host, "www.heon.com.co"))
              return "DIRECT";;
      if (dnsDomainIs(host, "saviasaludeps.com") || dnsDomainIs(host, "www.saviasaludeps.com"))
              return "DIRECT";;
              if (dnsDomainIs(host, "saviasaludeps.com") || dnsDomainIs(host, "www.saviasaludeps.com"))
              return "DIRECT";;
              if (dnsDomainIs(host, "outlook.office365.com") || dnsDomainIs(host, "www.outlook.office365.com"))
              return "DIRECT";;

      return "PROXY 192.168.1.254:3128";
      }

      ![Clipboarder.2016.09.02.png_thumb](/public/_imported_attachments_/1/Clipboarder.2016.09.02.png_thumb)
![Clipboarder.2016.09.02-002.png](/public/_imported_attachments_/1/Clipboarder.2016.09.02-002.png)
![Clipboarder.2016.09.02-002.png_thumb](/public/_imported_attachments_/1/Clipboarder.2016.09.02-002.png_thumb)
![Clipboarder.2016.09.02-003.png](/public/_imported_attachments_/1/Clipboarder.2016.09.02-003.png)
![Clipboarder.2016.09.02-003.png_thumb](/public/_imported_attachments_/1/Clipboarder.2016.09.02-003.png_thumb)
![Issue proxy.jpg](/public/_imported_attachments_/1/Issue proxy.jpg)
![Issue proxy.jpg_thumb](/public/_imported_attachments_/1/Issue proxy.jpg_thumb)

      Tutorials:

      https://www.mediafire.com/folder/v329emaz1e9ih/Tutorials

      1 Reply Last reply Reply Quote 0
      • A
        asterix
        last edited by

        Try this..

        Clear disk cache

        Hard disk cache system - aufs
        Level 1 dir - 64
        Memory cache size - 1024 MB (I would go for 1.5Gb)

        1 Reply Last reply Reply Quote 0
        • K
          killmasta93
          last edited by

          Thank you for the reply, I will immediately adjust it on your specification, I guess right now is wait until a user complains about it or is there a way I could check it?

          Thank you

          Tutorials:

          https://www.mediafire.com/folder/v329emaz1e9ih/Tutorials

          1 Reply Last reply Reply Quote 0
          • A
            asterix
            last edited by

            It shouldn't do it again. The ufs cache system most likely was causing the lag along with just 1MB mem cache size as that was way too low.

            1 Reply Last reply Reply Quote 0
            • K
              killmasta93
              last edited by

              Hi,
              Thank you for the reply, so unfortunately some users still complain about the lag not sure if its the proxy or the switches itself?

              This is a packet capture

              https://mega.nz/#!A9JhgCYS!i3OTZpKbvSpGN4ZWR617IkqO45bSDGKGXWRUFbAZW5k

              Thank you

              Tutorials:

              https://www.mediafire.com/folder/v329emaz1e9ih/Tutorials

              1 Reply Last reply Reply Quote 0
              • M
                Moscu
                last edited by

                Been having the same issue since yesterday with our web server wich is a shared hosting server.

                I've been getting connection timeout errors and lag when using the squid. I tested at the same time with another browser without proxy and the website loads inmediately.

                I've tried disabling caches, deleting cache, disabling squidguard. Also using WPAD. No other server seems affected by the lag.

                Using virtual pfsense 2.3 on xenserver 6.2, squid 3.5.19 with squidguard 1.4.15.

                1 Reply Last reply Reply Quote 0
                • KOMK
                  KOM
                  last edited by

                  I think I remember in the past that this issue could be fixed in some cases by checking the Resolve IPv4 DNS First checkbox in Squid's General Settings.

                  1 Reply Last reply Reply Quote 0
                  • M
                    Moscu
                    last edited by

                    @KOM:

                    I think I remember in the past that this issue could be fixed in some cases by checking the Resolve IPv4 DNS First checkbox in Squid's General Settings.

                    Yes, this was enabled since i installed the proxy.

                    1 Reply Last reply Reply Quote 0
                    • A
                      asterix
                      last edited by

                      I see you are on an earlier version of PF and most likely Squid as well. If you can sustain some down time or have a temporary backup, I would recommend you start out with a fresh clean install of the latest version of PF and Squid. Backup the config but do not restore the old config file. Instead, use the backed up config file for reference while manually setting up the new install.

                      Not sure why the SWAP space is being used since you have ample amount of free RAM. Also change MBUF Usage by setting kern.ipc.nmbclusters to 1000000 even though the current usage is not close to the max.

                      1 Reply Last reply Reply Quote 0
                      • K
                        killmasta93
                        last edited by

                        Thank you for the reply, As I have working on this for a while i thought to do some digging a packet capture on pfSense and then analyze on wireshark, I saw many TCP spurious Retransmission's, TCP Keep alive, TCP previous segment not capture and at one point shows tcp Zero window. Im guessing that the network card of the pfSense box cannot handled this much of traffic. I will keep checking what could be the issue. I will change the MBUF to see if it can handle worst case scenario buy a network card with more capacity?

                        Thank you

                        new packet capture–---  https://mega.nz/#!RsAh1CKA!gxYChuCWavhXIe-C9oBD50SLuq7XGscR4tm0MRJhb_I

                        Tutorials:

                        https://www.mediafire.com/folder/v329emaz1e9ih/Tutorials

                        1 Reply Last reply Reply Quote 0
                        • K
                          killmasta93
                          last edited by

                          Update: So i realized that unchecking auto detect proxy seems to speed everything up but then comes the dilemma of the WPAD….:(

                          Tutorials:

                          https://www.mediafire.com/folder/v329emaz1e9ih/Tutorials

                          1 Reply Last reply Reply Quote 0
                          • M
                            Moscu
                            last edited by

                            As last option i just bypassed the proxy when connecting with my web server (which i'm having issues with) using the wpad configuration with isInNet(dnsResolve(host), "IP", "255.255.255.255").

                            So the issue is gone but i hope it doesn't happen with other sites.

                            1 Reply Last reply Reply Quote 0
                            • K
                              killmasta93
                              last edited by

                              Thats true the WPAD ignores that IP goes though for a moment I thought it was WPAD now that I have been doing a deep analyze its not it. Not even sure what to do now…

                              Tutorials:

                              https://www.mediafire.com/folder/v329emaz1e9ih/Tutorials

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post
                              Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.