Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Squid slowness issues - also a bounty post

    Scheduled Pinned Locked Moved pfSense Packages
    48 Posts 17 Posters 28.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B
      ben
      last edited by

      @heiko:

      did you also checked the (RFC 1918) Checkbox?

      No. I didn't think that was needed for my network.

      1 Reply Last reply Reply Quote 0
      • H
        heiko
        last edited by

        please check this box and test again

        1 Reply Last reply Reply Quote 0
        • M
          mhab12
          last edited by

          @sullrich:

          The problem is not the squid package itself.
          We turned off write caching on hard drives which is surely causing your slowdowns:

          sysctl hw.ata.wc

          hw.ata.wc: 0
          Edit /etc/sysctl.conf and find this value and change to:
          hw.ata.wc=1

          I've played with this setting and have not seen any change in throughput, however mine is a SCSI box so perhaps the command is different.  Do you know off hand how to re-enable write caching for SCSI disks?

          I've also noticed that proxy throughput is fine for a few seconds just after reboot.  After reading through the system log, it appears that squid may be starting twice.  Please find below an excerpt from my cache.log

          2007/12/31 10:36:35| Starting Squid Cache version 2.6.STABLE5 for i386-portbld-freebsd6.1...
2007/12/31 10:36:35| Process ID 948
2007/12/31 10:36:35| With 7232 file descriptors available
2007/12/31 10:36:35| Using kqueue for the IO loop
2007/12/31 10:36:35| DNS Socket created at 0.0.0.0, port 9650, FD 5
2007/12/31 10:36:35| Adding nameserver 65.106.1.196 from /etc/resolv.conf
2007/12/31 10:36:35| Adding nameserver 65.106.7.196 from /etc/resolv.conf
2007/12/31 10:36:35| Unlinkd pipe opened on FD 10
2007/12/31 10:36:35| Swap maxSize 102400 KB, estimated 7876 objects
2007/12/31 10:36:35| Target number of buckets: 393
2007/12/31 10:36:35| Using 8192 Store buckets
2007/12/31 10:36:35| Max Mem  size: 8192 KB
2007/12/31 10:36:35| Max Swap size: 102400 KB
2007/12/31 10:36:35| Store logging disabled
2007/12/31 10:36:35| Rebuilding storage in /var/squid/cache (CLEAN)
2007/12/31 10:36:35| Using Least Load store dir selection
2007/12/31 10:36:35| Current Directory is /tmp
2007/12/31 10:36:35| Loaded Icons.
2007/12/31 10:36:35| Accepting proxy HTTP connections at 10.21.1.24, port 3128, FD 12.
2007/12/31 10:36:35| Accepting proxy HTTP connections at 192.168.1.1, port 3128, FD 13.
2007/12/31 10:36:35| Accepting transparently proxied HTTP connections at 127.0.0.1, port 80, FD 14.
2007/12/31 10:36:35| WCCP Disabled.
2007/12/31 10:36:35| Ready to serve requests.
2007/12/31 10:36:35| Done reading /var/squid/cache swaplog (162 entries)
2007/12/31 10:36:35| Finished rebuilding storage from disk.
2007/12/31 10:36:35|       162 Entries scanned
2007/12/31 10:36:35|         0 Invalid entries.
2007/12/31 10:36:35|         0 With invalid flags.
2007/12/31 10:36:35|       162 Objects loaded.
2007/12/31 10:36:35|         0 Objects expired.
2007/12/31 10:36:35|         0 Objects cancelled.
2007/12/31 10:36:35|         0 Duplicate URLs purged.
2007/12/31 10:36:35|         0 Swapfile clashes avoided.
2007/12/31 10:36:35|   Took 0.5 seconds ( 317.8 objects/sec).
2007/12/31 10:36:35| Beginning Validation Procedure
2007/12/31 10:36:35|   Completed Validation Procedure
2007/12/31 10:36:35|   Validated 162 Entries
2007/12/31 10:36:35|   store_swap_size = 568k
2007/12/31 10:36:36| storeLateRelease: released 0 objects
2007/12/31 10:36:40| Starting Squid Cache version 2.6.STABLE5 for i386-portbld-freebsd6.1...
2007/12/31 10:36:40| Process ID 1000
2007/12/31 10:36:40| With 7232 file descriptors available
2007/12/31 10:36:40| Using kqueue for the IO loop
2007/12/31 10:36:40| DNS Socket created at 0.0.0.0, port 3353, FD 5
2007/12/31 10:36:40| Adding nameserver 65.106.1.196 from /etc/resolv.conf
2007/12/31 10:36:40| Adding nameserver 65.106.7.196 from /etc/resolv.conf
2007/12/31 10:36:40| Unlinkd pipe opened on FD 10
2007/12/31 10:36:40| Swap maxSize 102400 KB, estimated 7876 objects
2007/12/31 10:36:40| Target number of buckets: 393
2007/12/31 10:36:40| Using 8192 Store buckets
2007/12/31 10:36:40| Max Mem  size: 8192 KB
2007/12/31 10:36:40| Max Swap size: 102400 KB
2007/12/31 10:36:40| Store logging disabled
2007/12/31 10:36:40| Rebuilding storage in /var/squid/cache (DIRTY)
2007/12/31 10:36:40| Using Least Load store dir selection
2007/12/31 10:36:40| Current Directory is /tmp
2007/12/31 10:36:40| Loaded Icons.
2007/12/31 10:36:40| Accepting proxy HTTP connections at 10.21.1.24, port 3128, FD 12.
2007/12/31 10:36:40| Accepting proxy HTTP connections at 192.168.1.1, port 3128, FD 13.
2007/12/31 10:36:40| Accepting transparently proxied HTTP connections at 127.0.0.1, port 80, FD 14.
2007/12/31 10:36:40| WCCP Disabled.
2007/12/31 10:36:40| Ready to serve requests.
2007/12/31 10:36:40| Done reading /var/squid/cache swaplog (162 entries)
2007/12/31 10:36:40| Finished rebuilding storage from disk.
2007/12/31 10:36:40|       162 Entries scanned
2007/12/31 10:36:40|         0 Invalid entries.
2007/12/31 10:36:40|         0 With invalid flags.
2007/12/31 10:36:40|       162 Objects loaded.
2007/12/31 10:36:40|         0 Objects expired.
2007/12/31 10:36:40|         0 Objects cancelled.
2007/12/31 10:36:40|         0 Duplicate URLs purged.
2007/12/31 10:36:40|         0 Swapfile clashes avoided.
2007/12/31 10:36:40|   Took 0.3 seconds ( 554.1 objects/sec).
2007/12/31 10:36:40| Beginning Validation Procedure
2007/12/31 10:36:40|   Completed Validation Procedure
2007/12/31 10:36:40|   Validated 162 Entries
2007/12/31 10:36:40|   store_swap_size = 568k
2007/12/31 10:36:41| storeLateRelease: released 0 objects

          I appreciate all the help thus far…

          1 Reply Last reply Reply Quote 0
          • M
            mhab12
            last edited by

            I am able to get 50+ mb/s when copying files to and from the pfSense box via WinSCP.  Perhaps this isn't a write caching issue after all.  I'm still ready to pay a bounty for a solution to this problem as it is impacting our business.

            1 Reply Last reply Reply Quote 0
            • M
              mhab12
              last edited by

              I just downloaded and installed 1.2RC3 release, not the latest snapshot and speeds were fine after installing squid via the GUI.  The issue must be in the pfsense code somewhere…

              1 Reply Last reply Reply Quote 0
              • S
                stevewm
                last edited by

                I am having this problem as well.

                I have a total of 6 pFsense boxen running 1.2 rc4 with the latest available Squid version via Packages.  All 6 boxen are experiencing this issue.

                When going through the proxy pages load about 30% to 40% slower.  I can test at 2800kbps on a speed test site without going through the proxy and only  2200kbps going through it.  I can reproduce this every time.

                I am using local authentication in normal mode, NOT transparent mode.

                I didn't try using Squid until RC4 was out, so cannot confirm if its a problem on an older version.

                1 Reply Last reply Reply Quote 0
                • A
                  acidrop
                  last edited by

                  Hello!

                  Are there any news with this issue?

                  1 Reply Last reply Reply Quote 0
                  • H
                    heiko
                    last edited by

                    Yes, please make a squid package update

                    1 Reply Last reply Reply Quote 0
                    • S
                      sullrich
                      last edited by

                      @acidrop:

                      Hello!

                      Are there any news with this issue?

                      Try the newest package.

                      1 Reply Last reply Reply Quote 0
                      • K
                        Kilian
                        last edited by

                        Hi, same problem here, 4 Mb downstream syncronous.

                        pfsense 1.2-RC3, and really serious problems with downstream using proxy, traffic sharper disabled.

                        squid version 2.6.18

                        can anyone help? is there a solution? i'm working with pfsense since a year ago and really proud of it, but no clue how to solve this issue :/

                        kind regards

                        1 Reply Last reply Reply Quote 0
                        • M
                          mhab12
                          last edited by

                          Please add to our bounty to fix the problem!

                          http://forum.pfsense.org/index.php/topic,7911.0.html

                          1 Reply Last reply Reply Quote 0
                          • Cry HavokC
                            Cry Havok
                            last edited by

                            With 1.2 release and the latest Squid package (and traffic shaper enabled) I'm seeing download speeds much the same as before the upgrades - between 16 and 17 Mb/s on my 20 Mb/s line.

                            1 Reply Last reply Reply Quote 0
                            • P
                              paulino
                              last edited by

                              Hi!

                              I am having the same squid slow transfer problems on my institution.
                              I have a dual-wan system: one 25 MBps cable connection on WAN for internet access and one institutional adsl 1MBps on WAN OPT3 (also has internet access using another gateway).
                              The OPT3 is connected to a 1GBps switch and has a public IP (we have 32 public ips for mail, webpage, etc… and a cisco router).
                              If I download a huge file from our servers (connected on the WAN OPT3 interface) I can easily get 11000 MBytes/sec from our internal lan's, passing through squid.
                              Nevertheless, accessing the internet (through the cable connection on WAN) I can get maximum 50Kbytes/sec using squid and 2000KBytes/sec using only NAT and proxy off.
                              Why using the same squid traffic is slow in one interface and normal on another?

                              Strangely if I discard the cable connection, deactivate WAN OPT3, and connect the 1MBps connection on WAN interface, I can get the full speed of it, downloading at about 200Kbytes/sec from the internet.
                              Can it be the different modems? On the cable I have a regular Motorola modem, and on the institucional I have a much higher quality adsl modem.
                              Can it be the NIC? When using cable we connect it to pfsense using a 100MBps 3Com card (xl driver) and the 1Mbps is on one VLAN that enters in pfsense throug a Intel PRO1000 (em driver) together with our internal networks...

                              Can anyone shed some light on this problem? I am liking a lot pfsense, but I cannot have squid turned on because of such slow transfers!

                              Yours,
                              Antonio Paulino

                              1 Reply Last reply Reply Quote 0
                              • Cry HavokC
                                Cry Havok
                                last edited by

                                Your choice of NICs will pretty certainly be at the core - Intel NICs are much better than the 3Com ones.  I suspect if you switch it for another Intel you'll find a big improvement.

                                1 Reply Last reply Reply Quote 0
                                • P
                                  paulino
                                  last edited by

                                  Hi!
                                  Thank you for your sugestion.
                                  I exchanged the 3Com NIC for a Intel Pro100 (fxp driver) but the problem stays…

                                  Since it is the first time I use pfsense, does anyone knows when the problem first appeared? I don't need many fancy features of pfsense, so probably a old version will just suit me. I just need a simple multi wan routing/NAT/proxy box.

                                  António Paulino

                                  1 Reply Last reply Reply Quote 0
                                  • M
                                    mhab12
                                    last edited by

                                    The problem appeared sometime after 1.2rc3.

                                    There was a link in another post to older versions.  I just checked it and it has been updated and now only includes 1.2 final.  If anyone knows where some old 1.2 RCs are located, that would be great.  I have a live cd iso for 1.2rc2, but I would hope that the community will respond to the bounty to fix the problem before we're stuck at rc3.

                                    Have you tried using only two interfaces (only LAN/WAN, disable/remove all opt interfaces) and see if your problem still exists?

                                    1 Reply Last reply Reply Quote 0
                                    • P
                                      paulino
                                      last edited by

                                      Hi!

                                      I found the old version (including 1.0.1) in one of the mirrors, named loquefaltaba.
                                      There is one directory "old" which old versions, full iso and upgrade packages.

                                      I haven't done it, but I will try, perhaps today, and then let you know.

                                      António Paulino

                                      1 Reply Last reply Reply Quote 0
                                      • E
                                        eri--
                                        last edited by

                                        Please can you guys provide the output of kldstat command or by any chance those that have issues are using spamd package?

                                        1 Reply Last reply Reply Quote 0
                                        • M
                                          mhab12
                                          last edited by

                                          From my working (fast) 1.2rc3 box:

                                          Id Refs Address    Size     Name
 1    3 0xc0400000 6df4ac   kernel
 2    1 0xc0ae0000 59e80    acpi.ko

                                          From my working (very slow) 1.2-Release box:

                                          Id Refs Address    Size     Name
 1    4 0xc0400000 7fb834   kernel
 2    1 0xc0bfc000 59e80    acpi.ko
 3    1 0xc6f56000 d000     ipfw.ko

                                          What do these outputs mean?

                                          1 Reply Last reply Reply Quote 0
                                          • S
                                            sullrich
                                            last edited by

                                            ipfw.ko is ipfw which is enabled for schedules and or captive portal.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.