Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Squid slowness issues - also a bounty post

    Scheduled Pinned Locked Moved pfSense Packages
    48 Posts 17 Posters 28.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • H Offline
      heiko
      last edited by

      I can duplicate this behavoiur, did you use the transparent squid settings? For me it was a problem with the squid and the caching mode, after i set squid to use the transparent mode, all was OK. It is not a DNS Problem.

      1 Reply Last reply Reply Quote 0
      • B Offline
        ben
        last edited by

        @heiko:

        I can duplicate this behavoiur, did you use the transparent squid settings? For me it was a problem with the squid and the caching mode, after i set squid to use the transparent mode, all was OK. It is not a DNS Problem.

        Yes, Squid was using transparent mode (transparent proxy checkbox was on).

        1 Reply Last reply Reply Quote 0
        • H Offline
          heiko
          last edited by

          did you also checked the (RFC 1918) Checkbox?

          1 Reply Last reply Reply Quote 0
          • B Offline
            ben
            last edited by

            @heiko:

            did you also checked the (RFC 1918) Checkbox?

            No. I didn't think that was needed for my network.

            1 Reply Last reply Reply Quote 0
            • H Offline
              heiko
              last edited by

              please check this box and test again

              1 Reply Last reply Reply Quote 0
              • M Offline
                mhab12
                last edited by

                @sullrich:

                The problem is not the squid package itself.
                We turned off write caching on hard drives which is surely causing your slowdowns:

                sysctl hw.ata.wc

                hw.ata.wc: 0
                Edit /etc/sysctl.conf and find this value and change to:
                hw.ata.wc=1

                I've played with this setting and have not seen any change in throughput, however mine is a SCSI box so perhaps the command is different.  Do you know off hand how to re-enable write caching for SCSI disks?

                I've also noticed that proxy throughput is fine for a few seconds just after reboot.  After reading through the system log, it appears that squid may be starting twice.  Please find below an excerpt from my cache.log

                2007/12/31 10:36:35| Starting Squid Cache version 2.6.STABLE5 for i386-portbld-freebsd6.1...
2007/12/31 10:36:35| Process ID 948
2007/12/31 10:36:35| With 7232 file descriptors available
2007/12/31 10:36:35| Using kqueue for the IO loop
2007/12/31 10:36:35| DNS Socket created at 0.0.0.0, port 9650, FD 5
2007/12/31 10:36:35| Adding nameserver 65.106.1.196 from /etc/resolv.conf
2007/12/31 10:36:35| Adding nameserver 65.106.7.196 from /etc/resolv.conf
2007/12/31 10:36:35| Unlinkd pipe opened on FD 10
2007/12/31 10:36:35| Swap maxSize 102400 KB, estimated 7876 objects
2007/12/31 10:36:35| Target number of buckets: 393
2007/12/31 10:36:35| Using 8192 Store buckets
2007/12/31 10:36:35| Max Mem  size: 8192 KB
2007/12/31 10:36:35| Max Swap size: 102400 KB
2007/12/31 10:36:35| Store logging disabled
2007/12/31 10:36:35| Rebuilding storage in /var/squid/cache (CLEAN)
2007/12/31 10:36:35| Using Least Load store dir selection
2007/12/31 10:36:35| Current Directory is /tmp
2007/12/31 10:36:35| Loaded Icons.
2007/12/31 10:36:35| Accepting proxy HTTP connections at 10.21.1.24, port 3128, FD 12.
2007/12/31 10:36:35| Accepting proxy HTTP connections at 192.168.1.1, port 3128, FD 13.
2007/12/31 10:36:35| Accepting transparently proxied HTTP connections at 127.0.0.1, port 80, FD 14.
2007/12/31 10:36:35| WCCP Disabled.
2007/12/31 10:36:35| Ready to serve requests.
2007/12/31 10:36:35| Done reading /var/squid/cache swaplog (162 entries)
2007/12/31 10:36:35| Finished rebuilding storage from disk.
2007/12/31 10:36:35|       162 Entries scanned
2007/12/31 10:36:35|         0 Invalid entries.
2007/12/31 10:36:35|         0 With invalid flags.
2007/12/31 10:36:35|       162 Objects loaded.
2007/12/31 10:36:35|         0 Objects expired.
2007/12/31 10:36:35|         0 Objects cancelled.
2007/12/31 10:36:35|         0 Duplicate URLs purged.
2007/12/31 10:36:35|         0 Swapfile clashes avoided.
2007/12/31 10:36:35|   Took 0.5 seconds ( 317.8 objects/sec).
2007/12/31 10:36:35| Beginning Validation Procedure
2007/12/31 10:36:35|   Completed Validation Procedure
2007/12/31 10:36:35|   Validated 162 Entries
2007/12/31 10:36:35|   store_swap_size = 568k
2007/12/31 10:36:36| storeLateRelease: released 0 objects
2007/12/31 10:36:40| Starting Squid Cache version 2.6.STABLE5 for i386-portbld-freebsd6.1...
2007/12/31 10:36:40| Process ID 1000
2007/12/31 10:36:40| With 7232 file descriptors available
2007/12/31 10:36:40| Using kqueue for the IO loop
2007/12/31 10:36:40| DNS Socket created at 0.0.0.0, port 3353, FD 5
2007/12/31 10:36:40| Adding nameserver 65.106.1.196 from /etc/resolv.conf
2007/12/31 10:36:40| Adding nameserver 65.106.7.196 from /etc/resolv.conf
2007/12/31 10:36:40| Unlinkd pipe opened on FD 10
2007/12/31 10:36:40| Swap maxSize 102400 KB, estimated 7876 objects
2007/12/31 10:36:40| Target number of buckets: 393
2007/12/31 10:36:40| Using 8192 Store buckets
2007/12/31 10:36:40| Max Mem  size: 8192 KB
2007/12/31 10:36:40| Max Swap size: 102400 KB
2007/12/31 10:36:40| Store logging disabled
2007/12/31 10:36:40| Rebuilding storage in /var/squid/cache (DIRTY)
2007/12/31 10:36:40| Using Least Load store dir selection
2007/12/31 10:36:40| Current Directory is /tmp
2007/12/31 10:36:40| Loaded Icons.
2007/12/31 10:36:40| Accepting proxy HTTP connections at 10.21.1.24, port 3128, FD 12.
2007/12/31 10:36:40| Accepting proxy HTTP connections at 192.168.1.1, port 3128, FD 13.
2007/12/31 10:36:40| Accepting transparently proxied HTTP connections at 127.0.0.1, port 80, FD 14.
2007/12/31 10:36:40| WCCP Disabled.
2007/12/31 10:36:40| Ready to serve requests.
2007/12/31 10:36:40| Done reading /var/squid/cache swaplog (162 entries)
2007/12/31 10:36:40| Finished rebuilding storage from disk.
2007/12/31 10:36:40|       162 Entries scanned
2007/12/31 10:36:40|         0 Invalid entries.
2007/12/31 10:36:40|         0 With invalid flags.
2007/12/31 10:36:40|       162 Objects loaded.
2007/12/31 10:36:40|         0 Objects expired.
2007/12/31 10:36:40|         0 Objects cancelled.
2007/12/31 10:36:40|         0 Duplicate URLs purged.
2007/12/31 10:36:40|         0 Swapfile clashes avoided.
2007/12/31 10:36:40|   Took 0.3 seconds ( 554.1 objects/sec).
2007/12/31 10:36:40| Beginning Validation Procedure
2007/12/31 10:36:40|   Completed Validation Procedure
2007/12/31 10:36:40|   Validated 162 Entries
2007/12/31 10:36:40|   store_swap_size = 568k
2007/12/31 10:36:41| storeLateRelease: released 0 objects

                I appreciate all the help thus far…

                1 Reply Last reply Reply Quote 0
                • M Offline
                  mhab12
                  last edited by

                  I am able to get 50+ mb/s when copying files to and from the pfSense box via WinSCP.  Perhaps this isn't a write caching issue after all.  I'm still ready to pay a bounty for a solution to this problem as it is impacting our business.

                  1 Reply Last reply Reply Quote 0
                  • M Offline
                    mhab12
                    last edited by

                    I just downloaded and installed 1.2RC3 release, not the latest snapshot and speeds were fine after installing squid via the GUI.  The issue must be in the pfsense code somewhere…

                    1 Reply Last reply Reply Quote 0
                    • S Offline
                      stevewm
                      last edited by

                      I am having this problem as well.

                      I have a total of 6 pFsense boxen running 1.2 rc4 with the latest available Squid version via Packages.  All 6 boxen are experiencing this issue.

                      When going through the proxy pages load about 30% to 40% slower.  I can test at 2800kbps on a speed test site without going through the proxy and only  2200kbps going through it.  I can reproduce this every time.

                      I am using local authentication in normal mode, NOT transparent mode.

                      I didn't try using Squid until RC4 was out, so cannot confirm if its a problem on an older version.

                      1 Reply Last reply Reply Quote 0
                      • A Offline
                        acidrop
                        last edited by

                        Hello!

                        Are there any news with this issue?

                        1 Reply Last reply Reply Quote 0
                        • H Offline
                          heiko
                          last edited by

                          Yes, please make a squid package update

                          1 Reply Last reply Reply Quote 0
                          • S Offline
                            sullrich
                            last edited by

                            @acidrop:

                            Hello!

                            Are there any news with this issue?

                            Try the newest package.

                            1 Reply Last reply Reply Quote 0
                            • K Offline
                              Kilian
                              last edited by

                              Hi, same problem here, 4 Mb downstream syncronous.

                              pfsense 1.2-RC3, and really serious problems with downstream using proxy, traffic sharper disabled.

                              squid version 2.6.18

                              can anyone help? is there a solution? i'm working with pfsense since a year ago and really proud of it, but no clue how to solve this issue :/

                              kind regards

                              1 Reply Last reply Reply Quote 0
                              • M Offline
                                mhab12
                                last edited by

                                Please add to our bounty to fix the problem!

                                http://forum.pfsense.org/index.php/topic,7911.0.html

                                1 Reply Last reply Reply Quote 0
                                • Cry HavokC Offline
                                  Cry Havok
                                  last edited by

                                  With 1.2 release and the latest Squid package (and traffic shaper enabled) I'm seeing download speeds much the same as before the upgrades - between 16 and 17 Mb/s on my 20 Mb/s line.

                                  1 Reply Last reply Reply Quote 0
                                  • P Offline
                                    paulino
                                    last edited by

                                    Hi!

                                    I am having the same squid slow transfer problems on my institution.
                                    I have a dual-wan system: one 25 MBps cable connection on WAN for internet access and one institutional adsl 1MBps on WAN OPT3 (also has internet access using another gateway).
                                    The OPT3 is connected to a 1GBps switch and has a public IP (we have 32 public ips for mail, webpage, etc… and a cisco router).
                                    If I download a huge file from our servers (connected on the WAN OPT3 interface) I can easily get 11000 MBytes/sec from our internal lan's, passing through squid.
                                    Nevertheless, accessing the internet (through the cable connection on WAN) I can get maximum 50Kbytes/sec using squid and 2000KBytes/sec using only NAT and proxy off.
                                    Why using the same squid traffic is slow in one interface and normal on another?

                                    Strangely if I discard the cable connection, deactivate WAN OPT3, and connect the 1MBps connection on WAN interface, I can get the full speed of it, downloading at about 200Kbytes/sec from the internet.
                                    Can it be the different modems? On the cable I have a regular Motorola modem, and on the institucional I have a much higher quality adsl modem.
                                    Can it be the NIC? When using cable we connect it to pfsense using a 100MBps 3Com card (xl driver) and the 1Mbps is on one VLAN that enters in pfsense throug a Intel PRO1000 (em driver) together with our internal networks...

                                    Can anyone shed some light on this problem? I am liking a lot pfsense, but I cannot have squid turned on because of such slow transfers!

                                    Yours,
                                    Antonio Paulino

                                    1 Reply Last reply Reply Quote 0
                                    • Cry HavokC Offline
                                      Cry Havok
                                      last edited by

                                      Your choice of NICs will pretty certainly be at the core - Intel NICs are much better than the 3Com ones.  I suspect if you switch it for another Intel you'll find a big improvement.

                                      1 Reply Last reply Reply Quote 0
                                      • P Offline
                                        paulino
                                        last edited by

                                        Hi!
                                        Thank you for your sugestion.
                                        I exchanged the 3Com NIC for a Intel Pro100 (fxp driver) but the problem stays…

                                        Since it is the first time I use pfsense, does anyone knows when the problem first appeared? I don't need many fancy features of pfsense, so probably a old version will just suit me. I just need a simple multi wan routing/NAT/proxy box.

                                        António Paulino

                                        1 Reply Last reply Reply Quote 0
                                        • M Offline
                                          mhab12
                                          last edited by

                                          The problem appeared sometime after 1.2rc3.

                                          There was a link in another post to older versions.  I just checked it and it has been updated and now only includes 1.2 final.  If anyone knows where some old 1.2 RCs are located, that would be great.  I have a live cd iso for 1.2rc2, but I would hope that the community will respond to the bounty to fix the problem before we're stuck at rc3.

                                          Have you tried using only two interfaces (only LAN/WAN, disable/remove all opt interfaces) and see if your problem still exists?

                                          1 Reply Last reply Reply Quote 0
                                          • P Offline
                                            paulino
                                            last edited by

                                            Hi!

                                            I found the old version (including 1.0.1) in one of the mirrors, named loquefaltaba.
                                            There is one directory "old" which old versions, full iso and upgrade packages.

                                            I haven't done it, but I will try, perhaps today, and then let you know.

                                            António Paulino

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.