• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Dismal squid performance

Scheduled Pinned Locked Moved 2.0-RC Snapshot Feedback and Problems - RETIRED
31 Posts 4 Posters 16.4k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • C
    clarknova
    last edited by Aug 15, 2010, 3:19 AM

    pfsense 2.0 beta4
    August 7 build
    AMD64

    Atom D510 dual-core
    4GB RAM
    dual Intel GBE (em)
    OCZ Vertex 2 240GB

    I used the squid package in 1.2.3 on a Soekris net5501 (500MHz, 512MB) with a 5400 rpm drive. Cached downloads would come in at 30-50 mbps, which is awesome when your internet connection is 6/1 mbps.

    Now I've upgraded to the hardware and software listed above, and my internet connection is 32/4. As a test file, I downloaded direct to ramdisk an Ubuntu iso from a local mirror using wget on a LAN host, and it came in at a steady 10-11 mbps.

    Then I logged into another LAN host and downloaded to ramdisk the same iso from the same mirror using wget. This time the download speed was a steady 3 mbps. I know it came out of squid's cache because pfsense reported no WAN traffic throughout the download.

    Here are some holdups that I've ruled out and my rationale:

    network: I've transferred files between vlans at up to 200 mbps and I have no doubt it can push a lot more than that. The LAN interface was virtually idle while I pulled the iso from cache.

    CPU: I ran top on pfsense during the download and the idle process hovered around 90%.

    disk: The vertex 2 is one of the fastest 2.5" SSDs available, whether for sequential or random data. The HDD activity light on the firewall flashed briefly every couple of seconds during the download. It looks far busier than that when booting pfsense.

    Things I think it could be:

    squid misconfiguration: I didn't change much from default. It's running in transparent mode and I changed the cache system to diskd. I set the memory cache size to 1.5GB and the hard disk cache size to 150GB. I set the level 1 subdirectories to 256. The only one of these options I haven't tried on other systems is diskd.

    kernel boot configuration: I added the following lines to /boot/loader.conf:

    kern.ipc.nmbclusters="32768"
    kern.maxfiles="65536"
    kern.maxfilesperproc="32768"
    net.inet.ip.portrange.last="65535"

    These were recommended in this forum and I had good performance with them in 1.2.3 on the net5501.

    software issues: Are there known issues with the squid package on the AMD64 pfsense platform? Does somebody have it running well?

    What could I look at to find what's causing the poor performance?

    Thanks.

    db

    1 Reply Last reply Reply Quote 0
    • J
      jimp Rebel Alliance Developer Netgate
      last edited by Aug 16, 2010, 1:29 PM

      Have you tried it on 32-bit?

      I'm not sure how well 64-bit is supposed to perform on the Atoms.

      I've tried squid on amd64 in a VM (Though it's on a core i5-750) and it was fine at the time.

      Remember: Upvote with the πŸ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

      Need help fast? Netgate Global Support!

      Do not Chat/PM for help!

      1 Reply Last reply Reply Quote 0
      • C
        clarknova
        last edited by Aug 16, 2010, 1:40 PM

        @jimp:

        Have you tried it on 32-bit?

        No, because I wanted to use all of my RAM. I may have to if I don't get some resolution though. Still, I would have expected to see something not working out if it wasn't well supported, like high CPU usage or so.

        Of course, now that I've done a fresh reinstall, squid won't even start, so maybe that's a sign.

        db

        1 Reply Last reply Reply Quote 0
        • J
          jimp Rebel Alliance Developer Netgate
          last edited by Aug 16, 2010, 1:51 PM

          Well it's hard to say what the exact cause might be in this case, I was just curious if the performance was the same or different on 32 vs 64. If they are both slow, it's probably more related to the network card or some other chipset issue.

          Remember: Upvote with the πŸ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

          Need help fast? Netgate Global Support!

          Do not Chat/PM for help!

          1 Reply Last reply Reply Quote 0
          • C
            clarknova
            last edited by Aug 16, 2010, 1:58 PM

            Right. I will give 32 bit a try.

            db

            1 Reply Last reply Reply Quote 0
            • J
              jimp Rebel Alliance Developer Netgate
              last edited by Aug 16, 2010, 3:38 PM

              Looks like amd64 packages are having a problem. Somehow some of them are being mixed up with their 32-bit counterparts, resulting in libs that won't load on amd64. I'm checking into it.

              Remember: Upvote with the πŸ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

              Need help fast? Netgate Global Support!

              Do not Chat/PM for help!

              1 Reply Last reply Reply Quote 0
              • J
                jimp Rebel Alliance Developer Netgate
                last edited by Aug 17, 2010, 2:05 AM

                32-bit and 64-bit squid should be OK now. Both were rebuilt today and I've tried them out and they are starting again.

                Remember: Upvote with the πŸ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                Need help fast? Netgate Global Support!

                Do not Chat/PM for help!

                1 Reply Last reply Reply Quote 0
                • C
                  clarknova
                  last edited by Aug 17, 2010, 4:32 PM

                  I reinstalled squid 64-bit in a vm today and performance is worse than before. For example, downloading an iso from http://mirror.csclub.uwaterloo.ca, without squid it comes in ~6mbps. Using squid (non-transparent mode) it's a steady 1.75+/-.02 mbps. Using squid on a 32-bit pfsense 1.2.3 vm, the same download comes out of cache ~200 mbps. Both VMs are on the same ESXi server using 2 Xeon 5150 CPU cores and Intel GBE. PF 2.0 has 6GB RAM and 1.2.3 has 3.0GB, so hardware should not be an issue. Squid settings are identical between the two.

                  db

                  1 Reply Last reply Reply Quote 0
                  • J
                    jimp Rebel Alliance Developer Netgate
                    last edited by Aug 18, 2010, 8:00 PM

                    What is the output of "ifconfig -a" when running on 64-bit?

                    If it says anything about TSO and/or LRO, go into the advanced options and make sure the options to disable those are checked, and you may as well disable checksums.

                    Remember: Upvote with the πŸ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                    Need help fast? Netgate Global Support!

                    Do not Chat/PM for help!

                    1 Reply Last reply Reply Quote 0
                    • C
                      clarknova
                      last edited by Aug 18, 2010, 8:06 PM Aug 18, 2010, 8:03 PM

                      $ ifconfig -a
                      em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
                      	options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:0c:29:59:fa:4e
                      	inet6 fe80::20c:29ff:fe59:fa4e%em0 prefixlen 64 scopeid 0x1 
                      	inet 172.21.252.1 netmask 0xfffffe00 broadcast 172.21.253.255
                      	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
                      	status: active
                      em1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
                      	options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:0c:29:59:fa:58
                      	inet 172.21.33.58 netmask 0xfffffe00 broadcast 172.21.33.255
                      	inet6 fe80::20c:29ff:fe59:fa58%em1 prefixlen 64 scopeid 0x2 
                      	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
                      	status: active
                      plip0: flags=8810 <pointopoint,simplex,multicast>metric 0 mtu 1500
                      lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
                      	options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000 
                      	inet6 ::1 prefixlen 128 
                      	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 
                      	nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=0<> metric 0 mtu 1460
                      	syncpeer: 224.0.0.240 maxupd: 128
                      pflog0: flags=100 <promisc>metric 0 mtu 33152
                      enc0: flags=0<> metric 0 mtu 1536</promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></pointopoint,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast>
                      

                      TSO and LRO were already disabled. I disabled checksum offloading and DL from cache is still 1.77mbps. This is on a VM.

                      db

                      1 Reply Last reply Reply Quote 0
                      • J
                        jimp Rebel Alliance Developer Netgate
                        last edited by Aug 18, 2010, 8:48 PM

                        Looks normal then in that regard.

                        Is it VMware?
                        If so, can you confirm if the normal VMware settings are still there?

                        /etc/sysctl.conf should have:

                        kern.timecounter.hardware=i8254
                        

                        /boot/loader.conf should have:

                        kern.hz="100"
                        

                        Remember: Upvote with the πŸ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                        Need help fast? Netgate Global Support!

                        Do not Chat/PM for help!

                        1 Reply Last reply Reply Quote 0
                        • C
                          clarknova
                          last edited by Aug 18, 2010, 8:51 PM

                          I'll try those. By default my sysctl.conf has no uncommented lines, and loader.conf is empty.

                          db

                          1 Reply Last reply Reply Quote 0
                          • C
                            clarknova
                            last edited by Aug 18, 2010, 8:57 PM

                            I copied /boot/loader.conf from another 2.0 machine, added those two lines from your post to their respective files, and rebooted. DL from cache is now 2.3 +/- .2 mbps. This is on ESXi, pfsense installed from pfsense.iso

                            db

                            1 Reply Last reply Reply Quote 0
                            • C
                              clarknova
                              last edited by Aug 18, 2010, 9:03 PM

                              Are you getting good performance from yours now? If so I will try reinstalling it on my home pfsense. I just set up the vm because of the crashing that happened during the 32/64 fiasco. I want to minimize testing on my home box but if there's a chance that my issues are specific to this vm then I'll drop it like a rock.

                              db

                              1 Reply Last reply Reply Quote 0
                              • J
                                jimp Rebel Alliance Developer Netgate
                                last edited by Aug 18, 2010, 9:14 PM

                                I don't have a VM setup "behind" it to test right now. Mine is also in VirtualBox so it's a bit different setup.

                                Remember: Upvote with the πŸ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                Need help fast? Netgate Global Support!

                                Do not Chat/PM for help!

                                1 Reply Last reply Reply Quote 0
                                • C
                                  clarknova
                                  last edited by Aug 20, 2010, 8:39 PM

                                  I installed squid on a fresh Beta4 i386 system this morning and….same problem. iso comes out of cache at 3.3 mbps. About halfway through the download it jumps to 10 mbps. No detectable signs of strain from any part of the system, just slow.

                                  db

                                  1 Reply Last reply Reply Quote 0
                                  • J
                                    jimp Rebel Alliance Developer Netgate
                                    last edited by Aug 21, 2010, 12:59 PM

                                    I tried it out on my amd64 box today, I moved an XP VM behind the amd64 pfSense VM with squid, and it came through at the full wire speed of my cable, 10Mbps.

                                    Remember: Upvote with the πŸ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                    Need help fast? Netgate Global Support!

                                    Do not Chat/PM for help!

                                    1 Reply Last reply Reply Quote 0
                                    • C
                                      clarknova
                                      last edited by Aug 21, 2010, 6:43 PM

                                      Uncached objects always download at the expected rate, it's the cached objects that are slow without apparent cause. Try deleting the download from your xp machine and then download it again. Assuming your squid cache was configured to cache objects of that size, you will see what I mean.

                                      The expected behaviour is that cached objects should download at a speed that is bounded only by CPU, LAN network, or disk speed. Try this exercise in 1.2.3 and see the difference.

                                      db

                                      1 Reply Last reply Reply Quote 0
                                      • J
                                        jimp Rebel Alliance Developer Netgate
                                        last edited by Aug 21, 2010, 7:41 PM

                                        Ah, well I noticed in my haste of configuring squid I left its cache size as 100mb, not very effective if I want to store ISOs while testing…

                                        So I bumped it up to 3GB, and tried again, and the download came to me at 17MByte/s (byte, not bit). I deleted the downloaded file and cleared firefox's cache between attempts. Squid's access log shows a TCP_HIT for the request so I know it was coming from the cache.

                                        I'll try it a few more times with different download locations to see if I get any differences.

                                        This is on an amd64 VM on a core i5-750, with the only tweak being the single loader.conf line tweaking nmbclusters to 32768.

                                        Remember: Upvote with the πŸ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                        Need help fast? Netgate Global Support!

                                        Do not Chat/PM for help!

                                        1 Reply Last reply Reply Quote 0
                                        • J
                                          jimp Rebel Alliance Developer Netgate
                                          last edited by Aug 21, 2010, 8:50 PM

                                          Tried a few more iso downloads and the lowest one I got was 10MByte/s. (Though it went back and forth between 10-11) and the fastest was that first one, about 17MB/s.

                                          I did notice that at least one place, Knoppix, looked like it was an http link on a mirror but actually redirected to an ftp link so it bypassed the cache entirely. (I only figured that out when I right clicked on the download in FF and copied the link to be sure I got the same thing twice).

                                          Remember: Upvote with the πŸ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                          Need help fast? Netgate Global Support!

                                          Do not Chat/PM for help!

                                          1 Reply Last reply Reply Quote 0
                                          8 out of 31
                                          • First post
                                            8/31
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                            This community forum collects and processes your personal information.
                                            consent.not_received