Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    In need of help to solve a bandwidth issue

    Scheduled Pinned Locked Moved General pfSense Questions
    61 Posts 11 Posters 20.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      Nachtfalke
      last edited by

      @luke240778:

      Thanks for that, great news.. Do you at all see any issues when your cache decides to update itself and use up all the bandwidth? That is what mine was doing..

      Yes, in the past I had the problem that my cache started downloading files but no client has an open download/connection. So it was just squid which was downloading. I had sometimes a high difference that WAN has downloaded 8GB and LAN only 6GB.

      But as I said in my post before I changed the custom options in squid and now it is working as it should: caching files but not downloading "useless" files.

      1 Reply Last reply Reply Quote 0
      • L
        luke240778
        last edited by

        Well, i have inserted that code and will see how things go for a couple days.. thanks again!!  it will indeed save alot of bandwidth.

        1 Reply Last reply Reply Quote 0
        • N
          Nachtfalke
          last edited by

          You could add this for apple updates:

          refresh_pattern -i .*apple\.com/.*\.(pkg|dmg) 129600 100% 129600 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;
          

          and this for ubuntu (an perhaps other linux distris) updates:

          refresh_pattern -i .*ubuntu\.com/.*\.(tar|bz|bz2|gpg|gz|zip|deb) 129600 100% 129600 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;
          

          and perhaps this will save you bandwidth when you want to save some iso files and multimedia:

          # refresh_pattern -i /.*\.(iso|wmv|mov|rm|avi|mp4|mpeg|mpg|divx|xvid|swf|flv|x-flv) 2880 100% 2880 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;
          

          But please be carful with the caching time.
          For caching OS updates I am using 90 days.
          For caching ISO files and so on just 2 days.

          Perhaps you could play with these settings/times.

          With these settings you could avoid that squid is caching wrong DNS entries longer than 5 seconds:

          negative_ttl 5 second;
          negative_dns_ttl 5 second;
          

          Good luck an dpost back how it is working :o)

          1 Reply Last reply Reply Quote 0
          • N
            Nachtfalke
            last edited by

            I did some (successful) changes for caching Windows 7 updates but in a way that squid isn't downloading files without any use.

            range_offset_limit 20 MB;
            

            This means:
            I f you have a download with 100MB and the download is starting at 35MB till 100MB (what windows 7 seems to do) than this file will NOT be cache by squid with my option above because the range which is not downloades (less than 35MB) is bigger than the size in the range_offset_limit.

            If the download of the 100MB file starts at 15MB till 100MB, that the range_offset_limit takes effect and squid is downloading the rest of the file before 15MB.

            I hope I could explain it with my small english skills.

            That this all will work it is neccessary that your "maximum object size" is high enough to cache the files you are downloading

            1 Reply Last reply Reply Quote 0
            • L
              luke240778
              last edited by

              I havent seen my bandwidth issue return since adding this code which is a good thing.. but i do see since that my Disk usage has gone from around 8-10% up to around 70%..  With squid running for over 5 months with all my clients it was only using around 10%..  Nos wince adding this code its way up… What can i do to troubleshoot this?

              1 Reply Last reply Reply Quote 0
              • N
                Nachtfalke
                last edited by

                Hi,

                of course you disk cache is increasing - that's a good sign because this means that squid is caching many files. And you want to cache many files because you would like to save i-net bandwidth.
                In "Cache Management" you find "Low- and High watermark" if more than 80% of the squid cache is used than squid starts to throw out "old" files. but this depends on your replacement algorithms.

                For memory replacement I am using: Heap GDSF
                For cache replacement I am using: Heap LFUDA

                1 Reply Last reply Reply Quote 0
                • L
                  luke240778
                  last edited by

                  Ill take a look at this. Out of interest, is there someway that i can see what files are in my cache? For example, not just websites, but would love to see what large files are in there, like music and movies and updates and stuff.. out of interest to see what my clients interests are to maybe try some other bandwidth saving techniques

                  1 Reply Last reply Reply Quote 0
                  • N
                    Nachtfalke
                    last edited by

                    Hi,

                    I am not sure where to find these information. But you can use cachemgr.cgi to analyze squid more in detail.

                    And I found this:
                    http://wiki.squid-cache.org/SquidFaq/SquidLogs#store.log

                    But you have to enable this in

                    /usr/local/pkg/squid.inc
                    

                    store.log is disabled by default. To enable this I think you just have to modify the line 624:
                    from

                    cache_store_log none
                    

                    to

                    /var/log/store.log
                    

                    I am NOT sure if the syntax is correct but it should work. After changing the squid.inc you have to click "Save" in "Services -> Proxy Server -> General" so that the squid.conf will be generated with the new options. But then please check syslog if there are any errors.

                    1 Reply Last reply Reply Quote 0
                    • L
                      luke240778
                      last edited by

                      Thanks for that, i have cachmgr.cgi setup, but where in there shows the files in cache and what they are?

                      Ill play with that other part now also

                      1 Reply Last reply Reply Quote 0
                      • N
                        Nachtfalke
                        last edited by

                        Hi,

                        I am sorry but I didn't play with this kind of information. Cachemgr.cgi was just to give you some information about how squid performs.

                        "All Cached Objects" shows you all objects in squid but they are "encrypted". Here are some information about these lines:
                        14.2.1.26 objects: All Cache Objects
                        http://tilt.lib.tsinghua.edu.cn/docs/books/squid/TheDefinitiveGuideChinese/0596001622/squid-CHP-14-SECT-2.html

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.