Trying to follow the Squid Package Tuning



  • Hi Everyone
    I'm working through the Squid Package Tuning doc here:

    http://doc.pfsense.org/index.php/Squid_Package_Tuning

    In particular, the Tips/Tricks section and caching of Windows and Apple updates.  It seems that you cannot only make the adjustments listed in the article, which are:

    Hard Disk Cache size
    Maximum object size

    Attempting to set these attributes must also trigger requirements in these:

    The field 'Hard disk cache location' is required.
    The field 'Memory cache size' is required.
    The field 'Minimum object size' is required.
    The field 'Maximum object size in RAM' is required.
    You must enter a valid value for 'memory_cache_size'
    You must enter a valid value for 'Minimum object size'

    My specific question is whether or not there are any recommendations/follow-up articles on the values for these options, being that the article is for performance/optimization tuning.  If I go ahead and choose "random" values (or at least less-informed values) for these, would they/could they invalidate any of the other tuning/performance values that the doc suggests?

    Has anyone else run across this?

    Thanks.



  • The questions would be:
    What do you want to realize with squid?
    Caching web content like windows updates to save bandwidth?
    Or just for filtering URLs and block them afterwards with dansguardian or squidguard?

    You say you want to "tune" squid - in general squid runs with the default settings. What do you want to improve ?

    So you can see it would be better if you tell us what you want to realize, tell us your system specs (RAM, HDD, which other packages/services are running on pfsense) and we can try to find the "best" values for you :-)



  • Hi - Thanks for your reply.  Let me explain -

    My end goal is to have squid run as optimally and efficiently as it can on my Pfsense hardware.  As most of us sysadmins know, lots of things run "fine" with default settings.  I also know that there are definitely benefits to tuning/adjusting these things too.  That's why I checked out the docs and decided to implement what makes sense for my configuration, which brings me to just that -

    I run squidguard for filtering and want to cache Windows and Apple updates.  I have over 30 devices/nodes connected to the network at any one time.  I like to tinker quite a bit, so there are lots of re-installs, re-patches, etc.  and all the devices obviously are bandwidth hungry for basic functionality and patches/updates.  Pfsense is running on a single P3 1GHz with 512MB of RAM with a 40GB MAXTOR 6L040J2 with a quad-port Intel Pro NIC.

    I've made the ipc.nmbclusters and vfs.read_max changes and have noticed a real, tangible performance boost from those in regards to throughput.

    Hope this helps.

    Thanks again.



  • Hi,

    Ok I see what you want to do and that it is working in general. Your amount of RAM would not allow a big RAM cache and not so a big HDD cache. My squid process is using ~6GB of RAM. I gave set it to 3GB RAM cache and 40GB HDD.

    I think this site will be the best place to calculate which values you should maximum set on HDD and RAM:
    http://wiki.squid-cache.org/SquidFaq/SquidMemory

    10 MB of memory per 1 GB on disk for 32-bit Squid

    10GB HDD = 100MB RAM
    32MB RAM cache
    256MB for pfsense itself and other processes - like squidguard

    So you will have around ~100MB RAM left. You can adjust HDD or RAM cache - I would probably increase HDD cache a little bit because you say that there are much reinstallations of computers, updates and service packs needed.

    So that would be my suggestion:

    • Hard disk cache size 15GB

    • Hard disk cache system aufs or ufs (increase vfs.read like you did already) Test which performs better

    • Memory cache size 32MB

    • Minimum object size 16KB (I set it to 64 KB because I do not want that so many small files will be cached. This increases HDD read/write and my WAN bandwidth allows that to download such small files directly from the web and not from squid)

    • Maximum object size 1GB ( ServicePacks do have sizeses until 1GB. If you do not want to cache all kind of service packs than probably a size of 256MB will be enough to get most of the available updates in cache)

    • Maximum object size in RAM 2MB (In RAM I would like to have only small files which could be served very fast. I do not want to have a 100MB file in RAM which wastes much space and could be served fast enough even from HDD)

    • Memory replacement policy LRU (RAM should cache actual objects)

    • Cache replacement policy Heap LFUDA (what to cache servicepacks and big files on HDD)

    I am using this custom options on squid2 to cache windows, apple and ubuntu updates for 180 days:

    
    refresh_pattern -i .*apple\.com/.*\.(exe|zip|rar|cab|tar|bz|bz2|msi|mst|msu|mzz|gz|pkg|dmg) 259200 100% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;
    
    refresh_pattern -i .*microsoft\.com/.*\.(exe|zip|rar|cab|tar|bz|bz2|msi|mst|msu|mzz|gz|msp) 259200 100% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;
    
    refresh_pattern -i .*windowsupdate\.com/.*\.(exe|zip|rar|cab|tar|bz|bz2|msi|mst|msu|mzz|gz|msp) 259200 100% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;
    
    refresh_pattern -i .*ubuntu\.com/.*\.(tar|bz|bz2|gpg|gz|zip|deb) 259200 100% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;
    
    

    Hope this will help you. Good luck! :)



  • This is great stuff, thank you very much.


Locked