Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Improve Custom refresh pattern

    Scheduled Pinned Locked Moved Cache/Proxy
    111 Posts 27 Posters 47.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • KOMK
      KOM
      last edited by

      Bottom line: caching Windows updates does indeed work

      I still have yet to see any evidence to back this claim.  A 19% hit rate for WU is crap.  And no, there aren't a zillion little 1K files skewing the ratio.

      Here is a proper test:

      take two Windows 7 clients that you installed fresh and put them both behind the proxy, leaving only 1 powered on.
      Install all updates until there are none left
      Power on the second unit and then do all the same updates
      Once complete, on pfSense run:

      squidclient -h LAN_IP -p 3128 mgr:info
      

      and post the output.  That will give a lot more detail about the recent session.

      1 Reply Last reply Reply Quote 0
      • A
        aGeekhere
        last edited by

        updated, thanks to Valeriy

        if there are any mistakes in the refresh pattern let me know

        Never Fear, A Geek is Here!

        1 Reply Last reply Reply Quote 0
        • V
          Valeriy
          last edited by

          KOM was right, in fact it was not working properly. Also, the results of bare counting of MISSes/HITs from not fully populated cache was not good measurement. Right now, any windowsupdate downloaded file (even partial) is fetched from cache, verified manually and monitored in access.log.

          Few days ago I have made some improvements to utilize store_rewrite module.
          They were tested, as suggested above by KOM, successfully.

          Indeed, some of WU were not cached properly since they were received from different CDN hosts.

          Config files are below (squid.conf is only part that is related to this issue)

          Apple and Symantec updates are cached well, too.

          Steam/Xbox (cs.steampowered.com / llnwd.net respective CDN) are not confirmed yet.

          The rest of refresh patterns and URL rewrites are still in experimental mode. Needs a lot of testing and research in order to make it done properly.

          Attached is screenshot from Lightsquid, you can clearly see changes after 3rd of March.

          P.S. Questions:

          • not quite sure if I need to list all possible domains under acl dstdomain directive, or just first level domain will be good enough?

          Comments:

          • I have total 1.3TB of swap available and 32GB on server running Pfsense+Squid+SquidGuard+Snort+PfBlocker. Configuration is optimized for ~2000 hosts on LAN.

          Current byte/hit rate is 20-45%

          /usr/local/etc/squid/squid.conf

          
          .......
          # No range offset limit for windowsupdate sites
          acl Windows_Update dstdomain windowsupdate.microsoft.com
          acl Windows_Update dstdomain update.microsoft.com
          acl Windows_Update dstdomain download.windowsupdate.com
          acl Windows_Update dstdomain windowsupdate.com
          acl Windows_Update dstdomain au.download.windowsupdate.com
          acl Windows_Update dstdomain bg.v4.pr.dl.ws.microsoft.com
          acl Windows_Update dstdomain wsus.ds.download.windowsupdate.com
          acl Windows_Update dstdomain au.b1.download.windowsupdate.com
          range_offset_limit -1  Windows_Update
          
          # Storeid_rewrite configuration
          
          acl updatesites dstdom_regex "/usr/local/etc/squid/updatesites.txt"
          
          store_id_access allow updatesites
          store_id_access deny all
          store_id_program /usr/local/libexec/squid/storeid_file_rewrite /usr/local/etc/squid/storeid_rewrite.conf
          store_id_children 200 startup=60 idle=1 concurrency=0
          
          # Refresh patterns
          # 525600 min is one year
          
          # Gaming CDN
          
          refresh_pattern -i \.llnwd.net 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
          
          refresh_pattern -i \.cs.steampowered.com 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
          
          #windows update
          
          refresh_pattern -i \.update.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
          
          refresh_pattern -i \.windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
          
          refresh_pattern -i \.download.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
          
          refresh_pattern -i \.ws.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
          
          # Symantec Updates
          # Only reload-into-ims works properly, other flags break update process 
          
          refresh_pattern -i \.symantecliveupdate.com/.*\.(zip|exe|z7)          525600 100% 525600 reload-into-ims
          
          

          /usr/local/etc/squid/updatesites.txt

          
          \.adobe.com
          \.java.com
          \.sun.com
          \.oracle.com
          \.apple.com
          \.microsoft.com
          \.windowsupdate.com
          \.ubuntu.com
          \.steampowered.com
          \.llnwd.net
          \.symantecliveupdate.com
          
          

          /usr/local/etc/squid/storeid_rewrite.conf

          
          ^http:\/\/.+?\.microsoft\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx)     http://wupdate.squid.local/$1
          ^http:\/\/.+?\.windowsupdate\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx) http://wupdate.squid.local/$1
          ^http:\/\/.+?\.cs\.steampowered\.com\/(.*)      http://steamupdates.squid.internal/$1
          ^http:\/\/.+?\.apple\.com\/(.*) http://appupdates.apple.squid.internal/$1
          ^http:\/\/.+?\.llnwd\.net\/(.*)                 http://llnwd.net.squid.internal/$1
          
          

          Regards,

          Valeriy

          1 Reply Last reply Reply Quote 0
          • V
            Valeriy
            last edited by

            Just a note: manually modifying squid.conf is not good idea, since it can and will be rewritten by PfSense later. With next post I ll try to summarize all the changes and how to incorporate them using GUI of pfSense.

            1 Reply Last reply Reply Quote 0
            • S
              SaschaITM
              last edited by

              This is interesting, I wish I had the time to test this thoroughly. Will do that as soon as I find the time.

              FWIW, I had massive problems with the "range_offset_limit -1" directive and Windows 10 updates, with "psf" added to the "refresh_pattern". Update download was extremely slow and used up all available bandwidth after a while. It looked like Win10 was aborting and restarting the update download every few seconds, while Squid was still downloading from previous requests. Commented out the "range_offset_limit -1", and updates were downloading "normally" again. The "-1" value isn't documented in the current Squid docs, but it's documented for Squid 2.7. I wonder if "range_offset_limit -1" is still valid/needed with recent Squid versions.

              1 Reply Last reply Reply Quote 0
              • S
                saluto
                last edited by

                It worked very well.

                1 Reply Last reply Reply Quote 0
                • S
                  saluto
                  last edited by

                  No caches of video, can you help?

                  1 Reply Last reply Reply Quote 0
                  • V
                    Valeriy
                    last edited by

                    Forget about video caching…
                    Most video services push content via HTTPS, which we cannt cache.

                    If you really need to have some video supplied to your users, then just download it and make is available via PLEX service, for instance or just regular network file system (SMB, NFS, etc.)

                    1 Reply Last reply Reply Quote 0
                    • P
                      pfsensation
                      last edited by

                      @Valeriy:

                      Forget about video caching…
                      Most video services push content via HTTPS, which we cannt cache.

                      If you really need to have some video supplied to your users, then just download it and make is available via PLEX service, for instance or just regular network file system (SMB, NFS, etc.)

                      Unless…You do MITM inspection.

                      1 Reply Last reply Reply Quote 0
                      • S
                        saluto
                        last edited by

                        @pfsensation:

                        @Valeriy:

                        Forget about video caching…
                        Most video services push content via HTTPS, which we cannt cache.

                        If you really need to have some video supplied to your users, then just download it and make is available via PLEX service, for instance or just regular network file system (SMB, NFS, etc.)

                        Unless…You do MITM inspection.

                        I do not know the MITM inspection.

                        1 Reply Last reply Reply Quote 0
                        • P
                          pfsensation
                          last edited by

                          @saluto:

                          @pfsensation:

                          @Valeriy:

                          Forget about video caching…
                          Most video services push content via HTTPS, which we cannt cache.

                          If you really need to have some video supplied to your users, then just download it and make is available via PLEX service, for instance or just regular network file system (SMB, NFS, etc.)

                          Unless…You do MITM inspection.

                          I do not know the MITM inspection.

                          I'm currently having issues with it after the latest Squid update, unsure as to why. However, you can do MITM inspection by enabling SSL filtering and deploying out a CA certificate to your clients.

                          1 Reply Last reply Reply Quote 0
                          • S
                            saluto
                            last edited by

                            Any update not refresh_pattern?

                            1 Reply Last reply Reply Quote 0
                            • V
                              Valeriy
                              last edited by

                              Caching video is bad idea, thats all I can say :)

                              1 Reply Last reply Reply Quote 0
                              • K
                                kikawala
                                last edited by

                                I have this for Apple iOS updates and iOS apps.

                                refresh_pattern -i appldnld\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate
                                
                                refresh_pattern -i phobos\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate
                                
                                refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate
                                
                                1 Reply Last reply Reply Quote 0
                                • A
                                  aGeekhere
                                  last edited by

                                  Thanks updated

                                  Never Fear, A Geek is Here!

                                  1 Reply Last reply Reply Quote 0
                                  • W
                                    waldopulanco
                                    last edited by

                                    how about youtube?

                                    1 Reply Last reply Reply Quote 0
                                    • P
                                      pfsensation
                                      last edited by

                                      @kikawala:

                                      I have this for Apple iOS updates and iOS apps.

                                      refresh_pattern -i appldnld\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate
                                      
                                      refresh_pattern -i phobos\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate
                                      
                                      refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate
                                      

                                      Doesn't apple block MITM using certificate pinning? Since their app store is over HTTPS.

                                      1 Reply Last reply Reply Quote 0
                                      • P
                                        PFsense Learner
                                        last edited by

                                        @Valeriy:

                                        KOM was right, in fact it was not working properly. Also, the results of bare counting of MISSes/HITs from not fully populated cache was not good measurement. Right now, any windowsupdate downloaded file (even partial) is fetched from cache, verified manually and monitored in access.log.

                                        Few days ago I have made some improvements to utilize store_rewrite module.
                                        They were tested, as suggested above by KOM, successfully.

                                        Indeed, some of WU were not cached properly since they were received from different CDN hosts.

                                        Config files are below (squid.conf is only part that is related to this issue)

                                        Apple and Symantec updates are cached well, too.

                                        Steam/Xbox (cs.steampowered.com / llnwd.net respective CDN) are not confirmed yet.

                                        The rest of refresh patterns and URL rewrites are still in experimental mode. Needs a lot of testing and research in order to make it done properly.

                                        Attached is screenshot from Lightsquid, you can clearly see changes after 3rd of March.

                                        P.S. Questions:

                                        • not quite sure if I need to list all possible domains under acl dstdomain directive, or just first level domain will be good enough?

                                        Comments:

                                        • I have total 1.3TB of swap available and 32GB on server running Pfsense+Squid+SquidGuard+Snort+PfBlocker. Configuration is optimized for ~2000 hosts on LAN.

                                        Current byte/hit rate is 20-45%

                                        /usr/local/etc/squid/squid.conf

                                        
                                        .......
                                        # No range offset limit for windowsupdate sites
                                        acl Windows_Update dstdomain windowsupdate.microsoft.com
                                        acl Windows_Update dstdomain update.microsoft.com
                                        acl Windows_Update dstdomain download.windowsupdate.com
                                        acl Windows_Update dstdomain windowsupdate.com
                                        acl Windows_Update dstdomain au.download.windowsupdate.com
                                        acl Windows_Update dstdomain bg.v4.pr.dl.ws.microsoft.com
                                        acl Windows_Update dstdomain wsus.ds.download.windowsupdate.com
                                        acl Windows_Update dstdomain au.b1.download.windowsupdate.com
                                        range_offset_limit -1  Windows_Update
                                        
                                        # Storeid_rewrite configuration
                                        
                                        acl updatesites dstdom_regex "/usr/local/etc/squid/updatesites.txt"
                                        
                                        store_id_access allow updatesites
                                        store_id_access deny all
                                        store_id_program /usr/local/libexec/squid/storeid_file_rewrite /usr/local/etc/squid/storeid_rewrite.conf
                                        store_id_children 200 startup=60 idle=1 concurrency=0
                                        
                                        # Refresh patterns
                                        # 525600 min is one year
                                        
                                        # Gaming CDN
                                        
                                        refresh_pattern -i \.llnwd.net 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
                                        
                                        refresh_pattern -i \.cs.steampowered.com 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
                                        
                                        #windows update
                                        
                                        refresh_pattern -i \.update.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
                                        
                                        refresh_pattern -i \.windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
                                        
                                        refresh_pattern -i \.download.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
                                        
                                        refresh_pattern -i \.ws.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600 reload-into-ims ignore-reload override-expire override-lastmod ignore-no-store ignore-private ignore-auth
                                        
                                        # Symantec Updates
                                        # Only reload-into-ims works properly, other flags break update process 
                                        
                                        refresh_pattern -i \.symantecliveupdate.com/.*\.(zip|exe|z7)          525600 100% 525600 reload-into-ims
                                        
                                        

                                        /usr/local/etc/squid/updatesites.txt

                                        
                                        \.adobe.com
                                        \.java.com
                                        \.sun.com
                                        \.oracle.com
                                        \.apple.com
                                        \.microsoft.com
                                        \.windowsupdate.com
                                        \.ubuntu.com
                                        \.steampowered.com
                                        \.llnwd.net
                                        \.symantecliveupdate.com
                                        
                                        

                                        /usr/local/etc/squid/storeid_rewrite.conf

                                        
                                        ^http:\/\/.+?\.microsoft\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx)     http://wupdate.squid.local/$1
                                        ^http:\/\/.+?\.windowsupdate\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx) http://wupdate.squid.local/$1
                                        ^http:\/\/.+?\.cs\.steampowered\.com\/(.*)      http://steamupdates.squid.internal/$1
                                        ^http:\/\/.+?\.apple\.com\/(.*) http://appupdates.apple.squid.internal/$1
                                        ^http:\/\/.+?\.llnwd\.net\/(.*)                 http://llnwd.net.squid.internal/$1
                                        
                                        

                                        Regards,

                                        Valeriy

                                        May I know.
                                        1. Is it just copy CODE in Table 1(/usr/local/etc/squid/squid.conf) and paste to "Custom refresh_patterns" of squid on Pfsense ? Or any other changes need on Table 1.
                                        2. How useful Table 2(/usr/local/etc/squid/updatesites.txt ) and Table 3(/usr/local/etc/squid/storeid_rewrite.conf)? Need to insert these tables on Table 1? If needed, guide me pls(I am Beginner)
                                        My aim is to get steam update cache(dota2)

                                        1 Reply Last reply Reply Quote 0
                                        • K
                                          kikawala
                                          last edited by

                                          @pfsensation:

                                          @kikawala:

                                          I have this for Apple iOS updates and iOS apps.

                                          refresh_pattern -i appldnld\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate
                                          
                                          refresh_pattern -i phobos\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate
                                          
                                          refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignore-must-revalidate
                                          

                                          Doesn't apple block MITM using certificate pinning? Since their app store is over HTTPS.

                                          Apps and iOS updates are over HTTP

                                          1 Reply Last reply Reply Quote 0
                                          • T
                                            trinitech
                                            last edited by

                                            Hi,

                                            Sorry for the stupid question..
                                            But where does the following code goes:

                                            #new refresh patterns 3
                                            acl Windows_Update dstdomain windowsupdate.microsoft.com
                                            acl Windows_Update dstdomain .update.microsoft.com
                                            acl Windows_Update dstdomain download.windowsupdate.com
                                            acl Windows_Update dstdomain www.download.windowsupdate.com
                                            acl Windows_Update dstdomain au.download.windowsupdate.com
                                            acl Windows_Update dstdomain bg.v4.pr.dl.ws.microsoft.com
                                            

                                            Does it go to the ACLs tab?
                                            Is it white list or does it go in local cache -> Custom refresh_patterns

                                            Thank you

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.