Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Caching Steam game downloads

    Scheduled Pinned Locked Moved Cache/Proxy
    12 Posts 9 Posters 15.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • W
      whiztech
      last edited by

      Hello..

      Its this topic again..

      I'm trying to get pfSense 2.2.2 and squid3 (0.2.8 according to pfsense package manager) to cache Steam game downloads

      I tested squid with virtualbox before buying real hardware

      We are on 4MBit adsl line (the fastest with no download quota available in my area, or we had to choose 3g network) and have 4 pcs playing dota 2 and csgo, so caching will greatly help us..

      from what i can see, steam download servers ranging from valve<insert number="" here="">.cs.steampowered.com, cdn.highwinds.cs.steampowered.com, cdn.akamai.cs.steampowered.com, cdn.level3.cs.steampowered.com and probably many more with parent subdomain being cs.steampowered.com

      squid is caching steam download without any problem, but i can only get cache hit if steam is downloading same chunks from same download server. how do i get squid to serve the cached download chunks even if steam request the same download chunks from different server?

      i can see the download chunks have ? at the end of the url if they are served from highwinds, akamai or level3 mirrors but that is not really the priority for now..i just need squid to work with valve*.cs.steampowered.com for the time being..

      heres are some of the access.log for steam game downloads

      
1434118773.348    171 192.168.1.6 TCP_MISS/200 16830 GET http://valve277.cs.steampowered.com/depot/346901/chunk/b5151492613a27dbfe2d51fec30daf2f42b65fba - ORIGINAL_DST/153.254.86.139 application/x-steam-chunk
1434124045.877   2777 192.168.1.6 TCP_MISS/200 16830 GET http://valve1000.cs.steampowered.com/depot/346901/chunk/b5151492613a27dbfe2d51fec30daf2f42b65fba - ORIGINAL_DST/103.10.124.15 application/x-steam-chunk
1434124408.798   1763 192.168.1.6 TCP_HIT/200 16838 GET http://valve1000.cs.steampowered.com/depot/346901/chunk/b5151492613a27dbfe2d51fec30daf2f42b65fba - HIER_NONE/- application/x-steam-chunk
1434040096.678   1798 192.168.1.6 TCP_MISS/200 10990 GET http://valve32.cs.steampowered.com/depot/363971/chunk/ffcbe0b4463fca9945cf0ddf026d83ac5fae04f0 - ORIGINAL_DST/103.10.124.14 application/x-steam-chunk
1434040535.834    652 192.168.1.6 TCP_MISS/200 10991 GET http://valve276.cs.steampowered.com/depot/363971/chunk/ffcbe0b4463fca9945cf0ddf026d83ac5fae04f0 - ORIGINAL_DST/153.254.86.138 application/x-steam-chunk
1434041122.497    563 192.168.1.6 TCP_HIT/200 10999 GET http://valve276.cs.steampowered.com/depot/363971/chunk/ffcbe0b4463fca9945cf0ddf026d83ac5fae04f0 - HIER_NONE/- application/x-steam-chunk
1434121207.307    216 192.168.1.6 TCP_MISS/200 11273 GET http://cdn.highwinds.cs.steampowered.com/depot/363971/chunk/ffcbe0b4463fca9945cf0ddf026d83ac5fae04f0? - ORIGINAL_DST/205.185.216.42 application/x-steam-chunk
1434121883.188   1507 192.168.1.6 TCP_MISS/200 10972 GET http://cdn.akamai.cs.steampowered.com/depot/363971/chunk/ffcbe0b4463fca9945cf0ddf026d83ac5fae04f0? - ORIGINAL_DST/1.9.56.35 application/x-steam-chunk
1434040044.067   2664 192.168.1.6 TCP_MISS/200 236538 GET http://cdn.highwinds.cs.steampowered.com/depot/363971/chunk/2ec3626658d13a51f3d8c2ba647815c2b9c846e1? - ORIGINAL_DST/205.185.216.10 application/x-steam-chunk
1434121192.315   1765 192.168.1.6 TCP_HIT/200 236548 GET http://cdn.highwinds.cs.steampowered.com/depot/363971/chunk/2ec3626658d13a51f3d8c2ba647815c2b9c846e1? - HIER_NONE/- application/x-steam-chunk
1434040083.059    706 192.168.1.6 TCP_MISS/200 230172 GET http://cdn.akamai.cs.steampowered.com/depot/363971/chunk/6701fe71974b73c6c6ab396b396cb6a5816ca8f3? - ORIGINAL_DST/1.9.56.154 application/x-steam-chunk
1434121223.129   2772 192.168.1.6 TCP_HIT/200 230182 GET http://cdn.akamai.cs.steampowered.com/depot/363971/chunk/6701fe71974b73c6c6ab396b396cb6a5816ca8f3? - HIER_NONE/- application/x-steam-chunk
1434040096.896   1966 192.168.1.6 TCP_MISS/200 10934 GET http://cdn.level3.cs.steampowered.com/depot/363971/chunk/619210ec5a674c95bfe81ca4fb32ba560ed09048? - ORIGINAL_DST/113.29.66.253 application/x-steam-chunk
1434121340.788    216 192.168.1.6 TCP_HIT/200 10944 GET http://cdn.level3.cs.steampowered.com/depot/363971/chunk/619210ec5a674c95bfe81ca4fb32ba560ed09048? - HIER_NONE/- application/x-steam-chunk

      What I've done so far:

      squid.conf (i put this in the Custom ACLS (Before_Auth) area)

      
refresh_pattern ^http:\/\/(.*)\.cs\.steampowered\.com\/(.*) 43200 100% 43200 reload-into-ims

acl steamcdn dstdomain .cs.steampowered.com
store_id_program /usr/pbi/bin/libexec/squid/storeid_file_rewrite /usr/pbi/bin/libexec/squid/storeid_rewrite.conf
store_id_children 20 startup=0 idle=1 concurrency=0
store_id_access allow steamcdn
store_id_access deny all

      /usr/pbi/bin/libexec/squid/storeid_rewrite.conf (chmod +x)

      
^http:\/\/(.*)\.cs\.steampowered\.com\/(.*)          http://steamcdn.cs.steampowered.com/$1

      i tried to use single tab and double tab as whitespace for storeid_rewrite.conf, but same result.

      other squid settings:
      Hard disk cache size: 18000mb
      Minimum object size: 0
      Maximum object size: 5000000kb
      Memory cache size: 512mb
      Maximum object size in RAM: 256mb
      Finish transfer if more than x % finished: 75%

      Transparent HTTP proxy enabled
      caching both http and https

      I've enabled Cache Dynamic Content (Youtube, Windows Update, Symantec, Avira, avast refresh pattern selected)

      my reference so far, basically my method are the combinations from these sites:
      http://wiki.squid-cache.org/Features/StoreID
      http://wiki.squid-cache.org/Features/StoreID/DB
      http://blog.thelifeofkenneth.com/2014/08/using-squid-storeids-to-optimize-steams.html (tried this to, installed bash on pfsense, but same result)
      http://wiki.sebeka.k12.mn.us/web_services:squid_update_cache (same result)

      any help is greatly appreciated.. thank you..

      p/s: im still new to *nix , regex and squid</insert>

      1 Reply Last reply Reply Quote 0
      • A
        agixdota
        last edited by

        sorry out of topic, can you teach me how to intergrated/install storeid squid 3.4 on pfsense 2.2

        1 Reply Last reply Reply Quote 0
        • KOMK
          KOM
          last edited by

          can you teach me how to intergrated/install to squid 3.4 on pfsense 2.2

          There are a million YouTube videos and other resources you can find through a search engine on how to do that.  Don't hijack someone's else's thread to ask something totally unrelated.

          1 Reply Last reply Reply Quote 0
          • A
            agixdota
            last edited by

            @KOM:

            can you teach me how to intergrated/install to squid 3.4 on pfsense 2.2

            There are a million YouTube videos and other resources you can find through a search engine on how to do that.  Don't hijack someone's else's thread to ask something totally unrelated.

            I'm so sorry sir, I'm mean storeid into squid 3.4  ;D

            1 Reply Last reply Reply Quote 0
            • W
              wagex
              last edited by

              a little late dunno if you are still around as that was your only post, i am working on a solution for this as well.

              i looked at the same pages you listed before and the issue with the best option is they didnt put the functions into squid until 3.4 and we are on 3.2 or something im not quite sure how to read the versions… squid3 v0.2.8.

              been searching all morning looking how to change the location of the caches so that valve1-200.steampowere.com all save to one folder and get cached / hit using the same name... im a complete noob with squid how to do this tho ive seen some stuff possibly using symlinks, though im not sure.

              right now im getting about a 50/50 chance the object has been cached which isnt terrible and will help a bit for my lan party. though i wish i could fix the cache so it would be a 100% hit rate.

              edit: i tink i might have found a solution, im going to have to read through it for a little bit and try to understand how it works completely. http://wiki.squid-cache.org/Features/StoreUrlRewrite

              1 Reply Last reply Reply Quote 0
              • M
                mutiny1
                last edited by

                Bump….?  I too would like to achieve this.  Is the store_rewrite function built in yet?

                1 Reply Last reply Reply Quote 0
                • W
                  whiztech
                  last edited by

                  @mutiny1:

                  Bump….?  I too would like to achieve this.  Is the store_rewrite function built in yet?

                  i gave up and use pfsense + lancache (http://blog.multiplay.co.uk/2014/04/lancache-dynamically-caching-game-installs-at-lans-using-nginx/)

                  1 Reply Last reply Reply Quote 0
                  • M
                    mutiny1
                    last edited by

                    Thanks for the reply.  I'm not smart enought to use that yet.  I tried but it didn't work out too well.

                    1 Reply Last reply Reply Quote 0
                    • C
                      C0RR0SIVE
                      last edited by

                      @whiztech:

                      @mutiny1:

                      Bump….?  I too would like to achieve this.  Is the store_rewrite function built in yet?

                      i gave up and use pfsense + lancache (http://blog.multiplay.co.uk/2014/04/lancache-dynamically-caching-game-installs-at-lans-using-nginx/)

                      Are you using Lancache on PFSense, or on a dedicated server?  If on PFSense, could you write a guide for those that wish to use this with pfsense?

                      1 Reply Last reply Reply Quote 0
                      • B
                        bonbonboi
                        last edited by

                        waiting for the guide  as well ::)

                        1 Reply Last reply Reply Quote 0
                        • W
                          wscott
                          last edited by

                          So I have been playing with this and not having any luck yet, but I thought I would post what I have learned.

                          Your rewrite patterns are wrong.

                          ^http:\/\/(.*)\.cs\.steampowered\.com\/(.*)          http://steamcdn.cs.steampowered.com/$1

                          In a regex, the $1 returns the first () in the pattern and $2 the second etc.  So you created a URL that looks like http://steamcdn.cs.streampowered.com/valve32

                          better would be something like this:

                          ^http:\/\/.*\.cs\.steampowered\.com\/(depot\/\d+\/chunk\/.*)          http://valveN.steamcdn.cs.steampowered.com.squid.internal/$1

                          I used more of the url to make sure we are only matching the actual game downloads and used squid.internal like this collection: http://wiki.squid-cache.org/Features/StoreID/DB

                          The next problem I had is that the /usr/pbi/bin/libexec/squid/storeid_file_rewrite program doesn't seem to actually work. When I run it directly from the shell it returns immediately. It should read URLs on stdin and write the results to stdout. Not seeing what I did wrong but when I use it I just get this in cache.log.

                          
2016/03/12 07:50:21 kid1| Starting new store_id helpers...
2016/03/12 07:50:21 kid1| WARNING: store_id #Hlpr0 exited
2016/03/12 07:50:21 kid1| WARNING: store_id #Hlpr0 exited
2016/03/12 07:50:21 kid1| WARNING: store_id #Hlpr0 exited
2016/03/12 07:50:22 kid1| WARNING: store_id #Hlpr0 exited
2016/03/12 07:50:22 kid1| WARNING: store_id #Hlpr0 exited
2016/03/12 07:50:22 kid1| WARNING: store_id #Hlpr0 exited
2016/03/12 07:50:22 kid1| WARNING: store_id #Hlpr0 exited
2016/03/12 07:50:22 kid1| WARNING: store_id #Hlpr0 exited
2016/03/12 07:50:22 kid1| WARNING: store_id #Hlpr0 exited

                          Also I tried using my own perl script but had troubles getting a perl to run.

                          I will keep poking as a have time but perhaps someone can shed light.

                          1 Reply Last reply Reply Quote 0
                          • R
                            rudiservo
                            last edited by

                            I tried to make it work with perl in pfsense 2.6 with little success.

                            anyways I did make a helper in PHP with and loaded with some configs and rewrite parterns

                            works pretty fine altough I had some hangs in steam do not know why.

                            https://github.com/rudiservo/pfsense_storeid

                            1 Reply Last reply Reply Quote 0
                            • First post
                              Last post
                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.