Caching Steam game downloads



  • Hello..

    Its this topic again..

    I'm trying to get pfSense 2.2.2 and squid3 (0.2.8 according to pfsense package manager) to cache Steam game downloads

    I tested squid with virtualbox before buying real hardware

    We are on 4MBit adsl line (the fastest with no download quota available in my area, or we had to choose 3g network) and have 4 pcs playing dota 2 and csgo, so caching will greatly help us..

    from what i can see, steam download servers ranging from valve<insert number="" here="">.cs.steampowered.com, cdn.highwinds.cs.steampowered.com, cdn.akamai.cs.steampowered.com, cdn.level3.cs.steampowered.com and probably many more with parent subdomain being cs.steampowered.com

    squid is caching steam download without any problem, but i can only get cache hit if steam is downloading same chunks from same download server. how do i get squid to serve the cached download chunks even if steam request the same download chunks from different server?

    i can see the download chunks have ? at the end of the url if they are served from highwinds, akamai or level3 mirrors but that is not really the priority for now..i just need squid to work with valve*.cs.steampowered.com for the time being..

    heres are some of the access.log for steam game downloads

    
    1434118773.348    171 192.168.1.6 TCP_MISS/200 16830 GET http://valve277.cs.steampowered.com/depot/346901/chunk/b5151492613a27dbfe2d51fec30daf2f42b65fba - ORIGINAL_DST/153.254.86.139 application/x-steam-chunk
    1434124045.877   2777 192.168.1.6 TCP_MISS/200 16830 GET http://valve1000.cs.steampowered.com/depot/346901/chunk/b5151492613a27dbfe2d51fec30daf2f42b65fba - ORIGINAL_DST/103.10.124.15 application/x-steam-chunk
    1434124408.798   1763 192.168.1.6 TCP_HIT/200 16838 GET http://valve1000.cs.steampowered.com/depot/346901/chunk/b5151492613a27dbfe2d51fec30daf2f42b65fba - HIER_NONE/- application/x-steam-chunk
    1434040096.678   1798 192.168.1.6 TCP_MISS/200 10990 GET http://valve32.cs.steampowered.com/depot/363971/chunk/ffcbe0b4463fca9945cf0ddf026d83ac5fae04f0 - ORIGINAL_DST/103.10.124.14 application/x-steam-chunk
    1434040535.834    652 192.168.1.6 TCP_MISS/200 10991 GET http://valve276.cs.steampowered.com/depot/363971/chunk/ffcbe0b4463fca9945cf0ddf026d83ac5fae04f0 - ORIGINAL_DST/153.254.86.138 application/x-steam-chunk
    1434041122.497    563 192.168.1.6 TCP_HIT/200 10999 GET http://valve276.cs.steampowered.com/depot/363971/chunk/ffcbe0b4463fca9945cf0ddf026d83ac5fae04f0 - HIER_NONE/- application/x-steam-chunk
    1434121207.307    216 192.168.1.6 TCP_MISS/200 11273 GET http://cdn.highwinds.cs.steampowered.com/depot/363971/chunk/ffcbe0b4463fca9945cf0ddf026d83ac5fae04f0? - ORIGINAL_DST/205.185.216.42 application/x-steam-chunk
    1434121883.188   1507 192.168.1.6 TCP_MISS/200 10972 GET http://cdn.akamai.cs.steampowered.com/depot/363971/chunk/ffcbe0b4463fca9945cf0ddf026d83ac5fae04f0? - ORIGINAL_DST/1.9.56.35 application/x-steam-chunk
    1434040044.067   2664 192.168.1.6 TCP_MISS/200 236538 GET http://cdn.highwinds.cs.steampowered.com/depot/363971/chunk/2ec3626658d13a51f3d8c2ba647815c2b9c846e1? - ORIGINAL_DST/205.185.216.10 application/x-steam-chunk
    1434121192.315   1765 192.168.1.6 TCP_HIT/200 236548 GET http://cdn.highwinds.cs.steampowered.com/depot/363971/chunk/2ec3626658d13a51f3d8c2ba647815c2b9c846e1? - HIER_NONE/- application/x-steam-chunk
    1434040083.059    706 192.168.1.6 TCP_MISS/200 230172 GET http://cdn.akamai.cs.steampowered.com/depot/363971/chunk/6701fe71974b73c6c6ab396b396cb6a5816ca8f3? - ORIGINAL_DST/1.9.56.154 application/x-steam-chunk
    1434121223.129   2772 192.168.1.6 TCP_HIT/200 230182 GET http://cdn.akamai.cs.steampowered.com/depot/363971/chunk/6701fe71974b73c6c6ab396b396cb6a5816ca8f3? - HIER_NONE/- application/x-steam-chunk
    1434040096.896   1966 192.168.1.6 TCP_MISS/200 10934 GET http://cdn.level3.cs.steampowered.com/depot/363971/chunk/619210ec5a674c95bfe81ca4fb32ba560ed09048? - ORIGINAL_DST/113.29.66.253 application/x-steam-chunk
    1434121340.788    216 192.168.1.6 TCP_HIT/200 10944 GET http://cdn.level3.cs.steampowered.com/depot/363971/chunk/619210ec5a674c95bfe81ca4fb32ba560ed09048? - HIER_NONE/- application/x-steam-chunk
    
    

    What I've done so far:

    squid.conf (i put this in the Custom ACLS (Before_Auth) area)

    
    refresh_pattern ^http:\/\/(.*)\.cs\.steampowered\.com\/(.*) 43200 100% 43200 reload-into-ims
    
    acl steamcdn dstdomain .cs.steampowered.com
    store_id_program /usr/pbi/bin/libexec/squid/storeid_file_rewrite /usr/pbi/bin/libexec/squid/storeid_rewrite.conf
    store_id_children 20 startup=0 idle=1 concurrency=0
    store_id_access allow steamcdn
    store_id_access deny all
    
    

    /usr/pbi/bin/libexec/squid/storeid_rewrite.conf (chmod +x)

    
    ^http:\/\/(.*)\.cs\.steampowered\.com\/(.*)          http://steamcdn.cs.steampowered.com/$1
    
    

    i tried to use single tab and double tab as whitespace for storeid_rewrite.conf, but same result.

    other squid settings:
    Hard disk cache size: 18000mb
    Minimum object size: 0
    Maximum object size: 5000000kb
    Memory cache size: 512mb
    Maximum object size in RAM: 256mb
    Finish transfer if more than x % finished: 75%

    Transparent HTTP proxy enabled
    caching both http and https

    I've enabled Cache Dynamic Content (Youtube, Windows Update, Symantec, Avira, avast refresh pattern selected)

    my reference so far, basically my method are the combinations from these sites:
    http://wiki.squid-cache.org/Features/StoreID
    http://wiki.squid-cache.org/Features/StoreID/DB
    http://blog.thelifeofkenneth.com/2014/08/using-squid-storeids-to-optimize-steams.html (tried this to, installed bash on pfsense, but same result)
    http://wiki.sebeka.k12.mn.us/web_services:squid_update_cache (same result)

    any help is greatly appreciated.. thank you..

    p/s: im still new to *nix , regex and squid</insert>



  • sorry out of topic, can you teach me how to intergrated/install storeid squid 3.4 on pfsense 2.2



  • can you teach me how to intergrated/install to squid 3.4 on pfsense 2.2

    There are a million YouTube videos and other resources you can find through a search engine on how to do that.  Don't hijack someone's else's thread to ask something totally unrelated.



  • @KOM:

    can you teach me how to intergrated/install to squid 3.4 on pfsense 2.2

    There are a million YouTube videos and other resources you can find through a search engine on how to do that.  Don't hijack someone's else's thread to ask something totally unrelated.

    I'm so sorry sir, I'm mean storeid into squid 3.4  ;D



  • a little late dunno if you are still around as that was your only post, i am working on a solution for this as well.

    i looked at the same pages you listed before and the issue with the best option is they didnt put the functions into squid until 3.4 and we are on 3.2 or something im not quite sure how to read the versions… squid3 v0.2.8.

    been searching all morning looking how to change the location of the caches so that valve1-200.steampowere.com all save to one folder and get cached / hit using the same name... im a complete noob with squid how to do this tho ive seen some stuff possibly using symlinks, though im not sure.

    right now im getting about a 50/50 chance the object has been cached which isnt terrible and will help a bit for my lan party. though i wish i could fix the cache so it would be a 100% hit rate.

    edit: i tink i might have found a solution, im going to have to read through it for a little bit and try to understand how it works completely. http://wiki.squid-cache.org/Features/StoreUrlRewrite



  • Bump….?  I too would like to achieve this.  Is the store_rewrite function built in yet?



  • @mutiny1:

    Bump….?  I too would like to achieve this.  Is the store_rewrite function built in yet?

    i gave up and use pfsense + lancache (http://blog.multiplay.co.uk/2014/04/lancache-dynamically-caching-game-installs-at-lans-using-nginx/)



  • Thanks for the reply.  I'm not smart enought to use that yet.  I tried but it didn't work out too well.



  • @whiztech:

    @mutiny1:

    Bump….?  I too would like to achieve this.  Is the store_rewrite function built in yet?

    i gave up and use pfsense + lancache (http://blog.multiplay.co.uk/2014/04/lancache-dynamically-caching-game-installs-at-lans-using-nginx/)

    Are you using Lancache on PFSense, or on a dedicated server?  If on PFSense, could you write a guide for those that wish to use this with pfsense?



  • waiting for the guide  as well ::)



  • So I have been playing with this and not having any luck yet, but I thought I would post what I have learned.

    Your rewrite patterns are wrong.

    ^http:\/\/(.*)\.cs\.steampowered\.com\/(.*)          http://steamcdn.cs.steampowered.com/$1
    

    In a regex, the $1 returns the first () in the pattern and $2 the second etc.  So you created a URL that looks like http://steamcdn.cs.streampowered.com/valve32

    better would be something like this:

    ^http:\/\/.*\.cs\.steampowered\.com\/(depot\/\d+\/chunk\/.*)          http://valveN.steamcdn.cs.steampowered.com.squid.internal/$1
    

    I used more of the url to make sure we are only matching the actual game downloads and used squid.internal like this collection: http://wiki.squid-cache.org/Features/StoreID/DB

    The next problem I had is that the /usr/pbi/bin/libexec/squid/storeid_file_rewrite program doesn't seem to actually work. When I run it directly from the shell it returns immediately. It should read URLs on stdin and write the results to stdout. Not seeing what I did wrong but when I use it I just get this in cache.log.

    
    2016/03/12 07:50:21 kid1| Starting new store_id helpers...
    2016/03/12 07:50:21 kid1| WARNING: store_id #Hlpr0 exited
    2016/03/12 07:50:21 kid1| WARNING: store_id #Hlpr0 exited
    2016/03/12 07:50:21 kid1| WARNING: store_id #Hlpr0 exited
    2016/03/12 07:50:22 kid1| WARNING: store_id #Hlpr0 exited
    2016/03/12 07:50:22 kid1| WARNING: store_id #Hlpr0 exited
    2016/03/12 07:50:22 kid1| WARNING: store_id #Hlpr0 exited
    2016/03/12 07:50:22 kid1| WARNING: store_id #Hlpr0 exited
    2016/03/12 07:50:22 kid1| WARNING: store_id #Hlpr0 exited
    2016/03/12 07:50:22 kid1| WARNING: store_id #Hlpr0 exited
    
    

    Also I tried using my own perl script but had troubles getting a perl to run.

    I will keep poking as a have time but perhaps someone can shed light.



  • I tried to make it work with perl in pfsense 2.6 with little success.

    anyways I did make a helper in PHP with and loaded with some configs and rewrite parterns

    works pretty fine altough I had some hangs in steam do not know why.

    https://github.com/rudiservo/pfsense_storeid


Log in to reply