Dynamic cache





  • Yes.  Services - Squid - Local Cache - Dynamic and Update Content - Cache Dynamic Content.  However, you're probably going to find it a challenge to convert those various config files to pfSense.



  • yes, I feel it difficult. Ins't There an App just for this? Because with that, It would be more easier.
    Or can also tell me places in squid conf.



  • No, there is no automatic program to do it for you.  The provided squid.conf can be converted to pfSense through a mix of GUI settings and advanced options, plus you will have to create a few files on pfSense itself that the squid.conf refers to.  And after all that work, do you even know if it will properly cache these objects?



  • :'( :'( :'( :'( :'(
    Thank you



  • Also to consider: the probability that more than one user will request the same YT video is very low, unless you're dealing with hundreds or thousands of clients. IMO, the only "large content" worth caching in nearly any scenario is OS updates.



  • for squid 3.4 above there's a store re writer storeid_id_program. should rewrite the url something like this:

    $|=1;
    while (<>) {
        chomp;
    	@X = split;
    
    	if (@X[0] =~ m/^(exit|quit|x|q)/) {
    		print STDERR "quiting helper quietly\n";
    		exit 0;
    	}
    
    if  ($X[0] =~ m/^.*\:\/\/.*(youtube|google).*(videoplayback|liveplay).*/){
    	@itag = m/[&?](itag=[\d]*)/;
    	@ids = m/[&?]id\=([^\&\s]*)/;
    	@mime = m/[&?](mime\=[^\&\s]*)/;
    	@cpn = m/[&?]cpn\=([^\&\s]*)/;
    	if (defined($cpn[0])) {
    		$fn = "/var/squid/logs/@cpn";
    		if (-e $fn) {
    			open FH,"<".$fn ;
    			$id  = <fh>;
    			chomp $id ;
    			close FH ;
    		} else {
    			$id = $ids[0] ;
    		}
    	} else {
    	$id = $ids[0] ;
    	}
    	@range = m/[&?](range=[^\&\s]*)/;
    	$out="http://video-srv.youtube.com.squid.internal/" . $id .  "?@itag?@mime?@range";
    
    		#for yimg.com with &sig=
    } elsif ($X[0] =~ m/^.*\:\/\/([^\.]*)\.yimg\.com\/(.*)/) {
            @y = ($1,$2);
            $y[0] =~ s/[a-z]+([0-9]+)?/cdn/;
            $y[1] =~ s/&sig=.*//;
            $out="http://" . $y[0] . ".yimg.com/"  . $y[1] ;
    
    } else {
            $out='';
    }
    
    if ($out) { 
    	print "OK store-id=$out\n" ;
    } else { 
    	print "ERR\n" ;
    }</fh>
    

    Am able to cache YT videos with that but not on html5 player. webm format was able to cache only if I download the video first then html5 can play the cache video.



  • Thank you



  • made few adjustments on the script. can now cache html5 playback but  got 40 percent hit only. flash player has better hits . htnl5 has more random ranges at playback. tested in firefox only and IDM. IDM will cache all format 100 percent.

    concatinating all the fragmented video of YT should improve the hits but have  no idea how.



  • We can improve together with our community.



  • ######cache Pfsense
    refresh_pattern -i .(gif|png|ico|jpg|jpeg|jp2|webp)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
    refresh_pattern -i .(jpx|j2k|j2c|fpx|bmp|tif|tiff|bif)$ 100000 90% 20000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
    refresh_pattern -i .(pcd|pict|rif|exif|hdr|bpg|img|jif|jfif)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
    refresh_pattern -i .(woff|woff2|eps|ttf|otf|svg|svgi|svgz|ps|ps1|acsm|eot)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
    #cache content
    refresh_pattern -i .(swf|js|ejs)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
    refresh_pattern -i .(wav|css|class|dat|zsci|ver|advcs)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
    #cache videos
    refresh_pattern -i .(mpa|m2a|mpe|avi|mov|mpg|mpeg|mpg3|mpg4|mpg5)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
    refresh_pattern -i .(m1s|mp2v|m2v|m2s|m2ts|mp2t|wmx|rm|rmvb|3pg|3gpp|omg|ogm|asf|war)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
    refresh_pattern -i .(asx|mp2|mp3|mp4|mp5|wmv|flv|mts|f4v|f4|pls|midi|mid)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
    refresh_pattern -i .(htm|html)$ 9440 90% 200000 reload-into-ims ignore-no-store ignore-private refresh-ims
    refresh_pattern -i .(xml|flow|asp|aspx)$ 0 90% 200000 refresh-ims
    refresh_pattern -i .(json)$ 0 90% 200000 refresh-ims
    refresh_pattern -i (/cgi-bin/|?) 0 90% 200000
    #cache binaries
    refresh_pattern -i .(app|bin|deb|rpm|drpm|exe|zip|zipx|tar|tgz|tbz2|tlz|iso|arj|cfs|dar|jar)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
    refresh_pattern -i .(bz|bz2|ipa|ram|rar|uxx|gz|msi|dll|lz|lzma|7z|s7z|Z|z|zz|sz)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
    refresh_pattern -i .(exe|msi)$ 0 90% 200000 refresh-ims
    refresh_pattern -i .(cab|psf|vidt|apk|wtex|hz|ova|ovf)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
    #cache microsoft and adobe and other documents
    refresh_pattern -i .(ppt|pptx|doc|docx|docm|docb|dot|pdf|pub|ps)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
    refresh_pattern -i .(xls|xlsx|xlt|xlm|xlsm|xltm|xlw|csv|txt)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims

    #cache specific sites
    refresh_pattern -i ^http://liveupdate.symantecliveupdate.com.(zip)$ 0 0% 0
    refresh_pattern -i ^http://premium.avira-update.com.
    (gz) 0 0% 0
    refresh_pattern -i microsoft.com/..(cab|exe|msi|msu|msf|asf|wma|dat|zip)$ 4320 80% 43200 reload-into-ims refresh-ims
    refresh_pattern -i windowsupdate.com/.
    .(cab|exe|msi|msu|msf|asf|wma|wmv)|dat|zip)$ 4320 80% 43200 reload-into-ims refresh-ims
    refresh_pattern -i windows.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip)$ 4320 80% 43200 reload-into-ims refresh-ims
    refresh_pattern -i apple.com/.
    .(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip|dist)$ 0 80% 43200 reload-into-ims refresh-ims

    Youtube Video

    refresh_pattern -i (get_video?|videoplayback?|videodownload?|.mp4|.webm|.flv|((audio|video)/(webm|mp4))) 241920 100% 241920 override-expire ignore-reload ignore-private ignore-no-store ignore-must-revalidate reload-into-ims ignore-auth store-stale
    refresh_pattern -i ^https?://..googlevideo.com/videoplayback.    10080 99% 43200 override-lastmod override-expire ignore-reload reload-into-ims ignore-private reload-into-ims ignore-auth store-stale
    refresh_pattern -i ^https?://..googlevideo.com/videoplayback.$    241920 100% 241920 override-expire ignore-reload ignore-private ignore-no-store ignore-must-revalidate reload-into-ims ignore-auth store-stale

    Image Youtube

    refresh_pattern -i (yimg|twimg).com.*        1440 100% 129600 override-expire ignore-reload reload-into-ims
    refresh_pattern -i (ytimg|ggpht).com.*        1440 80% 129600 override-expire override-lastmod ignore-auth ignore-reload reload-into-ims

    #images facebook
    refresh_pattern -i fbcdn.net/..((jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js)|(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js)(?|.$)) 241920 99% 241920 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-auth
    refresh_pattern -i pixel.facebook.com.
    .(jpg|png|gif|ico|css|js) 241920 80% 241920 override-expire ignore-reload reload-into-ims ignore-auth
    refresh_pattern -i .akamaihd.net..(jpg|png|gif|ico|css|js) 241920 80% 241920 override-expire ignore-reload reload-into-ims ignore-auth
    refresh_pattern -i ((facebook.com)|(85.131.151.39)).(jpg|png|gif) 241920 99% 241920 ignore-reload override-expire ignore-no-store store-stale
    refresh_pattern -i fbcdn.net/.
    .((jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js)|(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js)(?|.$)) 241920 99% 241920 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-auth
    refresh_pattern static.(xx|ak).fbcdn.net
    .(jpg|gif|png) 241920 99% 241920 ignore-reload override-expire ignore-no-store
    refresh_pattern ^https?://profile.ak.fbcdn.net*.(jpg|gif|png) 241920 99% 241920 ignore-reload override-expire ignore-no-store

    Video Facebook

    refresh_pattern -i .video.ak.fbcdn.net.*.(mp4|flv|mp3|amf)                    10080 80% 43200 override-expire ignore-reload reload-into-ims ignore-private ignore-no-store ignore-must-revalidate
    refresh_pattern (audio|video)/(webm|mp4) 129600 99% 129600 ignore-reload override-expire override-lastmod ignore-must-revalidate  ignore-private ignore-no-store ignore-auth store-stale
    refresh_pattern -i ^http://.squid.internal.  241920 100% 241920 override-lastmod override-expire ignore-reload ignore-must-revalidate ignore-private ignore-no-store ignore-auth store-stale