Dynamic cache

saluto

They use dynamic cache?
http://comastnet.blogspot.com.br/2016/06/squid-youtube-cache-2016-terbaru-suport.html
How do I if you managed?

KOM

Yes. Services - Squid - Local Cache - Dynamic and Update Content - Cache Dynamic Content. However, you're probably going to find it a challenge to convert those various config files to pfSense.

saluto

yes, I feel it difficult. Ins't There an App just for this? Because with that, It would be more easier.
Or can also tell me places in squid conf.

KOM

No, there is no automatic program to do it for you. The provided squid.conf can be converted to pfSense through a mix of GUI settings and advanced options, plus you will have to create a few files on pfSense itself that the squid.conf refers to. And after all that work, do you even know if it will properly cache these objects?

saluto

:'( :'( :'( :'( :'(
Thank you

SaschaITM

Also to consider: the probability that more than one user will request the same YT video is very low, unless you're dealing with hundreds or thousands of clients. IMO, the only "large content" worth caching in nearly any scenario is OS updates.

Kababayan

for squid 3.4 above there's a store re writer storeid_id_program. should rewrite the url something like this:

$|=1;
while (<>) {
    chomp;
	@X = split;

	if (@X[0] =~ m/^(exit|quit|x|q)/) {
		print STDERR "quiting helper quietly\n";
		exit 0;
	}

if  ($X[0] =~ m/^.*\:\/\/.*(youtube|google).*(videoplayback|liveplay).*/){
	@itag = m/[&?](itag=[\d]*)/;
	@ids = m/[&?]id\=([^\&\s]*)/;
	@mime = m/[&?](mime\=[^\&\s]*)/;
	@cpn = m/[&?]cpn\=([^\&\s]*)/;
	if (defined($cpn[0])) {
		$fn = "/var/squid/logs/@cpn";
		if (-e $fn) {
			open FH,"<".$fn ;
			$id  = <fh>;
			chomp $id ;
			close FH ;
		} else {
			$id = $ids[0] ;
		}
	} else {
	$id = $ids[0] ;
	}
	@range = m/[&?](range=[^\&\s]*)/;
	$out="http://video-srv.youtube.com.squid.internal/" . $id .  "?@itag?@mime?@range";

		#for yimg.com with &sig=
} elsif ($X[0] =~ m/^.*\:\/\/([^\.]*)\.yimg\.com\/(.*)/) {
        @y = ($1,$2);
        $y[0] =~ s/[a-z]+([0-9]+)?/cdn/;
        $y[1] =~ s/&sig=.*//;
        $out="http://" . $y[0] . ".yimg.com/"  . $y[1] ;

} else {
        $out='';
}

if ($out) { 
	print "OK store-id=$out\n" ;
} else { 
	print "ERR\n" ;
}</fh>

Am able to cache YT videos with that but not on html5 player. webm format was able to cache only if I download the video first then html5 can play the cache video.

saluto

Thank you

Kababayan

made few adjustments on the script. can now cache html5 playback but got 40 percent hit only. flash player has better hits . htnl5 has more random ranges at playback. tested in firefox only and IDM. IDM will cache all format 100 percent.

concatinating all the fragmented video of YT should improve the hits but have no idea how.

saluto

We can improve together with our community.

saluto

######cache Pfsense
refresh_pattern -i .(gif|png|ico|jpg|jpeg|jp2|webp)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
refresh_pattern -i .(jpx|j2k|j2c|fpx|bmp|tif|tiff|bif)$ 100000 90% 20000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
refresh_pattern -i .(pcd|pict|rif|exif|hdr|bpg|img|jif|jfif)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
refresh_pattern -i .(woff|woff2|eps|ttf|otf|svg|svgi|svgz|ps|ps1|acsm|eot)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
#cache content
refresh_pattern -i .(swf|js|ejs)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
refresh_pattern -i .(wav|css|class|dat|zsci|ver|advcs)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
#cache videos
refresh_pattern -i .(mpa|m2a|mpe|avi|mov|mpg|mpeg|mpg3|mpg4|mpg5)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
refresh_pattern -i .(m1s|mp2v|m2v|m2s|m2ts|mp2t|wmx|rm|rmvb|3pg|3gpp|omg|ogm|asf|war)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
refresh_pattern -i .(asx|mp2|mp3|mp4|mp5|wmv|flv|mts|f4v|f4|pls|midi|mid)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
refresh_pattern -i .(htm|html)$ 9440 90% 200000 reload-into-ims ignore-no-store ignore-private refresh-ims
refresh_pattern -i .(xml|flow|asp|aspx)$ 0 90% 200000 refresh-ims
refresh_pattern -i .(json)$ 0 90% 200000 refresh-ims
refresh_pattern -i (/cgi-bin/|?) 0 90% 200000
#cache binaries
refresh_pattern -i .(app|bin|deb|rpm|drpm|exe|zip|zipx|tar|tgz|tbz2|tlz|iso|arj|cfs|dar|jar)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
refresh_pattern -i .(bz|bz2|ipa|ram|rar|uxx|gz|msi|dll|lz|lzma|7z|s7z|Z|z|zz|sz)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
refresh_pattern -i .(exe|msi)$ 0 90% 200000 refresh-ims
refresh_pattern -i .(cab|psf|vidt|apk|wtex|hz|ova|ovf)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
#cache microsoft and adobe and other documents
refresh_pattern -i .(ppt|pptx|doc|docx|docm|docb|dot|pdf|pub|ps)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims
refresh_pattern -i .(xls|xlsx|xlt|xlm|xlsm|xltm|xlw|csv|txt)$ 100000 90% 200000 override-expire reload-into-ims ignore-no-store ignore-private refresh-ims

#cache specific sites
refresh_pattern -i ^http://liveupdate.symantecliveupdate.com.(zip)$ 0 0% 0
refresh_pattern -i ^http://premium.avira-update.com.(gz) 0 0% 0
refresh_pattern -i microsoft.com/..(cab|exe|msi|msu|msf|asf|wma|dat|zip)$ 4320 80% 43200 reload-into-ims refresh-ims
refresh_pattern -i windowsupdate.com/..(cab|exe|msi|msu|msf|asf|wma|wmv)|dat|zip)$ 4320 80% 43200 reload-into-ims refresh-ims
refresh_pattern -i windows.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip)$ 4320 80% 43200 reload-into-ims refresh-ims
refresh_pattern -i apple.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip|dist)$ 0 80% 43200 reload-into-ims refresh-ims

Youtube Video

refresh_pattern -i (get_video?|videoplayback?|videodownload?|.mp4|.webm|.flv|((audio|video)/(webm|mp4))) 241920 100% 241920 override-expire ignore-reload ignore-private ignore-no-store ignore-must-revalidate reload-into-ims ignore-auth store-stale
refresh_pattern -i ^https?://..googlevideo.com/videoplayback. 10080 99% 43200 override-lastmod override-expire ignore-reload reload-into-ims ignore-private reload-into-ims ignore-auth store-stale
refresh_pattern -i ^https?://..googlevideo.com/videoplayback.$ 241920 100% 241920 override-expire ignore-reload ignore-private ignore-no-store ignore-must-revalidate reload-into-ims ignore-auth store-stale

Image Youtube

refresh_pattern -i (yimg|twimg).com.* 1440 100% 129600 override-expire ignore-reload reload-into-ims
refresh_pattern -i (ytimg|ggpht).com.* 1440 80% 129600 override-expire override-lastmod ignore-auth ignore-reload reload-into-ims

#images facebook
refresh_pattern -i fbcdn.net/..((jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js)|(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js)(?|.$)) 241920 99% 241920 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-auth
refresh_pattern -i pixel.facebook.com..(jpg|png|gif|ico|css|js) 241920 80% 241920 override-expire ignore-reload reload-into-ims ignore-auth
refresh_pattern -i .akamaihd.net..(jpg|png|gif|ico|css|js) 241920 80% 241920 override-expire ignore-reload reload-into-ims ignore-auth
refresh_pattern -i ((facebook.com)|(85.131.151.39)).(jpg|png|gif) 241920 99% 241920 ignore-reload override-expire ignore-no-store store-stale
refresh_pattern -i fbcdn.net/..((jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js)|(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|css|js)(?|.$)) 241920 99% 241920 ignore-no-store ignore-private override-expire override-lastmod reload-into-ims ignore-auth
refresh_pattern static.(xx|ak).fbcdn.net.(jpg|gif|png) 241920 99% 241920 ignore-reload override-expire ignore-no-store
refresh_pattern ^https?://profile.ak.fbcdn.net*.(jpg|gif|png) 241920 99% 241920 ignore-reload override-expire ignore-no-store

Video Facebook

refresh_pattern -i .video.ak.fbcdn.net.*.(mp4|flv|mp3|amf) 10080 80% 43200 override-expire ignore-reload reload-into-ims ignore-private ignore-no-store ignore-must-revalidate
refresh_pattern (audio|video)/(webm|mp4) 129600 99% 129600 ignore-reload override-expire override-lastmod ignore-must-revalidate ignore-private ignore-no-store ignore-auth store-stale
refresh_pattern -i ^http://.squid.internal. 241920 100% 241920 override-lastmod override-expire ignore-reload ignore-must-revalidate ignore-private ignore-no-store ignore-auth store-stale