Squid video/gif/pics-cache for youtube/social media in 2019

nuclearstrength

there's plenty of posts on the forum but they're all old, referring to pre 3.5 squid versions, what's the situation now?

I found a couple of github projects that made ID helpers for squid 3.x to handle this but those projects are also old.

is it even possible to achieve this?
generally speaking do you guys manage to get a decent amount of hits besides windows update?
is it just enough to add some custom refresh_pattern rules or do we still need helpers and url_rewrite to make it work?

I know social media dynamic content makes it pretty hard, still the same situation as years ago?

specifically i was thinking about using this

https://github.com/hscbrasil/hsc-dynamic-cache

HSC Dynamic Cache delivers ready-to-use Squid-3.5 "Store ID" helper. To improve the cache keys, there is an URL patterns file which can be customized by the user.

Here is the list of patterns included:

IMDb trailers
Sourceforge downloads
R7 vídeos
Metacafe
MSN videos
Terra TV
Globo vídeos
Facebook
YouTube
Google Maps

Requirements

Perl
Squid >= 3.5
squidGuard or equivalent url_rewrite_program (optional)

KOM

I gave up on squid caching HTTPS years ago. My hit rate was never better than 7%. I didn't have the time to immerse myself in squid to figure out how to handle dynamic requests, store_ids and all that other stuff. Besides, this isn't the 90's anymore and bandwidth is generally plentiful, so caching isn't an important requirement anymore. I use squid now as a base for squidguard so I can filter URLs.

nuclearstrength

i get that, it's also basically useless since the cdn are doing that job already, the data is served from the closest point, I also see youtube has some dynamic adaptation going on too and for sure squid would mess with it.

still, i am getting a 0.03% hit rate with ssl inspection (full splice) enabled and working, I am a single user tho and it's been active for a day only.

You use no custom_refresh at all? I browse reddit a lot, anything reddit related that you guys know of?

KOM

No, I don't use custom_refresh. I don't even have a disk cache defined anymore. Also, I'm using squid in explicit mode with WPAD, not transparent with SSL interception.

nuclearstrength

thanks, i suspected as much given what people post about Squid, it's use as a cache it's kinda limited nowadays.

aGeekhere

this is the best I could do https://forum.netgate.com/topic/99613/improve-custom-refresh-pattern

KOM

@aGeekhere I'd be curious to hear what hit rates you're getting.

aGeekhere

it really depends, but like 5 to 10%

KOM

Hmm, seems hardly worth the effort. I was getting 4-7% with a basic config. I suppose it depends a lot on where you're going, their content, etc.

aGeekhere

It helps for OS updates, so if you have windows, mac os and linux. It would be really helpful if it could cache steam updates (but i have not been able to get that to work). My knowledge in refresh pattens is very limited, what would be an idea is have a community maintained custom refresh patten, where more skill users in the area could update the list and improve it. Maybe have an update button in squid to download the list when there is a new update.

kklouzal

Bandwidth is plentiful unless you're with a provider that has a strict bandwidth cap and high overage charges.

My household generally falls just short of our 1TB bandwidth cap every month by around 50GB. Occasionally we go over, and when we do it's by a lot causing a high penalty on our bill. It's not economical at this time to pay for an unlimited bandwidth plan, maybe if we start going over the 1TB cap more often then yes.

Nonetheless I have experimented with squid caching in the past but achieved a very low hit rate. This was with using WPAD to access HTTPS traffic.

Does anyone have more information on setting up a squid cache for more modern dynamic content like facebook, youtube, etc..? I think anything over a 10% hit rate would be worth the effort. You need to realize users don't view the same content day in and day out, they are always finding new content to keep themselves occupied therefore low hit rates are to be expected.

aGeekhere

Try https://forum.netgate.com/topic/99613/improve-custom-refresh-pattern

Leomarbatista

@nuclearstrength friend these rules is 5 years ago