Squid alternative for bandwidth control?

[Cabledude]

Hey all,
A couple of years ago, I heard about Squid and how it can cache websites, allowing to keep bandwidth down and make internet browsing feel more responsive.
Now that Im finally taking steps to deploy, I read about security vulnerabilities and that netgate apparently decided to discourage us from using Squid.

So two questions before I delve into this matter further:

1. If I use Squid exclusively for website caching, are the security issues relevant?
2. Is there a different way to perform website caching?
   My setup is SG-2100, Synology NAS’es (on site and offsite), unifi and macOS iOS clients.

Thank you

[JonathanLee]

@Cabledude said in Squid alternative for bandwidth control?:

I read about security vulnerabilities

The updated version of squid mitigated all the risk. just a FYI

[aGeekhere]

@Cabledude
I would first try https://lancache.net/ in a docker container for caching.

[Cabledude]

@JonathanLee said in Squid alternative for bandwidth control?:

@Cabledude said in Squid alternative for bandwidth control?:

I read about security vulnerabilities

The updated version of squid mitigated all the risk. just a FYI

Okay, thank you, I didn't know that. But then I wonder why Netgate still strongly discourages us to use it. I'll have to dig a little deeper or just ask Netgate why.

[Gertjan]

@Cabledude said in Squid alternative for bandwidth control?:

A couple of years ago, I heard about Squid and how it can cache websites, allowing to keep bandwidth down and make internet browsing feel more responsive.

First, let's make sure that this is what you want.
Add this rule on your LAN interface first :

and now wait for a day or so.
Be close to pfSense to undo this rule, as people will come after you, and things will probably go downhill fast.
You played with the idea to use Squid, so see this as what will happen more often in the future.

If nothing happens, you are ready for squid : you've proven that http sites are ok for you, and you don't need this modern https stuff.
( TCP port 443 = https)

Btw : I didn't say Squid can't handle https (TLS) traffic, it can, but you have to (baby) assists it constantly.
On reddit there are some threads about "Is squid still a good idea ?", look them up, and you'll understand that we, the common mortals, stay away from squid as far as possible. Life is just to short.

All the other traffic : streaming, most websites, apps etc etc etc etc can't be cached anymore. Nearly every web page is specially made for you, "for your eyes only" at the very moment you asked for it.
Moments later, no web browser will accept that cached content anymore.

Imho, the one and only cache that is still in use today is handled by your Microsoft OS : If you have more then one Windows OS on your LAN, then you can set them both up so that the can share what they have downloaded. This means that one PC downloads a big update, and the second and other can get the update from this device.

In short : It's not hard to deal with the bandwidth question.
If needed, allocate for every device a upper limit : Firewall > Traffic Shaper > By Interface
Need more ? call your ISP.

[Uglybrian]

A little more insight.
https://www.netgate.com/blog/deprecation-of-squid-add-on-package-for-pfsense-software

[aGeekhere]

@Cabledude If you do want to use squid for caching you can try this setup
Install squid
enable Transparent HTTP Proxy

NOTE: If you use both WPAD and Transparent proxy the DIRECT traffic from the WPAD will go to the Transparent proxy, you could leave Transparent HTTP Proxy off when using WPAD

Bypass Proxy for Private Address Destination
Enable SSL filtering
SSL/MITM Mode: Custom
CA: yourcrt
under show advanced Custom Options (SSL/MITM)

email_err_data off
# 1. Allow Squid to fetch the rest of the file in the background 
# even if the user only requested a part of it.
range_offset_limit 512 KB all  # or "none" only for specific ACLs if possible
quick_abort_min 0 KB
quick_abort_pct 95  # default; aborts if <5% left when client disconnects
# ===================== GLOBAL SETTINGS =====================
tls_outgoing_options min-version=1.2
sslproxy_cert_error allow all
on_unsupported_protocol tunnel all

# ===================== ACLs =====================
acl step1 at_step SslBump1
acl step2 at_step SslBump2
acl step3 at_step SslBump3

acl blockSites ssl::server_name "/home/customBlockUrls.txt"
acl bypassUsers src "/home/bypassUsers.txt"
acl excludeSites ssl::server_name "/home/excludeSites.txt"
acl bumpSites ssl::server_name "/home/bumpSites.txt"
acl bumpUsers src "/home/bumpUsers.txt"

# ===================== DIRECT DECISION =====================
# if you want excludeSites sites to go direct use this and remove ssl_bump splice step2 excludeSites
# always_direct allow excludeSites

# ===================== SSL BUMP FLOW =====================

# --- Step 1 ---
ssl_bump splice step1 bypassUsers
ssl_bump peek step1

# --- Step 2 ---
# 1. Block first
ssl_bump terminate step2 blockSites

# 2. IMMEDIATE SPLICE for Pinning (Fixes the NONE_NONE/200 errors)
# We splice these at Step 2 so Squid NEVER sends its own cert to them.
ssl_bump splice step2 excludeSites

# 3. STARE for the rest (This allows the "Splice on Error" logic for unknown sites)
ssl_bump stare step2 bumpUsers
ssl_bump stare step2 bumpSites

# Default for Step 2
ssl_bump splice step2 all

# --- Step 3 ---
ssl_bump bump step3 bumpUsers
ssl_bump bump step3 bumpSites

# Safe default
ssl_bump splice all

NOTE: make sure you make the files
"/home/customBlockUrls.txt"
"/home/bypassUsers.txt"
"/home/excludeSites.txt"
"/home/bumpSites.txt"
"/home/bumpUsers.txt"

For caching add domains to "/home/bumpSites.txt" that you want to try and bump
Use the following as a start

.githubusercontent.com
.7-zip.org
.us.download.nvidia.com
.smetrics.nvidia.com

Every time you make an edit here you need save squid and restart it.
Note you cannot just bump everything (you can test this by adding your ip to bumpUsers.txt ) because if you do a lot of sites will break. What is best is to see what content your users are downloading often (nvidia, intel, amd drivers etc) and only bump those domains after you have tested that they can be bumped without issues.

Go to Package Proxy Server: Cache Management Local Cache
set your Squid Hard Disk Cache Settings
enable Cache Dynamic Content
and use the new REFRESH PATTERNS that i made https://github.com/mmd123/squid-cache-dynamic_refresh-list/blob/master/refresh_pattern_squid_v7_3
It will also work for pfsense squid

I suggest to also use a WPAD, you can install the WPAD package at https://github.com/freitasbr/unofficial-pfsense-wpad and use this as a starter guide

function FindProxyForURL(url, host) 
{
    // No proxy for specific local domain
    if (shExpMatch(host, "pfsense.pfsensedomain.local")) return "DIRECT";

	
	// You may also want to bypass local/hostnames and local intranet:
	if (isPlainHostName(host) || shExpMatch(host, "*.local") || dnsDomainIs(host, ".localdomain")) 
	{
		return "DIRECT";
	}

    return "PROXY 192.168.1.1:3128";
}

Note you can add rules here to bypass the proxy for domains so that they do not even touch squid.

I would not block port 80 or 443 for now, see how this goes

Now to test, go and download the latest nvidia drivers and see if you get a hit you should see

TCP_HIT/200	https://us.download.nvidia.com/Windows/581.80/581.80-desktop-win10-win11-64bit-international-dch-whql.exe

If you set this all up with the WPAD then
On PC internet - Enable Automatically detect settings
On mobile devices use - Automatically detect settings - url http://192.168.1.1/wpad.dat

And there you have it, though as time goes on it may become harder to cache things.
I still recommend to also set up https://lancache.net/ as well as they can work together.

[Cabledude]

@Gertjan said in Squid alternative for bandwidth control?:

@Cabledude said in Squid alternative for bandwidth control?:

A couple of years ago, I heard about Squid

First, let's make sure that this is what you want.
Add this rule on your LAN interface first :

Okay, thank you for this heads up. My post was meant for exactly that: ask first before starting.

Based on your message and also those of @Uglybrian and @aGeekhere I will pass.

A special thanks to @aGeekhere for the writeup, so much work and now I don't even proceed setting it up, I'm half embarrassed half sorry to say.

So thank you everyone for providing illumination.

[JonathanLee]

@Cabledude remeber to block QUIC or udp over 443 with this setup also because traffic will quickly try to do that after.

also you can use access control lists for rep_mime_type too so you can block doh right when its seen with ssl/mitm mode

acl deny_rep_mime_doh rep_mime_type application/dns-message
acl deny_rep_mime_doh rep_mime_type text/dns
acl deny_rep_mime_doh rep_mime_type application/dns+json
http_reply_access deny deny_rep_mime_doh

acl doh_rfc8484 urlpath_regex -i ^/dns-query
acl doh_rfc8484 urlpath_regex -i dns=
acl doh_rfc8484 urlpath_regex -i ^/resolve
acl doh_group any-of deny_rep_mime_doh doh_rfc8484

then simply

acl terminate_group any-of deny_rep_mime_doh doh_rfc8484

and something like this with your bump order

ssl_bump terminate all block_hours
miss_access deny no_miss active_use
ssl_bump peek step1
ssl_bump splice no_bump_pfsense active_use
ssl_bump terminate terminate_group
ssl_bump splice splice_main active_use
ssl_bump bump bump_main active_use
acl activated note active_use true
ssl_bump terminate !activated

bingo no giant doh lists it will catch a ton of them its the kill short for doh abuse

[Cabledude]

@JonathanLee said in Squid alternative for bandwidth control?:

@Cabledude remeber to block QUIC or udp over 443 with this setup also because traffic will quickly try to do that after.

also you can use access control lists for rep_mime_type too so you can block doh right when its seen with ssl/mitm mode

acl deny_rep_mime_doh rep_mime_type 
...
bingo no giant doh lists it will catch a ton of them its the kill short for doh abuse

Thank you Jonathan, much appreciated, though I decided to pass on this tool, so it's pearls for the hogwarths, which is a dutch saying for gifts given to people that don't make the most of it (or even don't care what's given, but that doesn't apply to me: I appreciate your efforts!)