Squid + https

[techtester-m]

@Gertjan said in Squid + https:

On the squid side, everything is unpacked, and thus readable, and squid connects on your behalf to the real "your-bank.tld" site over SSL as normal. Replies that come back are unpacked on squid's side, and repacked in the SSL connections between squid and your device.

I understand the part where Squid is the middle man and everything but how does it know what the client has asked from the real website in order to ask it on his behalf and cache it (https)? The first time a webpage is visited by the client it goes to the real one, right? Unless all requests go through Squid/MITM?

What I fail to understand is where/when/how Squid can "decrypt/unpack" the https traffic of the client? Due to the fact that the client has installed that cert or...?

[DaddyGo]

@techtester-m

I see you are quite enthusiastic in Squid theme.
Do you want to learn?

I have a well-prepared acquaintance (Marcelo) here on the forum, can help you
(he recently wrote that he have more time, because the COVID anyway)

I can ping here @mcury... and if he have time you can learn a lot from him

[techtester-m]

@DaddyGo I just like to know everything (almost, within the reasonable limits of my understanding) about what I'm working on, using, dealing with etc. haha
I hate the "It just works, so don't touch it" approach :)

[DaddyGo]

@techtester-m said in Squid + https:

I hate the "It just works, so don't touch it" approach :)

Perfectly good approach.
I think Marcelo will show up soon, if only not spend his vacation now.

[techtester-m]

@DaddyGo That's the visualization of my question haha...

[DaddyGo]

@techtester-m

I understand your question, but now you are next...
the best way to learn, if you do everything yourself and just get guidance from colleagues

https://www.howtoforge.com/filtering-https-traffic-with-squid
https://wiki.squid-cache.org/Features/SslBump

of course have to run a test copy of Squid and practice the steps and watch what happens

if you get completely stuck, then it is our job to help

[techtester-m]

@DaddyGo Ok...I've read it. So in order to intercept Squid does exactly this:

Correct?

[DaddyGo]

@techtester-m

exactly, somehow so
now the test time can come
(if you use it at home, create a test environment)

you will run into a lot of problems, mainly with Android, iOS devices, so you’ll see what you need to shape

next dose of curriculum (for example):
http://www.webdnstools.com/articles/squid-proxy-whitelist

I give a sample file:
https://www.dropbox.com/sh/pp3m9reh2eikks2/AADIJmyBKZ4cZKOqs3A-eBGva?dl=0

[techtester-m]

@DaddyGo Ok...
Few more if you may :)

As I understand it so far it goes something like this (and please correct me where I'm wrong) -

(1) Client --> Squid interception of PK, aka MITM --> destination.

(2) It will do the above only for what is in its whitelist.

(3) A private self created CA has to be installed on every client that needs to use the Squid proxy, so it could dynamically generate certificates. Using Let's Encrypt or anything of the sort won't work here and also be a big no no.

(4) Everything in the whitelist will be cached (up to X G/MB limit), including files.

(5) A destination that is not on the whitelist will be blocked? If so then I think I prefer DNS blocking.

(6) Is their an option to cache/intercept only certain destinations (whitelist or something) and treat all the others as usual without "squidding" them?

[DaddyGo]

@techtester-m

1. Yes
2. no, these are exceptions as they require special rules and accesses
   (otherwise they do not work)
   note the whole world cannot be whitelisted
3. Yes, with this method... ("Using Let's Encrypt or anything of the sort won't work here and also be a big no no.") yes

4. half true, (including files - it would require awful storage capacity - depending on what file we are talking about)
5. No, the whitelist is necessary, because of the above, I also recommend DNS blocking in SOHO environment...
   (Squid is a big boys game, because of big systems and proxy capabilities)
   6.Yes by bypassing the proxy (ACL, PAC,etc)
   https://wiki.squid-cache.org/ConfigExamples/Authenticate/Bypass

[techtester-m]

@DaddyGo said in Squid + https:

no, these are exceptions as they require special rules and accesses
(otherwise they do not work)

So what do you do with exceptions? Let them bypass squid and go as usual or "fine tuning" them?

@DaddyGo said in Squid + https:

half true, (including files - it would require awful storage capacity - depending on what file we are talking about)

For the sake of learning let's say there's enough storage, even though an organized NAS would be MUCH better for any business/company. Does squid has the option to set a maximum X G/MB for caching? Sort of a circular caching where a new file would push and old one(s) when that maximum X is met.

@DaddyGo said in Squid + https:

No, the whitelist is necessary, because of the above

Perhaps it's only semantics here but the whitelist in this case is what you want Squid to authenticate for you (MITM) and cache and has nothing to do with restrictions, blocking etc.?

Ok...Semantics again. Squid does both access restrictions AND caching?

[Gertjan]

There are still some sites left on the Internet that accept http (NON SSL) requests.
Install Squid, and focus on the classic 'intercept port 80' or 'use a wpad file' and have your browser been 'redirected to a 8080 proxy port'.
This way, you can see how it works in the old days- I guess, for that type of set-up, many examples are available.

When you feel up to it, include 'https' in the addition.

Take note : it's an on-going battle. I wonder how one proxies the sites that have HSTS ( https://fr.wikipedia.org/wiki/HTTP_Strict_Transport_Security - it's yet another anti MITM ) activated. I use HSTS on all my sites.

[DaddyGo]

@Gertjan said in Squid + https:

Take note : it's an on-going battle. I wonder how one proxies the sites that have HSTS ( https://fr.wikipedia.org/wiki/HTTP_Strict_Transport_Security - it's yet another anti MITM ) activated. I use HSTS on all my sites.

exactly, I referred to problems such as those described by @Gertjan
and then there are the mobile OP systems....etc.

never give up, but I think Squid and similar solutions are coming to an end

or "fine tuning" them? - yes otherwise behind a proxy, WhatsApp, etc. will be dead
(because it is not only one encrypted process going on in the background)

like Apple stuff:

*.phobos.itunes-apple.com.akadns.net
*.gateway.push-apple.com.akadns.net
*.ax.itunes.apple.com
*.mesu.apple.com
*.phobos.apple.com
*.albert.gcsis-apple.com.akadns.net
*.ax.init.itunes.apple.com
*.init.itunes.apple.com
*.oscp.apple.com
*.deploy.static.akamaitechnologies.com
*.itunes.apple.com.edgekey.net
*.swcdn.apple.com
*.swdownload.apple.com
*.swquery.apple.com
*.swscan.apple.com
*.appldnld.apple.com
*.suconfig.apple.com
*.serverstatus.apple.com
*.gs.apple.com
*.apple.com
*.updates.cdn-apple.com

@techtester-m "For the sake of learning let's say there's enough storage, even though an organized NAS would be MUCH better for any business/company. Does squid has the option to set a maximum X G/MB for caching? Sort of a circular caching where a new file would push and old one(s) when that maximum X is met."

• just think of a multi-gig Windows update here

and has nothing to do with restrictions, blocking etc.?

don't think of Squid as pfBlockerNG for example
and we didn’t even talk about SquidGuard then

++++edit:
just watch the Squid cache directory...

[techtester-m]

@Gertjan @DaddyGo Ok...as I see it, the headache that comes with Squid/MITM solutions just in order to cache data, makes it not so worth it. A business would prefer solutions like NAS or a local server with let's say Elasticsearch or something like that. If an employee needs a certain file first search the local storage.

For simply caching web pages it's almost redundant and would make sense only in places without a broadband or for very large organizations.

[techtester-m]

@DaddyGo @Gertjan A MITM can intercept PKs easily, but he would need his CA to be installed on the client's side.
But what prevents a MITM to simply forward you the certificate it received from the original destination?

Such an attempt would fail because the client's browser would see the mismatch between the cert and the source address/domain of the MITM, right? Feels like such a basic question but still...

I just wanna make sure I fully understand it.

From what I've read the combination between RSA encryption/PKs & Certificate authentication is as secure as it gets so long the end user hasn't installed any CA other than what is already trusted by the browser.

[DaddyGo]

@techtester-m

nowadays, due to the development of physical hardware, this caching is not really interesting, on small-scale systems

a proxy can help for clusters with hundreds of users if you always have to load the same content

and behind a proxy, a lot of things can be well hidden

but as mentioned https and all its advantages, it becomes a disadvantage here
everyone is try hiding something

[techtester-m]

@DaddyGo said in Squid + https:

and behind a proxy, a lot of things can be well hidden

A VPN server would do the job without MITM stuff :)

[DaddyGo]

@techtester-m

In the past, I worked as a sysadmin for a company, with 2 800-2 900 direct workstations

well then and there the proxy was good, it kept the stupidity of the employees in check

the VPN also obscures everything, just a little differently than, for example, the intranet proxy on a campus network

[techtester-m]

@DaddyGo Is there a good way to prevent http access and force https without installing browser add-ons?

via pfSense, Pi-hole etc. maybe?

Also, things like squid are great when you wanna know the content of whatever your employees do centralized in one place, but I guess there's many software solutions to do that on each device owned by the company.

[DaddyGo]

@techtester-m said in Squid + https:

Is there a good way to prevent http access and force https without installing browser add-ons?
via pfSense, Pi-hole etc. maybe?

slowly wear out the "http" ...
disable port 80, though that would still be crazy, hahaha

I think the pi-hole is forgettable, here is the great Unbound + pfBlockerNG pair

@techtester-m "Also, things like squid are great when you wanna know the content of whatever your employees do centralized in one place, but I guess there's many software solutions to do that on each device owned by the company."

don't think, that...
big companies are the driest jobs, even though I worked as an outside worker...hmmmm

they invest in an IT structure and after that they think it will work forever

I have a friend who works for a very large company (their names are not public, because you would faint if you knew) and still runs Win7 on VM (hypv) enviroment with Citrix
as management regrets the money, for out of date Win7 replacements