Squid + https
-
I want to try squid on pfSense.
Since https connections encrypt everything including the URLs so none, not even the ISP or the local router, can see anything, how does squid manages to get the entire URL in order to cache it?
Sorry if it's a noob question...
-
hi,
Good Squid setup takes a lot of time and effort...
the answer to your question is....... an intermediate cert. uses for https - (MITM)like:
-
@DaddyGo But how does it do that exactly?
-
as you do, when you browse with Firefox/Chrome, etc. and then examine inside (https) and with intermediate Cert. pass behind the proxy
it has to be said that in many cases this only works with a problem, for example at bank and government sites
-
@DaddyGo But unlike the browser which has all the data, when it reaches the pfSense router it's all already encrypted. I'm sorry but I don't understand exactly how it's possible or allowed lol
Actually these all the questions (I hope) I have about Squid:
- How does squid's ssl interception/MITM work exactly?
- When a client asks for a cached https page squid needs to serve it with a new certificate. Could that be a trusted certificate, in order to not have to install certificates on every client?
- Does every request for a cached content needs a different certificate or something like that?
- What happens when a client received a cached content but now request or does something new in that website? Obviously the new request/action which is not cached will try to reach the actual website but with what certificate/key, since a different one was signed by squid. Will it simply create a new connection/session with that website and fetch new certs/keys etc.?
I'd appreciate it very much If you could take the time to answer these and bear with me for a little bit.
Thank you,
-
@techtester-m said in Squid + https:
Could that be a trusted certificate, in order to not have to install certificates on every client?
A (your !) cert has to be install don every device that have to use your proxy.
And worse.It could be any certificate,and set to be trusted on every users device.
This enables the browser to actually accept connections when it want to connect to "your-bank.tld" but it receives a reply from "yourpfsense'.tld". Normally, the browser would yell, and signal a big huge intrusion alarm, which is what happens.
Now, the browser accepts silently the fake certificate from the web server it connected to, the device where squid is running. On the squid side, everything is unpacked, and thus readable, and squid connects on your behalf to the real "your-bank.tld" site over SSL as normal. Replies that come back are unpacked on squid's side, and repacked in the SSL connections between squid and your device.Btw : these are my words. I think I'm not far of here.
- https traffic is always 'set' as non cacheable. Browser won't cache https traffic.
-
@Gertjan said in Squid + https:
On the squid side, everything is unpacked, and thus readable, and squid connects on your behalf to the real "your-bank.tld" site over SSL as normal. Replies that come back are unpacked on squid's side, and repacked in the SSL connections between squid and your device.
I understand the part where Squid is the middle man and everything but how does it know what the client has asked from the real website in order to ask it on his behalf and cache it (https)? The first time a webpage is visited by the client it goes to the real one, right? Unless all requests go through Squid/MITM?
What I fail to understand is where/when/how Squid can "decrypt/unpack" the https traffic of the client? Due to the fact that the client has installed that cert or...?
-
I see you are quite enthusiastic in Squid theme.
Do you want to learn?I have a well-prepared acquaintance (Marcelo) here on the forum, can help you
(he recently wrote that he have more time, because the COVID anyway)I can ping here @mcury... and if he have time you can learn a lot from him
-
@DaddyGo I just like to know everything (almost, within the reasonable limits of my understanding) about what I'm working on, using, dealing with etc. haha
I hate the "It just works, so don't touch it" approach :) -
@techtester-m said in Squid + https:
I hate the "It just works, so don't touch it" approach :)
Perfectly good approach.
I think Marcelo will show up soon, if only not spend his vacation now. -
@DaddyGo That's the visualization of my question haha...
-
I understand your question, but now you are next...
the best way to learn, if you do everything yourself and just get guidance from colleagueshttps://www.howtoforge.com/filtering-https-traffic-with-squid
https://wiki.squid-cache.org/Features/SslBumpof course have to run a test copy of Squid and practice the steps and watch what happens
if you get completely stuck, then it is our job to help
-
@DaddyGo Ok...I've read it. So in order to intercept Squid does exactly this:
Correct?
-
exactly, somehow so
now the test time can come
(if you use it at home, create a test environment)you will run into a lot of problems, mainly with Android, iOS devices, so you’ll see what you need to shape
next dose of curriculum (for example):
http://www.webdnstools.com/articles/squid-proxy-whitelistI give a sample file:
https://www.dropbox.com/sh/pp3m9reh2eikks2/AADIJmyBKZ4cZKOqs3A-eBGva?dl=0 -
@DaddyGo Ok...
Few more if you may :)As I understand it so far it goes something like this (and please correct me where I'm wrong) -
(1) Client --> Squid interception of PK, aka MITM --> destination.
(2) It will do the above only for what is in its whitelist.
(3) A private self created CA has to be installed on every client that needs to use the Squid proxy, so it could dynamically generate certificates. Using Let's Encrypt or anything of the sort won't work here and also be a big no no.
(4) Everything in the whitelist will be cached (up to X G/MB limit), including files.
(5) A destination that is not on the whitelist will be blocked? If so then I think I prefer DNS blocking.
(6) Is their an option to cache/intercept only certain destinations (whitelist or something) and treat all the others as usual without "squidding" them?
-
- Yes
- no, these are exceptions as they require special rules and accesses
(otherwise they do not work)
note the whole world cannot be whitelisted - Yes, with this method... ("Using Let's Encrypt or anything of the sort won't work here and also be a big no no.") yes
- half true, (including files - it would require awful storage capacity - depending on what file we are talking about)
- No, the whitelist is necessary, because of the above, I also recommend DNS blocking in SOHO environment...
(Squid is a big boys game, because of big systems and proxy capabilities)
6.Yes by bypassing the proxy (ACL, PAC,etc)
https://wiki.squid-cache.org/ConfigExamples/Authenticate/Bypass
-
@DaddyGo said in Squid + https:
no, these are exceptions as they require special rules and accesses
(otherwise they do not work)So what do you do with exceptions? Let them bypass squid and go as usual or "fine tuning" them?
@DaddyGo said in Squid + https:
half true, (including files - it would require awful storage capacity - depending on what file we are talking about)
For the sake of learning let's say there's enough storage, even though an organized NAS would be MUCH better for any business/company. Does squid has the option to set a maximum X G/MB for caching? Sort of a circular caching where a new file would push and old one(s) when that maximum X is met.
@DaddyGo said in Squid + https:
No, the whitelist is necessary, because of the above
Perhaps it's only semantics here but the whitelist in this case is what you want Squid to authenticate for you (MITM) and cache and has nothing to do with restrictions, blocking etc.?
Ok...Semantics again. Squid does both access restrictions AND caching? -
There are still some sites left on the Internet that accept http (NON SSL) requests.
Install Squid, and focus on the classic 'intercept port 80' or 'use a wpad file' and have your browser been 'redirected to a 8080 proxy port'.
This way, you can see how it works in the old days- I guess, for that type of set-up, many examples are available.When you feel up to it, include 'https' in the addition.
Take note : it's an on-going battle. I wonder how one proxies the sites that have HSTS ( https://fr.wikipedia.org/wiki/HTTP_Strict_Transport_Security - it's yet another anti MITM ) activated. I use HSTS on all my sites.
-
@Gertjan said in Squid + https:
Take note : it's an on-going battle. I wonder how one proxies the sites that have HSTS ( https://fr.wikipedia.org/wiki/HTTP_Strict_Transport_Security - it's yet another anti MITM ) activated. I use HSTS on all my sites.
exactly, I referred to problems such as those described by @Gertjan
and then there are the mobile OP systems....etc.never give up, but I think Squid and similar solutions are coming to an end
or "fine tuning" them? - yes otherwise behind a proxy, WhatsApp, etc. will be dead
(because it is not only one encrypted process going on in the background)like Apple stuff:
*.phobos.itunes-apple.com.akadns.net
*.gateway.push-apple.com.akadns.net
*.ax.itunes.apple.com
*.mesu.apple.com
*.phobos.apple.com
*.albert.gcsis-apple.com.akadns.net
*.ax.init.itunes.apple.com
*.init.itunes.apple.com
*.oscp.apple.com
*.deploy.static.akamaitechnologies.com
*.itunes.apple.com.edgekey.net
*.swcdn.apple.com
*.swdownload.apple.com
*.swquery.apple.com
*.swscan.apple.com
*.appldnld.apple.com
*.suconfig.apple.com
*.serverstatus.apple.com
*.gs.apple.com
*.apple.com
*.updates.cdn-apple.com@techtester-m "For the sake of learning let's say there's enough storage, even though an organized NAS would be MUCH better for any business/company. Does squid has the option to set a maximum X G/MB for caching? Sort of a circular caching where a new file would push and old one(s) when that maximum X is met."
- just think of a multi-gig Windows update here
and has nothing to do with restrictions, blocking etc.?
don't think of Squid as pfBlockerNG for example
and we didn’t even talk about SquidGuard then++++edit:
just watch the Squid cache directory... -
@Gertjan @DaddyGo Ok...as I see it, the headache that comes with Squid/MITM solutions just in order to cache data, makes it not so worth it. A business would prefer solutions like NAS or a local server with let's say Elasticsearch or something like that. If an employee needs a certain file first search the local storage.
For simply caching web pages it's almost redundant and would make sense only in places without a broadband or for very large organizations.
-
@DaddyGo @Gertjan A MITM can intercept PKs easily, but he would need his CA to be installed on the client's side.
But what prevents a MITM to simply forward you the certificate it received from the original destination?Such an attempt would fail because the client's browser would see the mismatch between the cert and the source address/domain of the MITM, right? Feels like such a basic question but still...
I just wanna make sure I fully understand it.
From what I've read the combination between RSA encryption/PKs & Certificate authentication is as secure as it gets so long the end user hasn't installed any CA other than what is already trusted by the browser.
-
nowadays, due to the development of physical hardware, this caching is not really interesting, on small-scale systems
a proxy can help for clusters with hundreds of users if you always have to load the same content
and behind a proxy, a lot of things can be well hidden
but as mentioned https and all its advantages, it becomes a disadvantage here
everyone is try hiding something -
@DaddyGo said in Squid + https:
and behind a proxy, a lot of things can be well hidden
A VPN server would do the job without MITM stuff :)
-
In the past, I worked as a sysadmin for a company, with 2 800-2 900 direct workstations
well then and there the proxy was good, it kept the stupidity of the employees in check
the VPN also obscures everything, just a little differently than, for example, the intranet proxy on a campus network
-
@DaddyGo Is there a good way to prevent http access and force https without installing browser add-ons?
via pfSense, Pi-hole etc. maybe?
Also, things like squid are great when you wanna know the content of whatever your employees do centralized in one place, but I guess there's many software solutions to do that on each device owned by the company.
-
@techtester-m said in Squid + https:
Is there a good way to prevent http access and force https without installing browser add-ons?
via pfSense, Pi-hole etc. maybe?slowly wear out the "http" ...
disable port 80, though that would still be crazy, hahahaI think the pi-hole is forgettable, here is the great Unbound + pfBlockerNG pair
@techtester-m "Also, things like squid are great when you wanna know the content of whatever your employees do centralized in one place, but I guess there's many software solutions to do that on each device owned by the company."
don't think, that...
big companies are the driest jobs, even though I worked as an outside worker...hmmmmthey invest in an IT structure and after that they think it will work forever
I have a friend who works for a very large company (their names are not public, because you would faint if you knew) and still runs Win7 on VM (hypv) enviroment with Citrix
as management regrets the money, for out of date Win7 replacements -
@DaddyGo said in Squid + https:
disable port 80, though that would still be crazy
That's too much haha. I want to do what "https everywhere" does but on the pfsense or other end equipment and without breaking internal pc/os stuff. On a DNS level basically but on the other hand how do you determine what comes from browsing and what not...
@DaddyGo said in Squid + https:
I think the pi-hole is forgettable, here is the great Unbound + pfBlockerNG pair
DNS is probably one of the only services I wouldn't mind to "outsource" from the firewall. Especially when Pi-hole is much more organized, visual and has a bunch of people who all they do everyday 24/7 is trying improve it.
An entire project dedicated solely to DNS.
pfBlockerNG is at the end of the day a package. I'd still use it as an IP blocker though :)@DaddyGo said in Squid + https:
their names are not public, because you would faint if you knew
Crazy! With all the money these companies have they still try to save as much as possible and be cheap, even when it's in their best interest to invest it hahaha unbelievable.
-
Hmmm HTTP...
with http you you can't start anything this is a webserver question, for example on a web server in the .htaccess file all http must be redirected to https
plus Let'sE certificate and that's aboutonly web masters are lazy, cca. 10 minutes job it
you don't think so,.. the Pi-hole is just popular and beautiful nothing more
(I apologize to all the Pi-hole fans) - perfect for raspberry piUnbound + DoT + DNSSEC + CF DNS servers and pfBlockerNG -devel, it is a professional solution
+++edit:
people tend to overcomplicate everything even though it is simple and everything is included in pfSense (more or less) -
@DaddyGo said in Squid + https:
Unbound + DoT + DNSSEC + CF DNS servers and pfBlockerNG
That's exactly what I use! But...except for the fact that it's AIO solution + IP blocking what would be another advantage of pfBlocker over Pi-Hole (DNS wise)? Also, I've seen more lists and just info in general about Pi-Hole and as I said it seems much more organized, visual etc.
One last question if you may.
What happens when/if a MITM has its PK signed by a trusted CA? Would it still break because the browser will check from which domain/address the data came from resulting in a mismatch between what the client requested and the source? -
pfBlockerNG does exactly the same thing as Pi-hole, only better
mainly because it does with a professional DNS resolver, which is Unbound
don't be fooled by the look...(GUI, etc) - Pi-holeprofessional stuff usually doesn't even have a GUI, just a CLI
just look at Brocade, Juniper, etc. switches or Cisco enterprise devices
pfBlockerNG has a clean design and what is good can be set even by those who don't understand it so muchthe browser only sees the Squid internal certificate behind the proxy
he does not even know the existence of the original certificate -
@DaddyGo said in Squid + https:
professional stuff usually doesn't even have a GUI, just a CLI
I know, but it's not only the GUI part here. It's an entire dedicated solution for one single task - "DNSing", but I'll search for some good lists for pfBlockerNG and see.
@DaddyGo said in Squid + https:
mainly because it does with a professional DNS resolver
I forward the requests anyway, because the 13 DNS root servers don't support DoT (yet).
With Pi-Hole it would be DoH because they don't support DoT yet for some reason. -
@DaddyGo @Gertjan Do any of you know how does a certificate validation works exactly, in technical details?
I just posted it on security.stackexchange but decided to try and ask you guys as well :)I've read this post - https://security.stackexchange.com/questions/135401/certificate-validation,
where it says this : "...an attacker can still take the whole signed content and present it to you but won't be able to change any details or the signature won't match."My questions are these:
(1) How does the browser validates the details of a certificate and see its content like: domain name etc.? Knowing that the cert is encrypted.
(2) If a MITM wanna change anything he would need to decrypt the cert, which is impossible, or simply change what he wants but then fail at the browser because, as mentioned above, the signature won't match?Thank you,
-
@techtester-m said in Squid + https:
but I'll search for some good lists for pfBlockerNG and see.
in the "devel" version have a lot of built-in lists, but I also have a lot of collected lists
don't be fooled by the word = DEVEL - this current / actual version
-
@techtester-m said in Squid + https:
Do any of you know how does a certificate validation works exactly, in technical details?
Yep.
Have a look at least 10 videos from this list https://www.youtube.com/results?search_query=https+certificate+how+it+worksAdded to that :
In a perfect world, DNS is non spoofable (because you use the Resolver and DNSSEC ;) ), your app will know for sure what the real IP of a site is.The important things :
In a certificate, you find the dates of validity,
The CA, used to certify that the certicate is valid. An OS, web browser, app, whatever, sues a build in list with known trusted CA's - and take note : you can add your own CA .... you can become a CA for your own network.
In a certificate is embedded the domain name or even a root domain name : -
@Gertjan @DaddyGo I already knew about the certificate itself and I myself have signed or got signed by a trusted one etc. All I wanted to know is the security mechanism(s) that prevent MITM. Yesterday I did more reading on the subject and on digital signatures as well and understood everything I wanted to understand and know. It took a few hours over 1-2 days but Hopefully it'll be stored in my head forever haha
As I understand it - Unless a MITM got the end user to trust his CA, the attack will fail.
It would fail because:
(A) MITM would try and send his own PK and the browser will give warning.
(B) MITM must send his own PubK in order to encrypt/decrypt the content with his Private Key which is the key to everything here basically.
(C) If MITM changes anything in the message that comes with the certificate, like the domain name etc. the signature verifying algorithm would fail and the browser will give warning.
(D) No CA would sign fraudulent certificate unless they wanna go bankrupt and get sued.
(E) Even if a MITM sends the end user the exact same certificate he himself got from let's say Google then the browser might think for the first time that it's talking to Google but the next communication would fail because it would be encrypted with Google's PubK and could be decrypted only with Google's Private-Key so the MITM won't be able to read anything, there will be a connection time out and the attack would fail.
(F) I'm sure there are few more reasons...Thank you guys for all your help, input and knowledge. A pleasure as always :)
-
Hello,
Can you help me find solution?
I've added CA of our domain controller in System > Cert Manager
But in squid SSL MITM i can't select this CA and enable SSL filtering. Only "None" can select.
How i can set up domain CA for enabling SSL filtering ?
Or pfsense can only accept self made cert from internal CA?
Thanks in advance. -
@viberua I'm no expert at all on the matter but it tells you to create your own on pfSense, because it needs to be able to create certs on demand and locally...I think, but again..no expert.
-
Never used squid before, but I guess a CA should be created first.
Here :Then, based on the CA, you create your certs :
These certs can be used in OpenVPN, FreeRadius, the pfSense GUI, etc.
CA's can't be use directly, except for signing (your own) certs. -
@Gertjan said in Squid + https:
Never used squid before, but I guess a CA should be created first.
Here :exactly,
use the pfSense certificate builder and then it will appear in Squid settingsthen you can also export it for installation on external devices
like:
-
@DaddyGo so if i don't want create new CA because i already have one, then i can't use this external CA cert in MITM?