Custom Options (SSL/MITM) best settings for local cache {RESOLVED ON END}

JonathanLee · Aug 19, 2022, 2:18 PM

Hello Fellow Netgate community members,

Can you please help?

What would be the best custom ssl settings a good local cache with Squid on a 2100 Max Netgate Firewall?

(Image: Hits only show for HTTP currently)

I have created and installed certificates and they seem to be working. I can see traffic generated however I see tunnel a lot in the live logs. When using the "custom option" on SSL you must configure the option in advanced. I am testing custom out. I am hoping you can shine some light on what would be the best settings in a home network, all devices have certificates except the Xbox.

(Image: Port 3129)

I noticed I can not use port 3129 for anything or the internet stops when that is turned on for proxy use, however it works if I still use the 3128 listener port.

(Image: Local settings still set to 3128)

(Image: Port 3128)

I still need transparent proxy for the Xbox use. Palo Alto firewalls can issues and perform certificate use similar to the Netgate system. However the image below shows that it is possible to get HTTPS hits.

Wpad is configured also.

ssl_bump peek all
ssl_bump all

ssl_bumb peek step1
ssl_splice all

ssl_bump peek all
ssl_bump splice all

My Goal is to cache and accelerate more traffic and have larger hit rates. I understand that TSL/SSL is a encrypted tunnel however one can say that encrypted data should still carry some hash values during transmission. Even if it is cached locally it is encrypted data and non readable without the browser and certificate that requested it, the data would be the same once it is delivered and used if required again. Wouldn't that hash value generated for the data that was carried in that encrypted tunnel be the same if that same data was requested again? Can we essentially cache HTTPS data.

Yes Squid is a HTTP, HTTPS, FTP based proxy, plus an accelerator.

I have noticed online that some users are able to cache HTTPS items. However I am still trying to get this to work correctly. Are they using wget?

(Image: Youtube Squid use with many hits for HTTPS for Facebook)

Any recommendations I use to think this was not possible however I have been seeing youtube posts of HTTPS caching working for Squid?

Can you CURL or WGET and save a website and deliver it after the request was made however store it encrypted in the cache? That would make the proxy check the website before it's delivered.

Are my certificates set up incorrectly?

I have also been reading about ecap use over icap for performance gains.

(Image: Squid's Web Page on ECAP use)

JonathanLee · Aug 17, 2022, 4:58 PM

(IMAGE: Best setting I have found so far to match above with HTTPS CACHE WARNING THIS MANY NOT WORK IN ALL AREAS, YOU MUST OWN DEVICES AND INSTALL CERTIFICATES ON THEM TO USE THIS FUNCTIONALITY THIS MIMICS ENTERPRISE FIREWALLS LIKE PALO ALTO SYSTEMS)

This was my best setting for high cache rates with HTTPS

The 192.168.20.11 is a device that can not install a certificate so it is marked as splice only.

(IMAGE: CERTIFICATES WORKING WITH CACHE)

JonathanLee · Aug 19, 2022, 12:46 AM

@jonathanlee

To add to this once you have this running you will not have any Windows updates however everything else works. Wait I found a work around while reading the Squid Forums,

This is the solution create a file inside of the Netgate firewall.

(IMAGE: SSL_BUMP FILE FOR SPLICE ONLY)

Splice only is needed for Windows update and Apple updates, they require their own custom certificate and if you are using a third party certificate like me at home that causes some issues. It's ok simply mark the update sites to splice when they see them, and it works again.

(IMAGE: Custom Options Squid)

My ACL are named Splice_only is based by source IP for my Xbox because it cannot install a certificate and for Amazon Tablet because it has some issues with the updates

My ACL "Access Control Lists" NOSSLIntercept is based off the file I created above. This allows windows updates to splice while still keeping the security of SSL bumping for all other sites.

(Image: Windows update working without errors)

(IMAGE: Make sure you add your proxy to Winhttp for the Windows update to also point to the proxy or it will not work still)

JonathanLee · Aug 19, 2022, 2:15 PM

@jonathanlee You will also need to make some custom adjustments to the Splice list, meaning that it skips SSL Interception for the following sites:

Addition:
You also need to add for apple products:
Use Apple products on enterprise networks
https://support.apple.com/en-us/HT210060

Android products:
Android Enterprise Network Requirements
https://support.google.com/work/android/answer/10513641?hl=en

My list looks like this: Android use and Mac Iphone Macafee Antivirus and some banking sites as an example.
They must skip SSL Interception

#Sites to be spliced
update.microsoft.com
update.microsoft.com.\akadns.net
apple.com
cdn-apple.com
icloud.com
icloud-content.com
itunes.com
mzstatic.com
play.google.com
android.com
google-analytics.com
googleusercontent.com
gstatic.com
gvt1.com
ggpht.com
dl.google.com
dl-ssl.google.com
android.clients.google.com
gvt2.com
gvt3.com
accounts.google.com
accounts.google.us
pki.google.com
clients1.google.com
clients2.google.com
clients3.google.com
clients4.google.com
clients5.google.com
clients6.google.com
connectivitycheck.android.com
mtalk.google.com
mtalk4.google.com
mtalk-staging.google.com
mtalk-dev.google.com
alt1-mtalk.google.com
alt2-mtalk.google.com
alt3-mtalk.google.com
alt4-mtalk.google.com
alt5-mtalk.google.com
alt6-mtalk.google.com
alt7-mtalk.google.com
alt8-mtalk.google.com
android.clients.google.com
device-provisioning.googleapis.com
connectivitycheck.gstatic.com
play.google.com
omahaproxy.appspot.com
payments.google.com
googleapis.com
googleapis.com
notifications.google.com
mservice.bankofamerica.com
bankofamerica.com
mcafee.com
crl.pki.google.com
ocsp.pki.google.com
clients1.google.com
payments.google.com
ogs.google.com
googleapis.com
androidmanagement.googleapis.com
appldnld.apple.com.edgesuite.net
entrust.net
digicert.com
digicert.cn
apple-cloudkit.com
apple-livephotoskit.com
gc.apple.com
icloud-content.com
olui2m.fs.ml.com
ml.com

JonathanLee · Aug 19, 2022, 2:17 PM

@jonathanlee

JonathanLee · Aug 19, 2022, 5:00 PM

@jonathanlee

If you allow Facebook on your network,

Facebook allows proxy cache to work if you do splice

edge-chat.facebook.com

That way it does not cache any messages in the SSL system. But allows the rest to scan for viruses and the proxy to function.

Zoom asks that you allow zoom.us to pass the SSL proxy also.

zoom.us

JonathanLee · Aug 19, 2022, 5:31 PM

@jonathanlee

(IMAGE: HITS)

aGeekhere · Aug 20, 2022, 12:39 AM

I wonder if something like this is possible

try SSL Interception first, if website returns error then try splice all

example of idea

if (SSLInterception() == true)
{
  return; //loaded without error
}
else if (spliceAll() == true)
{
  return; //loaded without error
}
else 
{
 byPassTraffic(); //both Interception and spliceAll are returning errors so bypass the traffic 
}

So instead of creating and maintaining big bypass lists just use logic to skip[ SSL Interception.

JonathanLee · Aug 20, 2022, 1:33 AM

@ageekhere it was amazing to see this work the first time the huge list of hits, there was something in a url called Zion from icloud that was running after cdn.china "content delivery" had a url extension with command injection and after it stopped working as well. I checked the logs and it lists something in a url made.in.china and something about a command injection. But it listed CNN.zion and a command injection there I swore I seen. Like I was in the Matrix city of Zion. Hahaha Software defined Networking just reload it and try again. Today Snort blocked many UDP command injections. It is amazing to see it run . It runs so fast you have to download the logs to see it run. Maybe I watched snort catch a bad guy.

It may just have been going so fast I mixed up some CNN official command inject that I saw pass by and Zions link.

(IMAGE: Items Found on first day with SSL working correctly checked this url inside of Virus total and it shows clean)

(IMAGE: Zion virus total url check shows clean)

(IMAGE: China's Content Delivery Network Running Inside CNN for some reason ran this inside of virus total and it shows clean)

(IMAGE: Made in China? no idea I checked that CDN site to see what it was maybe this was the result)

(IMAGE: After all this occurred Something called USA.MADE.1 came in and it stopped maybe related to Amazon tablet)

Keep in mind all I went to was CNN first and just watched all the lists of how interconnected everything is on the firewall as it searched for a virus. All of the rest was automatically run with cookies and news items.

JonathanLee · Aug 20, 2022, 12:52 AM

@ageekhere you would need a list of approved bypass urls. Apple, some android, Windows updates. . . Etc

If(list.contains) something like that? You as an administrator must have granular control. As well as approve trusted sources. A GUI would work better with just a button that says Apple, Android, Windows, to help create lists for bypass traffic.