• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Custom Options (SSL/MITM) best settings for local cache {RESOLVED ON END}

Scheduled Pinned Locked Moved Cache/Proxy
10 Posts 2 Posters 1.4k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • J
    JonathanLee
    last edited by JonathanLee Aug 19, 2022, 2:18 PM May 24, 2022, 4:38 PM

    Hello Fellow Netgate community members,

    Can you please help?

    What would be the best custom ssl settings a good local cache with Squid on a 2100 Max Netgate Firewall?

    Screen Shot 2022-05-24 at 9.25.24 AM.jpg

    (Image: Hits only show for HTTP currently)

    I have created and installed certificates and they seem to be working. I can see traffic generated however I see tunnel a lot in the live logs. When using the "custom option" on SSL you must configure the option in advanced. I am testing custom out. I am hoping you can shine some light on what would be the best settings in a home network, all devices have certificates except the Xbox.

    Screen Shot 2022-05-24 at 9.12.42 AM.png

    (Image: Port 3129)

    I noticed I can not use port 3129 for anything or the internet stops when that is turned on for proxy use, however it works if I still use the 3128 listener port.

    Screen Shot 2022-05-24 at 9.37.49 AM.png

    (Image: Local settings still set to 3128)

    Screen Shot 2022-05-24 at 9.12.31 AM.png

    (Image: Port 3128)

    I still need transparent proxy for the Xbox use. Palo Alto firewalls can issues and perform certificate use similar to the Netgate system. However the image below shows that it is possible to get HTTPS hits.

    Wpad is configured also.

    ssl_bump peek all
    ssl_bump all

    ssl_bumb peek step1
    ssl_splice all

    ssl_bump peek all
    ssl_bump splice all

    Screen Shot 2022-05-24 at 9.11.02 AM.png

    My Goal is to cache and accelerate more traffic and have larger hit rates. I understand that TSL/SSL is a encrypted tunnel however one can say that encrypted data should still carry some hash values during transmission. Even if it is cached locally it is encrypted data and non readable without the browser and certificate that requested it, the data would be the same once it is delivered and used if required again. Wouldn't that hash value generated for the data that was carried in that encrypted tunnel be the same if that same data was requested again? Can we essentially cache HTTPS data.

    Screen Shot 2022-05-24 at 9.31.05 AM.png

    Yes Squid is a HTTP, HTTPS, FTP based proxy, plus an accelerator.

    I have noticed online that some users are able to cache HTTPS items. However I am still trying to get this to work correctly. Are they using wget?

    Screen Shot 2022-05-24 at 9.21.48 AM.jpg

    (Image: Youtube Squid use with many hits for HTTPS for Facebook)

    Any recommendations I use to think this was not possible however I have been seeing youtube posts of HTTPS caching working for Squid?

    Can you CURL or WGET and save a website and deliver it after the request was made however store it encrypted in the cache? That would make the proxy check the website before it's delivered.

    Are my certificates set up incorrectly?

    I have also been reading about ecap use over icap for performance gains.

    Screen Shot 2022-05-24 at 9.34.41 AM.png

    (Image: Squid's Web Page on ECAP use)

    Make sure to upvote

    1 Reply Last reply Reply Quote 0
    • J
      JonathanLee
      last edited by Aug 17, 2022, 4:58 PM

      Screen Shot 2022-08-17 at 9.49.10 AM.png

      (IMAGE: Best setting I have found so far to match above with HTTPS CACHE WARNING THIS MANY NOT WORK IN ALL AREAS, YOU MUST OWN DEVICES AND INSTALL CERTIFICATES ON THEM TO USE THIS FUNCTIONALITY THIS MIMICS ENTERPRISE FIREWALLS LIKE PALO ALTO SYSTEMS)

      This was my best setting for high cache rates with HTTPS

      The 192.168.20.11 is a device that can not install a certificate so it is marked as splice only.

      Screen Shot 2022-08-17 at 9.56.00 AM.jpg

      (IMAGE: CERTIFICATES WORKING WITH CACHE)

      Make sure to upvote

      J 1 Reply Last reply Aug 19, 2022, 12:34 AM Reply Quote 0
      • J
        JonathanLee @JonathanLee
        last edited by JonathanLee Aug 19, 2022, 12:46 AM Aug 19, 2022, 12:34 AM

        @jonathanlee

        To add to this once you have this running you will not have any Windows updates however everything else works. Wait I found a work around while reading the Squid Forums,

        This is the solution create a file inside of the Netgate firewall.

        sites to splice.PNG
        (IMAGE: SSL_BUMP FILE FOR SPLICE ONLY)

        Splice only is needed for Windows update and Apple updates, they require their own custom certificate and if you are using a third party certificate like me at home that causes some issues. It's ok simply mark the update sites to splice when they see them, and it works again.

        custom.PNG
        (IMAGE: Custom Options Squid)

        My ACL are named Splice_only is based by source IP for my Xbox because it cannot install a certificate and for Amazon Tablet because it has some issues with the updates

        My ACL "Access Control Lists" NOSSLIntercept is based off the file I created above. This allows windows updates to splice while still keeping the security of SSL bumping for all other sites.

        update works again.PNG
        (Image: Windows update working without errors)

        windows netsh.PNG
        (IMAGE: Make sure you add your proxy to Winhttp for the Windows update to also point to the proxy or it will not work still)

        Make sure to upvote

        1 Reply Last reply Reply Quote 0
        • J
          JonathanLee
          last edited by JonathanLee Aug 19, 2022, 2:15 PM Aug 19, 2022, 2:13 PM

          @jonathanlee You will also need to make some custom adjustments to the Splice list, meaning that it skips SSL Interception for the following sites:

          Addition:
          You also need to add for apple products:
          Use Apple products on enterprise networks
          https://support.apple.com/en-us/HT210060

          Android products:
          Android Enterprise Network Requirements
          https://support.google.com/work/android/answer/10513641?hl=en

          My list looks like this: Android use and Mac Iphone Macafee Antivirus and some banking sites as an example.
          They must skip SSL Interception

          #Sites to be spliced
          update.microsoft.com
          update.microsoft.com.\akadns.net
          apple.com
          cdn-apple.com
          icloud.com
          icloud-content.com
          itunes.com
          mzstatic.com
          play.google.com
          android.com
          google-analytics.com
          googleusercontent.com
          gstatic.com
          gvt1.com
          ggpht.com
          dl.google.com
          dl-ssl.google.com
          android.clients.google.com
          gvt2.com
          gvt3.com
          accounts.google.com
          accounts.google.us
          pki.google.com
          clients1.google.com
          clients2.google.com
          clients3.google.com
          clients4.google.com
          clients5.google.com
          clients6.google.com
          connectivitycheck.android.com
          mtalk.google.com
          mtalk4.google.com
          mtalk-staging.google.com
          mtalk-dev.google.com
          alt1-mtalk.google.com
          alt2-mtalk.google.com
          alt3-mtalk.google.com
          alt4-mtalk.google.com
          alt5-mtalk.google.com
          alt6-mtalk.google.com
          alt7-mtalk.google.com
          alt8-mtalk.google.com
          android.clients.google.com
          device-provisioning.googleapis.com
          connectivitycheck.gstatic.com
          play.google.com
          omahaproxy.appspot.com
          payments.google.com
          googleapis.com
          googleapis.com
          notifications.google.com
          mservice.bankofamerica.com
          bankofamerica.com
          mcafee.com
          crl.pki.google.com
          ocsp.pki.google.com
          clients1.google.com
          payments.google.com
          ogs.google.com
          googleapis.com
          androidmanagement.googleapis.com
          appldnld.apple.com.edgesuite.net
          entrust.net
          digicert.com
          digicert.cn
          apple-cloudkit.com
          apple-livephotoskit.com
          gc.apple.com
          icloud-content.com
          olui2m.fs.ml.com
          ml.com

          Make sure to upvote

          1 Reply Last reply Reply Quote 0
          • J
            JonathanLee
            last edited by Aug 19, 2022, 2:17 PM

            @jonathanlee

            ACL.PNG

            Make sure to upvote

            1 Reply Last reply Reply Quote 0
            • J
              JonathanLee
              last edited by Aug 19, 2022, 5:00 PM

              @jonathanlee

              If you allow Facebook on your network,

              Facebook allows proxy cache to work if you do splice

              edge-chat.facebook.com

              That way it does not cache any messages in the SSL system. But allows the rest to scan for viruses and the proxy to function.

              Zoom asks that you allow zoom.us to pass the SSL proxy also.

              zoom.us

              Make sure to upvote

              J 1 Reply Last reply Aug 19, 2022, 5:31 PM Reply Quote 0
              • J
                JonathanLee @JonathanLee
                last edited by Aug 19, 2022, 5:31 PM

                @jonathanlee

                httpscache.PNG

                (IMAGE: HITS)

                Make sure to upvote

                1 Reply Last reply Reply Quote 0
                • A
                  aGeekhere
                  last edited by Aug 20, 2022, 12:39 AM

                  I wonder if something like this is possible

                  try SSL Interception first, if website returns error then try splice all

                  example of idea

                  if (SSLInterception() == true)
                  {
                    return; //loaded without error
                  }
                  else if (spliceAll() == true)
                  {
                    return; //loaded without error
                  }
                  else 
                  {
                   byPassTraffic(); //both Interception and spliceAll are returning errors so bypass the traffic 
                  }
                  

                  So instead of creating and maintaining big bypass lists just use logic to skip[ SSL Interception.

                  Never Fear, A Geek is Here!

                  J 1 Reply Last reply Aug 20, 2022, 12:47 AM Reply Quote 0
                  • J
                    JonathanLee @aGeekhere
                    last edited by JonathanLee Aug 20, 2022, 1:33 AM Aug 20, 2022, 12:47 AM

                    @ageekhere it was amazing to see this work the first time the huge list of hits, there was something in a url called Zion from icloud that was running after cdn.china "content delivery" had a url extension with command injection and after it stopped working as well. I checked the logs and it lists something in a url made.in.china and something about a command injection. But it listed CNN.zion and a command injection there I swore I seen. Like I was in the Matrix city of Zion. Hahaha 😂😆 Software defined Networking just reload it and try again. Today Snort blocked many UDP command injections. It is amazing to see it run 😁. It runs so fast you have to download the logs to see it run. Maybe I watched snort catch a bad guy.

                    It may just have been going so fast I mixed up some CNN official command inject that I saw pass by and Zions link.

                    injection.PNG
                    (IMAGE: Items Found on first day with SSL working correctly checked this url inside of Virus total and it shows clean)

                    zionapi.PNG
                    (IMAGE: Zion virus total url check shows clean)

                    cache.PNG
                    (IMAGE: China's Content Delivery Network Running Inside CNN for some reason ran this inside of virus total and it shows clean)

                    made.in.PNG
                    (IMAGE: Made in China? no idea I checked that CDN site to see what it was maybe this was the result)

                    usamade1.PNG
                    (IMAGE: After all this occurred Something called USA.MADE.1 came in and it stopped maybe related to Amazon tablet)

                    Keep in mind all I went to was CNN first and just watched all the lists of how interconnected everything is on the firewall as it searched for a virus. All of the rest was automatically run with cookies and news items.

                    Make sure to upvote

                    J 1 Reply Last reply Aug 20, 2022, 12:52 AM Reply Quote 0
                    • J
                      JonathanLee @JonathanLee
                      last edited by Aug 20, 2022, 12:52 AM

                      @ageekhere you would need a list of approved bypass urls. Apple, some android, Windows updates. . . Etc

                      If(list.contains) something like that? You as an administrator must have granular control. As well as approve trusted sources. A GUI would work better with just a button that says Apple, Android, Windows, to help create lists for bypass traffic.

                      Make sure to upvote

                      1 Reply Last reply Reply Quote 0
                      • First post
                        Last post
                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                        [[user:consent.lead]]
                        [[user:consent.not_received]]