• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception

Scheduled Pinned Locked Moved Cache/Proxy
46 Posts 4 Posters 12.6k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • D
    dkzsys
    last edited by dkzsys Sep 18, 2023, 3:39 AM Sep 18, 2023, 3:33 AM

    Greetings,

    I have been spending quite some time trying to make the Squid Proxy working in Transparent HTTP Proxy Mode + HTTPS/SSL Interception enabled.

    However, I keep running into issues with various apps, mainly Apple Store and Evernote. (Note that I haven't get around to other apps).

    The approach I implemented the Squid Proxy is:

    • Transparent HTTP Proxy Mode + HTTPS/SSL Interception enabled
    • Defined Firewall Alias
    • Include the defined firewall alias in the "Bypass Proxy for These Destination IPs"
    • Have the following config in "Custom Options (Before Auth)"
    acl step1 at_step SslBump1
    ssl_bump peek step1
    ssl_bump stare all
    

    Some screenshots here for ease of digestion:
    79dec074-bc2c-4506-aac7-df0b1c831631-image.png

    cca5275e-eb39-4104-9092-a8e1e16ffb06-image.png

    0fc2ea57-a63b-4e7f-9ba9-0309a3569df5-image.png

    e6c94dec-9b97-4900-a20a-0d481065173d-image.png

    My Challenge
    It seems that what I put in the Firewall Alias for destination IP bypass is not effective for some reason.

    Example: ocsp2.g.aaplimg.com - one of the apple URLs
    It's resolved to 17.253.121.201 and 17.253.121.202.

    ~ ❯❯❯ nslookup ocsp2.apple.com
    Server:         1.0.0.1
    Address:        1.0.0.1#53
    
    Non-authoritative answer:
    ocsp2.apple.com canonical name = ocsp2-lb.apple.com.akadns.net.
    ocsp2-lb.apple.com.akadns.net   canonical name = ocsp2.g.aaplimg.com.
    Name:   ocsp2.g.aaplimg.com
    Address: 17.253.121.202
    Name:   ocsp2.g.aaplimg.com
    Address: 17.253.121.201
    

    I have both the FQDN and the IP addresses in the Firewall Alias "squid_bypass"; however, I am still seeing the Squid access log, indicating it's not being bypassed.

    1695005343.772      3 10.0.1.11 TCP_MISS/410 586 GET https://ocsp2.apple.com/ocsp03-wwdr04/MEcwRTBDMEEwPzAHBgUrDgMCGgQUAOsMzPRxbLpGI6PVL5jUPfYwqR0EFIgnFwmpthhgi%2BzruvZHWcVSVKO3AggO61eH554JjQ%3D%3D - ORIGINAL_DST/17.253.121.201 text/html
    

    Any suggestion on how to resolve it, or setup Squid properly would be much appreciated.

    Perhaps I get can some help from @CaliPilot here?

    Thanks in advance.

    Including the "squid_bypass" list here for reference: squid_bypass bulk list.txt

    J P 2 Replies Last reply Sep 20, 2023, 5:50 AM Reply Quote 0
    • D dkzsys referenced this topic on Sep 18, 2023, 3:36 AM
    • D
      dkzsys
      last edited by dkzsys Sep 18, 2023, 8:12 AM Sep 18, 2023, 8:12 AM

      A separate problem - thought I'd add to the list.

      I am still unable to Update apps from mac osx App Store, even after I bypass destination IPs for all the FQDNs (excluding wildcard sub domains - is there a way to do the wildcard??) on this Apple page https://support.apple.com/en-us/HT210060.

      The following squid access log was captured when I attempted the update in app store.

      Apart from knowing they are Apple IP addreses, no idea which domain they are for.

      Any idea on

      • a) what happens here? (i.e. have I missed any host/FQDN? not showing in log though), and
        b) is there a way to do wildcard sub domain bypass?
      1695024181.929    713 10.0.1.11 NONE_NONE/000 0 CONNECT 17.36.202.158:443 - ORIGINAL_DST/17.36.202.158 -
      1695024182.889    733 10.0.1.11 NONE_NONE/000 0 CONNECT 17.36.202.158:443 - ORIGINAL_DST/17.36.202.158 -
      1695024183.749    692 10.0.1.11 NONE_NONE/000 0 CONNECT 17.36.202.158:443 - ORIGINAL_DST/17.36.202.158 -
      1695024184.573    680 10.0.1.11 NONE_NONE/000 0 CONNECT 17.36.202.158:443 - ORIGINAL_DST/17.36.202.158 -
      1695024185.221     87 10.0.1.11 NONE_NONE/000 0 CONNECT 23.206.199.19:443 - ORIGINAL_DST/23.206.199.19 -
      1695024185.442     75 10.0.1.11 NONE_NONE/000 0 CONNECT 23.206.199.19:443 - ORIGINAL_DST/23.206.199.19 -
      1695024185.606     74 10.0.1.11 NONE_NONE/000 0 CONNECT 23.206.199.19:443 - ORIGINAL_DST/23.206.199.19 -
      1695024185.797     80 10.0.1.11 NONE_NONE/000 0 CONNECT 23.206.199.19:443 - ORIGINAL_DST/23.206.199.19 -
      1695024185.967     73 10.0.1.11 NONE_NONE/000 0 CONNECT 23.206.199.19:443 - ORIGINAL_DST/23.206.199.19 -
      1695024189.007     82 10.0.1.11 NONE_NONE/000 0 CONNECT 203.87.122.80:443 - ORIGINAL_DST/203.87.122.80 -
      1695024189.144     66 10.0.1.11 NONE_NONE/000 0 CONNECT 23.34.237.181:443 - ORIGINAL_DST/23.34.237.181 -
      
      M 2 Replies Last reply Sep 18, 2023, 11:26 AM Reply Quote 0
      • M
        michmoor LAYER 8 Rebel Alliance @dkzsys
        last edited by michmoor Sep 18, 2023, 11:28 AM Sep 18, 2023, 11:26 AM

        @dkzsys I have the same set up except for any custom ssl options.
        It works 95% of the time but there are some apps that don’t like a proxy no matter what.
        My Microsoft teams (along with MSFT Authenticator) and certain apple domains such as those related to iCloud. So in my testing and experience there are two ways to solve this whole issue with transparent mode

        1. Switch to explicit proxy. This resolves all the quirky problems but you still will have to make exceptions. But it’s the best solution.
        2. While in transparent mode create an Alias so that certain IPs bypass proxy. That’s what I’ve done. You will have to be ok with certain devices not going through a proxy. Make sure you update the firewall rules to account for those bypassed IPs. Downside is you lose any reporting for those IPs as LightSquid won’t tell you about any sites visited.

        Firewall: NetGate,Palo Alto-VM,Juniper SRX
        Routing: Juniper, Arista, Cisco
        Switching: Juniper, Arista, Cisco
        Wireless: Unifi, Aruba IAP
        JNCIP,CCNP Enterprise

        D 1 Reply Last reply Sep 18, 2023, 12:10 PM Reply Quote 0
        • M
          michmoor LAYER 8 Rebel Alliance @dkzsys
          last edited by Sep 18, 2023, 11:33 AM

          @dkzsys
          “ b) is there a way to do wildcard sub domain bypass?”

          Funny you brought this up as this was going to be my post today. Don’t think there’s a way to use a wildcard domain so they won’t get decrypted. I would love that as there a bunch of MSFT domains that should not be accessed behind a proxy (per documentation). This is desperately needed in transparent or explicit mode. I know other vendors have an option that states “No decryption done on these domains”.

          The problem. Squid, from what I can tell, has no active maintainer in pfsense so asking for a feature request is pointless but you could submit a redmine.

          Firewall: NetGate,Palo Alto-VM,Juniper SRX
          Routing: Juniper, Arista, Cisco
          Switching: Juniper, Arista, Cisco
          Wireless: Unifi, Aruba IAP
          JNCIP,CCNP Enterprise

          1 Reply Last reply Reply Quote 1
          • D
            dkzsys @michmoor
            last edited by Sep 18, 2023, 12:10 PM

            @michmoor said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

            Thanks for your input, @michmoor!

            @dkzsys I have the same set up except for any custom ssl options.

            That's awesome - makes it much easier to explain and relate!

            I have just removed the configs under the custom ssl options. I used to have acl rules there to bypass source IP addresses - I've migrated that to "Bypass Proxy for These Source IPs", which is much more effective. And the specific peek and stare config doesn't seem to do much. I am reviewing the squid feature doc again in case I missed anything - https://wiki.squid-cache.org/Features/SslPeekAndSplice.

            It works 95% of the time but there are some apps that don’t like a proxy no matter what.
            My Microsoft teams (along with MSFT Authenticator) and certain apple domains such as those related to iCloud. So in my testing and experience there are two ways to solve this whole issue with transparent mode

            1. Switch to explicit proxy. This resolves all the quirky problems but you still will have to make exceptions. But it’s the best solution.

            I'll try this, as a plan B. So do you just change to explicit proxy, with the same firewall alias for destination IP exclusion, and update client with the proxy details?

            1. While in transparent mode create an Alias so that certain IPs bypass proxy. That’s what I’ve done. You will have to be ok with certain devices not going through a proxy. Make sure you update the firewall rules to account for those bypassed IPs. Downside is you lose any reporting for those IPs as LightSquid won’t tell you about any sites visited.

            That's exactly what I have done as well, i.e. source IP bypass. For context, I setup the Squid Proxy specifically to monitor my children's internet activity, on Mac and iPad. So I do need their devices enabled for monitoring or reporting (tail, or lightsquid)

            So I ended up source bypass all other devices (so I don't have to suffer the semi-broken transparent proxy with ssl_bump); but keep it enabled for my children's devices. Problem is that it's still breaking their gaming, such as Roblox. (why not..:P) And I have to manually source bypass during their gaming time. Not the end of the world for now.

            M 1 Reply Last reply Sep 18, 2023, 1:34 PM Reply Quote 0
            • M
              michmoor LAYER 8 Rebel Alliance @dkzsys
              last edited by michmoor Sep 18, 2023, 1:51 PM Sep 18, 2023, 1:34 PM

              @dkzsys I am right now using Squid as a Transparent proxy. I have used it as an explict proxy in years past and ive had no issues.
              Transparent is just a different beast. In theory sites shouldnt fail to load as there is no MITM going on but there is clearly some indication to the client that they are sitting behind a proxy. Dont know how they know but i surmise they do.

              @dkzsys said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

              That's exactly what I have done as well, i.e. source IP bypass. For context, I setup the Squid Proxy specifically to monitor my children's internet activity, on Mac and iPad. So I do need their devices enabled for monitoring or reporting (tail, or lightsquid)

              You might as well perform an explicit proxy setup then and load the firewall certiificate on those devices as you control them.

              Strangly, my childrens Roblox is unaffected. No issues. Just make sure all clients are pointed to pfSense as the DNS server and also block DoH servers as well. This is also a pain point for some. You have to make sure that all clients are getting the DNS response from pfsense and not from an external DoH or DoT server otherwise that breaks connectivity as well. So using pfBlocker i created a DoH alias list out of the TheGreatWall_DoH_IP feeds list. From there i applied that alias to a floating rule that matches all my interfaces. Now i block DoH on all my LANs. When a client cant reach its DoH server it falls back to my pfsense.
              Apple devices for example use 1.1.1.1 (cloudflare dns). It gets a different IP from a dns query than the pfsense does which leads to a break in communications at times (/409 error).

              @periko Is pretty good with Proxy setups on pfsense. Do you have any thoughts on this?

              Firewall: NetGate,Palo Alto-VM,Juniper SRX
              Routing: Juniper, Arista, Cisco
              Switching: Juniper, Arista, Cisco
              Wireless: Unifi, Aruba IAP
              JNCIP,CCNP Enterprise

              D 1 Reply Last reply Sep 19, 2023, 11:26 AM Reply Quote 0
              • D
                dkzsys @michmoor
                last edited by Sep 19, 2023, 11:26 AM

                @michmoor said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

                @dkzsys I am right now using Squid as a Transparent proxy. I have used it as an explict proxy in years past and ive had no issues.
                Transparent is just a different beast. In theory sites shouldnt fail to load as there is no MITM going on but there is clearly some indication to the client that they are sitting behind a proxy. Dont know how they know but i surmise they do.

                Out of curiosity, what's your rational of changing from explicit proxy to transparent?
                For the transparent mode, I have to fine-tune almost every app, Evernote, Discord, Roblox, even Grok (websocket)... It gets a bit painful and time consuming. And Apple App Store simply just doesn't work.

                You might as well perform an explicit proxy setup then and load the firewall certiificate on those devices as you control them.
                Strangly, my childrens Roblox is unaffected. No issues.

                That's interesting. I was troubleshooting Roblox earlier today... Finally working after including an IP address 184.24.220.74 in the destination bypass rule. Still no idea which domain, if at all, it's resolved from.

                1695093782.995    116 10.0.1.15 NONE_NONE/000 0 CONNECT 184.24.220.74:443 - ORIGINAL_DST/184.24.220.74 -
                

                Just make sure all clients are pointed to pfSense as the DNS server and also block DoH servers as well. This is also a pain point for some. You have to make sure that all clients are getting the DNS response from pfsense and not from an external DoH or DoT server otherwise that breaks connectivity as well. So using pfBlocker i created a DoH alias list out of the TheGreatWall_DoH_IP feeds list. From there i applied that alias to a floating rule that matches all my interfaces. Now i block DoH on all my LANs. When a client cant reach its DoH server it falls back to my pfsense.
                Apple devices for example use 1.1.1.1 (cloudflare dns). It gets a different IP from a dns query than the pfsense does which leads to a break in communications at times (/409 error).

                Interesting points. Re "... DNS response from pfsense and not from an external DoH or DoT server otherwise that breaks connectivity as well", question is if pfSense DNS configured in Forwarding mode (to cloudflare for example) and have DNS setup to also DoH to cloudflare, they are considered having the same DNS resolver with identical resolution, right?

                The way I set up my DNS is:

                • Pfsense DNS Resolver in Forwarding mode to Cloudflare.
                • Block DoH from clients to Cloudflare directly. (others DoH IPv4 from the TheGreatWall_DoH_IP feed to be added later today)
                • Main DNS using AdGuard set up on RaspberryPi, with DoH enabled to Cloudflare (explicit fw allow rule)
                • NAT DNS 53 to AdGuard (I wrote a script to fallback the NAT to pfsense, but never implement it)

                I use AdGuard mainly for the ease of toggle (e.g. Blocking Roblox service with 1 click and effective almost immediately) and domain block.

                You reckon this setup may be breaking things in Squid transparent mode?

                @periko Is pretty good with Proxy setups on pfsense. Do you have any thoughts on this?

                Would be keen to get some thoughts/inputs from @periko 👍

                M 1 Reply Last reply Sep 19, 2023, 2:32 PM Reply Quote 0
                • M
                  michmoor LAYER 8 Rebel Alliance @dkzsys
                  last edited by michmoor Sep 19, 2023, 2:35 PM Sep 19, 2023, 2:32 PM

                  @dkzsys said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

                  Out of curiosity, what's your rational of changing from explicit proxy to transparent?

                  I just didnt want to deal with loading certificates on every device and i just needed to do granular domain blocking which you cant do in pfblocker and I also wanted the LightSquid reporting.

                  I do not have any of the Roblox side issues you are seeing but in my experience its almost always that the client is using a different DNS server than what the proxy is using. For example, my iPad shows that my DNS is my pfsense but behind the scenes its using 1.1.1.1 on port 443 to reach out for DNS as well -- Thats Cloudflare.
                  So we know some machines are hard coded to reach out to different server IPs regardless of whats given to them in DHCP.
                  My Roku TV is the same. Reaches out to 8.8.4.4 even though through DHCP it gets my pfsense dns.
                  To be fair...IoT devices shouldnt be kept behind a proxy anyway. Too much work and not much to gain but i see some beneift in a home environment where you got kids you want to monitor a bit tightly.

                  My LANs all use pfsense. I block external DNS (53), i block as best i can DoH servers and I block DoT (853). The majority of my problems are solved now and Transparent proxy is pretty effective.
                  You will still run into these one-off issues where a /409 Conflict message will come up which indicates a discrepancy in what the clietn thinks the IP address is of the site and what pfsense thinks it is. In those cases I usually opt to bypass proxy for those IPs.

                  Firewall: NetGate,Palo Alto-VM,Juniper SRX
                  Routing: Juniper, Arista, Cisco
                  Switching: Juniper, Arista, Cisco
                  Wireless: Unifi, Aruba IAP
                  JNCIP,CCNP Enterprise

                  D 1 Reply Last reply Sep 19, 2023, 11:12 PM Reply Quote 0
                  • D
                    dkzsys @michmoor
                    last edited by Sep 19, 2023, 11:12 PM

                    I just didnt want to deal with loading certificates on every device and i just needed to do granular domain blocking which you cant do in pfblocker and I also wanted the LightSquid reporting.

                    @michmoor I thought after enabling HTTPS/SSL Interception, even in transparent mode, clients could still complain about CA cert? as per https://agix.com.au/pfsense-as-a-transparent-proxy/.

                    I do not have any of the Roblox side issues you are seeing but in my experience its almost always that the client is using a different DNS server than what the proxy is using. For example, my iPad shows that my DNS is my pfsense but behind the scenes its using 1.1.1.1 on port 443 to reach out for DNS as well -- Thats Cloudflare.
                    So we know some machines are hard coded to reach out to different server IPs regardless of whats given to them in DHCP.
                    My Roku TV is the same. Reaches out to 8.8.4.4 even though through DHCP it gets my pfsense dns.
                    To be fair...IoT devices shouldnt be kept behind a proxy anyway. Too much work and not much to gain but i see some beneift in a home environment where you got kids you want to monitor a bit tightly.

                    My LANs all use pfsense. I block external DNS (53), i block as best i can DoH servers and I block DoT (853). The majority of my problems are solved now and Transparent proxy is pretty effective.
                    You will still run into these one-off issues where a /409 Conflict message will come up which indicates a discrepancy in what the clietn thinks the IP address is of the site and what pfsense thinks it is. In those cases I usually opt to bypass proxy for those IPs.

                    I have modified my DNS implementation last night:

                    • Block DoH to all external servers as per https://github.com/Sekhan/TheGreatWall
                    • Block DoT (853) to WAN (I forgot to mention about this yesterday)
                    • Block DNS (53) to WAN (I forgot to mention about this yesterday)
                    • Allow DNS (53) to pfSense (probably redundant)
                    • Allow DoH to Cloudflare (1.1.1.1 and 1.0.0.1) from AdGuard Home (RaspberryPi LAN IP)
                    • AdGuard Home configured with DoH pointing to Cloudflare (1.1.1.1 and 1.0.0.1)
                    • pfSense DNS Server Setting to Cloudflare.
                    • pfSense DNS Resolver enabled, with "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers" and "Enable Forwarding Mode"
                    • Use NAT from LAN to DNS (53) to control the local DNS destination
                      ** Either pfSense, or
                      ** AdGuard Home on RaspberryPi

                    What you said about DNS resolution discrepancy makes sense. I have tried NATing to PfSense and AdGuard Home separately, but my Roblox problem is still there. (This time at a different glitch/break point... to 23.206.199.219 and 23.206.199.226. Another good indication of potential DNS issue for these missing IP addresses.)

                    NAT is pretty standard implementation; so thought NATing to pfSense would do the trick, but no luck. For your pfSense DNS setup, do you use similar approach?

                    • pfSense DNS Server Setting to Cloudflare (or one of your choice).
                    • pfSense DNS Resolver enabled, with "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers" and "Enable Forwarding Mode"
                    M 1 Reply Last reply Sep 20, 2023, 12:45 AM Reply Quote 0
                    • M
                      michmoor LAYER 8 Rebel Alliance @dkzsys
                      last edited by Sep 20, 2023, 12:45 AM

                      @dkzsys the whole point of a transparent proxy is that you dont have to load a certificate on any client. You cant do content inspection but for certain networks thats not really important.
                      Roblox works fine on the kids iPad or fire tablet so not sure whats the error your getting. You can review the Squid Real-Time logs for that.
                      As long as you block an external dns request that solves most of the problems. Not all. Like i mentioned already some applications do not like being behind a proxy. MSFT Teams or Office365. Thats easy to solve as you create an alias with those networks that you apply in the squid configuration.

                      0d24f17b-418d-4330-b526-624d53867af2-image.png

                      make sure you have a corresponding firewall rule to now match on that alias as well.

                      Firewall: NetGate,Palo Alto-VM,Juniper SRX
                      Routing: Juniper, Arista, Cisco
                      Switching: Juniper, Arista, Cisco
                      Wireless: Unifi, Aruba IAP
                      JNCIP,CCNP Enterprise

                      D 1 Reply Last reply Sep 20, 2023, 2:28 AM Reply Quote 0
                      • D
                        dkzsys @michmoor
                        last edited by Sep 20, 2023, 2:28 AM

                        @michmoor I did some more testing and found another contributing factors for roblox - the DNS resolution for setup.rbxcdn.com, which is used by Roblox on mac. The cname changes every so often - haven't seen any concrete patterns yet. So the timing of pfSense alias dns resolution update vs the cname and IP resolved at the time of execution can make a difference. Quite a rabbit hole there:)

                        006b9fb8-63c0-46df-898f-307d6381b9eb-image.png

                        1 Reply Last reply Reply Quote 1
                        • J
                          JonathanLee @dkzsys
                          last edited by JonathanLee Sep 20, 2023, 6:03 AM Sep 20, 2023, 5:50 AM

                          @dkzsys said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

                          acl step1 at_step SslBump1
                          ssl_bump peek step1
                          ssl_bump stare all

                          I noticed you said
                          "acl step1 at_step SslBump1
                          ssl_bump peek step1
                          ssl_bump stare all"

                          This is mine if it helps at all. I have some devices set to splice always like Smartphones etc.

                          acl splice_only src 192.168.1.7 #Jon Android
                          acl splice_only src 192.168.1.8 #Tasha Apple
                          acl splice_only src 192.168.1.11 #Amazon Fire
                          acl splice_only src 192.168.1.15 #Tasha HP
                          acl splice_only src 192.168.1.16 #iPad
                          acl splice_only src 192.168.1.18 #Xbox
                          acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"
                          acl step1 at_step SslBump1
                          ssl_bump peek step1 #peek at all at step 1 so the block list works in Squidguard or allow list works
                          ssl_bump splice splice_only # now splice the above LAN ips
                          ssl_bump splice NoSSLIntercept # now splice the URLS like banks Hulu etc
                          ssl_bump stare all # the rest are always MITM Intercepted.

                          Both Transparent and MITM work this way.

                          Yesterday I changed it from I would get tons of hits from the cache this way.
                          ssl_bump peek step1 #peek at all at step 1 so the block list works in Squidguard or allow list works
                          ssl_bump splice splice_only # now splice the above LAN ips
                          ssl_bump splice NoSSLIntercept # now splice the URLS like banks Hulu etc
                          ssl_bump stare step2 # the rest are always MITM Intercepted.
                          ssl_bump bump step3

                          However, this is the same as stare all includes bump on step 3.

                          Again, you can arrange it a number of different ways.

                          As for the DNS I noticed you are having it not respond with the Server as the firewall itself see below

                          58539fe6-f8e8-4b67-92ef-3ddc1e70768c-image.png
                          Notice 192.168.1.1

                          This works because of some NAT rules see below for example.

                          618c0d8f-d6da-4dcb-817f-f1e3a15232ad-image.png
                          Anything that is using port 53, 853, 953 that is not that is my negated rule ! going to the firewall network address translate it for me to the firewall 192.168.1.1 without the clients knowing. That way the request is served by way of the firewall. This way the firewalls loopback is not of concern as it is going to where it needs to be, without 127.0.0.1 the firewall will not work with the DNS, you need both in the NAT 127.0.0.1 and 192.168.1.1 if you are using IPV6 you also need that loopback.

                          I have the firewall and loopback in the aliases. See Below

                          9eb88645-4554-4d68-a100-8536d2bcda5d-image.png
                          I only use IPv4 with my ISP

                          51c97496-af9c-4f30-bef9-e9b2e3ea0d90-image.png
                          Ports that are redirected when they hit the firewall's NAT rule

                          c02bd019-0c1a-4caa-898d-09a42aa36253-image.png
                          I have it set to use port 853 I do not think it is any more secure but hey I was a cool think to learn about.

                          Check this out...
                          7e2713b6-2ab6-403f-b9c1-eeea81f86f19-image.png
                          All port :53 only is seen on the loopback and the firewall's lan address 192.168.1.1

                          Nothing leaving the LAN does anything but the firewall itself.

                          Try to NAT it

                          Make sure to upvote

                          D 1 Reply Last reply Sep 20, 2023, 7:06 AM Reply Quote 0
                          • D
                            dkzsys @JonathanLee
                            last edited by dkzsys Sep 20, 2023, 7:23 AM Sep 20, 2023, 7:06 AM

                            @JonathanLee said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

                            "acl step1 at_step SslBump1
                            ssl_bump peek step1
                            ssl_bump stare all"

                            Thank you for the detailed response, Jonathan @JonathanLee !

                            Re the Squid Custom Options (Before Auth) config, as I do not use SquidGaurd, it's less attractive to me to configure it. I ended up removing the config and fell back to the simpler IP address (source and destination) bypass.

                            acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"

                            I did try to implement ACL list to bypass destination URLs, but it wasn't effective for me.

                            As for the DNS I noticed you are having it not respond with the Server as the firewall itself see below

                            For DNS, I have configured 1.0.0.1 in DHCP and never bothered to remove that; but all DNS requests are NAT to AdGuard Home on RaspberryPi atm. I can easily update the NAT to pfSense.

                            J 1 Reply Last reply Sep 20, 2023, 7:26 AM Reply Quote 1
                            • J
                              JonathanLee @dkzsys
                              last edited by JonathanLee Sep 20, 2023, 7:28 AM Sep 20, 2023, 7:26 AM

                              @dkzsys

                              acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"

                              This is a file you would create with custom URLs that need to be spliced and not intercepted like banks etc you visit. If you have no file it won't work.
                              Mine is url.nobump

                              Try to let PfSense handle the DNS requests with the NAT that might fix it.

                              Make sure to upvote

                              D 1 Reply Last reply Sep 20, 2023, 7:36 AM Reply Quote 0
                              • D
                                dkzsys @JonathanLee
                                last edited by dkzsys Sep 20, 2023, 7:54 AM Sep 20, 2023, 7:36 AM

                                @JonathanLee said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

                                @dkzsys

                                acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"

                                This is a file you would create with custom URLs that need to be spliced and not intercepted like banks etc you visit. If you have no file it won't work.
                                Mine is url.nobump

                                I used to have this configured:

                                # SSL Bypass for Domains
                                acl bypass_ssl_domains ssl::server_name .apple.com
                                acl bypass_ssl_domains ssl::server_name .bag.itunes.apple.com
                                acl bypass_ssl_domains ssl::server_name .cdn-apple.com
                                acl bypass_ssl_domains ssl::server_name .icloud.com
                                acl bypass_ssl_domains ssl::server_name .icloud-content.com
                                acl bypass_ssl_domains ssl::server_name .itunes.com
                                acl bypass_ssl_domains ssl::server_name .mzstatic.com
                                acl bypass_ssl_domains ssl::server_name .realtime.groklearning.com
                                acl bypass_ssl_domains ssl::server_name .dualsite-terminal.comp.gl
                                acl bypass_ssl_domains ssl::server_name .evernote.com
                                
                                acl step1 at_step SslBump1
                                ssl_bump peek step1
                                ssl_bump splice bypass_ssl_domains
                                ssl_bump stare all
                                

                                I noticed you used "ssl::server_name_regex"; and I don't seem to find much reference to it with a quick search. I wonder what the difference between "ssl::server_name_regex" and "ssl::server_name"? Also, can you show me how you structured the URLs in your "url.nobump" file please?

                                Try to let PfSense handle the DNS requests with the NAT that might fix it.

                                I'll do a bit more testing with this.

                                1 Reply Last reply Reply Quote 1
                                • D
                                  dkzsys
                                  last edited by dkzsys Sep 20, 2023, 12:00 PM Sep 20, 2023, 11:54 AM

                                  @michmoor @JonathanLee - can I ask a favour from you please. Do you mind checking what you can see with your transparent or MITM/explicit proxy on, to this URL? https://www.tradingview.com/markets

                                  When I have mine on (tried both transparent and MITM/explicit), I don't get the chart and missing quite a few other info as per screenshot below; once I bypass my IP address, the site is working properly.

                                  Thanks.

                                  Squid Proxy On
                                  099f2a44-bb03-4882-8420-2a86728031ba-image.png

                                  Squid Proxy Bypassed
                                  9b889810-b125-4543-8431-682dc0e52372-image.png

                                  J 1 Reply Last reply Sep 20, 2023, 2:03 PM Reply Quote 0
                                  • J
                                    JonathanLee @dkzsys
                                    last edited by JonathanLee Sep 20, 2023, 2:32 PM Sep 20, 2023, 2:03 PM

                                    @dkzsys

                                    Here is the URL accessed with transparent mode...

                                    Screenshot_20230920-070225.png
                                    (SPLICED DEVICE)

                                    Screenshot 2023-09-20 at 7.08.56 AM.png
                                    (MITM DEVICE WITH ROOT AUTHORITY CERTIFICATES)

                                    The firewall or proxy must be blocking something on that site and or the site does not allow proxy use with root certificates.

                                    MITM devices you have to set some urls to splice as they sometimes do not work with the proxy, and some stuff you need to ethically set the urls to splice.

                                    Here is some of my spice list urls my "nobump" file

                                    Screenshot 2023-09-20 at 7.20.37 AM.png

                                    so if you trust a site you could just add it to a nobump file so it works all the time. I use bigcharts

                                    Make sure to upvote

                                    M 1 Reply Last reply Sep 20, 2023, 2:31 PM Reply Quote 0
                                    • M
                                      michmoor LAYER 8 Rebel Alliance @JonathanLee
                                      last edited by Sep 20, 2023, 2:31 PM

                                      I visited the site without issues. I am working with Transparent Mode only.

                                      2a26fca1-8f4c-4280-ae92-309cf99a4c8f-image.png

                                      Firewall: NetGate,Palo Alto-VM,Juniper SRX
                                      Routing: Juniper, Arista, Cisco
                                      Switching: Juniper, Arista, Cisco
                                      Wireless: Unifi, Aruba IAP
                                      JNCIP,CCNP Enterprise

                                      D 1 Reply Last reply Sep 21, 2023, 7:22 AM Reply Quote 0
                                      • D
                                        dkzsys
                                        last edited by Sep 21, 2023, 5:57 AM

                                        Thank you both, Mike @michmoor and Jonathan @JonathanLee!

                                        So the behaviour from Jonathan's setups matches exactly what I am experiencing.

                                        P.S. I have NAT all my DNS queries to pfSense now, to minimise the impacting factors.
                                        80d6cf8c-c9b5-4d97-8a33-06acafb7a27f-image.png

                                        Here is some of my spice list urls my "nobump" file

                                        Thanks, Jonathan. This is helpful.

                                        so if you trust a site you could just add it to a nobump file so it works all the time. I use bigcharts

                                        Good choice:)

                                        1 Reply Last reply Reply Quote 0
                                        • D
                                          dkzsys @michmoor
                                          last edited by dkzsys Sep 21, 2023, 7:24 AM Sep 21, 2023, 7:22 AM

                                          @michmoor said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

                                          I visited the site without issues. I am working with Transparent Mode only.

                                          That is great! Also makes me wonder why my setup (and Jonathan's) is having a very different result.

                                          I have tried deleting the CA certificate; and I immediate get a cert error in transparent mode. Fixed after re-installing the cert.
                                          e21b0f2a-8514-47a4-9658-3839145830b7-image.png

                                          Did you end up installing the CA Cert on your clients?

                                          Can you cast your eyes over my config to see if there is any obvious discrepancy to your setup please? BTW, which version of pfSense are you on? Mine is 23.05.1-RELEASE (amd64)

                                          5835044f-0f63-4ada-b389-686aebb0f73a-image.png
                                          745573c1-6cca-4a3d-a28f-1a236f02844c-image.png

                                          In terms of DNS config and the corresponding FW rules (block requests to WAN), I'm now pointed back to pfSense to alignment. One factor eliminated at least.

                                          J 2 Replies Last reply Sep 21, 2023, 10:35 PM Reply Quote 0
                                          20 out of 46
                                          • First post
                                            20/46
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                            This community forum collects and processes your personal information.
                                            consent.not_received