Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception

dkzsys

@michmoor said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

@dkzsys I am right now using Squid as a Transparent proxy. I have used it as an explict proxy in years past and ive had no issues.
Transparent is just a different beast. In theory sites shouldnt fail to load as there is no MITM going on but there is clearly some indication to the client that they are sitting behind a proxy. Dont know how they know but i surmise they do.

Out of curiosity, what's your rational of changing from explicit proxy to transparent?
For the transparent mode, I have to fine-tune almost every app, Evernote, Discord, Roblox, even Grok (websocket)... It gets a bit painful and time consuming. And Apple App Store simply just doesn't work.

You might as well perform an explicit proxy setup then and load the firewall certiificate on those devices as you control them.
Strangly, my childrens Roblox is unaffected. No issues.

That's interesting. I was troubleshooting Roblox earlier today... Finally working after including an IP address 184.24.220.74 in the destination bypass rule. Still no idea which domain, if at all, it's resolved from.

1695093782.995    116 10.0.1.15 NONE_NONE/000 0 CONNECT 184.24.220.74:443 - ORIGINAL_DST/184.24.220.74 -

Just make sure all clients are pointed to pfSense as the DNS server and also block DoH servers as well. This is also a pain point for some. You have to make sure that all clients are getting the DNS response from pfsense and not from an external DoH or DoT server otherwise that breaks connectivity as well. So using pfBlocker i created a DoH alias list out of the TheGreatWall_DoH_IP feeds list. From there i applied that alias to a floating rule that matches all my interfaces. Now i block DoH on all my LANs. When a client cant reach its DoH server it falls back to my pfsense.
Apple devices for example use 1.1.1.1 (cloudflare dns). It gets a different IP from a dns query than the pfsense does which leads to a break in communications at times (/409 error).

Interesting points. Re "... DNS response from pfsense and not from an external DoH or DoT server otherwise that breaks connectivity as well", question is if pfSense DNS configured in Forwarding mode (to cloudflare for example) and have DNS setup to also DoH to cloudflare, they are considered having the same DNS resolver with identical resolution, right?

The way I set up my DNS is:

Pfsense DNS Resolver in Forwarding mode to Cloudflare.
Block DoH from clients to Cloudflare directly. (others DoH IPv4 from the TheGreatWall_DoH_IP feed to be added later today)
Main DNS using AdGuard set up on RaspberryPi, with DoH enabled to Cloudflare (explicit fw allow rule)
NAT DNS 53 to AdGuard (I wrote a script to fallback the NAT to pfsense, but never implement it)

I use AdGuard mainly for the ease of toggle (e.g. Blocking Roblox service with 1 click and effective almost immediately) and domain block.

You reckon this setup may be breaking things in Squid transparent mode?

@periko Is pretty good with Proxy setups on pfsense. Do you have any thoughts on this?

Would be keen to get some thoughts/inputs from @periko

michmoor

@dkzsys said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

Out of curiosity, what's your rational of changing from explicit proxy to transparent?

I just didnt want to deal with loading certificates on every device and i just needed to do granular domain blocking which you cant do in pfblocker and I also wanted the LightSquid reporting.

I do not have any of the Roblox side issues you are seeing but in my experience its almost always that the client is using a different DNS server than what the proxy is using. For example, my iPad shows that my DNS is my pfsense but behind the scenes its using 1.1.1.1 on port 443 to reach out for DNS as well -- Thats Cloudflare.
So we know some machines are hard coded to reach out to different server IPs regardless of whats given to them in DHCP.
My Roku TV is the same. Reaches out to 8.8.4.4 even though through DHCP it gets my pfsense dns.
To be fair...IoT devices shouldnt be kept behind a proxy anyway. Too much work and not much to gain but i see some beneift in a home environment where you got kids you want to monitor a bit tightly.

My LANs all use pfsense. I block external DNS (53), i block as best i can DoH servers and I block DoT (853). The majority of my problems are solved now and Transparent proxy is pretty effective.
You will still run into these one-off issues where a /409 Conflict message will come up which indicates a discrepancy in what the clietn thinks the IP address is of the site and what pfsense thinks it is. In those cases I usually opt to bypass proxy for those IPs.

dkzsys

I just didnt want to deal with loading certificates on every device and i just needed to do granular domain blocking which you cant do in pfblocker and I also wanted the LightSquid reporting.

@michmoor I thought after enabling HTTPS/SSL Interception, even in transparent mode, clients could still complain about CA cert? as per https://agix.com.au/pfsense-as-a-transparent-proxy/.

I do not have any of the Roblox side issues you are seeing but in my experience its almost always that the client is using a different DNS server than what the proxy is using. For example, my iPad shows that my DNS is my pfsense but behind the scenes its using 1.1.1.1 on port 443 to reach out for DNS as well -- Thats Cloudflare.
So we know some machines are hard coded to reach out to different server IPs regardless of whats given to them in DHCP.
My Roku TV is the same. Reaches out to 8.8.4.4 even though through DHCP it gets my pfsense dns.
To be fair...IoT devices shouldnt be kept behind a proxy anyway. Too much work and not much to gain but i see some beneift in a home environment where you got kids you want to monitor a bit tightly.

My LANs all use pfsense. I block external DNS (53), i block as best i can DoH servers and I block DoT (853). The majority of my problems are solved now and Transparent proxy is pretty effective.
You will still run into these one-off issues where a /409 Conflict message will come up which indicates a discrepancy in what the clietn thinks the IP address is of the site and what pfsense thinks it is. In those cases I usually opt to bypass proxy for those IPs.

I have modified my DNS implementation last night:

Block DoH to all external servers as per https://github.com/Sekhan/TheGreatWall
Block DoT (853) to WAN (I forgot to mention about this yesterday)
Block DNS (53) to WAN (I forgot to mention about this yesterday)
Allow DNS (53) to pfSense (probably redundant)
Allow DoH to Cloudflare (1.1.1.1 and 1.0.0.1) from AdGuard Home (RaspberryPi LAN IP)
AdGuard Home configured with DoH pointing to Cloudflare (1.1.1.1 and 1.0.0.1)
pfSense DNS Server Setting to Cloudflare.
pfSense DNS Resolver enabled, with "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers" and "Enable Forwarding Mode"
Use NAT from LAN to DNS (53) to control the local DNS destination
** Either pfSense, or
** AdGuard Home on RaspberryPi

What you said about DNS resolution discrepancy makes sense. I have tried NATing to PfSense and AdGuard Home separately, but my Roblox problem is still there. (This time at a different glitch/break point... to 23.206.199.219 and 23.206.199.226. Another good indication of potential DNS issue for these missing IP addresses.)

NAT is pretty standard implementation; so thought NATing to pfSense would do the trick, but no luck. For your pfSense DNS setup, do you use similar approach?

pfSense DNS Server Setting to Cloudflare (or one of your choice).
pfSense DNS Resolver enabled, with "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers" and "Enable Forwarding Mode"

michmoor

@dkzsys the whole point of a transparent proxy is that you dont have to load a certificate on any client. You cant do content inspection but for certain networks thats not really important.
Roblox works fine on the kids iPad or fire tablet so not sure whats the error your getting. You can review the Squid Real-Time logs for that.
As long as you block an external dns request that solves most of the problems. Not all. Like i mentioned already some applications do not like being behind a proxy. MSFT Teams or Office365. Thats easy to solve as you create an alias with those networks that you apply in the squid configuration.

make sure you have a corresponding firewall rule to now match on that alias as well.

dkzsys

@michmoor I did some more testing and found another contributing factors for roblox - the DNS resolution for setup.rbxcdn.com, which is used by Roblox on mac. The cname changes every so often - haven't seen any concrete patterns yet. So the timing of pfSense alias dns resolution update vs the cname and IP resolved at the time of execution can make a difference. Quite a rabbit hole there:)

JonathanLee

@dkzsys said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump stare all

I noticed you said
"acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump stare all"

This is mine if it helps at all. I have some devices set to splice always like Smartphones etc.

acl splice_only src 192.168.1.7 #Jon Android
acl splice_only src 192.168.1.8 #Tasha Apple
acl splice_only src 192.168.1.11 #Amazon Fire
acl splice_only src 192.168.1.15 #Tasha HP
acl splice_only src 192.168.1.16 #iPad
acl splice_only src 192.168.1.18 #Xbox
acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"
acl step1 at_step SslBump1
ssl_bump peek step1 #peek at all at step 1 so the block list works in Squidguard or allow list works
ssl_bump splice splice_only # now splice the above LAN ips
ssl_bump splice NoSSLIntercept # now splice the URLS like banks Hulu etc
ssl_bump stare all # the rest are always MITM Intercepted.

Both Transparent and MITM work this way.

Yesterday I changed it from I would get tons of hits from the cache this way.
ssl_bump peek step1 #peek at all at step 1 so the block list works in Squidguard or allow list works
ssl_bump splice splice_only # now splice the above LAN ips
ssl_bump splice NoSSLIntercept # now splice the URLS like banks Hulu etc
ssl_bump stare step2 # the rest are always MITM Intercepted.
ssl_bump bump step3

However, this is the same as stare all includes bump on step 3.

Again, you can arrange it a number of different ways.

As for the DNS I noticed you are having it not respond with the Server as the firewall itself see below

Notice 192.168.1.1

This works because of some NAT rules see below for example.

Anything that is using port 53, 853, 953 that is not that is my negated rule ! going to the firewall network address translate it for me to the firewall 192.168.1.1 without the clients knowing. That way the request is served by way of the firewall. This way the firewalls loopback is not of concern as it is going to where it needs to be, without 127.0.0.1 the firewall will not work with the DNS, you need both in the NAT 127.0.0.1 and 192.168.1.1 if you are using IPV6 you also need that loopback.

I have the firewall and loopback in the aliases. See Below

I only use IPv4 with my ISP

Ports that are redirected when they hit the firewall's NAT rule

I have it set to use port 853 I do not think it is any more secure but hey I was a cool think to learn about.

Check this out...

All port :53 only is seen on the loopback and the firewall's lan address 192.168.1.1

Nothing leaving the LAN does anything but the firewall itself.

Try to NAT it

dkzsys

@JonathanLee said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

"acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump stare all"

Thank you for the detailed response, Jonathan @JonathanLee !

Re the Squid Custom Options (Before Auth) config, as I do not use SquidGaurd, it's less attractive to me to configure it. I ended up removing the config and fell back to the simpler IP address (source and destination) bypass.

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"

I did try to implement ACL list to bypass destination URLs, but it wasn't effective for me.

As for the DNS I noticed you are having it not respond with the Server as the firewall itself see below

For DNS, I have configured 1.0.0.1 in DHCP and never bothered to remove that; but all DNS requests are NAT to AdGuard Home on RaspberryPi atm. I can easily update the NAT to pfSense.

JonathanLee

@dkzsys

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"

This is a file you would create with custom URLs that need to be spliced and not intercepted like banks etc you visit. If you have no file it won't work.
Mine is url.nobump

Try to let PfSense handle the DNS requests with the NAT that might fix it.

dkzsys

@JonathanLee said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

@dkzsys

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"

This is a file you would create with custom URLs that need to be spliced and not intercepted like banks etc you visit. If you have no file it won't work.
Mine is url.nobump

I used to have this configured:

# SSL Bypass for Domains
acl bypass_ssl_domains ssl::server_name .apple.com
acl bypass_ssl_domains ssl::server_name .bag.itunes.apple.com
acl bypass_ssl_domains ssl::server_name .cdn-apple.com
acl bypass_ssl_domains ssl::server_name .icloud.com
acl bypass_ssl_domains ssl::server_name .icloud-content.com
acl bypass_ssl_domains ssl::server_name .itunes.com
acl bypass_ssl_domains ssl::server_name .mzstatic.com
acl bypass_ssl_domains ssl::server_name .realtime.groklearning.com
acl bypass_ssl_domains ssl::server_name .dualsite-terminal.comp.gl
acl bypass_ssl_domains ssl::server_name .evernote.com

acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump splice bypass_ssl_domains
ssl_bump stare all

I noticed you used "ssl::server_name_regex"; and I don't seem to find much reference to it with a quick search. I wonder what the difference between "ssl::server_name_regex" and "ssl::server_name"? Also, can you show me how you structured the URLs in your "url.nobump" file please?

Try to let PfSense handle the DNS requests with the NAT that might fix it.

I'll do a bit more testing with this.

dkzsys

@michmoor @JonathanLee - can I ask a favour from you please. Do you mind checking what you can see with your transparent or MITM/explicit proxy on, to this URL? https://www.tradingview.com/markets

When I have mine on (tried both transparent and MITM/explicit), I don't get the chart and missing quite a few other info as per screenshot below; once I bypass my IP address, the site is working properly.

Thanks.

Squid Proxy On

Squid Proxy Bypassed

JonathanLee

@dkzsys

Here is the URL accessed with transparent mode...

(SPLICED DEVICE)

Screenshot 2023-09-20 at 7.08.56 AM.png
(MITM DEVICE WITH ROOT AUTHORITY CERTIFICATES)

The firewall or proxy must be blocking something on that site and or the site does not allow proxy use with root certificates.

MITM devices you have to set some urls to splice as they sometimes do not work with the proxy, and some stuff you need to ethically set the urls to splice.

Here is some of my spice list urls my "nobump" file

Screenshot 2023-09-20 at 7.20.37 AM.png

so if you trust a site you could just add it to a nobump file so it works all the time. I use bigcharts

michmoor

I visited the site without issues. I am working with Transparent Mode only.

dkzsys

Thank you both, Mike @michmoor and Jonathan @JonathanLee!

So the behaviour from Jonathan's setups matches exactly what I am experiencing.

P.S. I have NAT all my DNS queries to pfSense now, to minimise the impacting factors.

Here is some of my spice list urls my "nobump" file

Thanks, Jonathan. This is helpful.

so if you trust a site you could just add it to a nobump file so it works all the time. I use bigcharts

Good choice:)

dkzsys

@michmoor said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

I visited the site without issues. I am working with Transparent Mode only.

That is great! Also makes me wonder why my setup (and Jonathan's) is having a very different result.

I have tried deleting the CA certificate; and I immediate get a cert error in transparent mode. Fixed after re-installing the cert.

Did you end up installing the CA Cert on your clients?

Can you cast your eyes over my config to see if there is any obvious discrepancy to your setup please? BTW, which version of pfSense are you on? Mine is 23.05.1-RELEASE (amd64)

In terms of DNS config and the corresponding FW rules (block requests to WAN), I'm now pointed back to pfSense to alignment. One factor eliminated at least.

JonathanLee

@dkzsys for MITM to work you must install certificates on your devices you run SSL intercept with.

When using both transparent and SSL intercept like you and I are, for transparent to work we must define what urls and or devices require splicing.

Not everything can be SSL intercepted, example being banks, private messages, and most smartphones. Think ethically, what should be spliced? It's powerful. It's unethical to open people's email or text messages so that would be an obvious spliced connection. However, the splice setting still allows you to perform basic URL filtering and blocking. Example, block all the HTTP GET requests for adult websites, or maybe block tictok access, finally you could install timers for YouTube for your kids and have the firewall block YouTube access after 10pm. You just won't inspect everything in those spliced connections or devices. Leading to, even with SSL intercept and your devices running a root certificate authority not all websites allow SSL interception. They could block you too from access.

I like to block sites that use AI tracking. Tracking like this evades privacy and thus it's not legal, by way of the new CCPA and GDPR laws that protect privacy. Most advertising companies ignore the privacy laws right now. They don't care if a user has set their browser to request no tracking. That's when URL blocking comes in.

Another example of devices that you can not do SSL interception with is an iPhone and Amazon fire tablets, you can not have them set to MITM ssl intercept, if you do none of the built in applications work only the browser will. I still install certificates anyway and set those devices as splice only.

For Windows 11 you must install the root certificates on the devices or it will not trust them, however transparent mode works without them as long as wpad or dhcp option 252 lands the device on the Proxy.

Transparent only mode allows basic URL filtering/inspection. Squid even has a YouTube API for age restrictions that works in transparent mode.

SSL intercept allows deep packet inspection and ClamAV works very effectively. It's harder to configure as you have to fine tune it.

Some sites will not allow SSL interception so splice them.

You are required to install certificates or it will show as untrusted for SSL intercept.

This is a security feature as you can only install root certificates on the devices you physically own. So Windows warns you when a certificate is being used that has not been marked trusted. It's warning the end users that traffic is going to a proxy.

Install the certificate and mark it as a trusted root authority to correct the issue you are seeing.

Here is another spliced device example: you bring home your work laptop, you can not legally install a certificate on that device because you don't own it. So, that device would be a splice only setup and use your proxy that way in transparent mode.

This is how most enterprise class firewalls work. They use certificates on the devices that in turn interact with the firewalls.

JonathanLee

@dkzsys if your not using a server for forwarding change this setting to transparent.

I had issues with this setting in the past.

dkzsys

@JonathanLee said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

@dkzsys for MITM to work you must install certificates on your devices you run SSL intercept with.
......
Here is another example, you bring home your work laptop, you can not legally install a certificate on that device so, that device would be a splice only setup.

Great context, Jonathan - adding the dimensions of ethic, commercial implementation and practicality.

My challenge with the SSL intercept is the continuous extension on the splicing as we start using new apps or web services. Ideally, I'd like to have it as set-and-forget and expect it to work most of the time, like 95%, and I am ok with the occasional manual effort. Just like what seems to be working for @michmoor. I am really curious on what the differences are in terms of implementation between his config and ours - the example for the TV charting data is quite revealing.

I'm also going to check out the squidGuard features today; to see how to best utilise it for (ad) blocking and access control. Do you have any recommendation on:

list itemany blacklist,
how best to implement the self-defined block list? (let's say roblox service. I had a quick play with Target Categories - toggling the redirect seems to take quite a long time; either I ran out of patient or it was ineffective. Tried different browsers. Ended up disabling squidGuard.)

JonathanLee

@dkzsys I use Squidguard, it takes time to configure.

But it does allow a URL blacklist that is pretty effective.

http://dsi.ut-capitole.fr/blacklists/download/blacklists_for_pfsense_reducted.tar.gz

I love this blacklist as it's reduced for my 2100

I made my Roblox list with help of their website for proxy/school district usage setup also approves my son's Roblox

I agree Mike has his set such that no certificates are required quicker and still effective. Again, I need the SSL intercept for viruses and major issues with my old job. I had a target on my back for years so bad that I couldn't even watch Netflix I would get hit with pings of death non stop and many other issues. I still find residual invasive actors and containers to this day even on my personal devices.

michmoor

@JonathanLee why set the XFF ito transparent?

michmoor

@JonathanLee said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

@dkzsys I use Squidguard, it takes time to configure.

But it does allow a URL blacklist that is pretty effective.

http://dsi.ut-capitole.fr/blacklists/download/blacklists_for_pfsense_reducted.tar.gz

I love this blacklist as it's reduced for my 2100

I made my Roblox list with help of their website for proxy/school district usage setup also approves my son's Roblox

I agree Mike has his set such that no certificates are required quicker and still effective. Again, I need the SSL intercept for viruses and major issues with my old job. I had a target on my back for years so bad that I couldn't even watch Netflix I would get hit with pings of death non stop and many other issues. I still find residual invasive actors and containers to this day even on my personal devices

This seems like a very interesting story I must hear one day.