Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception

michmoor

@dkzsys the whole point of a transparent proxy is that you dont have to load a certificate on any client. You cant do content inspection but for certain networks thats not really important.
Roblox works fine on the kids iPad or fire tablet so not sure whats the error your getting. You can review the Squid Real-Time logs for that.
As long as you block an external dns request that solves most of the problems. Not all. Like i mentioned already some applications do not like being behind a proxy. MSFT Teams or Office365. Thats easy to solve as you create an alias with those networks that you apply in the squid configuration.

make sure you have a corresponding firewall rule to now match on that alias as well.

dkzsys

@michmoor I did some more testing and found another contributing factors for roblox - the DNS resolution for setup.rbxcdn.com, which is used by Roblox on mac. The cname changes every so often - haven't seen any concrete patterns yet. So the timing of pfSense alias dns resolution update vs the cname and IP resolved at the time of execution can make a difference. Quite a rabbit hole there:)

JonathanLee

@dkzsys said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump stare all

I noticed you said
"acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump stare all"

This is mine if it helps at all. I have some devices set to splice always like Smartphones etc.

acl splice_only src 192.168.1.7 #Jon Android
acl splice_only src 192.168.1.8 #Tasha Apple
acl splice_only src 192.168.1.11 #Amazon Fire
acl splice_only src 192.168.1.15 #Tasha HP
acl splice_only src 192.168.1.16 #iPad
acl splice_only src 192.168.1.18 #Xbox
acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"
acl step1 at_step SslBump1
ssl_bump peek step1 #peek at all at step 1 so the block list works in Squidguard or allow list works
ssl_bump splice splice_only # now splice the above LAN ips
ssl_bump splice NoSSLIntercept # now splice the URLS like banks Hulu etc
ssl_bump stare all # the rest are always MITM Intercepted.

Both Transparent and MITM work this way.

Yesterday I changed it from I would get tons of hits from the cache this way.
ssl_bump peek step1 #peek at all at step 1 so the block list works in Squidguard or allow list works
ssl_bump splice splice_only # now splice the above LAN ips
ssl_bump splice NoSSLIntercept # now splice the URLS like banks Hulu etc
ssl_bump stare step2 # the rest are always MITM Intercepted.
ssl_bump bump step3

However, this is the same as stare all includes bump on step 3.

Again, you can arrange it a number of different ways.

As for the DNS I noticed you are having it not respond with the Server as the firewall itself see below

Notice 192.168.1.1

This works because of some NAT rules see below for example.

Anything that is using port 53, 853, 953 that is not that is my negated rule ! going to the firewall network address translate it for me to the firewall 192.168.1.1 without the clients knowing. That way the request is served by way of the firewall. This way the firewalls loopback is not of concern as it is going to where it needs to be, without 127.0.0.1 the firewall will not work with the DNS, you need both in the NAT 127.0.0.1 and 192.168.1.1 if you are using IPV6 you also need that loopback.

I have the firewall and loopback in the aliases. See Below

I only use IPv4 with my ISP

Ports that are redirected when they hit the firewall's NAT rule

I have it set to use port 853 I do not think it is any more secure but hey I was a cool think to learn about.

Check this out...

All port :53 only is seen on the loopback and the firewall's lan address 192.168.1.1

Nothing leaving the LAN does anything but the firewall itself.

Try to NAT it

dkzsys

@JonathanLee said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

"acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump stare all"

Thank you for the detailed response, Jonathan @JonathanLee !

Re the Squid Custom Options (Before Auth) config, as I do not use SquidGaurd, it's less attractive to me to configure it. I ended up removing the config and fell back to the simpler IP address (source and destination) bypass.

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"

I did try to implement ACL list to bypass destination URLs, but it wasn't effective for me.

As for the DNS I noticed you are having it not respond with the Server as the firewall itself see below

For DNS, I have configured 1.0.0.1 in DHCP and never bothered to remove that; but all DNS requests are NAT to AdGuard Home on RaspberryPi atm. I can easily update the NAT to pfSense.

JonathanLee

@dkzsys

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"

This is a file you would create with custom URLs that need to be spliced and not intercepted like banks etc you visit. If you have no file it won't work.
Mine is url.nobump

Try to let PfSense handle the DNS requests with the NAT that might fix it.

dkzsys

@JonathanLee said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

@dkzsys

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"

This is a file you would create with custom URLs that need to be spliced and not intercepted like banks etc you visit. If you have no file it won't work.
Mine is url.nobump

I used to have this configured:

# SSL Bypass for Domains
acl bypass_ssl_domains ssl::server_name .apple.com
acl bypass_ssl_domains ssl::server_name .bag.itunes.apple.com
acl bypass_ssl_domains ssl::server_name .cdn-apple.com
acl bypass_ssl_domains ssl::server_name .icloud.com
acl bypass_ssl_domains ssl::server_name .icloud-content.com
acl bypass_ssl_domains ssl::server_name .itunes.com
acl bypass_ssl_domains ssl::server_name .mzstatic.com
acl bypass_ssl_domains ssl::server_name .realtime.groklearning.com
acl bypass_ssl_domains ssl::server_name .dualsite-terminal.comp.gl
acl bypass_ssl_domains ssl::server_name .evernote.com

acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump splice bypass_ssl_domains
ssl_bump stare all

I noticed you used "ssl::server_name_regex"; and I don't seem to find much reference to it with a quick search. I wonder what the difference between "ssl::server_name_regex" and "ssl::server_name"? Also, can you show me how you structured the URLs in your "url.nobump" file please?

Try to let PfSense handle the DNS requests with the NAT that might fix it.

I'll do a bit more testing with this.

dkzsys

@michmoor @JonathanLee - can I ask a favour from you please. Do you mind checking what you can see with your transparent or MITM/explicit proxy on, to this URL? https://www.tradingview.com/markets

When I have mine on (tried both transparent and MITM/explicit), I don't get the chart and missing quite a few other info as per screenshot below; once I bypass my IP address, the site is working properly.

Thanks.

Squid Proxy On

Squid Proxy Bypassed

JonathanLee

@dkzsys

Here is the URL accessed with transparent mode...

(SPLICED DEVICE)

Screenshot 2023-09-20 at 7.08.56 AM.png
(MITM DEVICE WITH ROOT AUTHORITY CERTIFICATES)

The firewall or proxy must be blocking something on that site and or the site does not allow proxy use with root certificates.

MITM devices you have to set some urls to splice as they sometimes do not work with the proxy, and some stuff you need to ethically set the urls to splice.

Here is some of my spice list urls my "nobump" file

Screenshot 2023-09-20 at 7.20.37 AM.png

so if you trust a site you could just add it to a nobump file so it works all the time. I use bigcharts

michmoor

I visited the site without issues. I am working with Transparent Mode only.

dkzsys

Thank you both, Mike @michmoor and Jonathan @JonathanLee!

So the behaviour from Jonathan's setups matches exactly what I am experiencing.

P.S. I have NAT all my DNS queries to pfSense now, to minimise the impacting factors.

Here is some of my spice list urls my "nobump" file

Thanks, Jonathan. This is helpful.

so if you trust a site you could just add it to a nobump file so it works all the time. I use bigcharts

Good choice:)

dkzsys

@michmoor said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

I visited the site without issues. I am working with Transparent Mode only.

That is great! Also makes me wonder why my setup (and Jonathan's) is having a very different result.

I have tried deleting the CA certificate; and I immediate get a cert error in transparent mode. Fixed after re-installing the cert.

Did you end up installing the CA Cert on your clients?

Can you cast your eyes over my config to see if there is any obvious discrepancy to your setup please? BTW, which version of pfSense are you on? Mine is 23.05.1-RELEASE (amd64)

In terms of DNS config and the corresponding FW rules (block requests to WAN), I'm now pointed back to pfSense to alignment. One factor eliminated at least.

JonathanLee

@dkzsys for MITM to work you must install certificates on your devices you run SSL intercept with.

When using both transparent and SSL intercept like you and I are, for transparent to work we must define what urls and or devices require splicing.

Not everything can be SSL intercepted, example being banks, private messages, and most smartphones. Think ethically, what should be spliced? It's powerful. It's unethical to open people's email or text messages so that would be an obvious spliced connection. However, the splice setting still allows you to perform basic URL filtering and blocking. Example, block all the HTTP GET requests for adult websites, or maybe block tictok access, finally you could install timers for YouTube for your kids and have the firewall block YouTube access after 10pm. You just won't inspect everything in those spliced connections or devices. Leading to, even with SSL intercept and your devices running a root certificate authority not all websites allow SSL interception. They could block you too from access.

I like to block sites that use AI tracking. Tracking like this evades privacy and thus it's not legal, by way of the new CCPA and GDPR laws that protect privacy. Most advertising companies ignore the privacy laws right now. They don't care if a user has set their browser to request no tracking. That's when URL blocking comes in.

Another example of devices that you can not do SSL interception with is an iPhone and Amazon fire tablets, you can not have them set to MITM ssl intercept, if you do none of the built in applications work only the browser will. I still install certificates anyway and set those devices as splice only.

For Windows 11 you must install the root certificates on the devices or it will not trust them, however transparent mode works without them as long as wpad or dhcp option 252 lands the device on the Proxy.

Transparent only mode allows basic URL filtering/inspection. Squid even has a YouTube API for age restrictions that works in transparent mode.

SSL intercept allows deep packet inspection and ClamAV works very effectively. It's harder to configure as you have to fine tune it.

Some sites will not allow SSL interception so splice them.

You are required to install certificates or it will show as untrusted for SSL intercept.

This is a security feature as you can only install root certificates on the devices you physically own. So Windows warns you when a certificate is being used that has not been marked trusted. It's warning the end users that traffic is going to a proxy.

Install the certificate and mark it as a trusted root authority to correct the issue you are seeing.

Here is another spliced device example: you bring home your work laptop, you can not legally install a certificate on that device because you don't own it. So, that device would be a splice only setup and use your proxy that way in transparent mode.

This is how most enterprise class firewalls work. They use certificates on the devices that in turn interact with the firewalls.

JonathanLee

@dkzsys if your not using a server for forwarding change this setting to transparent.

I had issues with this setting in the past.

dkzsys

@JonathanLee said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

@dkzsys for MITM to work you must install certificates on your devices you run SSL intercept with.
......
Here is another example, you bring home your work laptop, you can not legally install a certificate on that device so, that device would be a splice only setup.

Great context, Jonathan - adding the dimensions of ethic, commercial implementation and practicality.

My challenge with the SSL intercept is the continuous extension on the splicing as we start using new apps or web services. Ideally, I'd like to have it as set-and-forget and expect it to work most of the time, like 95%, and I am ok with the occasional manual effort. Just like what seems to be working for @michmoor. I am really curious on what the differences are in terms of implementation between his config and ours - the example for the TV charting data is quite revealing.

I'm also going to check out the squidGuard features today; to see how to best utilise it for (ad) blocking and access control. Do you have any recommendation on:

list itemany blacklist,
how best to implement the self-defined block list? (let's say roblox service. I had a quick play with Target Categories - toggling the redirect seems to take quite a long time; either I ran out of patient or it was ineffective. Tried different browsers. Ended up disabling squidGuard.)

JonathanLee

@dkzsys I use Squidguard, it takes time to configure.

But it does allow a URL blacklist that is pretty effective.

http://dsi.ut-capitole.fr/blacklists/download/blacklists_for_pfsense_reducted.tar.gz

I love this blacklist as it's reduced for my 2100

I made my Roblox list with help of their website for proxy/school district usage setup also approves my son's Roblox

I agree Mike has his set such that no certificates are required quicker and still effective. Again, I need the SSL intercept for viruses and major issues with my old job. I had a target on my back for years so bad that I couldn't even watch Netflix I would get hit with pings of death non stop and many other issues. I still find residual invasive actors and containers to this day even on my personal devices.

michmoor

@JonathanLee why set the XFF ito transparent?

michmoor

@JonathanLee said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

@dkzsys I use Squidguard, it takes time to configure.

But it does allow a URL blacklist that is pretty effective.

http://dsi.ut-capitole.fr/blacklists/download/blacklists_for_pfsense_reducted.tar.gz

I love this blacklist as it's reduced for my 2100

I made my Roblox list with help of their website for proxy/school district usage setup also approves my son's Roblox

I agree Mike has his set such that no certificates are required quicker and still effective. Again, I need the SSL intercept for viruses and major issues with my old job. I had a target on my back for years so bad that I couldn't even watch Netflix I would get hit with pings of death non stop and many other issues. I still find residual invasive actors and containers to this day even on my personal devices

This seems like a very interesting story I must hear one day.

JonathanLee

@michmoor I had a container VM inside .trashes and the v100 spotlight folders that was all the sudden 1tb in size and that was running under within a system integrity environment. I got it removed and recovered my resource in the end, but come on, hide a VM in the .trashes folder. That was just last week. I kept seeing weird snort blocks and I started investigating it, I was very surprised.

JonathanLee

@michmoor the forum all the sudden was seeing my private LAN address getting passed a couple months ago with it set to on.

dkzsys

@JonathanLee said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

@dkzsys if your not using a server for forwarding change this setting to transparent.

I had issues with this setting in the past.

Good one - will try to update the setting.

But it does allow a URL blacklist that is pretty effective.

http://dsi.ut-capitole.fr/blacklists/download/blacklists_for_pfsense_reducted.tar.gz

I love this blacklist as it's reduced for my 2100
I made my Roblox list with help of their website for proxy/school district usage setup also approves my son's Roblox

Thanks - will try them out.

@ michmoor I had a container VM inside .trashes and the v100 spotlight folders that was all the sudden 1tb in size and that was running under within a system integrity environment. I got it removed and recovered my resource in the end, but come on, hide a VM in the .trashes folder. That was just last week. I kept seeing weird snort blocks and I started investigating it, I was very surprised.

Never short of creativity in this world!