Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception

dkzsys

Greetings,

I have been spending quite some time trying to make the Squid Proxy working in Transparent HTTP Proxy Mode + HTTPS/SSL Interception enabled.

However, I keep running into issues with various apps, mainly Apple Store and Evernote. (Note that I haven't get around to other apps).

The approach I implemented the Squid Proxy is:

Transparent HTTP Proxy Mode + HTTPS/SSL Interception enabled
Defined Firewall Alias
Include the defined firewall alias in the "Bypass Proxy for These Destination IPs"
Have the following config in "Custom Options (Before Auth)"

acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump stare all

Some screenshots here for ease of digestion:

My Challenge
It seems that what I put in the Firewall Alias for destination IP bypass is not effective for some reason.

Example: ocsp2.g.aaplimg.com - one of the apple URLs
It's resolved to 17.253.121.201 and 17.253.121.202.

~ ❯❯❯ nslookup ocsp2.apple.com
Server:         1.0.0.1
Address:        1.0.0.1#53

Non-authoritative answer:
ocsp2.apple.com canonical name = ocsp2-lb.apple.com.akadns.net.
ocsp2-lb.apple.com.akadns.net   canonical name = ocsp2.g.aaplimg.com.
Name:   ocsp2.g.aaplimg.com
Address: 17.253.121.202
Name:   ocsp2.g.aaplimg.com
Address: 17.253.121.201

I have both the FQDN and the IP addresses in the Firewall Alias "squid_bypass"; however, I am still seeing the Squid access log, indicating it's not being bypassed.

1695005343.772      3 10.0.1.11 TCP_MISS/410 586 GET https://ocsp2.apple.com/ocsp03-wwdr04/MEcwRTBDMEEwPzAHBgUrDgMCGgQUAOsMzPRxbLpGI6PVL5jUPfYwqR0EFIgnFwmpthhgi%2BzruvZHWcVSVKO3AggO61eH554JjQ%3D%3D - ORIGINAL_DST/17.253.121.201 text/html

Any suggestion on how to resolve it, or setup Squid properly would be much appreciated.

Perhaps I get can some help from @CaliPilot here?

Thanks in advance.

Including the "squid_bypass" list here for reference: squid_bypass bulk list.txt

dkzsys

A separate problem - thought I'd add to the list.

I am still unable to Update apps from mac osx App Store, even after I bypass destination IPs for all the FQDNs (excluding wildcard sub domains - is there a way to do the wildcard??) on this Apple page https://support.apple.com/en-us/HT210060.

The following squid access log was captured when I attempted the update in app store.

Apart from knowing they are Apple IP addreses, no idea which domain they are for.

Any idea on

a) what happens here? (i.e. have I missed any host/FQDN? not showing in log though), and
b) is there a way to do wildcard sub domain bypass?

1695024181.929    713 10.0.1.11 NONE_NONE/000 0 CONNECT 17.36.202.158:443 - ORIGINAL_DST/17.36.202.158 -
1695024182.889    733 10.0.1.11 NONE_NONE/000 0 CONNECT 17.36.202.158:443 - ORIGINAL_DST/17.36.202.158 -
1695024183.749    692 10.0.1.11 NONE_NONE/000 0 CONNECT 17.36.202.158:443 - ORIGINAL_DST/17.36.202.158 -
1695024184.573    680 10.0.1.11 NONE_NONE/000 0 CONNECT 17.36.202.158:443 - ORIGINAL_DST/17.36.202.158 -
1695024185.221     87 10.0.1.11 NONE_NONE/000 0 CONNECT 23.206.199.19:443 - ORIGINAL_DST/23.206.199.19 -
1695024185.442     75 10.0.1.11 NONE_NONE/000 0 CONNECT 23.206.199.19:443 - ORIGINAL_DST/23.206.199.19 -
1695024185.606     74 10.0.1.11 NONE_NONE/000 0 CONNECT 23.206.199.19:443 - ORIGINAL_DST/23.206.199.19 -
1695024185.797     80 10.0.1.11 NONE_NONE/000 0 CONNECT 23.206.199.19:443 - ORIGINAL_DST/23.206.199.19 -
1695024185.967     73 10.0.1.11 NONE_NONE/000 0 CONNECT 23.206.199.19:443 - ORIGINAL_DST/23.206.199.19 -
1695024189.007     82 10.0.1.11 NONE_NONE/000 0 CONNECT 203.87.122.80:443 - ORIGINAL_DST/203.87.122.80 -
1695024189.144     66 10.0.1.11 NONE_NONE/000 0 CONNECT 23.34.237.181:443 - ORIGINAL_DST/23.34.237.181 -

michmoor

@dkzsys I have the same set up except for any custom ssl options.
It works 95% of the time but there are some apps that don’t like a proxy no matter what.
My Microsoft teams (along with MSFT Authenticator) and certain apple domains such as those related to iCloud. So in my testing and experience there are two ways to solve this whole issue with transparent mode

Switch to explicit proxy. This resolves all the quirky problems but you still will have to make exceptions. But it’s the best solution.
While in transparent mode create an Alias so that certain IPs bypass proxy. That’s what I’ve done. You will have to be ok with certain devices not going through a proxy. Make sure you update the firewall rules to account for those bypassed IPs. Downside is you lose any reporting for those IPs as LightSquid won’t tell you about any sites visited.

michmoor

@dkzsys
“ b) is there a way to do wildcard sub domain bypass?”

Funny you brought this up as this was going to be my post today. Don’t think there’s a way to use a wildcard domain so they won’t get decrypted. I would love that as there a bunch of MSFT domains that should not be accessed behind a proxy (per documentation). This is desperately needed in transparent or explicit mode. I know other vendors have an option that states “No decryption done on these domains”.

The problem. Squid, from what I can tell, has no active maintainer in pfsense so asking for a feature request is pointless but you could submit a redmine.

dkzsys

@michmoor said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

Thanks for your input, @michmoor!

@dkzsys I have the same set up except for any custom ssl options.

That's awesome - makes it much easier to explain and relate!

I have just removed the configs under the custom ssl options. I used to have acl rules there to bypass source IP addresses - I've migrated that to "Bypass Proxy for These Source IPs", which is much more effective. And the specific peek and stare config doesn't seem to do much. I am reviewing the squid feature doc again in case I missed anything - https://wiki.squid-cache.org/Features/SslPeekAndSplice.

It works 95% of the time but there are some apps that don’t like a proxy no matter what.
My Microsoft teams (along with MSFT Authenticator) and certain apple domains such as those related to iCloud. So in my testing and experience there are two ways to solve this whole issue with transparent mode

Switch to explicit proxy. This resolves all the quirky problems but you still will have to make exceptions. But it’s the best solution.

I'll try this, as a plan B. So do you just change to explicit proxy, with the same firewall alias for destination IP exclusion, and update client with the proxy details?

While in transparent mode create an Alias so that certain IPs bypass proxy. That’s what I’ve done. You will have to be ok with certain devices not going through a proxy. Make sure you update the firewall rules to account for those bypassed IPs. Downside is you lose any reporting for those IPs as LightSquid won’t tell you about any sites visited.

That's exactly what I have done as well, i.e. source IP bypass. For context, I setup the Squid Proxy specifically to monitor my children's internet activity, on Mac and iPad. So I do need their devices enabled for monitoring or reporting (tail, or lightsquid)

So I ended up source bypass all other devices (so I don't have to suffer the semi-broken transparent proxy with ssl_bump); but keep it enabled for my children's devices. Problem is that it's still breaking their gaming, such as Roblox. (why not..:P) And I have to manually source bypass during their gaming time. Not the end of the world for now.

michmoor

@dkzsys I am right now using Squid as a Transparent proxy. I have used it as an explict proxy in years past and ive had no issues.
Transparent is just a different beast. In theory sites shouldnt fail to load as there is no MITM going on but there is clearly some indication to the client that they are sitting behind a proxy. Dont know how they know but i surmise they do.

@dkzsys said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

That's exactly what I have done as well, i.e. source IP bypass. For context, I setup the Squid Proxy specifically to monitor my children's internet activity, on Mac and iPad. So I do need their devices enabled for monitoring or reporting (tail, or lightsquid)

You might as well perform an explicit proxy setup then and load the firewall certiificate on those devices as you control them.

Strangly, my childrens Roblox is unaffected. No issues. Just make sure all clients are pointed to pfSense as the DNS server and also block DoH servers as well. This is also a pain point for some. You have to make sure that all clients are getting the DNS response from pfsense and not from an external DoH or DoT server otherwise that breaks connectivity as well. So using pfBlocker i created a DoH alias list out of the TheGreatWall_DoH_IP feeds list. From there i applied that alias to a floating rule that matches all my interfaces. Now i block DoH on all my LANs. When a client cant reach its DoH server it falls back to my pfsense.
Apple devices for example use 1.1.1.1 (cloudflare dns). It gets a different IP from a dns query than the pfsense does which leads to a break in communications at times (/409 error).

@periko Is pretty good with Proxy setups on pfsense. Do you have any thoughts on this?

dkzsys

@michmoor said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

@dkzsys I am right now using Squid as a Transparent proxy. I have used it as an explict proxy in years past and ive had no issues.
Transparent is just a different beast. In theory sites shouldnt fail to load as there is no MITM going on but there is clearly some indication to the client that they are sitting behind a proxy. Dont know how they know but i surmise they do.

Out of curiosity, what's your rational of changing from explicit proxy to transparent?
For the transparent mode, I have to fine-tune almost every app, Evernote, Discord, Roblox, even Grok (websocket)... It gets a bit painful and time consuming. And Apple App Store simply just doesn't work.

You might as well perform an explicit proxy setup then and load the firewall certiificate on those devices as you control them.
Strangly, my childrens Roblox is unaffected. No issues.

That's interesting. I was troubleshooting Roblox earlier today... Finally working after including an IP address 184.24.220.74 in the destination bypass rule. Still no idea which domain, if at all, it's resolved from.

1695093782.995    116 10.0.1.15 NONE_NONE/000 0 CONNECT 184.24.220.74:443 - ORIGINAL_DST/184.24.220.74 -

Just make sure all clients are pointed to pfSense as the DNS server and also block DoH servers as well. This is also a pain point for some. You have to make sure that all clients are getting the DNS response from pfsense and not from an external DoH or DoT server otherwise that breaks connectivity as well. So using pfBlocker i created a DoH alias list out of the TheGreatWall_DoH_IP feeds list. From there i applied that alias to a floating rule that matches all my interfaces. Now i block DoH on all my LANs. When a client cant reach its DoH server it falls back to my pfsense.
Apple devices for example use 1.1.1.1 (cloudflare dns). It gets a different IP from a dns query than the pfsense does which leads to a break in communications at times (/409 error).

Interesting points. Re "... DNS response from pfsense and not from an external DoH or DoT server otherwise that breaks connectivity as well", question is if pfSense DNS configured in Forwarding mode (to cloudflare for example) and have DNS setup to also DoH to cloudflare, they are considered having the same DNS resolver with identical resolution, right?

The way I set up my DNS is:

Pfsense DNS Resolver in Forwarding mode to Cloudflare.
Block DoH from clients to Cloudflare directly. (others DoH IPv4 from the TheGreatWall_DoH_IP feed to be added later today)
Main DNS using AdGuard set up on RaspberryPi, with DoH enabled to Cloudflare (explicit fw allow rule)
NAT DNS 53 to AdGuard (I wrote a script to fallback the NAT to pfsense, but never implement it)

I use AdGuard mainly for the ease of toggle (e.g. Blocking Roblox service with 1 click and effective almost immediately) and domain block.

You reckon this setup may be breaking things in Squid transparent mode?

@periko Is pretty good with Proxy setups on pfsense. Do you have any thoughts on this?

Would be keen to get some thoughts/inputs from @periko

michmoor

@dkzsys said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

Out of curiosity, what's your rational of changing from explicit proxy to transparent?

I just didnt want to deal with loading certificates on every device and i just needed to do granular domain blocking which you cant do in pfblocker and I also wanted the LightSquid reporting.

I do not have any of the Roblox side issues you are seeing but in my experience its almost always that the client is using a different DNS server than what the proxy is using. For example, my iPad shows that my DNS is my pfsense but behind the scenes its using 1.1.1.1 on port 443 to reach out for DNS as well -- Thats Cloudflare.
So we know some machines are hard coded to reach out to different server IPs regardless of whats given to them in DHCP.
My Roku TV is the same. Reaches out to 8.8.4.4 even though through DHCP it gets my pfsense dns.
To be fair...IoT devices shouldnt be kept behind a proxy anyway. Too much work and not much to gain but i see some beneift in a home environment where you got kids you want to monitor a bit tightly.

My LANs all use pfsense. I block external DNS (53), i block as best i can DoH servers and I block DoT (853). The majority of my problems are solved now and Transparent proxy is pretty effective.
You will still run into these one-off issues where a /409 Conflict message will come up which indicates a discrepancy in what the clietn thinks the IP address is of the site and what pfsense thinks it is. In those cases I usually opt to bypass proxy for those IPs.

dkzsys

I just didnt want to deal with loading certificates on every device and i just needed to do granular domain blocking which you cant do in pfblocker and I also wanted the LightSquid reporting.

@michmoor I thought after enabling HTTPS/SSL Interception, even in transparent mode, clients could still complain about CA cert? as per https://agix.com.au/pfsense-as-a-transparent-proxy/.

I do not have any of the Roblox side issues you are seeing but in my experience its almost always that the client is using a different DNS server than what the proxy is using. For example, my iPad shows that my DNS is my pfsense but behind the scenes its using 1.1.1.1 on port 443 to reach out for DNS as well -- Thats Cloudflare.
So we know some machines are hard coded to reach out to different server IPs regardless of whats given to them in DHCP.
My Roku TV is the same. Reaches out to 8.8.4.4 even though through DHCP it gets my pfsense dns.
To be fair...IoT devices shouldnt be kept behind a proxy anyway. Too much work and not much to gain but i see some beneift in a home environment where you got kids you want to monitor a bit tightly.

My LANs all use pfsense. I block external DNS (53), i block as best i can DoH servers and I block DoT (853). The majority of my problems are solved now and Transparent proxy is pretty effective.
You will still run into these one-off issues where a /409 Conflict message will come up which indicates a discrepancy in what the clietn thinks the IP address is of the site and what pfsense thinks it is. In those cases I usually opt to bypass proxy for those IPs.

I have modified my DNS implementation last night:

Block DoH to all external servers as per https://github.com/Sekhan/TheGreatWall
Block DoT (853) to WAN (I forgot to mention about this yesterday)
Block DNS (53) to WAN (I forgot to mention about this yesterday)
Allow DNS (53) to pfSense (probably redundant)
Allow DoH to Cloudflare (1.1.1.1 and 1.0.0.1) from AdGuard Home (RaspberryPi LAN IP)
AdGuard Home configured with DoH pointing to Cloudflare (1.1.1.1 and 1.0.0.1)
pfSense DNS Server Setting to Cloudflare.
pfSense DNS Resolver enabled, with "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers" and "Enable Forwarding Mode"
Use NAT from LAN to DNS (53) to control the local DNS destination
** Either pfSense, or
** AdGuard Home on RaspberryPi

What you said about DNS resolution discrepancy makes sense. I have tried NATing to PfSense and AdGuard Home separately, but my Roblox problem is still there. (This time at a different glitch/break point... to 23.206.199.219 and 23.206.199.226. Another good indication of potential DNS issue for these missing IP addresses.)

NAT is pretty standard implementation; so thought NATing to pfSense would do the trick, but no luck. For your pfSense DNS setup, do you use similar approach?

pfSense DNS Server Setting to Cloudflare (or one of your choice).
pfSense DNS Resolver enabled, with "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers" and "Enable Forwarding Mode"

michmoor

@dkzsys the whole point of a transparent proxy is that you dont have to load a certificate on any client. You cant do content inspection but for certain networks thats not really important.
Roblox works fine on the kids iPad or fire tablet so not sure whats the error your getting. You can review the Squid Real-Time logs for that.
As long as you block an external dns request that solves most of the problems. Not all. Like i mentioned already some applications do not like being behind a proxy. MSFT Teams or Office365. Thats easy to solve as you create an alias with those networks that you apply in the squid configuration.

make sure you have a corresponding firewall rule to now match on that alias as well.

dkzsys

@michmoor I did some more testing and found another contributing factors for roblox - the DNS resolution for setup.rbxcdn.com, which is used by Roblox on mac. The cname changes every so often - haven't seen any concrete patterns yet. So the timing of pfSense alias dns resolution update vs the cname and IP resolved at the time of execution can make a difference. Quite a rabbit hole there:)

JonathanLee

@dkzsys said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump stare all

I noticed you said
"acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump stare all"

This is mine if it helps at all. I have some devices set to splice always like Smartphones etc.

acl splice_only src 192.168.1.7 #Jon Android
acl splice_only src 192.168.1.8 #Tasha Apple
acl splice_only src 192.168.1.11 #Amazon Fire
acl splice_only src 192.168.1.15 #Tasha HP
acl splice_only src 192.168.1.16 #iPad
acl splice_only src 192.168.1.18 #Xbox
acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"
acl step1 at_step SslBump1
ssl_bump peek step1 #peek at all at step 1 so the block list works in Squidguard or allow list works
ssl_bump splice splice_only # now splice the above LAN ips
ssl_bump splice NoSSLIntercept # now splice the URLS like banks Hulu etc
ssl_bump stare all # the rest are always MITM Intercepted.

Both Transparent and MITM work this way.

Yesterday I changed it from I would get tons of hits from the cache this way.
ssl_bump peek step1 #peek at all at step 1 so the block list works in Squidguard or allow list works
ssl_bump splice splice_only # now splice the above LAN ips
ssl_bump splice NoSSLIntercept # now splice the URLS like banks Hulu etc
ssl_bump stare step2 # the rest are always MITM Intercepted.
ssl_bump bump step3

However, this is the same as stare all includes bump on step 3.

Again, you can arrange it a number of different ways.

As for the DNS I noticed you are having it not respond with the Server as the firewall itself see below

Notice 192.168.1.1

This works because of some NAT rules see below for example.

Anything that is using port 53, 853, 953 that is not that is my negated rule ! going to the firewall network address translate it for me to the firewall 192.168.1.1 without the clients knowing. That way the request is served by way of the firewall. This way the firewalls loopback is not of concern as it is going to where it needs to be, without 127.0.0.1 the firewall will not work with the DNS, you need both in the NAT 127.0.0.1 and 192.168.1.1 if you are using IPV6 you also need that loopback.

I have the firewall and loopback in the aliases. See Below

I only use IPv4 with my ISP

Ports that are redirected when they hit the firewall's NAT rule

I have it set to use port 853 I do not think it is any more secure but hey I was a cool think to learn about.

Check this out...

All port :53 only is seen on the loopback and the firewall's lan address 192.168.1.1

Nothing leaving the LAN does anything but the firewall itself.

Try to NAT it

dkzsys

@JonathanLee said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

"acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump stare all"

Thank you for the detailed response, Jonathan @JonathanLee !

Re the Squid Custom Options (Before Auth) config, as I do not use SquidGaurd, it's less attractive to me to configure it. I ended up removing the config and fell back to the simpler IP address (source and destination) bypass.

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"

I did try to implement ACL list to bypass destination URLs, but it wasn't effective for me.

As for the DNS I noticed you are having it not respond with the Server as the firewall itself see below

For DNS, I have configured 1.0.0.1 in DHCP and never bothered to remove that; but all DNS requests are NAT to AdGuard Home on RaspberryPi atm. I can easily update the NAT to pfSense.

JonathanLee

@dkzsys

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"

This is a file you would create with custom URLs that need to be spliced and not intercepted like banks etc you visit. If you have no file it won't work.
Mine is url.nobump

Try to let PfSense handle the DNS requests with the NAT that might fix it.

dkzsys

@JonathanLee said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

@dkzsys

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"

This is a file you would create with custom URLs that need to be spliced and not intercepted like banks etc you visit. If you have no file it won't work.
Mine is url.nobump

I used to have this configured:

# SSL Bypass for Domains
acl bypass_ssl_domains ssl::server_name .apple.com
acl bypass_ssl_domains ssl::server_name .bag.itunes.apple.com
acl bypass_ssl_domains ssl::server_name .cdn-apple.com
acl bypass_ssl_domains ssl::server_name .icloud.com
acl bypass_ssl_domains ssl::server_name .icloud-content.com
acl bypass_ssl_domains ssl::server_name .itunes.com
acl bypass_ssl_domains ssl::server_name .mzstatic.com
acl bypass_ssl_domains ssl::server_name .realtime.groklearning.com
acl bypass_ssl_domains ssl::server_name .dualsite-terminal.comp.gl
acl bypass_ssl_domains ssl::server_name .evernote.com

acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump splice bypass_ssl_domains
ssl_bump stare all

I noticed you used "ssl::server_name_regex"; and I don't seem to find much reference to it with a quick search. I wonder what the difference between "ssl::server_name_regex" and "ssl::server_name"? Also, can you show me how you structured the URLs in your "url.nobump" file please?

Try to let PfSense handle the DNS requests with the NAT that might fix it.

I'll do a bit more testing with this.

dkzsys

@michmoor @JonathanLee - can I ask a favour from you please. Do you mind checking what you can see with your transparent or MITM/explicit proxy on, to this URL? https://www.tradingview.com/markets

When I have mine on (tried both transparent and MITM/explicit), I don't get the chart and missing quite a few other info as per screenshot below; once I bypass my IP address, the site is working properly.

Thanks.

Squid Proxy On

Squid Proxy Bypassed

JonathanLee

@dkzsys

Here is the URL accessed with transparent mode...

(SPLICED DEVICE)

Screenshot 2023-09-20 at 7.08.56 AM.png
(MITM DEVICE WITH ROOT AUTHORITY CERTIFICATES)

The firewall or proxy must be blocking something on that site and or the site does not allow proxy use with root certificates.

MITM devices you have to set some urls to splice as they sometimes do not work with the proxy, and some stuff you need to ethically set the urls to splice.

Here is some of my spice list urls my "nobump" file

Screenshot 2023-09-20 at 7.20.37 AM.png

so if you trust a site you could just add it to a nobump file so it works all the time. I use bigcharts

michmoor

I visited the site without issues. I am working with Transparent Mode only.

dkzsys

Thank you both, Mike @michmoor and Jonathan @JonathanLee!

So the behaviour from Jonathan's setups matches exactly what I am experiencing.

P.S. I have NAT all my DNS queries to pfSense now, to minimise the impacting factors.

Here is some of my spice list urls my "nobump" file

Thanks, Jonathan. This is helpful.

so if you trust a site you could just add it to a nobump file so it works all the time. I use bigcharts

Good choice:)

dkzsys

@michmoor said in Squid Proxy "Bypass Proxy for These Destination IPs" Not Working?? Transparent HTTP Proxy Mode + HTTPS/SSL Interception:

I visited the site without issues. I am working with Transparent Mode only.

That is great! Also makes me wonder why my setup (and Jonathan's) is having a very different result.

I have tried deleting the CA certificate; and I immediate get a cert error in transparent mode. Fixed after re-installing the cert.

Did you end up installing the CA Cert on your clients?

Can you cast your eyes over my config to see if there is any obvious discrepancy to your setup please? BTW, which version of pfSense are you on? Mine is 23.05.1-RELEASE (amd64)

In terms of DNS config and the corresponding FW rules (block requests to WAN), I'm now pointed back to pfSense to alignment. One factor eliminated at least.