Setting low TTL to fix Squid issue

[michmoor]

@stephenw10
If Chrome was using some other external DoH i probably wouldve known -- maybe.
Im using a list in pfblocker to block DoH servers. Its possible that Chrome is using an IP not on the list.
Turning off the setting of "Use secure DNS" seems to have resolved the issue so far although it was set already to using my "current service provider".

The 409 Conflict error exists in lots of apps especially MSFT Teams and O365.
Even with using pfsense as the DNS Resolver. My suspicion is that perhaps the IPs are hard-coded within the apps but also the server names used are in DNS hence the conflict. Its possible. I dont know.

[JonathanLee]

Are you using IPv6 too?

[michmoor]

@JonathanLee nope.

Edit: you mean on pfsense or on client?

[JonathanLee]

@michmoor ISP side I had an issue where I needed to add advanced config into the resolver because of the same issue. Again I block DoH

[JonathanLee]

@michmoor

(I had to add specific DNS resolver settings to get around the issue you are having as my ISP as it does not provide ipv6)

Items kept requesting AAAA over just ipv4 records

(I also had to add a some Squidguard target categories to block DoH)

(I also added specific HTTPS blocks to the DNS resolvers I use)

(I also use port 853 for my resolver DNS over TLS SSL)

(Last I added a NAT for all DNS requests to send them to the fire wall)

Ref:
https://docs.netgate.com/pfsense/en/latest/recipes/dns-over-tls.html

This fixed all Google trying to bypass the proxy issues

ps.
Unbound resolver can also be configured to perform DoH resolving manually with advanced options as the GUI has not added it yet. Again, you have to point users to it. I got it running for a bit a while back on my 2100.

Ref:
https://unbound.docs.nlnetlabs.nl/en/latest/topics/privacy/dns-over-https.html
https://forum.netgate.com/topic/181338/feature-request-gui-options-to-unbound-resolver-s-new-doh-abilities
https://redmine.pfsense.org/issues/14558

[JonathanLee]

@michmoor did you manually block https for the DNS servers you use on the firewall?

Please let me know how you got around this firewall puzzle if you found another way with Squid / Squidguard use. I changed mine to stare all over stare step 2 bump step 3 it seemed to help. As I learned its essentially the same thing from our last research on Squid.

[michmoor]

@JonathanLee I block as much DoH as i can based on the pfblocker filter ive applied.
My upstream DNS is CloudFlare. I am not using my ISP dns servers.
I am already blocking external DNS and DoT.

I created a floating rule

I created an alias deny

Custom Unbound settings

[michmoor]

@michmoor said in Setting low TTL to fix Squid issue:
@JonathanLee

So a common problem which i really dont know why it happens is why there is a problem specifically with Chrome.
Chrome will clearly be in conflict with whats in the dns cache and here are the errors
For example, if i visit Twitter (X) i am unable to load any pictures or video.
Wireshark reveals the reason why.
Its always DNS. :)

Problem outlined: https://www.squid-cache.org/Doc/config/host_verify_strict/
Dont really know how to implement this host verify strict command..

[JonathanLee]

@michmoor

http://www.squid-cache.org/Doc/config/host_verify_strict/

Have you attempted to set this in advanced options to on or off? It's default is off. I am having this same issue with Apple music and mzstatic making it's own get requests.

https://forum.netgate.com/topic/182866/universal-procedure-pointers-upp-mzstatic-com-s-mode-of-access-redirector-question/

I am having the opposite it's apparently approving connection as splice with the same IP sometimes.

I think under advanced options is where it needs to be.

host_verify_strict on
host_verify_strict off

[michmoor]

@JonathanLee I did apply it under advanced options but doesnt seem to have any impact. I still get the /409 errors.
Do you know where the squid conf file is? I wonder if its really set..

i got so desperate i set it in each box lol

[michmoor]

But to my point about devices with hard coded dns servers

Look at this. My IoT television gets its DHCP from my pfsenese. Pfsense hands its IP out as the DNS server yet as you can see from pfblocker its still requesting a google dns.
Its hard coded in a lot of these devices which is an issue but dont think thats why squid breaks.

[JonathanLee]

@michmoor

Try

/usr/local/etc/squid/squid.conf

[JonathanLee]

@michmoor

I wanted to check with you on Netflix, I forgot to mention I have Hulu and other streaming services set to no cache. Are you attempting to cache Netflix?

I set them to never cache on Squid

Did you set a NAT for the DNS rules to force all devices to use the firewall?

That should help if you NAT it.

[JonathanLee]

@michmoor Are you blocking port 53 for 8.8.8.8? it shows a red lock, it should only block port 443 for 8.8.8.8, 53 is the standard or if you use dns over tls ssl 853 that might be the issue, TheGreatWall_DoH is blocking standard port 53 over just the 443 DoH access.

Create a NAT rule for all DNS requests that are not being sent to the firewall or it's loopback. And force it to go to the firewall.

[NollipfSense]

Interesting conversation here, indeed...thanks for sharing!

[michmoor]

@JonathanLee
Im blocking 53 and 443
The red lock in pfblocker should indicate that traffic is being blocked - sinkholed so dont think theres any worry about that.

Floating Rule

So in my case theres no reason i can think of to use Port Forards.

[JonathanLee]

@michmoor with pfBlocker don't you still need to redirect the clients that are ignoring the DNS settings still? That is interesting, I always have clients that will attempt to use a different DNS all the time with NAT it doesn't matter they go where I configured them too unless they use some new experimental protocol.

[michmoor]

@JonathanLee I will try adding a nat redirect just to test to see if it makes things better.

To your other question, i am not using Squid Proxy to cache anything.

[JonathanLee]

@michmoor I love the cache I have DSL/ copper it acts as an accelerator

[michmoor]

@JonathanLee ahhh
Im on 500/500 Fiber.