Setting low TTL to fix Squid issue

michmoor · Sep 18, 2023, 7:13 PM

Is there a way to set a low TTL for dns responses in unbound?

Backgroup: Im running Squid Proxy and every now and then google.com will stop responding. After some time it will come back but today is the day i decided to get curious as to why.
Squid logs show a /409 error so that got me thinking DNS related.
I then run a pcap and that confirms this is a DNS problem

I checked out the DNS queries and responses and the dns response IPs are different than from what my client is requesting
Client requesting 74.x.x.x
Pfsense resolves google.com to a 64.x.x.x

Ok now that i know the problem im wondering if there is a way to send a low TTL to clients. Im thinking Chrome is caching the response longer than what the TTL is when pfsense queries for the domain. So lets say pfsnse sees the TTL as 10, and chrome behind the scenes sets it to 20.

edit: To anyone wondering, im using pfsense as my DNS server and have done a pretty good job of blocking known DoH servers. Chrome is also configured to use pfsense
for dns.

michmoor · Sep 18, 2023, 7:52 PM

Thinking about it a bit more i dont think TTL is my issue.
Chrome itself is the issue. This problem does not appear on Edge or FireFox.
Reviewing the screenshot above, I disabled "Use secure DNS" and so far the problem is gone.

stephenw10 · Sep 18, 2023, 8:16 PM

If Chrome and Squid are not using the same DNS resolver that can happen. Usually you want both to be using Unbound in pfSense, the cached results there means both will get he same IP.

I would guess Chrome is using DoH or 8.8.8.8 directly.

Steve

michmoor · Sep 18, 2023, 8:39 PM

@stephenw10
If Chrome was using some other external DoH i probably wouldve known -- maybe.
Im using a list in pfblocker to block DoH servers. Its possible that Chrome is using an IP not on the list.
Turning off the setting of "Use secure DNS" seems to have resolved the issue so far although it was set already to using my "current service provider".

The 409 Conflict error exists in lots of apps especially MSFT Teams and O365.
Even with using pfsense as the DNS Resolver. My suspicion is that perhaps the IPs are hard-coded within the apps but also the server names used are in DNS hence the conflict. Its possible. I dont know.

JonathanLee · Sep 18, 2023, 9:02 PM

Are you using IPv6 too?

michmoor · Sep 18, 2023, 9:08 PM

@JonathanLee nope.

Edit: you mean on pfsense or on client?

JonathanLee · Sep 19, 2023, 1:43 AM

@michmoor ISP side I had an issue where I needed to add advanced config into the resolver because of the same issue. Again I block DoH

JonathanLee · Sep 19, 2023, 2:51 AM

@michmoor

(I had to add specific DNS resolver settings to get around the issue you are having as my ISP as it does not provide ipv6)

Items kept requesting AAAA over just ipv4 records

(I also had to add a some Squidguard target categories to block DoH)

(I also added specific HTTPS blocks to the DNS resolvers I use)

(I also use port 853 for my resolver DNS over TLS SSL)

(Last I added a NAT for all DNS requests to send them to the fire wall)

Ref:
https://docs.netgate.com/pfsense/en/latest/recipes/dns-over-tls.html

This fixed all Google trying to bypass the proxy issues

ps.
Unbound resolver can also be configured to perform DoH resolving manually with advanced options as the GUI has not added it yet. Again, you have to point users to it. I got it running for a bit a while back on my 2100.

Ref:
https://unbound.docs.nlnetlabs.nl/en/latest/topics/privacy/dns-over-https.html
https://forum.netgate.com/topic/181338/feature-request-gui-options-to-unbound-resolver-s-new-doh-abilities
https://redmine.pfsense.org/issues/14558

JonathanLee · Sep 19, 2023, 2:57 AM

@michmoor did you manually block https for the DNS servers you use on the firewall?

Please let me know how you got around this firewall puzzle if you found another way with Squid / Squidguard use. I changed mine to stare all over stare step 2 bump step 3 it seemed to help. As I learned its essentially the same thing from our last research on Squid.

michmoor · Sep 19, 2023, 2:25 PM

@JonathanLee I block as much DoH as i can based on the pfblocker filter ive applied.
My upstream DNS is CloudFlare. I am not using my ISP dns servers.
I am already blocking external DNS and DoT.

I created a floating rule

I created an alias deny

Custom Unbound settings

michmoor · Sep 19, 2023, 3:20 PM

@michmoor said in Setting low TTL to fix Squid issue:
@JonathanLee

So a common problem which i really dont know why it happens is why there is a problem specifically with Chrome.
Chrome will clearly be in conflict with whats in the dns cache and here are the errors
For example, if i visit Twitter (X) i am unable to load any pictures or video.
Wireshark reveals the reason why.
Its always DNS. :)

Problem outlined: https://www.squid-cache.org/Doc/config/host_verify_strict/
Dont really know how to implement this host verify strict command..

JonathanLee · Sep 19, 2023, 11:19 PM

@michmoor

http://www.squid-cache.org/Doc/config/host_verify_strict/

Have you attempted to set this in advanced options to on or off? It's default is off. I am having this same issue with Apple music and mzstatic making it's own get requests.

https://forum.netgate.com/topic/182866/universal-procedure-pointers-upp-mzstatic-com-s-mode-of-access-redirector-question/

I am having the opposite it's apparently approving connection as splice with the same IP sometimes.

I think under advanced options is where it needs to be.

host_verify_strict on
host_verify_strict off

michmoor · Sep 20, 2023, 12:38 AM

@JonathanLee I did apply it under advanced options but doesnt seem to have any impact. I still get the /409 errors.
Do you know where the squid conf file is? I wonder if its really set..

i got so desperate i set it in each box lol

michmoor · Sep 20, 2023, 5:32 AM

But to my point about devices with hard coded dns servers

Look at this. My IoT television gets its DHCP from my pfsenese. Pfsense hands its IP out as the DNS server yet as you can see from pfblocker its still requesting a google dns.
Its hard coded in a lot of these devices which is an issue but dont think thats why squid breaks.

JonathanLee · Sep 20, 2023, 3:10 AM

@michmoor

Try

/usr/local/etc/squid/squid.conf

JonathanLee · Sep 20, 2023, 5:32 AM

@michmoor

I wanted to check with you on Netflix, I forgot to mention I have Hulu and other streaming services set to no cache. Are you attempting to cache Netflix?

I set them to never cache on Squid

Did you set a NAT for the DNS rules to force all devices to use the firewall?

That should help if you NAT it.

JonathanLee · Sep 20, 2023, 6:54 AM

@michmoor Are you blocking port 53 for 8.8.8.8? it shows a red lock, it should only block port 443 for 8.8.8.8, 53 is the standard or if you use dns over tls ssl 853 that might be the issue, TheGreatWall_DoH is blocking standard port 53 over just the 443 DoH access.

Create a NAT rule for all DNS requests that are not being sent to the firewall or it's loopback. And force it to go to the firewall.

NollipfSense · Sep 20, 2023, 1:51 PM

Interesting conversation here, indeed...thanks for sharing!

michmoor · Sep 20, 2023, 2:36 PM

@JonathanLee
Im blocking 53 and 443
The red lock in pfblocker should indicate that traffic is being blocked - sinkholed so dont think theres any worry about that.

Floating Rule

So in my case theres no reason i can think of to use Port Forards.

JonathanLee · Sep 20, 2023, 2:42 PM

@michmoor with pfBlocker don't you still need to redirect the clients that are ignoring the DNS settings still? That is interesting, I always have clients that will attempt to use a different DNS all the time with NAT it doesn't matter they go where I configured them too unless they use some new experimental protocol.