Blocking DNS over HTTPS. Seems the only way is to fire a shotgun at it



  • Re: Any way to TRULY block DNS over https (doh)?

    I administer an enterprise network over multiple physical sites. DNS over HTTPS is becoming a problem that can no longer be ignored in a corporate environment, so I've been examining ways to deal with it. The problem has stepped up a gear since Google decided to enable it by default in Chrome. I won't list the reasons DNS over HTTPS is a problem for corporate network admins, those of you know know already know so let's move on. . .

    I noticed more and more workstations punching holes through our Cisco umbralla, Neustar blocking etc etc so looked into it. I blew and entire evening trying to figure out why blocking 1.1.1.1, 1.0.0.1 on ALL ports wasn't working for Cloudflare's DNS over HTTPS. I read through lots of forum posts and Google announcements, trying to find any mechanism that might allow me to turn off the new Chrome feature by policy but there are no clear instructions that I can find. . .

    In the end I fired up a packet capture. Chrome's requests to Cloudflare DNS appears to sequentially check every Cloudflare IP until one answers. It's simply not enough to block 1.1.1.1, 1.1.1.2 etc etc. You have to block Cloudflare's entire IP space. I do it with an Alias.

    Here's a screenie from my test environment. . .

    testenv.png

    As far as I can see, this crude but effective means of bringing it under control, at this time, seems the only way to do it. Until Cloudflare (and others?) offer some kind of mechanism to block this in enterprise networks, blocking their entire IP space is necessary. Obviously this means it is now impossible to visit a site behind Cloudflare but I can live with that.

    If anyone has a better way, please do chime in, because this problem is only going to get worse.



  • Interesting, didn't know chrome did this by default now.

    It looks like you already did this,
    https://docs.netgate.com/pfsense/en/latest/recipes/dns-redirect.html

    And this,
    https://docs.netgate.com/pfsense/en/latest/recipes/dns-block-external.html

    Did you do that last part with the custom resolve options?

    Also there is an option for it in pfblockerNG-Devel if you use that. Not sure if this works for Chrome too.
    8e639507-f18c-4bcb-b733-3967688038c7-image.png



  • The custom resolve option only works with Firefox AFAIK, and such measures would require compliance from all programs and apps, which Malware vendors would just ignore anyway. Chrome doesn't use a canary domain. Chrome devs themselves state that they use a 'unique' method.

    About pfBlocker I'll look into that.



  • Yes, good point.

    I think I see the option you mentioned in chrome now. It's not clear to me what "with your current service provider" means but I'm guessing my current DNS service provider which is pfSense?
    48535e77-b712-475c-9594-412ae296870d-image.png

    I found this though, not sure if it applies in your scenario.
    https://blog.chromium.org/2020/05/a-safer-and-more-private-browsing-DoH.html
    c6e7480e-a776-47b7-b042-54ed0ed9f7c7-image.png



  • I do have the default option enable as I showed above in Chrome, so I did some quick testing. Running a packet capture, I'm seeing normal DNS requests as I would expect within my network and the requests are going straight to pfSense over 53.
    f6962b63-90d0-4ffc-966c-1c3e9ca6f81c-image.png

    I am doing DNS redirection as mentioned in the first link I provided, I am also using the Firefox DoH blocking option in pfblockerNG-Devel. I don't know if either of those are actually making it work for me but I can confirm it's not an issue in my setup.



  • What OS are you using? I've been unable to reproduce this on my Linux box. I can only make it happen on a Windows 10 machine with Chrome 85 in our Bangkok office, which I am controlling via VNC.



  • @Lanna said in Blocking DNS over HTTPS. Seems the only way is to fire a shotgun at it:

    What OS are you using? I've been unable to reproduce this on my Linux box. I can only make it happen on a Windows machine in our Bangkok office, which I am controlling via VNC.

    I'm using Windows 10 pro.
    c4a584ae-bff2-4a0c-9aac-326e5eb3f8a9-image.png



  • Looking at the settings, this appears to only have support on Windows. With Linux variants of Chrome and Chromium, the options are greyed out for the time being.
    Also, it's clear that we will have to keep an eye on which DNS providers are supported by Chrome. See here. . .
    https://www.chromium.org/developers/dns-over-https

    I am now blocking all of those with Aliases


  • LAYER 8 Global Moderator

    @Lanna said in Blocking DNS over HTTPS. Seems the only way is to fire a shotgun at it:

    You have to block Cloudflare's entire IP space. I do it with an Alias.

    That would be throwing out the baby with the bath water sort of solution.. Clouldflare serves up a huge chunk of the internet, if you blocked all ports to all of cloudflare IP space many a site would not work.. Not just dns lookups.

    That clarly can not be the case for sure - because for doh to work, a specific fqdn is used.. They can not just shotgun all of their space, nor could they have their fqdn resolve to all of their IPs.



  • Do you know how I might find that FQDN (assuming you are correct about that)?. If they are doing anycast on that FQDN and thus using anything in their IP space to serve up the DNS, I see no other solution that just just block it in it's entirety, and accept that any Cloudflare protected site is gone.

    To be clear, if I allow any specific Cloudflare subnet, DNS over HTTPS starts working again.


  • LAYER 8 Global Moderator

    @Lanna said in Blocking DNS over HTTPS. Seems the only way is to fire a shotgun at it:

    If they are doin anycast on that FQDN and thus using anything in their IP space

    anycast is not going to be ALL of their IP space... DOH operates on a fqdn, while sure it could go to anycast IP(s) there is not possible way it could go to all of their space..

    Yeah your solution works - sure, but you washed the baby, and then threw it out as well with the dirty water ;)

    Your going to sure have some unhappy users if you block all of cloudflare.. Easier solution from an enterprise standpoint is force use of proxy..



  • I know it's a crude solution, but it's all I have until something else is suggested.


  • LAYER 8 Global Moderator

    I hear you, and agree with you what the companies are doing is utter shitstorm... Looking out for the users my ass.. They just want the dns queries sent to them.. And they want them from each of their products directly so they can better track every single user vs a bunch of users hiding behind a caching server..

    These companies are not trying to better anything - they are finding more ways to monetize user data..

    This whole dot, doh is just one large shit show... That is is for damn sure.

    If your going to roll it out, it sure and the hell should be mandatory opt-in, and it should check for a canary that the local enterprise can put in place to make sure its turned off on any browser on the corp network, where dhcp handed the OS a corp dns server.

    Atleast with dot, port 853 its easy enough to block. Hiding it inside 443 is just more sneaky bs..

    Here is list of doh IPs I am using
    https://raw.githubusercontent.com/Sekhan/TheGreatWall/master/TheGreatWall_ipv4

    This it the IPs they list for cloudflare doh

    # Cloudflare
    1.1.1.1
    1.0.0.1
    104.16.248.249
    104.16.249.249
    104.18.2.55
    104.18.3.55
    104.18.27.128
    104.18.26.128
    

    Also when they control your dns - pretty difficult to block ads based on dns..

    Anyone that thinks this is doing anything but giving these companies more control and more info is blinded by the BS.. Trust us we will make you safer my F'ing ASS! ;)

    You can look here for a list of doh fqdn
    https://github.com/curl/curl/wiki/DNS-over-HTTPS

    And they have a script to help you parse it.



  • @johnpoz Thanks for that list, I'll study that. In fact I'll evaluate the efficacy of that list in place of the blanket block I currently test.


  • LAYER 8 Global Moderator

    I also have put in some host overrides to resolve most of these fqdn to local IP that I block, and log - so I can see what IP might be trying to hit it

    local-zone: "use-application-dns.net"  always_nxdomain
    local-zone: "local."  always_nxdomain
    local-data: "dns.adguard.com. 120 IN A 172.19.19.19"
    local-data: "dns-family.adguard.com. 120 IN A 172.19.19.19"
    local-data: "dns.google. 120 IN A 172.19.19.19"
    local-data: "cloudflare-dns.com. 120 IN A 172.19.19.19"
    local-data: "dns.quad9.net. 120 IN A 172.19.19.19"
    local-data: "dns9.quad9.net. 120 IN A 172.19.19.19"
    local-data: "dns10.quad9.net. 120 IN A 172.19.19.19"
    

    It is much longer than that - but really need to work out a more elegant way than just entries in unbound.. Just haven't gotten around to it yet.. And nothing has hit any of my rules.. I always make sure sure its turned off any browser I use..

    see my edit above for a github list that lists many of the fqdn used..



  • I tested the list at https://raw.githubusercontent.com/Sekhan/TheGreatWall/master/TheGreatWall_ipv4
    Unfortunately, Chrome immediately started sending queries to 162.158.161.161 in Singapore and bypassing my countermeasures.



  • I realise Cloudflare cannot be using their entire IP space to serve up DNS, but they're clearly using a lot of IPs embedded in many, many subnets, either as a part of their design, or deliberately to obfuscate the target server for network admins.



  • Hi-
    How do you feel about using this list in PF Blocker https://heuristicsecurity.com/dohservers.txt.

    I know not everyone uses PF blocker, but how does a list of the DNS ip work for blocking when the query is sent out FQDN?



  • I'm now playing with a host override in my DNS resolver, pointing cloudflare-dns.com at local IPs to monitor, as you suggest above. However, I am seeing completely different IPs being queried from Chrome, also with DNS leak test websites. If I do a DNS lookup from the gateway itself, on those Cloudflare FQDNs, the IPs returned are in the blocklist. IPs queried from Chrome are not in the blocklist. Chrome must be using a different, unknown FQDN


  • LAYER 8 Global Moderator

    What exactly are you settings in chrome? So you have it on purpose set to try and use doh, and your trying to block it?

    You have it set like this

    setlikethis.png

    If so, I can do that and look to see what its doing.. Logging all traffic coming from the machine.. with a sniff.

    And see if dns works.



  • @johnpoz Yes, Chrome DoH set to use system DNS, host machine set to use 1.0.0.1 and 1.1.1.1

    This particular machine in Bangkok keeps using IP 162.158.161.161 when using DNS leak test website


  • LAYER 8 Global Moderator

    Show me the setting you have set, like I have above - you have the other setting set..

    And how your seeing that IP is from a leaktest.. I think your not understanding how those tests work.. Then.. Just because you see an IP there doesn't mean your client talked to that IP..



  • @johnpoz I have experimented with all the setting variants i.e. like in your screenshot above, and the other "current provider" setting. It appears to have the same result. If I choose Google, or CleanBrowsing, my countermeasures work. However, with Cloudflare, it is extremely difficult to block as far as I can see, without blocking all of their IPs.



  • @johnpoz said in Blocking DNS over HTTPS. Seems the only way is to fire a shotgun at it:

    I think your not understanding how those tests work.. Then.. Just because you see an IP there doesn't mean your client talked to that IP..

    Perhaps so. I am merely using that leak test site as an easy reference to see if that endpoint is using the DNS provider I specify in the gateway, or Cloudflare. It's ALWAYS Cloudflare without the blanket ban on Cloudflare IPs in place. You are correct in that I'm not understanding why this is so. I am trying to understand so I can remedy it.


  • LAYER 8 Global Moderator

    chrome is using this

    chrome.cloudflare-dns.com

    With the setting I had above..

    added that to my block list, and no more chrome working for anything with that setting.

    dontwork.png

    If you want to know what its doing, and what IPs it talking to - vs those stupid leak tests.. Just sniff.. See right away where its going

    clienthello.png

    Those leak tests don't show you what IP the client talked to, they show you what IP ended up resolving the test fqdn they used... So it could be some IP upstream of where you asked that actually resolved it... Those tests are pointless scare tactics to get users to be scared -- OMG its "leak" without clue one to what actually is going on..

    It never shows you 1.1.1.1 or 9.9.9.9 in those stupid tests.. It might show you your lame ISP dns if your using that - which NS uses the same IP to resolve with as it listens for queries on.. Small setups not enterprise or CDN setups..

    The real problem here is users don't actually even understand what dns is or how it works - and if someone says hey your "leaking" they jump!!! OMG.... the man knows what I did a dns query for... The black helicopters are coming.. Without clue one to the basics of how any of it works in the first place... They can not tell you the difference between a forwarder or resolver, etc..

    Sorry for the rant... Those dns leak tests don't do anything other than scare users to be honest.



  • @johnpoz Kudos!!! So it was indeed a previously unknown FQDN. That's sure going to make things easier for me.


  • LAYER 8 Global Moderator

    Oh you got me started - sorry... The above example where I show how easy it is to see where you went in a simple sniff..

    Should show these users.. They are so worried omg my ISP knows what websites I am going to... Hiding your dns doesn't stop them from knowing that.. Even encrypting it and sending it all to whereever..

    They still see the IPs you go to, and right there in the freaking hello is what fqdn you were trying to hit.. Exact same info dns gives them..

    So what are you doing other than handing all your dns to someone else, along with your ISP still having the info, and making your dns slower to boot.. But OMG a freaking leak<rolleyes>



  • https://redmine.pfsense.org/issues/10969 - feature request for adding https://github.com/Sekhan/TheGreatWall feeds to pfBlockerNG



  • Just to update this topic, setting the following in my resolver's custom options. . .

    server:
    local-zone: "use-application-dns.net" always_nxdomain
    local-zone: "cloudflare-dns.com" static
    

    . . . and adding the following IP lists to the firewall as blocked aliases. . .

    https://public-dns.info/nameservers.txt
    https://raw.githubusercontent.com/Sekhan/TheGreatWall/master/TheGreatWall_ipv4

    . . . completely hamstrings Firefox and Chrome's attempts to use DoH. I'm sure they will find new ways to screw with network admins, but for the time being, this appears to be highly effective, while keeping things pretty neat and tidy. This is what I am deploying on my production network.

    NOTE: Anyone reading this, don't just throw this into your config and forget. You MUST also have the DNS redirects to your local resolver/forwarder in place first.


  • LAYER 8 Global Moderator

    @Lanna said in Blocking DNS over HTTPS. Seems the only way is to fire a shotgun at it:

    local-zone: "cloudflare-dns.com" static

    That is a great solution.. Since you set it static, unbound will not try to resolve any subdomains of that be it the Mozilla or the chrome one..



  • @Raffi_
    Yea, that "managed environment..." seems to work for domain aware devices. I imported their Active Directory settings into my home domain (Windows Server 2016), and it comes up as disabled by default on domain members. I did turn off the Chrome DNS function via their policy additions anyway (and disabled DOT in Firefox too using their extensions). I then turned my attention to the non-domain stuff so added a NAT redirect for 53 on my IOT VLAN to catch all the 53 to 8.8.8.8 and redirect to my DNS, and don't allow 853 to the internet. DOH from the non-domain-joined IOT was still an issue, so I just setup Lanna's suggestion of the two block lists and the local-zone setting.

    This seems like a lot of work to stop software from doing something against my wishes. I was using DOT for a bit but decided I was still handing over my my browsing history to some company so I am just letting the router do the resolving to root servers now.

    Feels like a cat and mouse game, or wack a mole...


  • LAYER 8 Global Moderator

    @Tzvia said in Blocking DNS over HTTPS. Seems the only way is to fire a shotgun at it:

    Feels like a cat and mouse game, or wack a mole...

    Concur - its really no better than the spammer changing their tactics to find a way to get their spam to users through corp filtering.. Now its the likes of google and cloudflare.. We will get your users data someway, no matter what you say corp IT..

    They really want to send us their data, honest they do because we told them you were spying on their dns.. You know on the network you own and run, and them using the device you gave them to work with.. They clearly need to be able to resolve shop.tld

    Oh you don't really want that to happen corp IT.. Here

    hoop.jpg

    JUMP!



  • Anyway, TheGreatWall feeds are added to the latest version of pfBlockerNG-devel:

    Screenshot from 2020-10-16 08-23-24.png
    Screenshot from 2020-10-16 08-25-13.png
    Screenshot from 2020-10-16 08-25-26.png



  • @Lanna said in Blocking DNS over HTTPS. Seems the only way is to fire a shotgun at it:

    . . . and adding the following IP lists to the firewall as blocked aliases. . .

    Trying to wrap my head around this one ...
    Are you blocking everything to these IP's , or just 443 ??

    Are you pointing the alias to the listfiles via this one ??

    c491e146-5dd8-4da6-b124-aa6e9b008030-image.png

    Thanx for doing this

    I have setup my pfSense (unbound) to use (forward) all queries to use two Linux Bind9 servers i have locally (vlan100) , doing all the resolving.

    They have to have "access to the root servers" UDP 53 , if i enable (dns) portforwarding on vlan 100 , can i make an exception for these two so they're not redirected ?

    I'm already handing out pfSense IF as DNS via dhcp to clients , and blocking
    53/853 to other(s). No UDP 53 portredirect yet.

    I'm not that intertested in pfblocker-ng , i use Pihole (also vlan 100) for "scrubbing" my mobile devices.

    So i suppose i have 4 local ip's i'd like to prevent from being redirected.

    local DNS1 - A root server access
    local DNS2 - A root server access

    pihole - Allow dns from Phone vlan + Mmedia Vlan

    Express-VPN ATV DNS - Allow dns to this one from my ATV's on Mmedia vlan

    /Bingo



  • @bingo600 said in Blocking DNS over HTTPS. Seems the only way is to fire a shotgun at it:

    @Lanna said in Blocking DNS over HTTPS. Seems the only way is to fire a shotgun at it:

    Are you blocking everything to these IP's , or just 443 ??

    I am blocking all ports to those IPs, but adjust to your liking

    Are you pointing the alias to the listfiles via this one ??

    That's right, I am using the URL Table option


Log in to reply