Different ways to setup DNS over TLS

[pfsvrb]

@operations
I see a few things to try. The goal here is to just get you to a baseline config to see if DNSSEC starts working.

First off, disable Harden DNSSEC Data in advanced settings by unchecking this box:

Next, get rid of the custom options just to rule out any config issues there. Other than the included .conf, all of those other settings should already be configured for you by pfSense when you enable outbound DoT and have an IP address and DNS name specified in System/General (which you do already have setup).

[Operations]

@pfsvrb

I tried both things you said but no joy... :(

I also switched to Quad9 (9.9.9.9 - dns.quad9.net) to make our situations a bit more the same / make it maybe a bit easier to troubleshoot.

but now the first test "secure dns" also fails:

You may not be using secure DNS.

We weren’t able to detect whether you were using a DNS resolver over secure transport. Contact your DNS provider or try using 1.1.1.1 for fast & secure DNS.

after switching to Quad9.

[pfsvrb]

@operations
Based on the screenshots it looks like you're using a VPN? Is it possible to select your WAN gateway instead of the EXTRAIP tunnel, just for testing purposes?

Also, it appears that cloudflare-dns.com doesn't nslookup to 1.1.1.1 anymore. Or at least, when I check it, it doesn't show that address. Can you also try "1dot1dot1dot1.cloudflare-dns.com" for the hostname?

Rerun your DNSSEC tests after that and see?

If this continues to not work, I would try Quad9 DNS or Google DNS just to verify if it's a CloudFlare issue, or a localized issue.

Just to confirm, you've disabled the DNS services within your browser right? Otherwise the browser you test with may not use the local router for lookups.

[Operations]

@pfsvrb i have edited my previous comment :)

Plus it is not an VPN but a GRE tunnel.
Do you mean DNS over HTTPS? That is not enabled. The other things about FF using own DNS, it is coming back with WoodyNet now when i check. So that should be okee right? How do i disable FF using own DNS (network.ttr.mode = 5?)

When i do this:
Also, it appears that cloudflare-dns.com doesn't nslookup to 1.1.1.1 anymore. Or at least, when I check it, it doesn't show that address. Can you also try "1dot1dot1dot1.cloudflare-dns.com" for the hostname?

The Cloudflare test also tells me Secure DNS = not working. When i use Cloudflare-dns.com like before Secure DNS is marked green.
Just to be sure, you meant i use 1.1.1.1.cloudflare-dns.com or did you really wanted me to use "dot" ?

[pfsvrb]

@operations
Exactly, in the case of FF disabling TRR mode should do it. I almost hate to ask but, on the PC that you are doing the testing from, it is configured to use the pfSense router for DNS correct? If it's configured for an external source, obviously that will also impact the results. You'll have the same skewed results if the DHCP server is configured to hand out a DNS address other than that of the router. So if the DHCP server is handing out leases and there's a setting in that lease to use 1.1.1.1, clients will bypass using the local pfSense resolver and send their lookups directly to the DNS provider.

If the above doesn't work out then I'm really not sure what else to have you try, maybe just try a different DNS provider as a final test?

I'll past screenshots of my setup below just in case it helps. But I think we're using a relatively identical config with the exception of the different providers.

My General/Setup:

My DNS resolver/General:

My DNS resolver/advanced:

[Operations]

@pfsvrb i am using DHCP. And the computer gets the IP of the Domain Controller (+DNS) 1 en 2.

But the DNS is configured to use PFSense as a DNS forward. When i the PFSense IP directly as DNS it does not make a difference.

We have a couple of settings differently (advanced tab), why did you change them? I also did that not that it made a difference :)

By the way, "Encrypted SNI" is also not marked green. Shouldn't this option also work when using Cloudflare (or Quad9)?

[pfsvrb]

@operations
I'm not sure on SNI but my understanding is that it's only implemented at the browser level when using the browser configured DNS service. In our case, we're offloading DNS to a local resolver on our network, so I don't think we will see SNI. I could be wrong about this but, that's the basic understand that I have on the SNI part.

I only posted my advanced settings to show my full config. I've noticed a small performance bump in some DNS test scores by messing around with them but it is very slight. I did notice a nice increase by using the "serve expired" option as this serves the last known good IP while also updating the DNS cache in the background. The risk is that it could also give you stale data. Use at your own risk but, it can be helpful to speed up DNS, especially with using DoT that has more overhead.

I'm sorry I don't have any other tips for you regarding the lack of DNSSEC. I'm sort of wondering if somehow the GRE tunnel is breaking it? But I have no proof of that and I personally am not familiar with them so I don't have a way to troubleshoot it.

As a last try, maybe post the output of this command and see if we notice anything out of the ordinary: cat /var/unbound/unbound.conf

[Operations]

@pfsvrb

Okee i get what you are saying. Here is the output:

##########################

Unbound Configuration

##########################

Server configuration

server:
local-zone: ".168.192.in-addr.arpa" typetransparent
local-zone: ".168.192.in-addr.arpa" typetransparent
local-zone: "*.*172.in-addr.arpa" typetransparent

chroot: /var/unbound
username: "unbound"
directory: "/var/unbound"
pidfile: "/var/run/unbound.pid"
use-syslog: yes
port: 53
verbosity: 1
hide-identity: yes
hide-version: yes
harden-glue: yes
do-ip4: yes
do-ip6: yes
do-udp: yes
do-tcp: yes
do-daemonize: yes
module-config: "validator iterator"
unwanted-reply-threshold: 0
num-queries-per-thread: 512
jostle-timeout: 200
infra-host-ttl: 900
infra-cache-numhosts: 10000
outgoing-num-tcp: 20
incoming-num-tcp: 20
edns-buffer-size: 1432
cache-max-ttl: 86400
cache-min-ttl: 0
harden-dnssec-stripped: no
msg-cache-size: 250m
rrset-cache-size: 500m
qname-minimisation: yes

num-threads: 4
msg-cache-slabs: 4
rrset-cache-slabs: 4
infra-cache-slabs: 4
key-cache-slabs: 4
outgoing-range: 4096
#so-rcvbuf: 4m
auto-trust-anchor-file: /var/unbound/root.key
prefetch: yes
prefetch-key: yes
use-caps-for-id: no
serve-expired: yes

Statistics

Unbound Statistics

statistics-interval: 0
extended-statistics: yes
statistics-cumulative: yes

TLS Configuration

tls-cert-bundle: "/etc/ssl/cert.pem"
tls-port: 853
tls-service-pem: "/var/unbound/sslcert.crt"
tls-service-key: "/var/unbound/sslcert.key"

Interface IP(s) to bind to

interface: 192.168..1
interface: 192.168..1@853
interface: 172...1
interface: 172...1@853
interface: 172...1
interface: 172...1@853
interface: 10...1
interface: 10...1@853
interface: 192.168..1
interface: 192.168..1@853
interface: fe80::feb6:126e%vmx1
interface: fe80::f:feb6:126e%vmx1@853
interface: fe80::feb6:126e%vmx1.20
interface: fe80::26e%vmx1.20@853
interface: fe8feb6:126e%vmx1.145
interface: fe80::250:126e%vmx1.145@853
interface: fe80::feb6:126e%vmx1.200
interface: fe80::feb6:126e%vmx1.200@853
interface: fe80::250::126e%vmx1.101
interface: fe80::250::126e%vmx1.101@853
interface: 127.0.0.1
interface: 127.0.0.1@853
interface: ::1
interface: ::1@853

Outgoing interfaces to be used

DNS Rebinding

For DNS Rebinding prevention

private-address: 127.0.0.0/8
private-address: 10.0.0.0/8
private-address: ::ffff:a00:0/104
private-address: 172.16.0.0/12
private-address: ::ffff:ac10:0/108
private-address: 169.254.0.0/16
private-address: ::ffff:a9fe:0/112
private-address: 192.168.0.0/16
private-address: ::ffff:c0a8:0/112
private-address: fd00::/8
private-address: fe80::/10

Set private domains in case authoritative name server returns a Private IP address

private-domain: ".168.192.in-addr.arpa"
domain-insecure: ".168.192.in-addr.arpa"
private-domain: ".168.192.in-addr.arpa"
domain-insecure: ".168.192.in-addr.arpa"
private-domain: "..172.in-addr.arpa"
domain-insecure: "..172.in-addr.arpa"
private-domain: "ad.domain.nl"
domain-insecure: "ad.domain.nl"

Access lists

include: /var/unbound/access_lists.conf

Static host entries

include: /var/unbound/host_entries.conf

dhcp lease entries

include: /var/unbound/dhcpleases_entries.conf

Domain overrides

include: /var/unbound/domainoverrides.conf

Forwarding

forward-zone:
name: "."
forward-tls-upstream: yes
forward-addr: 1.1.1.1@853#Cloudflare-dns.com

Unbound custom options

server:
include: /var/unbound/pfb_dnsbl.*conf

Remote Control Config

include: /var/unbound/remotecontrol.conf

[pfsvrb]

@operations
This config file still looks weird. It has the module config as "validator iterator". Do you have any other custom options defined? It seems like something is still over riding the pfSense Web GUI selections that were made for the DNS Resolver.

Have you edited /var/unbound/unbound.conf?

[Operations]

@pfsvrb

To be completely honest, i don't even know how to do that. I have installed PFBlocker, is that maybe the source of the problem?

By the way how do i edit that or other conf files? By using SSH?

Edit; also no other custom options. Where would i do that?

[pfsvrb]

@operations
I was asking about directly editing it, such as with vi.

I don't use PFBlocker, so I'm not sure what impact it may be having here.

The "validator iterator" in the module config basically means that DNSSEC is still enabled on the pfSense resolver. This doesn't make sense because in the screenshots you show this is unchecked. But I suspect that is what is interfering with your DNSSEC test results. However I'm not sure what else might be forcing that feature to stay on.

[Operations]

@pfsvrb

I have turn it on/off plus reboot now it says:

do-daemonize: yes
module-config: "iterator"
unwanted-reply-threshold: 0
num-queries-per-thread: 512

Is this correct? Cloudflare website + DNSSEC check still fails.

[johnpoz]

How are you going to do any queries when you don't have any outbound interfaces listed? Be it you forward or resolve?

[Operations]

@johnpoz

I am sorry but i am not sure how to answer / what you are asking. Could you "dumb it down" a bit? / or tell me if i made a mistake what to change?

[johnpoz]

The config you posted list no outbound interfaces - so how and the hell would unbound query anything?

Oh you prob have ALL selected..

[Operations]

@johnpoz

Yes i have selected ALL. Should i have done that differently? (Apart from my DNSSEC problem)

[johnpoz]

I personally like to just use localhost as outbound.. This will nat to whatever interface your using outbound, be it you have multiple or vpn, etc. Without having to bind to any specific interface - incase the interface is not up when unbound starts..

[Operations]

@johnpoz

I am not sure what you mean, not binding to a specific interface (by using ALL) will that not have the same result as "use localhost as outbound.. This will nat to whatever interface your using outbound" ?

ALL = whatever interface? Or am i missing something / not getting it.

[Operations]

@pfsvrb

Any other ideas maybe?

[pfsvrb]

@operations
Sorry I'm not sure what else to try. I can't replicate the results you're seeing. I get DNSSEC validated results whether I used Quad9 or CloudFlare with the same configs that I've posted in my screenshots.

[Operations]

@pfsvrb said in Different ways to setup DNS over TLS:

@operations
Sorry I'm not sure what else to try. I can't replicate the results you're seeing. I get DNSSEC validated results whether I used Quad9 or CloudFlare with the same configs that I've posted in my screenshots.

@pfsvrb
I use a Windows domain so my client looks at DC for DNS and the DC (with DNS role) looks at PFSense.

When i use PFSense directly it does work. So the problem is my DC. Sorry i didnt mention this before.

[johnpoz]

What doesn't work? Exactly? You have your AD DNS forwarding to unbound on pfsense. You can not look up anything, only dnssec stuff fails? What?

Pfsense doesn't care if its a client asking or another NS.. Unless you AD dns is downstream of pfsense networks, and the acl is blocking.. unbound can not tell the difference between just some client asking for www.google.com or your DNS on your AD asking for - to unbound its just another client.

[bmeeks]

@operations said in Different ways to setup DNS over TLS:

@pfsvrb said in Different ways to setup DNS over TLS:

@operations
Sorry I'm not sure what else to try. I can't replicate the results you're seeing. I get DNSSEC validated results whether I used Quad9 or CloudFlare with the same configs that I've posted in my screenshots.

@pfsvrb
I use a Windows domain so my client looks at DC for DNS and the DC (with DNS role) looks at PFSense.

When i use PFSense directly it does work. So the problem is my DC. Sorry i didnt mention this before.

I had some random issues with DNS lookups on my 2012 R2 AD DNS server when I enabled DNSSEC. Most things worked fine, but every now and then I would get seemingly random DNS errors in the browser (mostly). The one that broke the camel's back, as they say, was a persistent issue with www.cdc.gov. @johnpoz determined that the CDC DNS entries are borked and that was causing the problem. But unbound would resolve it just fine. Only the Windows AD DNS got tripped up.

So I wound up turning off DNSSEC on the Windows AD DNS and changing it to forwarding mode with my pfSense box as the forwarder. So in unbound on pfSense I put in a domain override for my local AD domain that sends unbound to my AD server for those lookups (like pfSense log and ARP entries, for example). Local clients still point to the AD DNS server for DNS. Now everything just seems to work reliably and I have not had a single "random" DNS issue since.

So my takeaway is Windows AD DNS (at least in 2012 R2, which is old I admit) can get tripped up with DNSSEC now and then. That may be your problem, too. Since you state you are forwarding to unbound, just make sure you have turned off DNSSEC on the Windows DNS and let unbound on pfSense take care of the DNSSEC part when resolving external hosts or domains.

[johnpoz]

He already is I thought?

I use a Windows domain so my client looks at DC for DNS and the DC (with DNS role) looks at PFSense.

If he is forwarding - then yeah enabling dnssec is going to cause issues..

I have been over this atleast 100 times it seems... If you forward.. You do not set dnssec, you forward to a resolver that does dnssec. If you want dnssec, if you don't then forward to a resolver that doesn't do it.. There is no forwarding and asking for dnssec that ever makes sense.. Ever.. Only a resolver does dnssec.

No wonder he could having issues if that is what he is doing.

[bmeeks]

@johnpoz said in Different ways to setup DNS over TLS:

He already is I thought?

I use a Windows domain so my client looks at DC for DNS and the DC (with DNS role) looks at PFSense.

If he is forwarding - then yeah enabling dnssec is going to cause issues..

I have been over this atleast 100 times it seems... If you forward.. You do not set dnssec, you forward to a resolver that does dnssec. If you want dnssec, if you don't then forward to a resolver that doesn't do it.. There is no forwarding and asking for dnssec that ever makes sense.. Ever.. Only a resolver does dnssec.

No wonder he could having issues if that is what he is doing.

He did say forwarding, but did not say whether he disabled DNSSEC on the Windows side (if you had it on).

Yeah, I originally had Windows AD DNS using the root servers. Later I enabled DNSSEC and that's when the random failures started. They were very sporadic. Most of the time I just attributed them to maybe actual issues with the host domains or something. But the CDC site thing was persistent, and after the info you posted I decided to just stop DNSSEC with Windows and let unbound do it. So I switched the Windows DNS to forwarding and sent it to pfSense for external lookups. I have unbound doing the DNSSEC when resolving. And since then, not a single "random" hiccup from DNS.

[johnpoz]

If ad dns was resolving - then yeah you can set dnssec. But sure with something like what is going on with cdc.gov.. Its going to be luck of the draw.. I haven't seen a domain so F'd up from a major player for this long in a long time..

Its really sad.. As I mentioned earlier whoever is in charge of their dns shouldn't be doing dns..

There was a link to some cloudlfare thread where cloudlfare was putting in work arounds to try and make sure issue doesn't keep stuff from resolving.. That is the WRONG thing to do.. If their dnssec is borked, then it shouldn't resolve.. And companies like cloudflare and googledns, quad9 should call them out on it vs trying to work around their mess.

[bmeeks]

@johnpoz said in Different ways to setup DNS over TLS:

If ad dns was resolving - then yeah you can set dnssec. But sure with something like what is going on with cdc.gov.. Its going to be luck of the draw.. I haven't seen a domain so F'd up from a major player for this long in a long time..

Its really sad.. As I mentioned earlier whoever is in charge of their dns shouldn't be doing dns..

Similar to this, the Social Security web site would not work with IPv6 for weeks and weeks like a couple of years ago. They returned IPv6 addresses, but you couldn't connect. I would have to disable my HE IPv6 tunnel in order to access the site (or go through all the steps to get Windows and Chrome to prefer IPv4 first).

Maybe it's the old incompetent government employee thing ... .

[johnpoz]

I personally don't think ipv6 is really ready for prime time ;) While I have it on the network, and do even provide services to ntp pool on it. I set "my" machines to use IPv4... Until such time there is a resource that only can be gotten to via IPv6 that I actually want/need to get to - it not prime time.

That is many many years off it seems ;) I will be retired from the biz pretty sure.. Even though I would love nothing more for it not to be that.. But for now mobile devices are really the only prime time users of ipv6, and they go through ipv6 to ipv4 gateways to get to most of the net ;)

[bmeeks]

I'm with you on IPv6. I was trying it out, though, with the HE tunnel. Finally abandoned it for now. Kept getting in the way of the grandkids trying to watch cartoons on their iPads via Netflix. And I didn't want to keep jumping through all the policy routing, DNS mods and ASN aliases to keep their iPads going out via IPv4.

Like you, I have no real need for IPv6, so I just turned off my tunnel for now. If my cable provider ever offers the service, I will enable it on my network. That way I won't have to fight the streaming services blocking Hurricane Electric space. Really that's the biggest reason I disabled the HE tunnel.

[johnpoz]

Just leave it on 1 or 2 vlans for you to play with.. That is what I do ;)

My wifi and roku(netflix) vlans don't have ipv6 enabled. Its on my dmz so I can serve up ntp via it, and its on my main lan.. So I can just toggle it on my main PC if I want to play with something via IPv6.

And pfsense (unbound) can use it for resolving stuff, that sort of thing.

I can understand netflix problem with HE though.. There is nothing stopping someone from just using one of the many global tunnel endpoints to look like your in whatever country you want.. Think of it a free vpn for geoip restriction circumvention ;)

[Gertjan]

@bmeeks said in Different ways to setup DNS over TLS:

That way I won't have to fight the streaming services blocking Hurricane Electric space. Really that's the biggest reason I disabled the HE tunnel.

I'm using he.net for years now, it works .... well.
Two major downsides, as you stated : Netflix saw my IPv6 (geo located in Paris) as some kind of VPN type of access. So I could access Netflix, but as soon as I pressed Play, an obscure error message showed : "Do not use a VPN".
This changed a couple of weeks ago : no more issues.
The other one, for me, was Apple's icloud : the access is ok, but impossible to see uploaded photos. they refused to show up in the browser. I presume that it was some silly 'javascript' issue that went ko on IPv6 addresses as Apple should be IPv6 for years now. I don't think Apple has peering issues with Huricane neither.
But icloud works fine now , since ... a couple of weeks.

Anyway, 'NoAAAA' exists as a Python extension for unbound to block listed AAAA domains, which helped. The same NoAAAA - as it is special kind of DNSBL - is now integrated in pfBlockerNG now. So if some site has IPv6 difficulties, it can be excluded from DNS.

Btw : I love this cdc.org DNNSEC graph ....how on earth admin people can actually let such a situation sustain ? Resolvers that do DNSSEC checking will -as they should - fail on DNSSEC enabled sites with broken DNSSEC. I presume a site as "cdc" is rather important these days.

Using he.net is actually slowing down my overall network performances, as close to 3k accounts are using the he.net POP in Paris. This can't be good for performance, as IPv6 traffic is preferred above IPv4.

@Operations : sorry for going way out of subject. If you have questions : ask ;)