DNSSEC and DNS over TLS Problems with Resolver [RESOLVED]

emce · Dec 30, 2018, 2:09 PM

Update: Saving below for posterity but narrowing down in this post: https://forum.netgate.com/topic/139153/dnssec-and-dns-over-tls-problems-with-resolver/8

Hello,

I'm using the DNS Resovler with forwarding enabled and the Quad9 servers. By default (without either DNSSEC or DNS over TLS) this works correctly. My understanding is that Quad9 supports both of these options.

I can enable DNSSEC in the resolver, however, this has mixed results. My clients can seemingly randomly resolve hosts. Most will work, but often I'll get an unresolved name error. A refresh (in browser) or requery (using nslookup) resolves this. It seems to be totally random.

On the other hand, I cannot enable DNS over TLS. When I do, PFSense can still resolve hosts (via the DNS Lookup tool) but clients cannot. This implies to me there's something going on within the resolver.

No errors are showing up in the Resolver logs so it's as if PFSense is just dropping the client requests.

I've attached a few screenshots of relevant settings and would love any thoughts/advice. Thanks!

DNS Servers:

DNS Firewall Rules:

DNS Ports:

DNS Resolver Settings:

bcruze · Dec 30, 2018, 2:35 PM

I can’t get dns to travel over 853. Only 53.

I bet if you remove 853 from the ports and just use 53 it will work better

Either way I am following this thread to hopefully learn from it

emce · Dec 30, 2018, 2:50 PM

@bcruze said in DNSSEC and DNS over TLS Problems with Resolver:

I can’t get dns to travel over 853. Only 53.

I bet if you remove 853 from the ports and just use 53 it will work better

I've actually tried disabling those firewall rules all together but with the same results. PFSense definitely resolves upstream over 853, it just stops handling client requests correctly when I enable DNS over TLS.

emce · Dec 30, 2018, 4:26 PM

More info... I dialed up logging for the resolver and I'm seeing the error:

debug: tcp error for address 9.9.9.9 port 53

without DNS over TLS. This seems to coincide with failures when DNSSEC is enabled. With DNS over TLS, I'm seeing something similar (obviously with the TLS port):

debug: tcp error for address 9.9.9.9 port 853

I've also tried the cloudflare servers and experienced the same results.

johnpoz · Dec 30, 2018, 5:27 PM

enanbling dnssec when you forward is pretty pointless.. When you forward you want the end resolver to be doing dnssec, forwarding and asking for dnssec doesn't do anything worth anything.. Since your trusting the forwarder anyway.. They could in theory send you anything they wanted to send you - even the dnssec info, since they control everything you get..

Set quad 9 in your dns, click the use forwarder and tls.. unbound should still be listening on 53!!! So your clients can do normal queries too it.. there is ZERO reason to use tls over your local lan to your NS on pfsense.. Is your LAN hostile?

Here is sniff while doing queries

11:22:04.462690 IP 149.112.112.112.853 > 64.53.xxx.xxx.40881: tcp 31
11:22:04.601517 IP 9.9.9.9.853 > 64.53.xxx.xxx.6760: tcp 31
11:22:04.670454 IP 149.112.112.112.853 > 64.53.xxx.xxx.10217: tcp 31
11:22:04.794574 IP 149.112.112.112.853 > 64.53.xxx.xxx.20562: tcp 31
11:22:04.942879 IP 9.9.9.9.853 > 64.53.xxx.xxx.31852: tcp 31
11:22:09.242253 IP 149.112.112.112.853 > 64.53.xxx.xxx.58126: tcp 31
11:22:10.587677 IP 9.9.9.9.853 > 64.53.xxx.xxx.18540: tcp 31
11:22:10.805740 IP 9.9.9.9.853 > 64.53.xxx.xxx.48851: tcp 31
11:22:13.843248 IP 64.53.xxx.xxx.3675 > 149.112.112.112.853: tcp 0
11:22:13.854159 IP 149.112.112.112.853 > 64.53.xxx.xxx.3675: tcp 0
11:22:13.854207 IP 64.53.xxx.xxx.3675 > 149.112.112.112.853: tcp 0
11:22:13.854355 IP 64.53.xxx.xxx.3675 > 149.112.112.112.853: tcp 311
11:22:13.878315 IP 149.112.112.112.853 > 64.53.xxx.xxx.3675: tcp 1448
11:22:13.878357 IP 64.53.xxx.xxx.3675 > 149.112.112.112.853: tcp 0
11:22:13.879187 IP 149.112.112.112.853 > 64.53.xxx.xxx.3675: tcp 1448
11:22:13.879219 IP 64.53.xxx.xxx.3675 > 149.112.112.112.853: tcp 0
11:22:13.879223 IP 149.112.112.112.853 > 64.53.xxx.xxx.3675: tcp 11
11:22:13.879249 IP 64.53.xxx.xxx.3675 > 149.112.112.112.853: tcp 0
11:22:13.882067 IP 64.53.xxx.xxx.3675 > 149.112.112.112.853: tcp 126
11:22:13.894695 IP 149.112.112.112.853 > 64.53.xxx.xxx.3675: tcp 51
11:22:13.894744 IP 64.53.xxx.xxx.3675 > 149.112.112.112.853: tcp 0
11:22:13.894982 IP 64.53.xxx.xxx.3675 > 149.112.112.112.853: tcp 80
11:22:13.914490 IP 149.112.112.112.853 > 64.53.xxx.xxx.3675: tcp 1043
11:22:13.914526 IP 64.53.xxx.xxx.3675 > 149.112.112.112.853: tcp 0
11:22:13.914738 IP 64.53.xxx.xxx.3675 > 149.112.112.112.853: tcp 31
11:22:13.914912 IP 64.53.xxx.xxx.3675 > 149.112.112.112.853: tcp 0
11:22:13.925887 IP 149.112.112.112.853 > 64.53.xxx.xxx.3675: tcp 31
11:22:13.925951 IP 64.53.xxx.xxx.3675 > 149.112.112.112.853: tcp 0
11:22:13.926122 IP 149.112.112.112.853 > 64.53.xxx.xxx.3675: tcp 0
11:22:18.318623 IP 149.112.112.112.853 > 64.53.xxx.xxx.40881: tcp 31
11:22:18.331472 IP 9.9.9.9.853 > 64.53.xxx.xxx.6760: tcp 31
11:22:18.638330 IP 149.112.112.112.853 > 64.53.xxx.xxx.10217: tcp 31
11:22:18.862021 IP 9.9.9.9.853 > 64.53.xxx.xxx.31852: tcp 31
11:22:20.806554 IP 64.53.xxx.xxx.35160 > 149.112.112.112.853: tcp 0
11:22:20.806941 IP 64.53.xxx.xxx.53607 > 9.9.9.9.853: tcp 0
11:22:20.817595 IP 149.112.112.112.853 > 64.53.xxx.xxx.35160: tcp 0
11:22:20.817661 IP 64.53.xxx.xxx.35160 > 149.112.112.112.853: tcp 0
11:22:20.817806 IP 64.53.xxx.xxx.35160 > 149.112.112.112.853: tcp 311
11:22:20.823095 IP 9.9.9.9.853 > 64.53.xxx.xxx.53607: tcp 0
11:22:20.823146 IP 64.53.xxx.xxx.53607 > 9.9.9.9.853: tcp 0
11:22:20.823248 IP 64.53.xxx.xxx.53607 > 9.9.9.9.853: tcp 311
11:22:20.838495 IP 9.9.9.9.853 > 64.53.xxx.xxx.53607: tcp 1448
11:22:20.838542 IP 64.53.xxx.xxx.53607 > 9.9.9.9.853: tcp 0
11:22:20.839372 IP 9.9.9.9.853 > 64.53.xxx.xxx.53607: tcp 1448
11:22:20.839411 IP 64.53.xxx.xxx.53607 > 9.9.9.9.853: tcp 0
11:22:20.839416 IP 9.9.9.9.853 > 64.53.xxx.xxx.53607: tcp 13
11:22:20.839447 IP 64.53.xxx.xxx.53607 > 9.9.9.9.853: tcp 0
11:22:20.839564 IP 149.112.112.112.853 > 64.53.xxx.xxx.35160: tcp 1448
11:22:20.839608 IP 64.53.xxx.xxx.35160 > 149.112.112.112.853: tcp 0
11:22:20.840368 IP 149.112.112.112.853 > 64.53.xxx.xxx.35160: tcp 1448
11:22:20.840406 IP 64.53.xxx.xxx.35160 > 149.112.112.112.853: tcp 0
11:22:20.840561 IP 149.112.112.112.853 > 64.53.xxx.xxx.35160: tcp 11
11:22:20.840595 IP 64.53.xxx.xxx.35160 > 149.112.112.112.853: tcp 0
11:22:20.842724 IP 64.53.xxx.xxx.53607 > 9.9.9.9.853: tcp 126
11:22:20.843621 IP 64.53.xxx.xxx.35160 > 149.112.112.112.853: tcp 126
11:22:20.856426 IP 9.9.9.9.853 > 64.53.xxx.xxx.53607: tcp 51
11:22:20.856470 IP 64.53.xxx.xxx.53607 > 9.9.9.9.853: tcp 0
11:22:20.856755 IP 64.53.xxx.xxx.53607 > 9.9.9.9.853: tcp 91
11:22:20.856782 IP 149.112.112.112.853 > 64.53.xxx.xxx.35160: tcp 51
11:22:20.856825 IP 64.53.xxx.xxx.35160 > 149.112.112.112.853: tcp 0
11:22:20.857098 IP 64.53.xxx.xxx.35160 > 149.112.112.112.853: tcp 91
11:22:20.892438 IP 9.9.9.9.853 > 64.53.xxx.xxx.53607: tcp 243
11:22:20.892482 IP 64.53.xxx.xxx.53607 > 9.9.9.9.853: tcp 0
11:22:20.892749 IP 64.53.xxx.xxx.31106 > 9.9.9.9.853: tcp 0
11:22:20.892822 IP 64.53.xxx.xxx.53607 > 9.9.9.9.853: tcp 31
11:22:20.893021 IP 64.53.xxx.xxx.53607 > 9.9.9.9.853: tcp 0
11:22:20.907268 IP 149.112.112.112.853 > 64.53.xxx.xxx.35160: tcp 0
11:22:20.909487 IP 9.9.9.9.853 > 64.53.xxx.xxx.31106: tcp 0
11:22:20.909536 IP 64.53.xxx.xxx.31106 > 9.9.9.9.853: tcp 0
11:22:20.909659 IP 64.53.xxx.xxx.31106 > 9.9.9.9.853: tcp 311
11:22:20.909760 IP 9.9.9.9.853 > 64.53.xxx.xxx.53607: tcp 31
11:22:20.909807 IP 64.53.xxx.xxx.53607 > 9.9.9.9.853: tcp 0
11:22:20.910508 IP 9.9.9.9.853 > 64.53.xxx.xxx.53607: tcp 0
11:22:20.915565 IP 149.112.112.112.853 > 64.53.xxx.xxx.35160: tcp 193
11:22:20.915603 IP 64.53.xxx.xxx.35160 > 149.112.112.112.853: tcp 0
11:22:20.915828 IP 64.53.xxx.xxx.20167 > 149.112.112.112.853: tcp 0
11:22:20.915892 IP 64.53.xxx.xxx.35160 > 149.112.112.112.853: tcp 31
11:22:20.916075 IP 64.53.xxx.xxx.35160 > 149.112.112.112.853: tcp 0
11:22:20.929712 IP 9.9.9.9.853 > 64.53.xxx.xxx.31106: tcp 1448
11:22:20.929758 IP 64.53.xxx.xxx.31106 > 9.9.9.9.853: tcp 0
11:22:20.930560 IP 9.9.9.9.853 > 64.53.xxx.xxx.31106: tcp 1448
11:22:20.930597 IP 64.53.xxx.xxx.31106 > 9.9.9.9.853: tcp 0
11:22:20.930754 IP 9.9.9.9.853 > 64.53.xxx.xxx.31106: tcp 13
11:22:20.930788 IP 64.53.xxx.xxx.31106 > 9.9.9.9.853: tcp 0
11:22:20.933847 IP 64.53.xxx.xxx.31106 > 9.9.9.9.853: tcp 126
11:22:20.934386 IP 149.112.112.112.853 > 64.53.xxx.xxx.35160: tcp 0
11:22:20.934916 IP 149.112.112.112.853 > 64.53.xxx.xxx.35160: tcp 31
11:22:20.934956 IP 64.53.xxx.xxx.35160 > 149.112.112.112.853: tcp 0
11:22:20.935163 IP 149.112.112.112.853 > 64.53.xxx.xxx.35160: tcp 0
11:22:20.936141 IP 149.112.112.112.853 > 64.53.xxx.xxx.20167: tcp 0
11:22:20.936187 IP 64.53.xxx.xxx.20167 > 149.112.112.112.853: tcp 0
11:22:20.936310 IP 64.53.xxx.xxx.20167 > 149.112.112.112.853: tcp 311
11:22:20.947270 IP 9.9.9.9.853 > 64.53.xxx.xxx.31106: tcp 51
11:22:20.947313 IP 64.53.xxx.xxx.31106 > 9.9.9.9.853: tcp 0
11:22:20.947601 IP 64.53.xxx.xxx.31106 > 9.9.9.9.853: tcp 119
11:22:20.954975 IP 149.112.112.112.853 > 64.53.xxx.xxx.20167: tcp 1448
11:22:20.955018 IP 64.53.xxx.xxx.20167 > 149.112.112.112.853: tcp 0
11:22:20.955816 IP 149.112.112.112.853 > 64.53.xxx.xxx.20167: tcp 1448
11:22:20.955852 IP 64.53.xxx.xxx.20167 > 149.112.112.112.853: tcp 0
11:22:20.956007 IP 149.112.112.112.853 > 64.53.xxx.xxx.20167: tcp 12

What version of pfsense are you using.. What does the status show? This is really clickity clickity on the latest version of pfsense

emce · Dec 30, 2018, 5:44 PM

Hey there, thank you very much for the insights! Here are responses to some of your questions:

Version of PFSense: 2.4.4-RELEASE-p1

I just tried disabling DNSSEC Support while enabling DNS over TLS... rebooted PFSense just for good measure, but still seeing the same result (ie, no clients can resolve DNS through PFSense). I've never enabled TLS for local resolutions but I confirmed that was still off as well.

When attempting to resolve via clients, through nslookup, I'm always receiving either a timeout while connecting to the DNS sever (my PFSense box) or the error:

** server can't find google.com: SERVFAIL

Of interest is that the DNS Resolver status is showing these timeouts as well, even though doing lookups directly via the PFSense DNS Lookup tool do work:

bcruze · Dec 30, 2018, 6:43 PM

here is my resolver settings:

server:
forward-zone:
name: "."
forward-ssl-upstream: yes
forward-addr: 9.9.9.9@853

i turned off DNSSEC and restarted the resolver and instantly noticed a change in websites loading FASTER. my ping to 9.9.9.9 is roughly 100Ms. i am surprised its loading faster...

also from JohnPoz screen shot i unchecked Disable DNS Forwarder . so it lists 127.0.0.1 as a DNS name server. but the ping fluctuates tremendously.

i'll have to test this a little longer to see if everything works as it should before i only had 9.9.9.9 listed as a DNS server under DNS servers on the main menu

emce · Jan 6, 2019, 3:24 PM

Ok, update with a bit more info. Simplifying this such that the end goal is just DNS over TLS for now (I'll get to DNSSEC later).

So current status: the DNS Resolver doesn't resolve any client queries when DNS over TLS is enabled. PFSense itself can resolve these queries just fine (via the DNS Lookup tool.) No problems at all when DNS over TLS is disabled. Client queries do resolve just find when DNS over TLS is disabled in basic forwarding mode.

I've cleaned up firewall rules to defaults (ie, removed any trying to capture and redirect external DNS requests from clients).

Current config:

PFSense 2.4.4-p1 running on Netgate SG-4860
Upstream DNS: Quad9
DNS Resolver:
- Forwarding mode enabled
- DNSSEC disabled
- Outbound requests on WAN
- Internal requests on LAN+Localhost

I've dialed up log verbosity on the DNS Resolver service, but I'm not seeing anything. I can see from the Resolver status that PFSense is successfully connected to the Quad9 servers on port 853. But every client query, from any client, simply fails.

dig netgate.com

; <<>> DiG 9.11.3-1ubuntu1.3-Ubuntu <<>> netgate.com
;; global options: +cmd
;; connection timed out; no servers could be reached

More details on this below from tcpdump:

tcpdump -i eno1 -n "host 10.0.0.1 and port 53" -v

07:19:23.626582 IP (tos 0x0, ttl 64, id 19275, offset 0, flags [DF], proto UDP (17), length 68)
    10.0.0.10.51305 > 10.0.0.1.53: 59820+ [1au] A? netgate.com. (40)

...

07:19:44.632385 IP (tos 0x0, ttl 64, id 4805, offset 0, flags [none], proto UDP (17), length 66)
    10.0.0.1.53 > 10.0.0.10.55540: 8284 ServFail 0/0/0 (38)

Also, nmap from client showing accessibility on port 53:

nmap -sTU -p 53 10.0.0.1

Starting Nmap 7.60 ( https://nmap.org ) at 2019-01-06 08:01 MST
Nmap scan report for ... (10.0.0.1)
Host is up (0.00011s latency).

PORT   STATE    SERVICE
53/tcp filtered domain
53/udp open     domain
MAC Address: ... (ADI Engineering)

Nmap done: 1 IP address (1 host up) scanned in 0.79 seconds

Any thoughts? Thanks!

emce · Jan 14, 2019, 12:38 PM

I finally resolved this using the brute force method... I rebuilt the box.

Rather than using a backup I manually recreated my entire config. I had always suspected something had gone wrong with my certificate and cryptographic layer, but was never able to get to the bottom of it. The other symptom I had is that authing over SSH via public key had stopped working as well, while other things, such as HTTPS for the web configurator and my OpenVPN server, still worked correctly. Bizarre.

Coincidence or causation - the one thing I could pinpoint is that the DNS related issues started after installing PFBlockerNG, and unfortunately didn't start working again after I uninstalled it. This all broke some time ago (I think around the initial release of PFSense 2.4) so perhaps there was a bug or incompatibility at the time?

In any case - local DNS caching, DNSSEC, and DNS over TLS all work perfectly now. Sorry this was the resolution if anyone else runs into this :)