DNS over TLS issues (Cloudflare)

aptalca

Hi,

I have been using DNS over TLS with Cloudflare (1.1.1.1 and 1.0.0.1) for some time on the latest stable pfsense. It worked great for the most part. I set it up by putting the addresses into System/General Settings and also enabling Use SSL/TLS for outgoing DNS Queries to Forwarding Servers under DNS Resolver/General Settings.

About a week ago, I realized I could no longer connect to Nvidia GeForce Now (GFN). After a lot of troubleshooting, I realized it was due to DNS issues.

GFN makes a bunch of calls to nvidiagrid.net. Unfortunately, pfsense through Cloudflare is unable to resolve that address. If I query Cloudflare directly over TLS, it resolves, just not through pfsense. If I add QUAD dns to the list (also over TLS), it correctly resolves.

Since it works via direct query to Cloudflare, and via pfsense query to QUAD, it leads me to believe something is getting messed up when pfsense queries Cloudflare over TLS, but I can't figure it out. I was hoping you guys might be able to.

Here's the nslookup fail through pfsense (192.168.14.1 is my pfsense address):

root@18c52b6a02a3:/# nslookup nvidiagrid.net
Server:         192.168.14.1
Address:        192.168.14.1:53

** server can't find nvidiagrid.net: SERVFAIL

** server can't find nvidiagrid.net: SERVFAIL

Here's querying Cloudflare directly over TLS (using this tool)

root@18c52b6a02a3:/# nslookupot nvidiagrid.net
Address:        1.1.1.1#853
--
Name:           nvidiagrid.net
Name:           netbox-ngnipam-430a023de8492df0.elb.us-east-2.amazonaws.com
Ttl:            300

Name:           nvidiagrid.net
Address:        3.20.174.62
Ttl:            60

Name:           nvidiagrid.net
Address:        3.12.40.190
Ttl:            60

And here's regular nslookup resolving nvidia.com (just to show that dns settings in pfsense work fine)

root@18c52b6a02a3:/# nslookup nvidia.com
Server:         192.168.14.1
Address:        192.168.14.1:53

Non-authoritative answer:
Name:   nvidia.com
Address: 216.228.121.209

Non-authoritative answer:

And after I add QUAD dns to the dns server list (9.9.9.9), it is resolved correctly again:

root@18c52b6a02a3:/# nslookup nvidiagrid.net
Server:         192.168.14.1
Address:        192.168.14.1:53

Non-authoritative answer:
nvidiagrid.net  canonical name = netbox-ngnipam-430a023de8492df0.elb.us-east-2.amazonaws.com
Name:   netbox-ngnipam-430a023de8492df0.elb.us-east-2.amazonaws.com
Address: 3.12.40.190
Name:   netbox-ngnipam-430a023de8492df0.elb.us-east-2.amazonaws.com
Address: 3.20.174.62

Non-authoritative answer:
nvidiagrid.net  canonical name = netbox-ngnipam-430a023de8492df0.elb.us-east-2.amazonaws.com

Thanks

aptalca

I did some more troubleshooting and got weird results.

When I ssh in to pfsense and do nslookup for that address, here's the output:

[2.4.4-RELEASE][admin@pfSense.localdomain]/root: nslookup nvidiagrid.net
;; Got SERVFAIL reply from 127.0.0.1, trying next server
Server:         1.1.1.1
Address:        1.1.1.1#53

Non-authoritative answer:
nvidiagrid.net  canonical name = netbox-ngnipam-430a023de8492df0.elb.us-east-2.amazonaws.com.
Name:   netbox-ngnipam-430a023de8492df0.elb.us-east-2.amazonaws.com
Address: 3.12.40.190
Name:   netbox-ngnipam-430a023de8492df0.elb.us-east-2.amazonaws.com
Address: 3.20.174.62

And here's the dig output:

[2.4.4-RELEASE][admin@pfSense.localdomain]/root: dig nvidiagrid.net

; <<>> DiG 9.12.2-P1 <<>> nvidiagrid.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 27023
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;nvidiagrid.net.                        IN      A

;; Query time: 728 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Feb 28 08:49:26 EST 2020
;; MSG SIZE  rcvd: 43

It's strange that the nslookup shows:

;; Got SERVFAIL reply from 127.0.0.1, trying next server
Server:         1.1.1.1
Address:        1.1.1.1#53

Shouldn't 127.0.0.1 forward the request to 1.1.1.1 and return the results from there?

Is there anyway I can get detailed logs for the dns resolver?

Thanks

bcruze

i don't believe you have set it up properly.

the following thread shows the 2nd option to check on the resolver page:

https://forum.netgate.com/topic/139771/setup-dns-over-tls-on-pfsense-2-4-4-p2-guide

aptalca

Oh, I see what you're referring to now. When 127.0.0.1 fails, it tries the next server at 1.1.1.1#53, which should have been 1.1.1.1#853. Could that be a pfsense bug because I have it set up exactly as described in that post. Or perhaps just a side effect of trying nslookup on the pfsense box, which has 3 ips listed (although the cloudflare ones really should be tls only, not udp over 53)

The status page for dns resolver clearly shows that it is set up to use tls over port 853.
When I follow step 3 from that post and go to diagnostics/states and filter for 1.1.1.1, I see tcp 853, and
Also checked packet capture and see the dns requests go to cloudflare on port 853

Just an update, as of today, 127.0.0.1 on pfsense resolves nvidiagrid.net. I'm thinking perhaps nvidia had some configuration issue on their dns. But then again, it baffles me why direct queries to 1.1.1.1 over tls resolved, whereas dns resolver forwarded requests to it failed.