DNS over TLS issues (Cloudflare)
-
Hi,
I have been using DNS over TLS with Cloudflare (1.1.1.1 and 1.0.0.1) for some time on the latest stable pfsense. It worked great for the most part. I set it up by putting the addresses into
System/General Settings
and also enablingUse SSL/TLS for outgoing DNS Queries to Forwarding Servers
underDNS Resolver/General Settings
.About a week ago, I realized I could no longer connect to Nvidia GeForce Now (GFN). After a lot of troubleshooting, I realized it was due to DNS issues.
GFN makes a bunch of calls to
nvidiagrid.net
. Unfortunately, pfsense through Cloudflare is unable to resolve that address. If I query Cloudflare directly over TLS, it resolves, just not through pfsense. If I add QUAD dns to the list (also over TLS), it correctly resolves.Since it works via direct query to Cloudflare, and via pfsense query to QUAD, it leads me to believe something is getting messed up when pfsense queries Cloudflare over TLS, but I can't figure it out. I was hoping you guys might be able to.
Here's the nslookup fail through pfsense (
192.168.14.1
is my pfsense address):root@18c52b6a02a3:/# nslookup nvidiagrid.net Server: 192.168.14.1 Address: 192.168.14.1:53 ** server can't find nvidiagrid.net: SERVFAIL ** server can't find nvidiagrid.net: SERVFAIL
Here's querying Cloudflare directly over TLS (using this tool)
root@18c52b6a02a3:/# nslookupot nvidiagrid.net Address: 1.1.1.1#853 -- Name: nvidiagrid.net Name: netbox-ngnipam-430a023de8492df0.elb.us-east-2.amazonaws.com Ttl: 300 Name: nvidiagrid.net Address: 3.20.174.62 Ttl: 60 Name: nvidiagrid.net Address: 3.12.40.190 Ttl: 60
And here's regular nslookup resolving
nvidia.com
(just to show that dns settings in pfsense work fine)root@18c52b6a02a3:/# nslookup nvidia.com Server: 192.168.14.1 Address: 192.168.14.1:53 Non-authoritative answer: Name: nvidia.com Address: 216.228.121.209 Non-authoritative answer:
And after I add QUAD dns to the dns server list (9.9.9.9), it is resolved correctly again:
root@18c52b6a02a3:/# nslookup nvidiagrid.net Server: 192.168.14.1 Address: 192.168.14.1:53 Non-authoritative answer: nvidiagrid.net canonical name = netbox-ngnipam-430a023de8492df0.elb.us-east-2.amazonaws.com Name: netbox-ngnipam-430a023de8492df0.elb.us-east-2.amazonaws.com Address: 3.12.40.190 Name: netbox-ngnipam-430a023de8492df0.elb.us-east-2.amazonaws.com Address: 3.20.174.62 Non-authoritative answer: nvidiagrid.net canonical name = netbox-ngnipam-430a023de8492df0.elb.us-east-2.amazonaws.com
Thanks
-
I did some more troubleshooting and got weird results.
When I ssh in to pfsense and do nslookup for that address, here's the output:
[2.4.4-RELEASE][admin@pfSense.localdomain]/root: nslookup nvidiagrid.net ;; Got SERVFAIL reply from 127.0.0.1, trying next server Server: 1.1.1.1 Address: 1.1.1.1#53 Non-authoritative answer: nvidiagrid.net canonical name = netbox-ngnipam-430a023de8492df0.elb.us-east-2.amazonaws.com. Name: netbox-ngnipam-430a023de8492df0.elb.us-east-2.amazonaws.com Address: 3.12.40.190 Name: netbox-ngnipam-430a023de8492df0.elb.us-east-2.amazonaws.com Address: 3.20.174.62
And here's the dig output:
[2.4.4-RELEASE][admin@pfSense.localdomain]/root: dig nvidiagrid.net ; <<>> DiG 9.12.2-P1 <<>> nvidiagrid.net ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 27023 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;nvidiagrid.net. IN A ;; Query time: 728 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Fri Feb 28 08:49:26 EST 2020 ;; MSG SIZE rcvd: 43
It's strange that the nslookup shows:
;; Got SERVFAIL reply from 127.0.0.1, trying next server Server: 1.1.1.1 Address: 1.1.1.1#53
Shouldn't 127.0.0.1 forward the request to 1.1.1.1 and return the results from there?
Is there anyway I can get detailed logs for the dns resolver?
Thanks
-
i don't believe you have set it up properly.
the following thread shows the 2nd option to check on the resolver page:
https://forum.netgate.com/topic/139771/setup-dns-over-tls-on-pfsense-2-4-4-p2-guide
-
Oh, I see what you're referring to now. When 127.0.0.1 fails, it tries the next server at 1.1.1.1#53, which should have been 1.1.1.1#853. Could that be a pfsense bug because I have it set up exactly as described in that post. Or perhaps just a side effect of trying nslookup on the pfsense box, which has 3 ips listed (although the cloudflare ones really should be tls only, not udp over 53)
- The status page for dns resolver clearly shows that it is set up to use tls over port 853.
- When I follow step 3 from that post and go to diagnostics/states and filter for 1.1.1.1, I see tcp 853, and
- Also checked packet capture and see the dns requests go to cloudflare on port 853
Just an update, as of today, 127.0.0.1 on pfsense resolves nvidiagrid.net. I'm thinking perhaps nvidia had some configuration issue on their dns. But then again, it baffles me why direct queries to 1.1.1.1 over tls resolved, whereas dns resolver forwarded requests to it failed.