[SOLVED] Weird DNS Problem

EveningStarNM · Nov 11, 2018, 8:49 AM

This is a test environment with a fresh installation of pfsense 2.4.4, minimal configuration changes from the defaults, DNS Resolver works. The only firewall rules other than the defaults allow all traffic from LAN to WAN and DNS from LAN to pfsense and from pfsense to its forwarders 1.1.1.1/1.0.0.1 only. Clients on the test.local LAN run Windows 7, Windows 10, Linux Mint 19, and iOS12.1, including an iPad and an iPhone. Some of the clients are configured with static IPs and gateways, some use DHCP, but all use only the pfsense machine for DNS resolution.

All of the clients can access most domains. However, DNS for some domains resolves when using Diagnostics/DNS Lookup on pfsense, but the resolution doesn't pass through to any of the clients. For instance, robgillen.com resolves to an IP address using DNS lookup in pfsense, but none of the clients receive that address. For example, on Windows 10, I run this from the command line:

nslookup robgillen.com
Server:  pfSense.test.local
Address:  192.168.254.1

DNS request timed out.
    timeout was 2 seconds.
*** Request to pfSense.test.local timed-out

But if I specify the same external DNS server that pfsense uses, it resolves on that same client:

nslookup robgillen.com 1.1.1.1
Server:  one.one.one.one
Address:  1.1.1.1

Non-authoritative answer:
Name:    argodev.github.io
Addresses:  185.199.111.153
          185.199.108.153
          185.199.109.153
          185.199.110.153
Aliases:  robgillen.com

On the other hand, this works just fine when using pfsense's DNS Resolover:

nslookup www.google.com
Server:  pfSense.eis.local
Address:  192.168.254.1

Non-authoritative answer:
Name:    www.google.com
Addresses:  2607:f8b0:400f:805::2004
          172.217.1.196

If I change the clients' configurations to use external DNS servers, there are no problems, but if they use pfsense for DNS, that domain does not resolve.

This happens on all of the clients, regardless of operating system or configuration. As I said, most domains resolve without problems, but a few don't, even though pfsense can resolve the IP addresses for those domains in DNS lookup. Disabling the firewall doesn't fix the problem.

I'm baffled. This is the simplest and cleanest configuration I can imagine for pfsense and the test network. I'll appreciate any suggestions that anyone can offer.

UPDATE and Solution: I think that I misunderstandd how DNS forwarding works on pfsense. It appears to work differently than it does on Windows servers. Our lab has a direct connection to the internet, and I'd configured pfsense to forward DNS resolution requests for external domains to Cloudflare's DNS servers by checking "Enable Forwarding Mode" under Services/DNS Resolver/General Settings. Removing that check mark does not stop requests from being forwarded to the DNS servers configured under Settings/General Setup, and the few sites that I was having problems with now resolve. Nevertheless, even though the problem is solved, I'd like to know what is different about those domains that was causing the problem. Maybe one day I'll have the time to research it.

stephenw10 · Nov 11, 2018, 1:52 PM

Did you have DNSSec enabled?

In resolving mode Unbound talks directly to the DNS root servers which reliably support DNSSec. If you put it into forwarding mode it's usually better to disable DNSSec as it's likely something in the chain of forwarded servers will not support DNSSec and will be ignored. Since that can be different for different sites that could explain what you were seeing.

Steve

johnpoz · Nov 11, 2018, 2:06 PM

@eveningstarnm said in [SOLVED] Weird DNS Problem:

from LAN to WAN

That his not default the default is lan to ANY... There is a HUGE difference... Wan is not the internet - its just the wan network.. Users that set such a rule will break any sort of internet access...

stephenw10 · Nov 11, 2018, 2:10 PM

A "DNS from LAN to pfsense" rule should allow that though.

Steve

EveningStarNM · Mar 3, 2019, 12:25 PM

@stephenw10 Yes, I thought that, too, and I tried it, but it didn't work. In fact, I tried opening up everything between pfsense and LAN, yet still no joy. But thanks for that observation about how Unbound works in supporting DNSSec. I think you're right: That's probably what caused the problem.

4o4rh · Mar 3, 2019, 12:30 PM

@eveningstarnm did you ever get the solution to this? I have in production at home, and on the whole the dns works but i have just discovered this same problem with the download-c.huawei.com in debug logging, i can see the request from the client and the requests resolves on the server correctly. also from the pfsense diag screen. however, on a win10 machine, i receive DNS Request timeout, timeout was 2sec.

the machine originally had comodo dns servers set, but they are trapped and redirected on pfsense.
nevertheless, i change the dns to on the client to pfsense and it makes no difference.

p.s. i might add, if i switch DNS Query Forwarding on. it works.

johnpoz · Mar 4, 2019, 9:31 AM

What a cluster of cnames

;; QUESTION SECTION:
;download-c.huawei.com.         IN      A

;; ANSWER SECTION:
download-c.huawei.com.  3598    IN      CNAME   download-c.huawei.com.c.cdnhwc1.com.
download-c.huawei.com.c.cdnhwc1.com. 3600 IN CNAME download-c.huawei.com.wscdns.com.
download-c.huawei.com.wscdns.com. 3600 IN CNAME dp6opavwln80.cloudfront.net.

But not having any problems resolving that with unbound

They have a HUGE mess!!
http://dnsviz.net/d/download-c.huawei.com/dnssec/

4o4rh · Mar 3, 2019, 12:51 PM

like i said, if i use dns lookup on the pfsense fiag box, it resolve per yours. (both with/without forwarding)
Only the client side, it resolves with forwarding and does not without forwarding..i am using unbound

very strange indeed..... hence the original poster's Weird DNS Problem - but he didn't write how he solved it

johnpoz · Mar 3, 2019, 12:52 PM

That resolves just fine on my win 10 box.. My guess is their nonsense with no glue for their IPv6 and your trying to access via IPv6? Their DNS is freaking HORRIBLE!!! That anyone can get there is just freaking amazing ;)

even going through pihole

$ nslookup download-c.huawei.com
Server:  pi-hole.local.lan
Address:  192.168.3.10

Non-authoritative answer:
Name:    dp6opavwln80.cloudfront.net
Addresses:  99.84.240.136
          99.84.240.45
          99.84.240.175
          99.84.240.78
Aliases:  download-c.huawei.com
          download-c.huawei.com.c.cdnhwc1.com
          download-c.huawei.com.wscdns.com

4o4rh · Mar 3, 2019, 12:52 PM

@johnpoz i have IPv6 disabled on everything inc my clients

johnpoz · Mar 3, 2019, 1:02 PM

so do either a debug on nslookp or grab dig and do a trace and see what is failing or just getting there?

If I click what you posted I get

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

So have no idea what you think is suppose to be there or what is suppose to get there?

edit:
Not sure what his issue was.. He rule to any to wan wouldn't of even allowed internet at all, etc. His domain resolved just fine as well - but again a nightmare of errors... cname mess as well

Maybe he had qminimization on? Have no idea with the lack of info given.

If pfsense resolves it fine, and client asks pfsense then pfsense will hand to client what it got... If you want help in finding your issue going to have to give a bit more than client can not resolve.. For all we know your clients using some proxy, etc. Do you get a time out from pfsense? Do you get a NX, etc. etc..

stephenw10 · Mar 3, 2019, 1:02 PM

Ha, seems legit!

johnpoz · Mar 3, 2019, 1:10 PM

Yeah that any legit company would have such a NIGHTMARE MESS for their dns is just SAD!!!

But it does resolve - so without some more info.. there is nothing can tell you. On what your actual issue might or might not be?

You sure your pointing pfsense to itself.. See it all the time where clients point pfsense to some external when they are running resolver.. When you resolve you might have issue resolving if you can not actually talk to the NS for the domain, and with so many cnames your going to have to talk to many NS ;) failure or time out to any one of them in that chain could cause you grief in resolving.

Do a simple +trace to see where you might be having issue.

Here is trace - but its only 1st leg, since there are 2 more cnames!! Arrrggh

$ dig download-c.huawei.com +trace                                                                                 
                                                                                                                   
; <<>> DiG 9.12.3-P1 <<>> download-c.huawei.com +trace                                                             
;; global options: +cmd                                                                                            
.                       450361  IN      NS      d.root-servers.net.                                                
.                       450361  IN      NS      k.root-servers.net.                                                
.                       450361  IN      NS      j.root-servers.net.                                                
.                       450361  IN      NS      l.root-servers.net.                                                
.                       450361  IN      NS      c.root-servers.net.                                                
.                       450361  IN      NS      f.root-servers.net.                                                
.                       450361  IN      NS      a.root-servers.net.                                                
.                       450361  IN      NS      i.root-servers.net.                                                
.                       450361  IN      NS      e.root-servers.net.                                                
.                       450361  IN      NS      m.root-servers.net.                                                
.                       450361  IN      NS      h.root-servers.net.                                                
.                       450361  IN      NS      g.root-servers.net.                                                
.                       450361  IN      NS      b.root-servers.net.                                                
.                       450361  IN      RRSIG   NS 8 0 518400 20190315140000 20190302130000 16749 . ShqK2j6F+OJAQI+
sTMG6oms moskB4VtcKYv+CwtAxmuWDnVAVJz35hcj4rkdXBv6Wz423JmuesZsl4j SUMl4xFUT42faNiHG/OZybi+uxwXSggIrQHhM9kg9WlamZ+Y+
5/i+ifoWJGMUZOXp0SIkJJGXHeFI2P2 vfReHTB6/QUQNMFPHGjMhvM37AFruXs0vktZWZ9aAjj+uXvA0kvgZYIO SKcRsbULeCQNlnvSncTZFCM+YE
phwQ==                                                                                                             
;; Received 525 bytes from 192.168.3.10#53(192.168.3.10) in 3 ms                                                   
                                                                                                                   
com.                    172800  IN      NS      a.gtld-servers.net.                                                
com.                    172800  IN      NS      b.gtld-servers.net.                                                
com.                    172800  IN      NS      c.gtld-servers.net.                                                
com.                    172800  IN      NS      d.gtld-servers.net.                                                
com.                    172800  IN      NS      e.gtld-servers.net.                                                
com.                    172800  IN      NS      f.gtld-servers.net.                                                
com.                    172800  IN      NS      g.gtld-servers.net.                                                
com.                    172800  IN      NS      h.gtld-servers.net.                                                
com.                    172800  IN      NS      i.gtld-servers.net.                                                
com.                    172800  IN      NS      j.gtld-servers.net.                                                
com.                    172800  IN      NS      k.gtld-servers.net.                                                
com.                    172800  IN      NS      l.gtld-servers.net.                                                
com.                    172800  IN      NS      m.gtld-servers.net.                                                
com.                    86400   IN      DS      30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF 
com.                    86400   IN      RRSIG   DS 8 1 86400 20190316050000 20190303040000 16749 . FqDz8iA8R/26+mJO
5vVsdnr XEiNTUA+aO8xsAEzNfctse3SY7XLmTnMI4o/YgL4cfwLnQeUR+7fF1qj +pUvL5H+qXk3NRRWFkDhtYoaBEo4Vgg9yjZN2bSWThBelY/QD5
yWzrR5yLJfknD9OhewaHp120kCWxkn m0YnRtiZ0xUjlNm1IcmAv9CuliXTyMCTU41VqCFXfjFIoKRyVnEIMUos Q+wJ9gk1XRM4LUI3T+1+Jcobftk
3pA==                                                                                                              
;; Received 1181 bytes from 198.97.190.53#53(h.root-servers.net) in 35 ms                                          
                                                                                                                   
huawei.com.             172800  IN      NS      nsallsec.huawei.com.                                               
huawei.com.             172800  IN      NS      nsall.huawei.com.                                                  
huawei.com.             172800  IN      NS      nsall3rd.huawei.com.                                               
huawei.com.             172800  IN      NS      nsall4th.huawei.com.                                               
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q1GIN43N1ARRC9OSM6QPQR81H5M9A NS SOA RRSIG DNSKEY N
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20190308054542 20190301043542 16883 com. FgR1J
rnGB/qZQvAwOFwglFi sfO86z794cvm/6B4zzRDMFDLJKvhPyvYblmRCUCj2iMgZVksTqxDYbGR 2/2MFK9HwQpGr79itYpKMexrS1duvbogm/xNgiL
96PTCKH6PDT3DG589UG3MN0NA6LG5MTJ.com. 86400 IN NSEC3 1 1 0 - 96PU6P2QEB574VGKGIUQGM47GT2D04SQ NS DS RRSIG          
96PTCKH6PDT3DG589UG3MN0NA6LG5MTJ.com. 86400 IN RRSIG NSEC3 8 2 86400 20190309063008 20190302052008 16883 com. g36Vi
BL3b29EVq9rNyutbGT qdpfOBgD5JHsZHDf76qb0cKd7p+nzjee2UAFhd0Z5EgvRUv+3s0PM6qq evdbQx4tQA56b6NwLqipwe7SUduKpIhqAF7P0CY
;; Received 720 bytes from 192.35.51.30#53(f.gtld-servers.net) in 30 ms                                            
                                                                                                                   
download-c.huawei.com.  600     IN      CNAME   download-c.huawei.com.c.cdnhwc1.com.                               
;; Received 96 bytes from 122.96.104.66#53(nsall4th.huawei.com) in 255 ms

4o4rh · Mar 3, 2019, 5:37 PM

the huawei site is the download site for their windows application that HiSuite that connects their phones.

this is what i get on my linux client, with DNS forwarding enabled.

nslookup download-c.huawei.com
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
download-c.huawei.com	canonical name = download-c.huawei.com.c.cdnhwc1.com.
download-c.huawei.com.c.cdnhwc1.com	canonical name = download-c.huawei.com.wscdns.com.
download-c.huawei.com.wscdns.com	canonical name = dp6opavwln80.cloudfront.net.
Name:	dp6opavwln80.cloudfront.net
Address: 143.204.208.172
Name:	dp6opavwln80.cloudfront.net
Address: 143.204.208.221
Name:	dp6opavwln80.cloudfront.net
Address: 143.204.208.117
Name:	dp6opavwln80.cloudfront.net
Address: 143.204.208.36

this is what i get, if i uncheck DNS forwarding and no other changes.

nslookup download-c.huawei.com
;; connection timed out; no servers could be reached

to be clear, with forwarding unchecked, straight after the previous nslookup, i do a google and is ok.

nslookup google.com
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	google.com
Address: 172.217.17.110
Name:	google.com
Address: 2a00:1450:400e:806::200e

4o4rh · Mar 3, 2019, 5:44 PM

@johnpoz said in [SOLVED] Weird DNS Problem:

dig download-c.huawei.com +trace

using your command, this is what i get with forwarding unchecked

dig download-c.huawei.com +trace

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> download-c.huawei.com +trace
;; global options: +cmd
;; Received 28 bytes from 127.0.0.53#53(127.0.0.53) in 0 m

this is the result with forwarding checked, and i note the result is the same.

dig download-c.huawei.com +trace

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> download-c.huawei.com +trace
;; global options: +cmd
;; Received 28 bytes from 127.0.0.53#53(127.0.0.53) in 0

probably need some help from you with the commands, to do any deeper analysis. but seems very odd to me, that the nslookup fails simply by changing the forwarding mode

stephenw10 · Mar 3, 2019, 5:46 PM

You seem to be testing against localhost. What do you have running on the client? How is that configured?

What do you see if you test against the pfSense interface IP?

Steve

4o4rh · Mar 3, 2019, 6:07 PM

that is what it returns. both the windows and linux clients are DHCP with the gateway pointing to the LAN interface address. for the DNS, the linux points to the LAN interface address and for windows, it has static comodo firewall servers defined.

on the pfsense box

NAT is
LAN UDP * * !LOCAL DNS 127.0.0.1 DNS
** it didn't work at all when i used the LAN interface for the NAT IP. It only works if i use 127

Rules are
IPv4 TCP/UDP LAN * This Firewall DNS, VPN Gateway
** if i change the gateway to default, the results are the same

the only thing strange i notice on the windows client, (forwarding enabled)

nslookup download-c.huawei.com
Server:  UnKnown
Address:  156.154.70.22

Non-authoritative answer:
Name:    dp6opavwln80.cloudfront.net
Addresses:  143.204.98.14
          143.204.98.132
          143.204.98.156
          143.204.98.173
Aliases:  download-c.huawei.com
          download-c.huawei.com.c.cdnhwc1.com
          download-c.huawei.com.wscdns.com

forwarding disabled

nslookup download-c.huawei.com
DNS request timed out.
    timeout was 2 seconds.
Server:  UnKnown
Address:  156.154.70.22

DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
*** Request to UnKnown timed-out

in both cases, windows says server unknown...not sure if that is normal

4o4rh · Mar 3, 2019, 6:09 PM

@johnpoz said in [SOLVED] Weird DNS Problem:

Server: pi-hole.local.lan
I see in John's windows lookup above, he gets Server: pi-hole.local.lan returned, where as mine is unknown.

could it be something wrong, with my NAT or Rules above?

stephenw10 · Mar 3, 2019, 6:15 PM

Well try dig @pfSense_interface and see how that changes between modes.

Steve

4o4rh · Mar 3, 2019, 6:19 PM

this is from forwarding enabled

dig @192.168.20.5

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @192.168.20.5
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20423
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;.				IN	NS

;; ANSWER SECTION:
.			463269	IN	NS	a.root-servers.net.
.			463269	IN	NS	e.root-servers.net.
.			463269	IN	NS	l.root-servers.net.
.			463269	IN	NS	i.root-servers.net.
.			463269	IN	NS	j.root-servers.net.
.			463269	IN	NS	c.root-servers.net.
.			463269	IN	NS	k.root-servers.net.
.			463269	IN	NS	b.root-servers.net.
.			463269	IN	NS	m.root-servers.net.
.			463269	IN	NS	d.root-servers.net.
.			463269	IN	NS	h.root-servers.net.
.			463269	IN	NS	g.root-servers.net.
.			463269	IN	NS	f.root-servers.net.

;; Query time: 169 msec
;; SERVER: 192.168.20.5#53(192.168.20.5)
;; WHEN: Sun Mar 03 19:15:54 CET 2019
;; MSG SIZE  rcvd: 239

this is forwarding disabled.

$ nslookup download-c.huawei.com
;; connection timed out; no servers could be reached

$ dig @192.168.20.5

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @192.168.20.5
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12658
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;.				IN	NS

;; ANSWER SECTION:
.			518383	IN	NS	a.root-servers.net.
.			518383	IN	NS	b.root-servers.net.
.			518383	IN	NS	c.root-servers.net.
.			518383	IN	NS	d.root-servers.net.
.			518383	IN	NS	e.root-servers.net.
.			518383	IN	NS	f.root-servers.net.
.			518383	IN	NS	g.root-servers.net.
.			518383	IN	NS	h.root-servers.net.
.			518383	IN	NS	i.root-servers.net.
.			518383	IN	NS	j.root-servers.net.
.			518383	IN	NS	k.root-servers.net.
.			518383	IN	NS	l.root-servers.net.
.			518383	IN	NS	m.root-servers.net.

;; Query time: 3 msec
;; SERVER: 192.168.20.5#53(192.168.20.5)
;; WHEN: Sun Mar 03 19:18:05 CET 2019
;; MSG SIZE  rcvd: 239