Unbound not resolving quad9.net nameservers
-
This is weird - unbound seems to have a problem when trying to resolve nameserver names from quad9.net, EVEN when quad9 DNS is the primary DNS for pfSense!!! (But same broken behaviour no matter what upstream DNS servers are set).
This seems to be a bug? in unbound module - what appears to be happening is that unbound adds the local domain (as defined in pfSense) to the FQDN of these quad9.net hosts, but does not try to resolve the names just the FQDN being queried. In these cases no address is returned (as you would expect).
Behaviour with hosts in other domains seems to be to do a query on both the FQDN as entered as well as the FQDN concatenated with the local domain.
Interestingly, www.quad9.net returns the correct response -so DNS is not broken for this domain, only for certain hosts!
e.g. The quad9 queries only seem to query the local domain, but other domains try both local domain and the actual domain.
Shell Output - unbound-control -c /var/unbound/unbound.conf dump_cache | grep "dns9" msg dns9.quad9.net.mydomain.intranet. IN CNAME 33155 1 3564 5 0 3 0 msg dns9.quad9.net.mydomain.intranet. IN AAAA 33155 1 3564 5 0 3 0 msg dns9.quad9.net.mydomain.intranet. IN A 33155 1 3564 5 0 3 0Shell Output - unbound-control -c /var/unbound/unbound.conf dump_cache | grep "dns11" msg dns11.quad9.net.mydomain.intranet. IN AAAA 33155 1 3596 5 0 3 0 msg dns11.quad9.net.mydomain.intranet. IN CNAME 33155 1 3596 5 0 3 0 msg dns11.quad9.net.mydomain.intranet. IN A 33155 1 3596 5 0 3 0Shell Output - unbound-control -c /var/unbound/unbound.conf dump_cache|grep "dns9.netgate" msg dns9.netgate.com. IN CNAME 33155 1 55 3 0 1 0 msg dns9.netgate.com.mydomain.intranet. IN AAAA 33155 1 3595 5 0 3 0 msg dns9.netgate.com. IN A 33155 1 54 3 0 1 0 msg dns9.netgate.com.mydomain.intranet. IN CNAME 33155 1 3595 5 0 3 0 msg dns9.netgate.com. IN AAAA 33155 1 55 3 0 1 0 msg dns9.netgate.com.mydomain.intranet. IN A 33155 1 3594 5 0 3 0Here is one that returns the correct IPs
www.quad9.net. 650 IN AAAA 2620:0:871:9000::77 www.quad9.net. 650 IN RRSIG <signature removed for clarity> www.quad9.net. 1189 IN A 216.21.3.77 www.quad9.net. 1189 IN RRSIG <signature removed for clarity> msg www.quad9.net. IN CNAME 33152 1 51 3 0 2 0 msg www.quad9.net. IN AAAA 33152 1 650 3 1 0 0 www.quad9.net. IN AAAA 0 msg www.quad9.net.mydomain.intranet. IN CNAME 33155 1 3051 5 0 3 0 msg www.quad9.net. IN A 33152 1 1189 3 1 0 0 www.quad9.net. IN A 0At first I suspected that the 9 in the hostname might be causing a problem, but as can be seen for the netgate domain example, that would resolve if such a host existed.
- I have tried with pfBlocker disabled and enabled - same results.
- As noted, I have tried changing upstream forwarding - same results.
- I have tried with Hide Identity disabled - same results.
- I have powered off & re-started the firewall - same results.
- If I add a DNS over-ride for , say, dns9.quad9.net then it resolves as expected.
Related Settings:
- Unbound set to forwarding mode, with upstream DNS set to 9.9.9.9 and 1.1.1.1 (but also tried with 1.1.1.1 and 1.0.0.1 and 8.8.8.8)
- Python module enabled
- No DNS over-rides in place.
- Query Name Minimization is disabled
- Serve Expired is disabled
- local domain - mydomain.intranet
-
@rossm
As seen here : see the pastebin unbound log. -
@gertjan
Sorry - I don't understand how that relates to my issue.I don't have a slow DNS response - all the working DNS response times are fine.
The issue is that unbound is not even MAKING an external request for these hosts, and it seems to be because it is adding the local domain as a suffix to the requested DNS query.
If I try to force a root query by adding a traling dot (i.e. dns9.quad9.net. rather than dns9.quad9.net) then the query attempt does not even make it into the DNS logs.
Further troubleshooting shows that the issue is in the unbound Python module. If I turn off the Python module in DNS Resolver, all works correctly.
This definitely seems to be a bug in the Python module.
How do I go about reporting a bug?
-
I'm seeing the same thing :
but this is exactly what I want, as i've checked everything here :
I also had a list with DoH servers blocked :
When I removed all this, dns9.quad9.net gets resolved just fine :
I'm using unbound with the default settings + pfBlockerNG (latest) with the Python extension activated.
When I removed all IP and DNSBL feeds that could reference 9.9.9.9 or dns.quand9.net, I saw :
So, The unbound, and the pythn script for that matter, saw my request, and handled it.
@rossm said in Unbound not resolving quad9.net nameservers:
The issue is that unbound is not even MAKING an external request for these hosts
Possible reasons :
Unbound never sees the request coming from device. Does the DNS request arrive @pfSEnse port 53 UDP ?
Does unbound listen on that interface ?
Firewall rules that block your device (DNS requests) ?
Etc. -
@gertjan
Thanks - I do think you are right about it using the DNS over HTTPS/TLS Blocking list. However this should not be applied to DNS over HTTPS/TLS requests from the firewall itself !! If this is designed in behavior then I think its very poor design. I can't see why you would want to block the firewall from making those secure DNS requests. If its designed behavior, then there should at least be the option to exclude the firewall from this list.Or maybe there is a setting somewhere that fixes this issue & I can't find it?!? I'm happy to admit my lack of in-depth knowledge of pfSense.
Unbound never sees the request coming from device. Does the DNS request arrive @pfSEnse port 53 UDP ?
Does unbound listen on that interface ?For both above - Yes - if it did not then I would not be seeing the results I posted above in the pfSense logs!
As you can see from above, the logs show that unbound is concatenating the local domain to the requested fully qualified name, but only for certain hosts.Firewall rules that block your device (DNS requests) ?
Not relevant - its not a general failure of DNS - its a failure of how the system is handling these specific requests from the firewall itself. (In light of the above, likely how system is mis-using the DNS over HTTPS/TLS Block list to block requests from the firewall )
-
@rossm said in Unbound not resolving quad9.net nameservers:
unbound is concatenating the local domain to the requested fully qualified name
No it doesn't do that.. Your client is what send a query with search suffix or not.. unbound isn't going to alter what was actually asked for by the client.
set debug on your nslookup and you will set it doing it..
$ nslookup Default Server: pi.hole Address: 192.168.3.10 > set debug > www.google.com Server: pi.hole Address: 192.168.3.10 ------------ Got answer: HEADER: opcode = QUERY, id = 2, rcode = NXDOMAIN header flags: response, auth. answer, want recursion, recursion avail. questions = 1, answers = 0, authority records = 0, additional = 0 QUESTIONS: www.google.com.local.lan, type = A, class = IN ------------ ------------ Got answer: HEADER: opcode = QUERY, id = 3, rcode = NXDOMAIN header flags: response, auth. answer, want recursion, recursion avail. questions = 1, answers = 0, authority records = 0, additional = 0 QUESTIONS: www.google.com.local.lan, type = AAAA, class = IN ------------ ------------ Got answer: HEADER: opcode = QUERY, id = 4, rcode = NOERROR header flags: response, want recursion, recursion avail. questions = 1, answers = 1, authority records = 0, additional = 0 QUESTIONS: www.google.com, type = A, class = IN ANSWERS: -> www.google.com internet address = 142.250.190.4 ttl = 1976 (32 mins 56 secs)My local domain is local.lan - so yeah client doing set to append search suffix will add that..
if that is slowing down your queries because they are being forwarded, and taking time to get a nx response - or bad something you don't want because whatever you adding to the search suffix actually responds on the public. Turn off that feature in your OS.
Or in unbound set your local zone to static vs transparent - so now if say www.google.com.local.lan doesn't exist in my local records for local.lan - it will not forward or try to resolve those.
https://nlnetlabs.nl/documentation/unbound/unbound.conf/
It defaults to transparent - I have mine set to static, not because of any performance issues - but because there is no reason to try and resolve anything.local.lan on the public internet ever..
Now if you were using same domain name locally that was used publicly - that might be different story.
-
@johnpoz
But, but but - my client is the firewall!The shell output in my post above is from the pfSense console.
So maybe its not unbound - but something in the firewall is messing with these queries (Likely to do with the DNS over HTTPS/TLS Blocking list)
-
Ah - so hmmm, I just enabled query logging, yeah when I put in www.testlookup.com in the dns host gui, www.testlookup.com.local.lan is being asked for..
But if set to static for your zone it would end right there.. Prob set in the resolv.conf for pfsense..
yup!
[21.05.1-RELEASE][admin@sg4860.local.lan]/: cat /etc/resolv.conf nameserver 127.0.0.1 search local.lan [21.05.1-RELEASE][admin@sg4860.local.lan]/:Not sure if anyway to turn that off, not in the gui.. have to look.. Would be a good feature request I would think, if you edit the file, at some point prob just put it back..
-
With regard to the DNS over HTTPS/TLS Blocking list:
It does seem that this list stops these hosts from being resolved in DNS. And that seems a pretty blunt instrument.
It would be better if we could simply stop HTTPS traffic to this list (as we can handle DNS & DNS over TLS with simple firewall & port forward rules. Blocking HTTPS to these hosts would then stop encrypted DNS queries, but still allow the hosts to be resolved)
I guess I could build this manually, but would then have to keep it updated manually as well.
The reason this surfaced was that a whole bunch of machine updates broke on my network, as the software was doing an Internet connectivity check by trying to ping quad9 name-servers. As pfSense would not resolve the addresses, the updates kept failing.
-
I suppose my other option would be to manually maintain host over-rides in DNS for all the hosts in the HTTPS/TLS Blocking list. Again, not an ideal scenario.
I really assumed that HTTPS/TLS Blocking list would have been smarter & been blocking the appropriate ports for each host in the list (53, 443, 853) rather than just stopping name resolution. Blocking name resolution is pretty brain-dead, as its so easily subverted by vendors - eg Google could code the IPs for its name servers in Android OS. Same for any software/IoT vendor.
-
@rossm welcome to the bane that is doh, atleast with dot its just block port 853..
I block based on IP, there are lists you can pull..
but also block them from resolving as well
;; ANSWER SECTION: dns9.quad9.net. 2 IN A 0.0.0.0doh is HORRIBLE solution to a non issue in the first place.. And doesn't hide anything from anyone anyway until such time there is actual encrypted sni, its just a bane for network admins trying control their networks.. If you ask me its a way for them to serve more ads by circumvention of local dns.. And have users send them data that they can leverage for their whatever needs..
-
@johnpoz said in Unbound not resolving quad9.net nameservers:
@rossm welcome to the bane that is doh, atleast with dot its just block port 853..
I block based on IP, there are lists you can pull..
Hmm.. so is it possible to build a dynamic firewall rule based on a list? That would fix the issue for me - just block HTTPS to hosts in the list (which is how I was expecting the built in feature to work)
-
@rossm here is I am using
https://raw.githubusercontent.com/oneoffdallas/dohservers/master/iplist.txt
You can just create an alias with that in pfblocker - and then use in firewall you set.. That is what I do..
I use a port alias as well - because there are some that are using other than 443 port as well.. They all SUCK!! Its like spam - they always find a new trick to get through your filters, doh providers no better to be honest..
-
@johnpoz
Thanks John,I'll look into that in the morning (well - later in the morning - its 3am here now so I shouldn't still be in front of the computer - bed time!)
-
@rossm said in Unbound not resolving quad9.net nameservers:
I can't see why you would want to block the firewall from making those secure DNS requests.
No list will block root, tld or domain name servers.
For me, the secure part is :
Getting back a answer that I can trust : when unbound asks for : what is A for abcd.tld, I can get back an answer that is DNSSEC checked. DNSSEC only works over port 53, using the 'classic' root servers etc. I resolve myself using unbound, not askling some one to do it for me.
The fact that 'abcd.tld' goes over the net in clear, I don't mind. -
@gertjan said in Unbound not resolving quad9.net nameservers:
No list will block root, tld or domain name servers.
I can tell you categorically that some part of the pfSense DNS resolver is blocking requests made by the pfSense machine itself, so that statement is incorrect. Based on your (helpful - thanks) post earlier I tested the DNS over HTTPS/TLS Blocking List as the likely culprit for my issue. When it is turned on DNS queries, made by the pfSense machine itself, are being blocked. Turn it off & these queries resolve just fine.
I don't think this is helpful behavior at all. Sure - I want to block devices on my network from making queries direct to the DNS servers in this block list. But I DO want the pfSense resolver to be able to resolve the addresses of these machines.
As I stated above, just blocking name resolution for the servers in this block list is unhelpful & easily circumvented. A much better solution would be to block HTTPS & TLS traffic to these servers, but that is not what pfSense is doing - its simply refusing to resolve the IP addresses of these server names.
I already have rules to port forward all DNS & TLS traffic originating on my local network to unbound on pfSense. I think I will turn off the (to my mind, somewhat broken) DNS over HTTPS/TLS Blocking and use the list as an alias for a rule that blocks HTTPS to these hosts.
Maybe, as johnpoz says, some devices are using non-standard ports to initiate DoH queries. However I suspsct that if this is true these are edge cases & the majority of DoH traffic would be blocked by my strategy above. If I find any devices using non-standard ports, these can also be added to a block rule.
-
@rossm said in Unbound not resolving quad9.net nameservers:
blocking requests made by the pfSense machine itself,
pfsense asking unbound is no different than any other client - the query just come sin on localhost - there is no difference to unbound. Unless you have some ACL setup.
But if you put servers in normal dns listing under general - then pfsense can and will just ask them via normal 53.. Unless you change the setting..
Users have a hard time grasping this it seems.. The only way traffic is going to be forwarded and encrypted is if unbound is asked, be it this is via client on your network doing the query to 192.168.1.1 for example or pfsense asking via 127.0.0.1 but if you have dns setup in general - it can and will just ask say 8.8.8.8 via normal 53.
Pfsense is able to talk to anything it wants - unless you put in a floating rule blocking some ports outbound on the wan, etc..
-
@johnpoz
Yup - I understand that & have set up DNS for the firewall to only use localhost (unbound) for its queries. i.e. - its set up just the way your screenshot shows, and I have unbound set to use TLS.My mistake was (naively) believing that pfBlocker was doing something other than simply sink-holing DNS resolution for the hosts in the " DoH/DoT Blocking List". I did not expect that unbound would be unable to resolve these hosts due to this block list.
I also expected that unbound would be allowed to make outbound DNS queries (53 & 853) despite any other firewall rules in place. Is this the case - e.g. if I made a firewall rule blocking all outbound port 53 traffic, would that block unbound or would it still be able to resolve addresses using DNS port 53?
Is there any document that shows the traffic flow within pfSense for the various sub-systems/modules? That would be really helpful for troubleshooting, to understand where stuff might get blocked.
-
@rossm said in Unbound not resolving quad9.net nameservers:
I can tell you categorically that some part of the pfSense DNS resolver is blocking requests made by the pfSense machine itself, so that statement is incorrect
You're right.
While blocking root servers will shut down the entire DNS resolve process, blocking tld servers (the ones that know all about dot com dot org dot net dot us or dot net etc is) actually an option for pfBlockerNG.Refusing a DNS request is always a pfBlockerNG thing, not unbound, as unbound, before resolving starts, uses pfBlockerNG to check (compare) the domain name with known lists.
-
@rossm said in Unbound not resolving quad9.net nameservers:
if I made a firewall rule blocking all outbound port 53 traffic, would that block unbound or would it still be able to resolve addresses using DNS port 53?
If you did it on floating and outbound direction, then that blocks all 53 going outbound.. Doesn't matter what processes was creating the traffic.