DNS Forwarder changed behavior after upgrade to 2.7.0

kiokoman · Aug 23, 2023, 6:02 PM

I've noticed that some services (telegraf / zabbix ) were unable to resolve internal hostname
i have DNS Forwarder enable and this settings on general:

but dns forwarder is asking root servers instead of my own dns

[2.7.0-RELEASE][root@DTLFWSRV02.datalab.local]/root: cat /etc/resolv.conf
nameserver 127.0.0.1
nameserver 192.168.8.230
nameserver 192.168.8.234
search datalab.local

[2.7.0-RELEASE][root@DTLFWSRV02.datalab.local]/root: dig +trace www.google.it A

; <<>> DiG 9.18.14 <<>> +trace www.google.it A
;; global options: +cmd
.                       75354   IN      NS      h.root-servers.net.
.                       75354   IN      NS      a.root-servers.net.
.                       75354   IN      NS      i.root-servers.net.
.                       75354   IN      NS      b.root-servers.net.
.                       75354   IN      NS      l.root-servers.net.
.                       75354   IN      NS      g.root-servers.net.
.                       75354   IN      NS      e.root-servers.net.
.                       75354   IN      NS      j.root-servers.net.
.                       75354   IN      NS      c.root-servers.net.
.                       75354   IN      NS      k.root-servers.net.
.                       75354   IN      NS      m.root-servers.net.
.                       75354   IN      NS      d.root-servers.net.
.                       75354   IN      NS      f.root-servers.net.
;; Received 268 bytes from 127.0.0.1#53(127.0.0.1) in 1 ms

it.                     172800  IN      NS      nameserver.cnr.it.
it.                     172800  IN      NS      dns.nic.it.
it.                     172800  IN      NS      r.dns.it.
it.                     172800  IN      NS      a.dns.it.
it.                     172800  IN      NS      m.dns.it.
it.                     172800  IN      NS      d.dns.it.
it.                     86400   IN      DS      41901 10 2 47F7F7BA21E48591F6172EED13E35B66B93AD9F2880FC9BADA64F68C E28EBB90
it.                     86400   IN      RRSIG   DS 8 1 86400 20230905050000 20230823040000 11019 . jVohD3VlXJvQzLiYfwmEsau8yg33fbXvwxRKdDtIzIeGsF3yyBNGH0M6 iq4eLmNAZVaORMrZt6taHC7wB+3kLyIqLhfC9LLJ/AOsh9c0vx0jU8/Y 60Hzu2ERonaYvg8rfTeau4XfSPwpEWx+9/FzDeaerSC1oBaAUkW2O2c7 vHyn5uvZ/l4Xy1vdBcHGaM6TmIhCJBkmOYb2iCl/xHVPjhm3Mcj6RFKl 1/hH0qxNnGH6imEJT/RKBZO8VprSlYs9jDHndxQnUKSJIRBlq4J0Tj/1 6QuzP69zOjSKRBqBRKp1baeR7DzUv/hlfU0hXbF6zixpfOYzhIoL8XVF u1SBEQ==
;; Received 796 bytes from 192.33.4.12#53(c.root-servers.net) in 26 ms

It was working fine with 2.6.0
am i missing something? did something change in the meantime or is it a regression?

[2.7.0-RELEASE][root@DTLFWSRV02.datalab.local]/root: ps auxwwwww | grep dns
nobody 69987   0.0  0.0   17828   4872  -  S    19:27      0:00.45 /usr/local/sbin/dnsmasq -C /dev/null --rebind-localhost-ok --stop-dns-rebind --dhcp-hostsfile=/etc/hosts --listen-address=192.168.8.7 --listen-address=192.168.4.1 --listen-address=192.168.17.1 --listen-address=192.168.12.1 --listen-address=192.168.13.1 --listen-address=192.168.14.1 --listen-address=192.168.15.1 --listen-address=192.168.93.1 --listen-address=192.168.95.1 --listen-address=192.168.94.1 --listen-address=192.168.8.5 --listen-address=192.168.17.3 --listen-address=192.168.12.3 --listen-address=192.168.13.3 --listen-address=192.168.14.3 --listen-address=192.168.15.3 --listen-address=127.0.0.1 --bind-interfaces --server=/+++++.+++++++.com/192.168.100.171 --no-resolv --server=192.168.8.230 --server=192.168.8.234 --rebind-domain-ok=/++++++.++++++.com/ --all-servers --dns-forward-max=5000 --cache-size=10000 --local-ttl=1

[2.7.0-RELEASE][root@DTLFWSRV02.datalab.local]/root: dig @192.168.8.230 ********.datalab.local A

; <<>> DiG 9.18.14 <<>> @192.168.8.230 *******.datalab.local A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64560
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;***********.datalab.local.  IN      A

;; ANSWER SECTION:
*******.datalab.local. 3600 IN   A       192.168.8.184

;; Query time: 0 msec
;; SERVER: 192.168.8.230#53(192.168.8.230) (UDP)
;; WHEN: Wed Aug 23 19:57:33 CEST 2023
;; MSG SIZE  rcvd: 73

[2.7.0-RELEASE][root@DTLFWSRV02.datalab.local]/root: dig @127.0.0.1 *******.datalab.local A

; <<>> DiG 9.18.14 <<>> @127.0.0.1 **********.datalab.local A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2018
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; EDE: 15 (Blocked)
;; QUESTION SECTION:
;***********.datalab.local.  IN      A

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Wed Aug 23 19:57:45 CEST 2023
;; MSG SIZE  rcvd: 63

johnpoz · Aug 23, 2023, 6:29 PM

@kiokoman said in DNS Forwarder changed behavior after upgrade to 2.7.0:

but dns forwarder is asking root servers instead of my own dns

what do think is going to happen with +trace?? trace is too the roots, always!

+[no]trace
    Toggle tracing of the delegation path from the root name servers for the name being looked up. Tracing is disabled by default. When tracing is enabled, dig makes iterative queries to resolve the name being looked up. It will follow referrals from the root servers, showing the answer from each server that was used to resolve the lookup.

BTW -- notice this

;; WARNING: .local is reserved for Multicast DNS

using .local is bad choice for your local domain

kiokoman · Aug 23, 2023, 7:45 PM

@johnpoz

you are right, sorry, but the problem is with the dig below

dig @192.168.8.230 ********.datalab.local A
.....
;; ANSWER SECTION:
*******.datalab.local. 3600 IN A 192.168.8.184

dig @127.0.0.1 *******.datalab.local A
;; QUESTION SECTION:
;***********.datalab.local. IN A

maybe 10 years ago there wasn't this warning ... -> ;; WARNING: .local is reserved for Multicast DNS
is it something introduced with the new version? i don't think a warning can make this query not working anyway bc if i query my dns server i have the warning but the also the expected answer

i also have domain override not working for a .com for the moment i have disabled DNS Forwarder for the localhost interface

johnpoz · Aug 23, 2023, 8:23 PM

@kiokoman .local has been a horrible choice for tld for years, and years!! Has to be at least 10 years since the mdns came out and MS broke .local for normal use ;)

dig has started warning about it now.

Not sure what your showing there.. when you query localhost it doesn't answer? Do you have some acl set..

I have no such issue.. You get noerror, so its not like its not answering listening on.. Did you setup views? Or a RPZ?

your hiding what you did the query for - for all I know your asking for something that has no record.. But if that was the case you should get back a soa, is your zone set for transparent, not sure what happens with .local and you attempt to find something that isn't a local record and your transparent. You should get back a soa.. If I had to guess you setup a acl that doesn't include localhost, or some sort of view or rpz?

[23.05.1-RELEASE][admin@sg4860.local.lan]/root: dig nas.home.arpa

; <<>> DiG 9.18.13 <<>> nas.home.arpa
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35528
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;nas.home.arpa.                 IN      A

;; ANSWER SECTION:
nas.home.arpa.          3600    IN      A       192.168.9.10

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Wed Aug 23 15:15:59 CDT 2023
;; MSG SIZE  rcvd: 58

[23.05.1-RELEASE][admin@sg4860.local.lan]/root: dig @192.168.9.253 nas.home.arpa

; <<>> DiG 9.18.13 <<>> @192.168.9.253 nas.home.arpa
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49402
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;nas.home.arpa.                 IN      A

;; ANSWER SECTION:
nas.home.arpa.          3600    IN      A       192.168.9.10

;; Query time: 0 msec
;; SERVER: 192.168.9.253#53(192.168.9.253) (UDP)
;; WHEN: Wed Aug 23 15:16:11 CDT 2023
;; MSG SIZE  rcvd: 58

[23.05.1-RELEASE][admin@sg4860.local.lan]/root:

kiokoman · Aug 23, 2023, 8:44 PM

@johnpoz
i think the problem is here,
if i enable dns forwarder with "use remote DNS Server, ignore local dns"
i have this
[2.7.0-RELEASE][root@DTLFWSRV02.datalab.local]/root: cat /etc/resolv.conf
nameserver 127.0.0.1
nameserver 192.168.8.230
nameserver 192.168.8.234
search datalab.local

but with DNS Resolver with forwarding enable and the same "use remote DNS Server, ignore local dns"
i have this and everything is working.
[2.7.0-RELEASE][root@DTLFWSRV02.datalab.local]/root: cat /etc/resolv.conf
nameserver 192.168.8.230
nameserver 192.168.8.234
search datalab.local

i don't remember if there was that "nameserver 127.0.0.1" before the upgrade

dns resolver is answering my query coming from the lan with nslookup in any case

johnpoz · Aug 23, 2023, 8:53 PM

@kiokoman 127.0.0.1 is your local host.. This is what is in my resolv.conf

[23.05.1-RELEASE][admin@sg4860.local.lan]/root: cat /etc/resolv.conf 
nameserver 127.0.0.1
search local.lan
[23.05.1-RELEASE][admin@sg4860.local.lan]/root:

Maybe that setting does something with the ACLs, I don't have auto acls set, or something else so unbound doesn't answer when you ask it on local host? If you were not listening on localhost, then you would get a timeout, not a no error.

I really need to fully move away from local.lan and change that to home.arpa - I am like half my stuff uses home.arpa and other half still using my old domain local.lan

I would never set that remote only, not local - maybe that seeing does more than just edit your resolv.conf ? Why would I not want pfsense to ask itself for local resouces? googdns or quad9 sure and the hell not going to be able to resolve your local resources..

Hmmm if your only asking remote, maybe a rebind has to do with it.. If you forward to something and get back a rfc1918, pfsense isn't going to give you an answer..

I bet you that is it, your ask some other NS, 192.168.8.23X and he responds with 192.168.x.x for whatever.yourdomain.tld, but if you ask the local unbound and it forwards to 192.168.8.23X and it hands back a 192.168.x.x address or 10.x or a 172.16-31.x.x that would be a rebind..

Try setting your datalab.local as a private domain.. That would be in the unbound advanced custom box.. Or just turn off rebind protection in pfsense.

https://docs.netgate.com/pfsense/en/latest/services/dns/rebinding.html

It didn't really dawn on me that you were forwarding to different NS on your network until you posted the .230 and .234 directly.. Bet you a beer that is your problem ;)

kiokoman · Aug 23, 2023, 9:21 PM

@johnpoz
omfg it was that rebind .. mf.. protection
thanks

johnpoz · Aug 23, 2023, 9:47 PM

@kiokoman sorry took so long to spot it.. I was thinking that 192.168.8.x you were asking was just the pfsense IP.. Doh! if would of dawned on me that is was some other NS on your network rebind would been right away.. Sorry took a few posts for me to notice that, glad you got it sorted.