DNS Resolver (unbound) not updating ?

Lumberjack · Mar 20, 2019, 5:04 PM

Hi;

I have a pf controllilin 1 WAN and 7 subnets where;
a) One subnet (only one) uses DNS resolver. Primary to use "Host overides" for some stuff here.
b) Some subnets uses a Windows AD DNS server. This dns is "DNS server 1" in pf's "general settings".
c) ...and some subnets uses only ISP DNS servers.

A computer was "moved" from subnet a) to one of the b) subnets. (Actually, the computer was virtualized, so MAC changed). The windows AD DNS updated accordingly.

But the a) subnet - with DNS resolver - still gives the old IP back for that computer name, the IP it had when it was a physical machine on the "a)" net.

It has done this for months, and yes, pfsense has been booted in the meantime.

Why is this ?

Note;
Only setting enabled on DNS Resolver is "Enable DNSSEC support" (apart from a few host overrides).

johnpoz · Mar 20, 2019, 5:32 PM

well remove your override then... Or where unbound would be resolving it from.. Did you have unbound set to load dhcp reservations.

Lumberjack · Mar 21, 2019, 9:36 AM

The computer that moved between subnets, lets refer to it as CLIENT5, is not on the host override list. It has never been. The host overrides are working fine for the subnet.

That is the question really, exactly where are unbound resolving the CLIENT5 name from ?
The name of CLIENT5 is set to another IP in the DNS configurated as "DNS server 1" in pf's "general settings". (On the subnets not using unbound the name resolves correctly.)
The name of CLIENT5 does not exist in the ISP's DNS server (the only other DNS's mentioned on pf)

No, "Register DHCP leases in the DNS Resolver" is not set. "Register DHCP static mappings in the DNS Resolver" are also unchecked.
(Note:, the name of CLIENT5 exist in the DHCP leases list - with the correct IP.)

To me it seems like unbound picked up the name and IP combo for CLIENT5 a long time ago (back when it actually was correct) and has never updated itself since. So it keeps answering CLIENT5 name request with the wrong IP. Any ideas why ?

johnpoz · Mar 21, 2019, 10:40 AM

unbound doesn't pick up anything.. It resolves..

Entries are only the cache for the length of the ttl on the record.. A restart of unbound would flush the cache.

Please post exactly what is happening when you query the specific FQDN from unbound... client5 is not a fqdn, and will not resolve.. But you client could be adding its own search suffix and resolving that from??

It is returning rfc1918 space right?

Where does unbound say it would lookup the fqdn ?
/usr/local/sbin/unbound-control -c /var/unbound/unbound.conf lookup client5.domain.tld

What is in the local data, dump it and check - for example here I look for local data of fqdn of box of mine called pi.local.lan

[2.4.4-RELEASE][admin@sg4860.local.lan]/: /usr/local/sbin/unbound-control -c /var/unbound/unbound.conf list_local_data | grep pi.local.lan 
pi.local.lan.   3600    IN      A       192.168.3.31
31.3.168.192.in-addr.arpa.      3600    IN      PTR     pi.local.lan.
[2.4.4-RELEASE][admin@sg4860.local.lan]/:

Lumberjack · Mar 21, 2019, 12:13 PM

Result of unbound-control (Note: i masked the real FQDN with x's in the snippet below):

The following name servers are used for lookup of xxxxx.xxx.xx.
The noprime stub servers are used:
Delegation with 0 names, of which 0 can be examined to query further addresses.
It provides 1 IP addresses.
10.100.102.11   	NoDNSSEC rto 12 msec, ttl 121, ping 0 var 3 rtt 50, tA 0, tAAAA 0, tother 0, EDNS 0 probed.

... the IP returned/mentioned above is the wrong one (it has been in use by this FQDN previously, but thats a long time ago).

johnpoz · Mar 21, 2019, 12:28 PM

noprime stub servers..

so you did a lookup for host.domain.tld and you got back the NS its going to ask.. This 10.100.102.11 box..

For example if I ask unbound what will it use to lookup www.netgate.com

[2.4.4-RELEASE][admin@sg4860.local.lan]/: /usr/local/sbin/unbound-control -c /var/unbound/unbound.conf lookup www.netgate.com
The following name servers are used for lookup of www.netgate.com.
;rrset 2982 2 0 7 3
netgate.com.    2982    IN      NS      ns1.netgate.com.
netgate.com.    2982    IN      NS      ns2.netgate.com.
;rrset 2982 1 0 8 3
ns2.netgate.com.        2982    IN      A       162.208.119.38
;rrset 2982 1 0 3 3
ns1.netgate.com.        2982    IN      A       208.123.73.80
;rrset 2982 1 0 3 3
ns1.netgate.com.        2982    IN      AAAA    2610:160:11:11::80
Delegation with 2 names, of which 1 can be examined to query further addresses.
It provides 3 IP addresses.
2610:160:11:11::80      rto 339 msec, ttl 282, ping 15 var 81 rtt 339, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
208.123.73.80           rto 226 msec, ttl 282, ping 14 var 53 rtt 226, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
162.208.119.38          rto 338 msec, ttl 282, ping 6 var 83 rtt 338, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
[2.4.4-RELEASE][admin@sg4860.local.lan]/:

Its says it will use NS 2610, 208 and 162 that will provide the answer.

That is not the answer... So your unbound is asking the 10.100.102.11 box for that fqdn.. So it is the NS that is returning the wrong answer.

You have a domain override setup for whatever domain your asking about or your forwarding?

Look at your unbound.conf file - PM it to me and can take a look see.
example here is mine

[2.4.4-RELEASE][admin@sg4860.local.lan]/: cat /var/unbound/unbound.conf 
##########################
# Unbound Configuration
##########################

##
# Server configuration
##
server:

chroot: /var/unbound
username: "unbound"
directory: "/var/unbound"
pidfile: "/var/run/unbound.pid"
use-syslog: yes
port: 53
verbosity: 3
hide-identity: no
hide-version: no
harden-glue: yes
do-ip4: yes
do-ip6: yes
do-udp: yes
do-tcp: yes
do-daemonize: yes
module-config: "validator iterator"
unwanted-reply-threshold: 0
num-queries-per-thread: 512
jostle-timeout: 200
infra-host-ttl: 900
infra-cache-numhosts: 50000
outgoing-num-tcp: 20
incoming-num-tcp: 20
edns-buffer-size: 4096
cache-max-ttl: 86400
cache-min-ttl: 3600
harden-dnssec-stripped: yes
msg-cache-size: 20m
rrset-cache-size: 40m

num-threads: 4
msg-cache-slabs: 4
rrset-cache-slabs: 4
infra-cache-slabs: 4
key-cache-slabs: 4
outgoing-range: 4096
#so-rcvbuf: 4m
auto-trust-anchor-file: /var/unbound/root.key
prefetch: yes
prefetch-key: yes
use-caps-for-id: yes
serve-expired: yes
# Statistics
# Unbound Statistics
statistics-interval: 0
extended-statistics: yes
statistics-cumulative: yes

# SSL Configuration

# Interface IP(s) to bind to
interface: 192.168.3.253
interface: 2001:470:snipped::253
interface: 192.168.9.253
interface: 192.168.2.253
interface: 192.168.6.253
interface: 192.168.4.253
interface: 192.168.7.253
interface: 127.0.0.1
interface: ::1

# Outgoing interfaces to be used
outgoing-interface: 2001:470:snipped::2
outgoing-interface: 64.53.snipped

# DNS Rebinding
# For DNS Rebinding prevention
private-address: 10.0.0.0/8
private-address: ::ffff:a00:0/104
private-address: 172.16.0.0/12
private-address: ::ffff:ac10:0/108
private-address: 169.254.0.0/16
private-address: ::ffff:a9fe:0/112
private-address: 192.168.0.0/16
private-address: ::ffff:c0a8:0/112
private-address: fd00::/8
private-address: fe80::/10
# Set private domains in case authoritative name server returns a Private IP address
private-domain: "home.lan"
domain-insecure: "home.lan"


# Access lists
include: /var/unbound/access_lists.conf

# Static host entries
include: /var/unbound/host_entries.conf

# dhcp lease entries
include: /var/unbound/dhcpleases_entries.conf



# Domain overrides
include: /var/unbound/domainoverrides.conf


# Unbound custom options
server:
private-domain: "plex.direct"
so-reuseport: no
log-queries: yes
log-replies: yes
#private-address: ::/0  # filters out all AAAA !


###
# Remote Control Config
###
include: /var/unbound/remotecontrol.conf
[2.4.4-RELEASE][admin@sg4860.local.lan]/:

I only snipped out my public interface IPs

Lumberjack · Mar 21, 2019, 1:03 PM

Wow, You are fast!

And you are correct, of course, the IP 10.100.102.11 mentioned is to the DNS server (DNS on a windows AD). Its NOT the IP returned when the CLIENT5 name are resolved. My mistake.

(note: When doing unbound-control with www.netgate.com I get beck pretty much the same as You).

So it seems the problem is somewhere within the windows AD DNS then ?

The AD DNS somehow resolves the FQDN of CLIENT5 to be 10.100.12.8 when "going through" unbound, but the correct 10.100.105.79 when it set (by DHCP) to be used as the primary DNS ... ?

johnpoz · Mar 21, 2019, 1:10 PM

Do you have views setup on your AD setup, or conditional forwarders, etc.

Without more info to your setup of your dns in AD, not sure.. What version of windows you running 2k12, 2k16, 2003? etc.. Over the years they have made changes to how DNS works in windows for features... Your not still using wins are you?

Are you syncing DNS across multiple servers in your AD.. I assume you have it directory-integrated, etc.

to be used as the primary DNS ... ?

Not a fan when users state such a thing... While you can point clients to more than 1 NS.. you have to be SURE!!!! that these NS are going to resolve/forward and return the same info..

For example - you can not point a client to your local AD dns, while also listing say 8.8.8.8 be it 2nd NS or not.. You will have a bad day if you setup this way.. Since you can never be sure where a client will ask if there is more than 1 NS listed. So sure you could point to multiple AD dns that all return the same info.. But you can not point a client to say

NS1 10.x.x.x
NS2 8.8.8.8

Without running into problems at some point..

Lumberjack · Mar 21, 2019, 2:54 PM

The dc used here (10.100.102.11) are 2012 R2 std. Not wins, no. There exist "backup" dc's. It's a single site domain, no forest's or other "advanced" stuff used.

I have often used multiple DNS's in DHCP, but I see your point.

However, to clearify a bit (Note: i masked the real FQDN's with x's in the snippet below);

When on a windows client in the "outbound subnet", the DNS settings are as follows;

[D:\Source\Code\phy]ipconfig /all

Windows IP Configuration
   ...
   DNS Servers . . . . . . . . . . . : 10.100.0.1
                                       10.100.102.11
   ...

The 10.100.0.1 is the pfSense, DNS resolver enabled here (to enable som "host overrides"). So we ask to resolve;

[D:\Source\Code\phy]nslookup client5.xxx.xx
Server:  pfSense.xxx.xx
Address:  10.100.0.1

Name:    client5.xxx.xx
Address:  10.100.12.8

Now, earlier we established that outbound on pfSense are actually using 10.100.102.11

The following name servers are used for lookup of client5.xxx.xx.
The noprime stub servers are used:
Delegation with 0 names, of which 0 can be examined to query further addresses.
It provides 1 IP addresses.
10.100.102.11   	NoDNSSEC rto 12 msec, ttl 366, ping 0 var 3 rtt 50, tA 0, tAAAA 0, tother 0, EDNS 0 probed.

But when we resolve through 10.100.102.11 directly;

[D:\Source\Code\phy]nslookup client5.xxx.xx 10.100.102.11
Server:  pdc03.xxx.xx
Address:  10.100.102.11

Name:    client5.xxx.xx
Address:  10.100.105.79

We get another IP. This is the correct one.

I did a quick check on the DNS setup on 10.100.102.11: exporting everything to a .txt file and the a search, but 10.100.12.8 was not found anywhere.

johnpoz · Mar 21, 2019, 3:44 PM

@Lumberjack said in DNS Resolver (unbound) not updating ?:

Windows IP Configuration
...
DNS Servers . . . . . . . . . . . : 10.100.0.1
10.100.102.11
...

That is going to be problematic..

Your clients should point to AD if they are members of the AD or need to resolve stuff in AD... You would then forward from your AD to pfsense if you want AD to resolve stuff in pfsense overrides... But why would you not just put the overrides in AD in the first place?

Where are you doing that direct query from? Same subnet as pfsense does it query to your AD?

Just turn on logging in unbound and you can see exactly where your answer coming from..

server:
log-queries: yes
log-replies: yes

Put that in the custom options box on unbound setup..

Then set logging to level 3 or higher

Then query pfsense for this client and you can see exactly where it comes from... So I have a domain override setup for testing to windows dns... for home.lan 192.168.2.220

So when I query pfsense for host.home.lan it goes and asks the windows dns.. You can see exactly what unbound is doing with logging turned on.

Mar 21 10:30:23 	unbound 	6448:1 	info: 192.168.9.100 host.home.lan. A IN NOERROR 0.759713 0 58
Mar 21 10:30:23 	unbound 	6448:1 	info: query response was ANSWER
Mar 21 10:30:23 	unbound 	6448:1 	info: reply from <home.lan.> 192.168.2.220#53
Mar 21 10:30:23 	unbound 	6448:1 	info: response for host.home.lan. A IN
Mar 21 10:30:23 	unbound 	6448:1 	info: iterator operate: query host.home.lan. A IN
Mar 21 10:30:23 	unbound 	6448:1 	debug: sending to target: <home.lan.> 192.168.2.220#53
Mar 21 10:30:23 	unbound 	6448:1 	info: sending query: host.home.lan. A IN
Mar 21 10:30:23 	unbound 	6448:1 	info: processQueryTargets: host.home.lan. A IN
Mar 21 10:30:23 	unbound 	6448:1 	info: iterator operate: query host.home.lan. A IN
Mar 21 10:30:23 	unbound 	6448:1 	debug: sending to target: <home.lan.> 192.168.2.220#53
Mar 21 10:30:23 	unbound 	6448:1 	info: sending query: host.home.lan. A IN
Mar 21 10:30:23 	unbound 	6448:1 	info: processQueryTargets: host.home.lan. A IN
Mar 21 10:30:23 	unbound 	6448:1 	info: resolving host.home.lan. A IN
Mar 21 10:30:23 	unbound 	6448:1 	info: validator operate: query host.home.lan. A IN
Mar 21 10:30:23 	unbound 	6448:1 	info: 192.168.9.100 host.home.lan. A IN

You can see my client from 192.168.9.100 ask.. Unbound goes and ask 192.168.2.220 and gets answer

You see what my client gets back.

$ dig @192.168.9.253 host.home.lan

; <<>> DiG 9.12.3-P1 <<>> @192.168.9.253 host.home.lan
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12831
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;host.home.lan.                 IN      A

;; ANSWER SECTION:
host.home.lan.          3600    IN      A       10.10.10.10

;; Query time: 760 msec
;; SERVER: 192.168.9.253#53(192.168.9.253)
;; WHEN: Thu Mar 21 10:30:23 Central Daylight Time 2019
;; MSG SIZE  rcvd: 58

When I ask unbound where it would lookup host.home.lan I get

[2.4.4-RELEASE][admin@sg4860.local.lan]/: /usr/local/sbin/unbound-control -c /var/unbound/unbound.conf lookup host.home.lan
The following name servers are used for lookup of host.home.lan.
forwarding request:
Delegation with 0 names, of which 0 can be examined to query further addresses.
It provides 1 IP addresses.
192.168.2.220           rto 284 msec, ttl 558, ping 0 var 71 rtt 284, tA 0, tAAAA 0, tother 0, EDNS 0 probed.
[2.4.4-RELEASE][admin@sg4860.local.lan]/:

Why when you did for your client you see noprime stub? Please post your unbound config.. Do you have unbound set as stub? vs default transparent You cearly have something more than the gui configurations for unbound going on when it says your using stub zone and not prime.

You are not just forwarding to your AD to lookup specific domain... You or it would say forwarding like mine above when setup domain override to go ask windows dns at 192.168.2.220 for home.lan

Lumberjack · Mar 22, 2019, 10:05 AM

Detected that the site currently is running pfsense 2.3-RELEASE (amd64), FreeBSD 10.3-RELEASE. There is no 'log_replies' unbound option in that freebsd version.

I have scheduled a upgrade. Reporting back after that.

And I detected that unbound is set with "domain override" of the domain (eg. xxx.xx of the client5.xxx.xx). This possibly explains missing calls to prime zone's.

johnpoz · Mar 22, 2019, 10:31 AM

2.3?? Dude that is no longer supported, end of life... You need to upgrade to current..

Let us know after the upgrade.. But its still asking your AD for that..

What version of unbound is it running btw?

Lumberjack · Mar 23, 2019, 6:06 PM

Ok, pf on 2.4.4 now.

I believe I have found source of the the problem. it's this:
https://forum.netgate.com/topic/135409/unbound-dhcp-registration-checkbox-bug

First, I enabled the unbound log (as suggested) an this was the result:

Mar 23 17:44:16 unbound 70805:7 info: 10.100.12.21 client5.xxx.xx. AAAA IN NOERROR 0.000000 1 31 
Mar 23 17:44:16 unbound 70805:7 info: 10.100.12.21 client5.xxx.xx. AAAA IN 
Mar 23 17:44:16 unbound 70805:7 info: 10.100.12.21 client5.xxx.xx. A IN NOERROR 0.000000 1 47 
Mar 23 17:44:16 unbound 70805:7 info: 10.100.12.21 client5.xxx.xx. A IN 
Mar 23 17:44:16 unbound 70805:7 info: 10.100.12.21 1.0.100.10.in-addr.arpa. PTR IN NOERROR 0.000000 1 69 
Mar 23 17:44:16 unbound 70805:7 info: 10.100.12.21 1.0.100.10.in-addr.arpa. PTR IN

That's all. No contact with AD DNS.

So, after some thinkering, i did a cat of the unbound config and found this ;

...
# dhcp lease entries
include: /var/unbound/dhcpleases_entries.conf
...

...whut ?! thats not even enabled?

Then I did a cat on dhcpleases_entries.conf and, yes, i found client5.xxx.xx listed.
...along with approx other 10 names registered, where only 2 still exists today.

Somebody has probably, a long time ago, enabled/checked "DHCP Registration" in the pf GUI. Then unchecked it again a short time after enabling it. But the names collected got stuck in unbound. Not good.

Should I just truncate the dhcpleases_entries.conf or do there exist a more proper way to get rid of it ?

johnpoz · Mar 23, 2019, 6:08 PM

Just delete all the entries in there should be fine.

Lumberjack · Mar 23, 2019, 6:32 PM

Done. It worked, the correct IP are returned now.

One remark;

Something for the devs to consider, This;

usr/local/sbin/unbound-control -c /var/unbound/unbound.conf lookup client5.domain.tld

... should really give some kind of indication that the name in question do exist in a local table.

Grimson · Mar 23, 2019, 8:30 PM

@Lumberjack said in DNS Resolver (unbound) not updating ?:

Something for the devs to consider, This;

usr/local/sbin/unbound-control -c /var/unbound/unbound.conf lookup client5.domain.tld

... should really give some kind of indication that the name in question do exist in a local table.

Talk to the unbound devs then: https://nlnetlabs.nl

Gertjan · Mar 23, 2019, 10:22 PM

@Lumberjack said in DNS Resolver (unbound) not updating ?:

...whut ?! thats not even enabled?

Hummm. This rings a bell to me.
An issue that was solved a couple of updates ago ... emptying the dhcp leases file when dhcpleases process is shut down.

You 'missed' that upgrade I guess ^^

luckman212 · Jun 25, 2020, 6:46 PM

This bug is still present in 2.4.4-p3 (and possibly 2.4.5-p1, 2.5.0) and was frustrating to track down today.

Seems like the problem has 2 potential fixes:

make sure the contents of /var/unbound/dhcpleases_entries.conf is 100% cleared when the "Register DHCP leases in the DNS Resolver" checkbox feature is disabled.

or

when said checkbox is cleared, comment out the include: /var/unbound/dhcpleases_entries.conf line in /var/unbound/unbound.conf

I feel like #2 is the better fix, since that way DHCP leases would resolve properly if somone toggles the box off & then back on by mistake.